A build webhook works only when a build starts
Our build webhooks answered with success and started nothing. Now each one is proved the only way that counts: fire it and watch a build start.
The setup
For each project whose code lives upstream, in a repository of its own, we keep two repositories in Forgejo. One is a read-only mirror of the code, which checks the original every ten minutes. The other holds the build workflow: it clones the mirror, builds the images and publishes them to the private registry.
Out of the box, that workflow ran every six hours, or when someone started it by hand. Developers would push a commit, go to deploy it, and find the build was one version behind. Nothing connected “the mirror has new code” to “build it now”.
Why a webhook, not a push trigger
The obvious fix is to make the workflow run on every push. But the workflow lives in the build repository, and a push trigger there fires only on pushes to that repository, such as edits to the workflow itself. It never sees a commit arriving in the mirror.
The link has to come from the mirror. So the mirror carries a push webhook, and the webhook asks the build repository to run its workflow.
Two ways a webhook is born dead
We found both in the field, and both are silent.
The first is the address. Forgejo is set to deliver webhooks only to outside addresses, as a guard against requests aimed at internal services. A webhook pointed at the build service’s internal address is dropped, with no error. The fix is the public address, which on the same host simply passes through the proxy at the edge, with no real trip across the internet. An earlier version of our own runbook recommended the internal address, which is how the mistake spread.
The second is the token. Starting a workflow needs authorisation. Without the token, the request is accepted with a success code and no build is created. Put the token in the wrong field of the webhook and Forgejo drops it, just as quietly.
Reading the webhook back does not help. The stored token is hidden, so a webhook with no token and one with the right token look the same.
What the audit found
When we fired every mirror’s webhook and watched for a real build, none of them had ever worked. Every project had been riding the six-hour schedule all along.
Each webhook was corrected to the public address with its token, then fired again until a build appeared. The onboarding script that sets up new projects now creates the webhook the same way, and running it again corrects an existing webhook instead of adding a second one.
How we check now
A webhook counts as working only when a build starts. So the check is the real thing: fire the webhook’s test delivery and look for a new build run that it started, with a run number higher than the last one.
Two details matter. A new run can take between 30 and 90 seconds to appear, so a short wait proves nothing. And a success code from the test delivery means the delivery was queued, not that a build fired.
What it means for you
A push reaches a build without waiting for a schedule. The remaining wait is the mirror’s own check, at most ten minutes. The new image then goes to the private registry, and services labelled for automatic updates pick it up within the hour. Services that opt out are deployed by hand, on purpose, and we check the label before assuming a new image will roll out.
It is a small fix. What it changed is that “pushed” and “built” now mean the same thing, and we can show it, not just say it.
What we don’t claim
A webhook that fires means a build starts. Whether the build succeeds is the workflow’s business, and a build can still fail for its own reasons. The chain is not instant: the mirror’s check and the hourly update both add time. And not every service moves to a new image on its own; some are held back until someone deploys them.
Running into this yourself? Ask the assistant about it, or tell us what has to keep running.