Keeping builds moving on a slow line
Why our builds fetch apt, pip and npm packages through a pull-through cache, and what changed when the line to public registries was slow.
The problem
Every container build fetches packages: system packages through apt, Python packages through pip, JavaScript packages through npm. All of them come from public registries on the far side of the internet.
On the line our build runner sits behind, those registries were unreliable at busy hours. Connections timed out, downloads crawled, and now and then a registry answered with a server error. A build that took 30 seconds one day failed after 35 minutes the next, with no change to the code. The variable was always the network.
What did not work
The obvious fix was a nearby public mirror, so we tried the local mirror services first. The Python mirror was empty. The npm mirror refused most packages. The Debian mirror was more than two months out of date, stale enough to break the standard build tools. None of them could be trusted with a release.
A cache we control
What worked was a pull-through cache of our own: Nexus Repository, sitting between the build runner and the public registries. It proxies Debian packages and their security updates, the Python package index, and the npm registry.
The first time a build asks for a package, Nexus fetches it from the public registry, as slowly as the line allows. Every later request for that package is served from Nexus’s own disk. The public registries stay the source of truth. Nexus only remembers what it has already fetched.
How a build finds the cache
Three things have to line up. The build repository carries settings that name the cache for apt, pip and npm. The build workflow passes them into the image build. And the Dockerfile uses them.
New Dockerfiles follow a template that declares those settings. For projects whose Dockerfile we do not own, the build step adds the cache settings itself, after each base image line, and leaves a marker so it never adds them twice. The project’s own source is not changed.
If the settings are empty, everything falls back to the public registries. The same project still builds on a laptop or in another CI system, with no dependency on our cache.
Tuning for a flaky line
Two defaults had to change. Nexus normally blocks an upstream that keeps failing; on a flaky line that tripped on healthy registries, so automatic blocking is off. A single fetch can still fail, but the cache does not shut itself off. And pip gives up on a request after 15 seconds, which is too short for a large package on a cold cache, so builds raise the timeout and allow retries.
When one package keeps failing to come through, it can be fetched into the cache once by hand, and builds stop waiting on it.
What it changed
On a cache hit the difference is large. One web build went from 35 minutes cold to well under a minute warm. After one warm build per project, builds are limited by Docker’s own layer cache, not by the line.
The cost is paid once. The first build after the cache was set up took more than half an hour, because every package was a cold fetch.
What it means for you
A release no longer depends on how the line to the public registries is doing that afternoon. If your network is slow or filtered, builds fetch through the same kind of cache, and what was fetched once stays fetched.
What we don’t claim
The cache covers build time only. Running services pull their images from the private registry directly, and Nexus plays no part in that. It speeds up the second fetch, never the first. It covers apt, pip and npm; other package formats need their own proxy added. It is reachable only on the build host’s local network, so a runner somewhere else would need its own access or its own cache. And the timings above are one project’s, not a promise for yours.
Running into this yourself? Ask the assistant about it, or tell us what has to keep running.