Why Docker tags are mutable, how Node.js images broke yarn, and how to work with immutable Docker digests instead.
- On Saturday, official Node.js images all suffered from broken
yarnsupport - This highlighted that Docker tags may and can change at any time
- Users need to use digest hashes to identify source images, instead of referencing mutable tags
- Tools exist to automate digest pinning and updating so it does not need to be done manually
What happened?
yarn support was broken in all latest Node.js Docker images due to a mistake in a Dockerfile refactor. The Node.js version hadn’t changed, so the image tags also remained unchanged, however the new image pushed for each tag was broken. This meant that Docker images that had previously worked (e.g. node:9 or node:8.10.0-alpine) suddenly stopped working the next time somebody or some machine such as CI pulled them.
This was a classic “Works on my machine” moment
It was far from a “break the internet” incident like left-pad but you can observe from the issue that the result was the same for those affected – builds broke, couldn’t deploy from CI, etc.
This was a classic “works on my machine” / “what changed??” moment that you get used to if you don’t practice immutable builds.
Who was impacted?
The incident affected anybody using yarn with the most recent versions of the official Docker Node.js images, e.g. 9, 8, 9.8, 8.10, 9.8.0, 8.10.0, etc. Anyone pulling one of those images during that time would have got a non-working image/build.
What did users see?
For most people, they would have seen some variant of “command not found” for yarn. In Renovate’s case, the app’s automated builds on Docker Hub started erroring.
Why did it affect so many for so long?
The reason this took effect so quickly is because (a) most people didn’t really “opt in” to this upgrade because it’s wasn’t an upgrade – it was an image change for an existing tag, and (b) most people didn’t know how to roll back to the image that previously worked for them, because Docker digests are challenging to work with.
As a result, pretty much everyone was left waiting for “somebody to do something”. Unlike the left-pad incident though, nobody had removed any images or builds from Docker Hub. Perfectly working images were still there, ready to be pulled if you knew how to ask for them. The only thing that had changed was where the tags pointed to, like symbolic links.
Some users helped by identifying working tags in the registry:
However, this is not always viable solution. What if node:8.10.0 contains functionality you required, or was even a security fix? The solution clearly wouldn’t be to roll back to an insecure Node.js version just because the image pointed to by the node:8.10.0 tag was broken. The point is: rolling back to an outdated version of Node.js just to fix a broken build in the current version of Node.js would not always be viable.
This is a Teachable Moment
This is a great chance to talk about the mutability of Docker image tags and what you as a user can do to “protect” yourself. You should have more control over your builds and what goes into them and it’s only something you can do, not Docker or Node.js or anyone else you consume images from.
What you should know is this:
- This type of mutable “tag change” behaviour is by design for Docker
- This type of problem (a tag that once worked then doesn’t) will eventually happen again
- You can protect yourself against these changes
- Future breaks do not need to leave you waiting hours for “fixes”
- You can automate the process
Docker image tags are intentionally changeable
Just because it walks like a semver and quacks like a semver, doesn’t make it
a semverimmutable
Any Docker image tag – even ones that are based on semver – can change at any moment, whether for nefarious reasons (a hack) or for accidental ones like this yarn problem.
This is known as mutability/immutability. Mutable identifiers like Docker image tags can change “what they point to” at any time. Immutable identifiers – like npm’s semver versions – by policy do not.
Although everyone probably expects that a tag like node:latest or node:8 will change, you might mistakenly expect that a tag like node:8.10.0 or node:8.10.0-alpine will not change, because it looks like a semver – but this is not the case. Any Docker tag can change what it points to.
Sometimes tags need to be mutated
As an example, consider the case where an important patch is released for the base alpine image. You would want that fix, but what should happen to your node:8.10.0-alpine image? It clearly can’t become node:8.10.1-alpine, because the embedded semver refers to the node version. You could try something like node:8.10.0.1-alpine but now it’s “not semver” and probably leads to more confusion.
You could try a build number (e.g. node:8.10.0_2-alpine) but maybe that’s also confusing to people, especially if there are different build numbers between alpine and other variants of v8.10.0. Instead, the solution right now is to push a new image and update the node:8.10.0-alpine to point to the new one.
In this case, the Node.js team wanted to refactor some ways they build the images, while keeping the same versions/tags. Again, this is perfectly valid and nothing wrong with it, although ideally you don’t break things in refactors.
Breaking Docker tags will happen again
Even if you “pin” your base images to what looks like semvers (e.g. node:8.10.0), the reality is that one day you will probably experience a break again. Docker’s tags are intentionally mutable, partly because those who build images need the flexibility to change, as described above.
Insure against breaking Docker tags
If you’re coming from an immutable dependency source like npmjs and now the reality of mutable Docker tags is hitting you, you might think this approach is not feasible.
But do you know that Docker also supports immutable image identifiers too?
Docker supports Pulling an image by digest (immutable identifier)
The Docker image “digest” is a sha256 “hash” that can be assured to point to the same code as it always has. But it’s not particularly human-friendly:
$ docker pull node@sha256:06ebd9b1879057e24c1e87db508ba9fd0dd7f766bbf55665652d31487ca194eb
One useful trick know is that you don’t have to remove the tag if you want to add a digest. If both are present then the tag is ignored, so you can leave it in for human readability, e.g.
FROM node:8.10.0-alpine@sha256:06ebd9b1879057e24c1e87db508ba9fd0dd7f766bbf55665652d31487ca194eb
The important thing is that if everyone had been using digests in their builds, then they would have been protected against this weekend’s problem and maybe saved themselves or their teams hours of wasted time. Even if you were hit by the problem like me (e.g. you upgraded to the new digest before finding out that your build broke), you could easily roll back to the old digest and be up and running in seconds or minutes.
How can you use Docker digests in a user-friendly way?
Automating Docker image digest updates
Renovate – my tool and this website – is an open source dependency automation tool that you can either self-host yourself, or install as a GitHub App.
If you’re not interested in another tool or automation for Docker updates, you can finish reading now, and I hope the above was informative and worth your time.
How does it work?
Renovate scans your repository for every Dockerfile it can find, and looks for all FROM lines. Here’s an example PR from Renovate’s own repository:
Specify a semver-like tag if possible
Specify your version completely: e.g. use node:8.10.0 and not node:8.10 or node:8. It still works if you don’t do this, but you’ll gain less visibility into upgrades and not be sure of exactly which Node.js version you’re running.
You may leave off the digest for now because Renovate will handle that for you with a PR later.
Pin Docker Digests
Renovate will raise PRs for any Docker images that need pinning to a digest: e.g. FROM node:8.10.0-alpine becomes FROM node:8.10.0-alpine@sha256:a55d3e87802b2a8464b3bfc1f8c3c409f89e9b70a31f1dccce70bd146501f1a0
Merge the PRs you receive.
Merge Renovate’s digest or version update PRs
If a source tag is updated with a new semver version, you will receive a PR with a new FROM line that looks something like FROM node:8.10.1@sha256:eks9abc802b2a8464b3bfc1f8c3c409f89e9b70a31f1dccce70bd14mxwris. i.e. tag has changed and digest has changed.
If only the digest is updated, you would see that too (and the PR will tell you that it’s a digest update, not a version update). This is like what happened on Saturday.
Embarrassing disclosure: I accepted that (broken) update to Renovate itself, but at least I could roll it back once the break was pointed out to me!
Roll back commits if there’s ever a break
If you had been using Renovate before Saturday, it would have been very obvious that a digest update had broken your base Node image, and also obvious how to roll it back. No waiting 8 hours for someone on the West Coast USA to wake up and merge a PR.
Docker digests gives you immutabile builds. Renovate PRs give you automation, visibility and control of what you’re updating and when.
Feedback
Please contact me on Twitter or via email if you think I’ve missed or misrepresented anything.
Also note: some Renovate Docker features are still in the pipeline, such as Customizable file names for Dockerfiles or Docker Compose support. Please add your voice or vote to those issues to let me know that there’s demand to implement them.
