DevOps on Man-You

Multi-arch Docker builds in GitHub Actions

Thu, 19 Feb 2026 10:00:00 +0200

We needed ARM64 containers. Our Python services run on mixed infrastructure: amd64 in CI and some production clusters, arm64 on newer nodes. Building on one arch and emulating the other with QEMU was painfully slow and broke native extensions. So we added proper multi-arch builds to our GitHub Actions CI.

It took a week to get right. Then five more fixes over two weeks as each failure mode revealed itself in production. This is what went wrong and how we fixed it.

The setup

We use reusable workflows in a shared Woosmap/.github repo. Individual service repos call these. A build-helper repo (checked out at runtime) contains the actual logic as a single Node.js action that routes to TypeScript modules based on an action input:

build-helper/src/main.ts

switch (action) {
 case 'build': await build(await get_build_info()); break
 case 'publish': await publish(await get_build_info()); break
 case 'release': await release(await get_build_info()); break
 case 'create_manifest':
 await createManifest(await get_build_info()); break
 case 'create_release_manifest':
 await createReleaseManifests(await get_build_info()); break
}

The multi-arch matrix is generated dynamically in the workflow:

.github/workflows/sonar_python_bender_build.yml

strategy:
 matrix:
 include: ${{ fromJson(inputs.multi_arch &&
 '[{"runner":"ubuntu-24.04","arch":"amd64"},
 {"runner":"ubuntu-24.04-arm","arch":"arm64"}]'
 || '[{"runner":"ubuntu-24.04","arch":"amd64"}]') }}
runs-on: ${{ matrix.runner }}

When multi_arch is true, two jobs run in parallel on native runners. No QEMU. An ARCH_SUFFIX environment variable (-amd64 or -arm64) drives platform selection downstream:

build-helper/src/build.ts

export function getPlatformForArch(): string {
 const archSuffix = getArchSuffix()
 return archSuffix === '-arm64' ? 'linux/arm64' : 'linux/amd64'
}

ARM64 jobs skip tests, coverage, and SonarCloud. Only amd64 does those. Build + push only for arm64.

Problem 1: buildx attestations

Buildx 0.10+ defaults to pushing attestation manifests (SBOM, provenance) alongside your image. When you docker push myimage:tag, it doesn’t push a plain image, it pushes a manifest list containing the image plus attestation layers.

This is fine if you just pull and run. But we need to stitch arch-specific images into a multi-arch manifest:

docker manifest create myimage:pr_10 \
 --amend myimage:sha-amd64 \
 --amend myimage:sha-arm64

docker manifest create expects plain images as sources, not manifest lists. When sha-amd64 is itself a manifest list (because of attestations), the stitching fails.

The fix: disable attestations with BUILDX_NO_DEFAULT_ATTESTATIONS=1.

Our first attempt was conditional:

# Broken: empty string "" fails strconv.ParseBool
BUILDX_NO_DEFAULT_ATTESTATIONS: ${{ inputs.multi_arch && '1' || '' }}

When multi_arch is false, this evaluates to "". Buildx tries to parse it as a boolean via Go’s strconv.ParseBool, which fails on empty string. This broke every non-multi-arch build across the organization.

The fix was trivial, always set it to 1:

BUILDX_NO_DEFAULT_ATTESTATIONS: 1

Attestations aren’t needed regardless of arch mode. An env var being set to empty is different from not being set at all: a subtle but organization-wide footgun.

Problem 2: PR number resolution

Each arch job publishes its image with a tag. We need a consistent tag so the manifest step can find both. The original design used pr_X-amd64 / pr_X-arm64, where the PR number came from an API call:

build-helper/src/utils.ts (before)

export async function get_pr(): Promise<number> {
 const {data: commit_prs} =
 await oktokit.rest.repos.listPullRequestsAssociatedWithCommit({
 owner: repo[0], repo: repo[1], commit_sha: sha
 })
 if (commit_prs[0] === undefined) {
 throw new Error('Commit has no PR')
 }
 return commit_prs[0].number
}

The listPullRequestsAssociatedWithCommit API sometimes returned empty results: race conditions, merge commits, event timing. When it failed, get_publish_tag() silently fell back to the raw SHA:

build-helper/src/utils.ts (silent fallback)

export async function get_publish_tag(): Promise<string> {
 try {
 return `pr_${await get_pr()}`
 } catch (e) {
 return process.env['GITHUB_SHA'] || '' // No warning!
 }
}

The amd64 job would tag as pr_42-amd64 (API succeeded), the arm64 job as abc123def (API failed). The manifest step looked for pr_42-arm64, which didn’t exist.

Two fixes:

Read PR number from the event payload, always available, zero API calls:

build-helper/src/utils.ts (after)

export async function get_pr(): Promise<number> {
 const con = new Context()
 if (con.payload.pull_request) {
 return con.payload.pull_request.number
 }
 // Fallback for push events only
 const {data: commit_prs} = await oktokit.rest.repos
 .listPullRequestsAssociatedWithCommit({...})
 // ...
}

Use commit SHA for arch-suffixed images, the SHA is always the same across parallel jobs:

build-helper/src/publish.ts

export async function docker(image: string, name: string): Promise<void> {
 const archSuffix = getArchSuffix()
 const sha = process.env['GITHUB_SHA'] || ''
 const tag = archSuffix ? `${sha}${archSuffix}` : await get_publish_tag()
 // ...
}

The manifest step bridges them:

build-helper/src/manifest.ts

const manifestTag = await get_publish_tag() // "pr_X"
const sha = process.env['GITHUB_SHA'] || ''
const archTags = ARCHITECTURES.map(arch => `${sha}-${arch}`)
await createAndPushManifest(name, manifestTag, archTags)

So the flow is: {sha}-amd64 + {sha}-arm64 → manifest pr_X. No API race, no silent fallback.

Problem 3: deploy ordering

The Leela admin service deploys on every PR build. It was wired inline in the build job:

- name: Deploy
 if: matrix.arch == 'amd64'
 uses: ./helper/.github/actions/deploy

Problem: in multi-arch mode, the deploy step ran before the manifest existed. It tried to pull pr_X, which only gets created in a separate create_manifest job that depends on both arch builds completing.

The fix adds a separate downstream job for multi-arch deploys:

.github/workflows/sonar_python_bender_build.yml

# Inline deploy only for single-arch
- name: Deploy
 if: matrix.arch == 'amd64' && !inputs.multi_arch
 uses: ./helper/.github/actions/deploy

# Multi-arch: deploy after manifest is ready
pr_deploy:
 if: inputs.multi_arch
 needs: create_manifest
 runs-on: ubuntu-24.04
 steps:
 - name: Deploy
 uses: ./helper/.github/actions/deploy

Same pattern for the release workflow: deploy only after create_release_manifest completes.

Problem 4: release version calculation

The createReleaseManifests function creates the multi-arch manifest for release tags. It originally called get_split_version_tag() to determine the version:

build-helper/src/release.ts (before)

const tag = await get_split_version_tag() // Fetches latest release, bumps version
const version = `${tag[0]}.${tag[1]}.${tag[2]}`

get_split_version_tag() fetches the latest GitHub release and bumps the version based on PR labels. But createReleaseManifests runs after the release jobs complete. The release already happened. So get_latest_tag() now returns v5.5.3 (the new release), and bumping it gives v5.5.4, which doesn’t exist as a tag.

The fix: just use get_latest_tag() directly.

build-helper/src/release.ts (after)

const version = await get_latest_tag()
const parts = version.split('.')
const archTags = ARCHITECTURES.map(arch => `${version}-${arch}`)

await createAndPushManifest(name, version, archTags) // v1.2.3
await createAndPushManifest(name, `${parts[0]}.${parts[1]}`, archTags) // v1.2
await createAndPushManifest(name, parts[0], archTags) // v1

Three manifest tags per release. Semver consumers can pin to major, minor, or patch.

The full tagging flow

After all the fixes, the image lifecycle looks like this:

PR Build:
{sha}-amd64, {sha}-arm64 → manifest: pr_X
Release (per-arch):
pull pr_X (Docker resolves correct arch)
push v1.2.3-amd64, v1.2.3-arm64
Release Manifests:
v1.2.3-amd64 + v1.2.3-arm64 → manifests: v1.2.3, v1.2, v1

Only the amd64 job creates the GitHub release to avoid duplicates:

build-helper/src/release.ts

if (!archSuffix || archSuffix === '-amd64') {
 await create_release(version)
} else {
 core.info(`Skipping GitHub release creation on ${archSuffix} build`)
}

Timeline

Feb 4: Initial multi-arch support. Matrix strategy, arch suffix, manifest creation.
Feb 5: Deploy ordering fix. Leela deploying before manifests existed.
Feb 16: Same fix for the release/merge workflow.
Feb 17: BUILDX_NO_DEFAULT_ATTESTATIONS to fix attestation-poisoned manifests.
Feb 18: SHA-based arch tagging + PR number from event payload.
Feb 19: BUILDX_NO_DEFAULT_ATTESTATIONS empty string crash fix.
Feb 19: get_latest_tag() instead of get_split_version_tag() in release manifests.

Five fixes in two weeks after the initial rollout. Each one discovered in production by a different failure mode.

What I’d do differently

Test the non-multi-arch path. The empty string strconv.ParseBool crash only affected repos that hadn’t opted into multi-arch. We tested the new feature, not the old path.

Use SHA tags from the start. The PR-number-based approach was elegant but fragile. The SHA is always consistent across parallel jobs, no API calls, no race conditions.

Make the manifest step a required downstream job. The deploy-before-manifest problem was obvious in retrospect. Any step that depends on a multi-arch manifest should be in a job that needs: the manifest creation job. No exceptions, no “it works for single-arch” shortcuts.

Docker’s multi-arch story is good once it works. Getting there from a single-arch CI that evolved organically over years is where the pain lives. Every assumption about “one build produces one image” breaks in interesting ways.

Migrating Python containers to Wolfi and uv

Tue, 10 Feb 2026 09:00:00 +0200

Our Python services ran on ubuntu:24.04 with pip-installed dependencies. It worked, but the images carried hundreds of packages we never used, Trivy scans were noisy with OS-level CVEs, and builds were slower than they needed to be. Over a couple of months we migrated to Chainguard’s Wolfi base image and uv for dependency management. This is how it went for the maps service, the one that renders static map tiles with a C++/Python hybrid stack.

Cleaning up dependency management with uv

Before touching the base image, we sorted out the dependency mess. The maps service had drifted. pyproject.toml declared one set of versions, uv.lock referenced another, and a private package (maparazzo) was pinned to a version that didn’t exist on the registry anymore.

The first PR synced the lock file with reality and bumped maparazzo to a published version. The second one removed version constraints from pyproject.toml entirely, keeping only the lock file as the source of truth:

pyproject.toml (before)

dependencies = [
 "application_kit[fastapi]",
 "fastapi>=0.115.0",
 "maparazzo==0.4.0",
 "aiosqlite>=0.20.0",
 "valkey-glide==2.2.5",
 "sniffio>=1.3.1",
 # Dev tools
 "types-redis>=4.6.0",
 "coverage[toml]>=7.6.0",
 "pytest>=8.3.0",
 "pytest-asyncio>=0.24.0",
 "pytest-cov>=6.0.0",
 "respx>=0.22.0",
 "ruff>=0.8.0",
 "mypy>=1.14.0",
]

pyproject.toml (after)

dependencies = [
 "application_kit[fastapi]",
 "maparazzo==2.0.0",
 "aiosqlite",
 "valkey-glide==2.2.5",
 "sniffio",
]

[dependency-groups]
dev = [
 "types-redis",
 "coverage[toml]",
 "pytest",
 "pytest-asyncio",
 "pytest-cov",
 "respx",
 "ruff",
 "mypy",
]

Two changes happened here. First, lower bounds like >=0.115.0 were dropped: the lock file already pins exact versions, so the constraints in pyproject.toml were just noise that could silently go stale. Second, dev tools moved to a proper [dependency-groups] section. This matters for the multi-stage build that comes next: uv sync --no-dev now skips pytest, ruff, mypy, and the rest. Previously they all shipped in the production image.

Running uv lock --upgrade after removing the constraints updated 23 packages in under 3 seconds. That’s the part of uv that makes the biggest day-to-day difference: resolving a 60-package graph is nearly instant.

The Ubuntu Dockerfile

Here’s what we had before:

Dockerfile (Ubuntu-based)

FROM ubuntu:24.04 AS base
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
ENV UV_PYTHON_INSTALL_DIR=/python
ENV UV_PYTHON_PREFERENCE=only-managed

COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

RUN uv python install 3.12

WORKDIR /usr/src/app
ENV UV_PROJECT_ENVIRONMENT=/usr/src/venv

RUN --mount=type=cache,target=/root/.cache/uv \
 --mount=type=bind,source=uv.lock,target=uv.lock \
 --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
 uv sync --frozen --no-install-project --no-dev

ADD . /usr/src/app

RUN --mount=type=cache,target=/root/.cache/uv \
 uv sync --frozen --no-dev

FROM ubuntu:24.04

ARG DEBIAN_FRONTEND="noninteractive"
ENV PACKAGES="ca-certificates libopengl0 libegl1 curl"

RUN apt-get update && apt-get install --no-install-recommends -y \
 ${PACKAGES} && apt-get clean && apt-get autoclean && apt-get autoremove \
 && rm -rf /var/lib/apt/lists/*

COPY --from=base --chown=python:python /python /python
COPY --from=base --chown=app:app /usr/src/venv /usr/src/venv
COPY --from=base --chown=app:app /usr/src/app /usr/src/app

WORKDIR /usr/src/app
ENV PATH="/usr/src/venv/bin:$PATH"

RUN python ./minify_styles.py

EXPOSE 8000

This was already using uv and multi-stage builds, which was good. But the production stage started from a fresh ubuntu:24.04, which pulls in apt, dpkg, systemd fragments, locales, and hundreds of other packages that a Python web service never touches. The apt-get update && apt-get install dance also meant the image wasn’t reproducible: two builds a week apart could get different package versions.

The COPY --from=ghcr.io/astral-sh/uv:latest was another problem. latest means the uv version changes silently. If a new uv release changes lock file behavior or introduces a bug, your CI breaks with no obvious diff.

Switching to Wolfi

Wolfi is a Linux distribution built specifically for containers. No package manager bloat, no shell login infrastructure, minimal base. Chainguard maintains it and publishes daily CVE-patched images.

The new Dockerfile:

Dockerfile (Wolfi-based)

FROM cgr.dev/chainguard/wolfi-base AS base

RUN apk add --no-cache \
 mesa-egl \
 mesa-gl \
 mesa-gbm \
 libgcc \
 libstdc++ \
 bash \
 ca-certificates-bundle

ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
ENV UV_PYTHON_INSTALL_DIR=/python
ENV UV_PYTHON_PREFERENCE=only-managed

COPY --from=ghcr.io/astral-sh/uv:0.9.18 /uv /uvx /bin/

RUN uv python install 3.12

WORKDIR /usr/src/app
ENV UV_PROJECT_ENVIRONMENT=/usr/src/venv

RUN --mount=type=cache,target=/root/.cache/uv,id=uv-base \
 --mount=type=bind,source=uv.lock,target=uv.lock \
 --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
 uv sync --frozen --no-install-project --no-dev

COPY . /usr/src/app

RUN --mount=type=cache,target=/root/.cache/uv,id=uv-base \
 uv sync --frozen --no-dev

RUN /usr/src/venv/bin/python ./minify_styles.py


FROM base AS build

RUN --mount=type=cache,target=/root/.cache/uv,id=uv-build \
 uv sync --frozen

ENV PATH="/usr/src/venv/bin:$PATH"
ENV RUFF_CACHE_DIR=/tmp/.ruff_cache
ENV MYPY_CACHE_DIR=/tmp/.mypy_cache
ENV COVERAGE_FILE=/tmp/.coverage


FROM base AS prod

ENV PATH="/usr/src/venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1

USER 65532

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
 CMD ["/usr/src/app/healthcheck.sh"]

A few things to unpack.

Three stages instead of two

The old Dockerfile had base (build everything) and a final production stage that copied artifacts. The new one has three:

Stage	Purpose	Packages
`base`	Install runtime deps, sync production packages	Runtime only
`build`	Extends base, adds dev deps for testing in CI	Runtime + dev
`prod`	Extends base, sets non-root user and healthcheck	Runtime only

The build stage is what CI uses to run pytest, ruff, and mypy. The prod stage is what ships. Both extend base, so there’s no duplication of the dependency installation step.

The build stage redirects tool caches to /tmp because the app directory is read-only for the non-root user:

Tool cache redirection for CI

ENV RUFF_CACHE_DIR=/tmp/.ruff_cache
ENV MYPY_CACHE_DIR=/tmp/.mypy_cache
ENV COVERAGE_FILE=/tmp/.coverage

Pinned uv version

uv:latest became uv:0.9.18. One line, but it eliminates an entire class of “works on my machine” issues. When we want to upgrade uv, we do it explicitly in a PR.

Non-root production

The prod stage runs as UID 65532, matching the convention from distroless images. The base stage still runs as root (needed for apk add and uv python install), but the production image drops privileges before the entrypoint.

Minification at build time

The old Dockerfile ran python ./minify_styles.py in the production stage at container startup (well, at build time of the final stage, but from a fresh Ubuntu). The new one runs it in the base stage right after syncing dependencies. This means the minified assets are baked in and shared by both build and prod.

Build cache isolation

The --mount=type=cache directives got id=uv-base and id=uv-build suffixes. Without these, Docker can share cache mounts between stages that run concurrently in BuildKit, which causes lock contention.

The s5cmd surprise

After deploying the Wolfi image, the tile generation job broke. It ran a shell script that downloaded s5cmd (an S3-compatible file transfer tool) at runtime:

tile.sh (runtime download, broken)

if ! [ -x "$(command -v s5cmd)" ]; then
 curl -L \
 https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz \
 > s5cmd.tar.gz
 tar -xzvf ./s5cmd.tar.gz --directory /usr/local/bin/
 rm ./s5cmd.tar.gz
fi

Two problems. First, curl wasn’t in the image anymore; we’d removed it during the Wolfi migration because the application code doesn’t need it. Second, even if curl was there, extracting to /usr/local/bin/ would fail because the container runs as non-root.

The fix was one line in the Dockerfile and deleting the download script:

Installing s5cmd via apk

RUN apk add --no-cache \
 mesa-egl \
 mesa-gl \
 mesa-gbm \
 libgcc \
 libstdc++ \
 bash \
 s5cmd \
 ca-certificates-bundle

Wolfi has s5cmd v2.3.0 in its package repository, newer than the v2.2.2 being downloaded. Installing it at build time is faster, more reliable, and gets security updates through the normal apk channel.

This is the kind of thing that falls out of a migration like this. Runtime downloads of binaries are a pattern that works on fat base images but breaks the moment you tighten things up. And they should break. Downloading unsigned tarballs into a running container is a supply chain risk hiding in plain sight.

Scan results

After the migration, Trivy against the production image:

Severity	Before (Ubuntu)	After (Wolfi)
CRITICAL	0	0
HIGH	3	0
MEDIUM	12	0
LOW	27	0

The Ubuntu image had 42 findings, all in OS packages we never used. The Wolfi image has zero because it only contains packages we explicitly installed. The Python dependency layer had 2 findings in both cases (a starlette version that needed bumping), but the OS layer went from noisy to clean.

Dockle (Dockerfile best-practices linter) reports one false positive: it flags settings.py as a potential credential file because of the filename. Suppressed with dockle -af settings.py.

What made it work

A few things that helped this go smoothly:

uv’s dependency groups. Separating dev tools from runtime dependencies in pyproject.toml is what makes --no-dev actually useful. Before, pytest and ruff shipped in production because they were in the same flat dependency list.

Wolfi’s package repository. Having s5cmd, Mesa GL libraries, and a recent bash available via apk meant we didn’t need to maintain custom installation scripts. The packages are rebuilt daily with CVE patches.

The three-stage pattern. base splitting into build and prod is now our standard for Python services. CI targets build, production targets prod, and both share the same dependency installation.

Pinning uv. Sounds trivial but it removed a real source of non-determinism. uv:latest in a Dockerfile is the same antipattern as pip install package without a version: it works until it doesn’t, and you won’t know why.