Skip to content

bugfix: a release can run different image versions on amd64 and arm64 - #28

Merged
csandanov merged 2 commits into
masterfrom
fix/pin-arch-image-digests
Nov 18, 2026
Merged

csandanov merged 2 commits into
masterfrom
fix/pin-arch-image-digests

Conversation

@csandanov

@csandanov csandanov commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Problem

A release of wodby/python can publish a multi-arch image whose amd64 and arm64 images come from different builds, so the two architectures run different versions. This happened to wodby/mariadb r19, which shipped MariaDB 10.11.19 on amd64 and 10.11.20 on arm64. This repository publishes the same way.

Cause

Each architecture is built on its own runner and pushed to a shared per-architecture tag such as <version>-amd64. The publish step then joins whatever those tags hold. Release tag runs often start alongside master runs, and every run pushes the same per-architecture tags. When runs overlap, one run can replace a tag just before another run publishes, so the release picks up the other run's image.

Solution

Each build now remembers the exact image it pushed, and the publish step joins those remembered images. Another run's push can no longer end up in a release. If this run did not push an image, publishing stops with an error instead of using the shared tag. Pushes to the same branch also wait for each other, so an older master run cannot finish after a newer one and move floating tags back. Image names, tags, and local make targets stay the same.

Changes

  • Build jobs push with wodby/images/.github/actions/arch-image-push (from bugfix: multi-arch image releases can run different versions on amd64 and arm64 images#71) instead of make push. It runs docker push on the same make image-ref image that was tested and scanned, then records the resulting digest.
  • Publish jobs run wodby/images/.github/actions/arch-image-pins before ./.github/actions. Makefile resolves each buildx imagetools create source through the new arch-image lookup, which fails when this run recorded no digest for the source.
  • Workflow-level concurrency: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} with cancel-in-progress: false. Tag runs are unaffected because each tag is its own group. The event is part of the group because GitHub replaces a pending run with a newer one in the same group, and a scheduled or manual run must not replace a queued push run.

Validation

  • actionlint passes.
  • The wodby/images publisher check (tests/check_image_publishers.py from bugfix: multi-arch image releases can run different versions on amd64 and arm64 images#71) renders every release case with records and confirms that each source becomes its recorded wodby/python@sha256:… digest. It also confirms that a missing record stops publishing. Without ARCH_IMAGE_PINS, the sources stay the per-architecture tags.
  • The pushed image is the one the build step tested and scanned: it is make image-ref, which prints $(REPO):$(TAG), the same reference make push pushed, with the same step environment.
  • The pull request build runs the build, test, and scan jobs and resolves the pinned actions. Pushing and publishing run only on master and tags.
  • After the first master run, check that the publish log combines wodby/python@sha256:… sources and that every architecture of each published tag reports the same version.

Deployment

Merge wodby/images#71 first. The pin references its branch commit, which a merge commit keeps reachable.

Every workflow run pushes the same per-architecture tags, and the publish
job combined whatever they held, so an overlapping run could replace one
architecture before a release was published. wodby/mariadb r19 shipped
different MariaDB versions on amd64 and arm64 this way.

Build jobs now push through wodby/images arch-image-push, which records
the pushed digest. Publish jobs load those records with arch-image-pins,
and the Makefile resolves each buildx imagetools create source to its
recorded digest, failing when this run did not push it. A per-ref
concurrency group keeps runs of one branch or tag from overlapping, so an
older master run cannot finish last and move floating tags back.
GitHub replaces a pending run with a newer one in the same concurrency
group. With only the ref in the group, a scheduled or manual run on master
could replace a queued push run, and that commit would never be built.
The group now includes the event name, so only a newer run of the same
event supersedes a pending one. This also restores the blank line between
jobs that the previous commit dropped after the push step.
@csandanov
csandanov merged commit 664ab0d into master Nov 18, 2026
27 of 33 checks passed
@csandanov
csandanov deleted the fix/pin-arch-image-digests branch November 18, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant