Repository navigation
fix(oci): convert layer whiteouts into overlayfs whiteouts - #283
Merged
Merged
Conversation
mkfs.erofs --tar=f without --aufs keeps OCI whiteout entries (.wh.<name>, .wh..wh..opq) as plain empty files, so the guest overlayfs never hides the paths a later layer deletes: they reappear in every VM built from such an image, together with the .wh.* files themselves. Pass --aufs (available in the enforced erofs-utils >= 1.8) so whiteouts become 0:0 char devices and opaque markers become overlay opaque dirs. Blobs converted before this change are cached by layer digest and would keep being reused, so blob keys now hash the conversion format with the layer digest: pulls and imports miss the stale blobs and convert again, a pull of an unchanged manifest no longer counts as up to date, images and VMs still referencing old blobs keep working, and the existing image GC collects the old blobs once nothing references them.
Two imports of the same tar convert with different filesystem UUIDs, and the second commit's rename replaced the blob the first had published, so a VM created from the first image later opened a different file. Under the digest lock, a valid blob at the destination now wins and the temp blob is left to the work dir cleanup.
start.sh ran the same mkfs.erofs --tar=f conversion without --aufs, so the local harness kept the whiteout bug; Ubuntu 24.04's erofs-utils 1.7.1 already supports --aufs. TestBlobHex checked sha256 properties instead of the derivation, so a silent re-key of every blob passed; it now pins a golden value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every OCI layer tar is converted with
mkfs.erofs --tar=f ...and the guest stacks the blobs as overlayfs lowerdirs. Without--aufs, mkfs.erofs keeps OCI whiteout entries (.wh.<name>,.wh..wh..opq) as plain empty regular files, which overlayfs does not treat as whiteouts. So any path deleted by a later layer reappears in the VM, along with the.wh.*files themselves. This hits bothcocoon image pullandcocoon image import(both go throughrunErofsConversion).Fix
Pass
--aufsto mkfs.erofs. File whiteouts and dir whiteouts become 0:0 char devices, and opaque markers become overlay opaque directories.--aufsis available in the enforced erofs-utils >= 1.8 floor.Cache invalidation: blobs are cached by layer digest, so already-converted (broken) blobs would keep being reused. The blob key is now
sha256("aufs:" + layerDigestHex)(blobHexinimages/oci/erofs.go), used by both pull and import:isUpToDatealso requires the entry's layer keys to match the expected keys, so re-pulling an unchanged manifest converts again instead of reporting "Already up to date";A new blob extension or path was not used: GC scans
*.erofsstems and removesBlobPath(hex), andhypervisor.BlobHexFromPathtrimsfilepath.Ext. Old blobs would leak, or VM pins would break.Commit keeps an already published blob.
cocoon image importderives the temporary filesystem UUID from the label, so two imports of the same tar produce different bytes. The second commit's rename replaced the blob the first had published, and a VM created from the first image later opened a different file. Under the digest lock, a valid blob at the destination now wins, and the temp blob is left to the work dir cleanup.os-image/start.shruns the same conversion, so it passes--aufstoo.Hot-path cost: one SHA-256 over ~70 bytes per layer during pull/import; nothing on VM create/start.
Upgrade notes
VerifyBaseFileswith a missing base layer. This is intended: the snapshot's guest memory was captured against the old layer content.Tests
TestRunErofsConversionWhiteoutsconverts a layer containing a file whiteout, a dir whiteout and an opaque marker, then checks the result withdump.erofs --ls.CI skips this test: ubuntu-latest has no erofs-utils, and Ubuntu 24.04's packaged 1.7.1 is below the 1.8 floor. The evidence for it is the Linux container run below.
TestBlobHexpins the blob key derivation with a golden value.TestCommitBlobsKeepsPublishedBlobcommits two different temp blobs under one key and checks that the first one stays published.isUpToDategets a case where the stored entry is keyed by raw layer digests (pre-fix image), which must not count as up to date.Evidence
Unit test in a Linux container (golang:1.27-trixie + erofs-utils)
Negative control, same container, with
"--aufs"removed from the args:Whole package at the current head, same container image:
start.shflags on Ubuntu 24.04's packaged erofs-utilsubuntu:24.04withapt-get install erofs-utils. The layer has.wh.b,a/.wh.x,c/.wh..wh..opqandc/keep, and is converted with the exactstart.shflags (--tar=f --aufs -zlz4hc -C16384 -T0 -U <uuid>), through both the gzip pipe and the plain tar path:End to end on a Linux KVM host (erofs-utils 1.9.4, Cloud Hypervisor)
Isolated
COCOON_ROOT_DIR/COCOON_RUN_DIR/COCOON_LOG_DIR. One binary is built from this branch, the other from masterb3b40f3.crane export ghcr.io/cocoonstack/cocoon/ubuntu:24.04(flattened rootfs).usr/share/.wh.doc(dir whiteout),etc/.wh.issue.net(file whiteout),etc/update-motd.d/.wh..wh..opq(opaque) andetc/update-motd.d/99-whiteout-test(new file).This branch:
master
b3b40f3:Upgrade path: the master binary converted the tars first, then this branch's binary ran against the same root.
Pull path (
ghcr.io/cocoonstack/cocoon/ubuntu:24.04): master pulled first, then this branch pulled the same manifest digest.The third pull reports up to date, and the image boots.
Gates (
GOWORK=off)make test(vet + race) in the Linux container with erofs-utils 1.8.6:meta/json, which hit the Makefile's 120 s timeout in its storm test at the default worker count.