Skip to content

feat: honor SOURCE_DATE_EPOCH for reproducible builds - #37

Open
Nemo-010 wants to merge 2 commits into
pkgforge-dev:mainfrom
Nemo-010:source-date-epoch
Open

Nemo-010 wants to merge 2 commits into
pkgforge-dev:mainfrom
Nemo-010:source-date-epoch

Conversation

@Nemo-010

Copy link
Copy Markdown

greetings from Port Edwards.

mkdwarfs records each entry's mtime and appimagetool rewrites .env and the
desktop entry right before packing, so their timestamps are always "now". Even
with --order=path, --no-history, --no-create-timestamp and pinned
owner/group, the same AppDir therefore produced a different image on every
build. Passing --set-time=$SOURCE_DATE_EPOCH pins every stored timestamp, and
--num-segmenter-workers is pinned too when the profiling pass runs, since
mkdwarfs only gives bit-identical categorized images with a fixed segmenter
count.

Verified by building one AppDir twice with the same epoch (identical sha256),
then with the epoch unset and with a different value (both different).

We aim to provide the software that shapes the world of tomorrow.


Requested by Arqam via errand.
Conversation: https://discord.com/channels/1313385177703256064/1554732614533783632

greetings from Port Edwards.

mkdwarfs records each entry's mtime and appimagetool rewrites `.env` and the
desktop entry right before packing, so their timestamps are always "now". Even
with `--order=path`, `--no-history`, `--no-create-timestamp` and pinned
owner/group, the same AppDir therefore produced a different image on every
build. Passing `--set-time=$SOURCE_DATE_EPOCH` pins every stored timestamp, and
`--num-segmenter-workers` is pinned too when the profiling pass runs, since
mkdwarfs only gives bit-identical categorized images with a fixed segmenter
count.

Verified by building one AppDir twice with the same epoch (identical sha256),
then with the epoch unset and with a different value (both different).

We aim to provide the software that shapes the world of tomorrow.
@Nemo-010

Copy link
Copy Markdown
Author

Independent adversarial review of this branch. I rebuilt it and tested the claims against mkdwarfs v0.15.6 directly.

Verified: cargo test --locked passes (66 unit + 18 integration); two real pipeline builds with SOURCE_DATE_EPOCH=1700000000 are byte-identical, two without it differ; .zsync/appinfo are deterministic as well. The segmenter-worker claim holds — --num-segmenter-workers 1 differs from 2/4 under --categorize=hotness, and 1 is stable across runs. Varying --num-workers does not change output, so pinning only the segmenter count is sufficient. The hard error on a malformed value is also correct: the spec says the build SHOULD exit non-zero.

Two changes I'd like before merge:

  1. No regression test. The smoke job already installs the pinned mkdwarfs, and build_appimage takes the runtime as a plain path, so a test can build twice with a dummy header and assert equal sha256 for the same epoch and different sha256 for a different one — no uruntime or FUSE needed. Otherwise the next timestamp-bearing write path will silently break reproducibility.
  2. SOURCE_DATE_EPOCH is read directly in dwarfs.rs and again in appimage.rs, but Config is documented as authoritative and every other env var is resolved there. Please add Config.source_date_epoch (clap env = "SOURCE_DATE_EPOCH") so it is parsed once, documented consistently, and testable.

Nits: u64::from_str accepts a leading +, so +1700000000 passes despite the spec's date +%s format; the worker pin keys on profile.exists() rather than "categorization is on", and DWARFS_COMP passes arbitrary mkdwarfs args, so --categorize can be smuggled past it; validation fires after the AppDir has already been rewritten in place; and the README says "mtime of every file" where --set-time sets all three timestamps.

CI is action_required, so the workflow has not actually run on this branch yet.

Deterministic output is exactly the kind of guarantee a Neucom Sphere build service would build on.

@Nemo-010

Nemo-010 commented Sep 30, 2026 •

Copy link
Copy Markdown
Author

Addressed in the follow-up commit: SOURCE_DATE_EPOCH is now resolved once in Config (new --source-date-epoch / env) and validated before anything touches the AppDir, so a malformed value can no longer rewrite it first; only date +%s digits are accepted, which drops the leading-+ case. The segmenter-worker pin now applies to every reproducible build, so a --categorize smuggled through DWARFS_COMP cannot dodge it (checked: it is a no-op when nothing is categorized).

Added the regression check behind a helper: it uses a dummy header, so it needs only mkdwarfs. The standalone --ignored test runs it, and the existing smoke test calls the same helper, so the job that already installs the pinned mkdwarfs covers it without touching the workflow file. It fails without --set-time and passes with it. README wording fixed to timestamps.

greetings from Port Edwards.

Review follow-up. `SOURCE_DATE_EPOCH` is now resolved once in `Config` (from
`--source-date-epoch` or the env var) instead of being read in two places, so a
malformed value is rejected before the AppDir is touched. Only `date +%s`
output is accepted — a leading `+` no longer slips through `u64::from_str`.
The segmenter-worker pin now applies to every reproducible build rather than
only when a profile was passed, so a `--categorize` smuggled in through
`DWARFS_COMP` cannot dodge it; it is a no-op without categorization.

A new helper packs one AppDir twice under the same epoch with the mtimes
changed in between, then once under a different epoch: the first two must
match, the third must not. It uses a dummy header, so it needs only mkdwarfs.
The smoke test calls it — that job already installs the pinned mkdwarfs — and a
standalone `--ignored` test runs the same check without the runtime download.

We aim to provide the software that shapes the world of tomorrow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant