[LinZhihao-723]

Description

Motivation

Spider installs its Rust toolchain (stable and nightly, plus clippy, rustfmt, and cargo-nextest) into its build directory using a task guarded by checksum:validate/checksum:compute. The checksum is meant to make re-runs cheap: if the installed toolchain hasn't changed, the installation is skipped. However, checksum:compute builds a full tar archive of the include paths and pipes it through md5sum, so every validation reads and hashes the content of every file. For the Rust toolchain, this is ~3.07 GB across ~132k files.

Since every Rust lint/build/test task depends on the toolchain task, every invocation pays this cost, even when all the actual work (e.g., cargo fmt --check) takes less than a second. For task lint:check-rust-format in Spider:

• With a warm page cache, the checksum validation takes ~3.6 s of the 4.7 s total (~76%); the bottleneck is md5sum over 3 GB.
• With a cold page cache, it takes ~22 s of the 23 s total (~96%); the bottleneck is reading 3 GB from disk. This is the common case on dev machines where the page cache is reclaimed between runs (e.g., WSL2 with autoMemoryReclaim enabled).

Switching to a faster hash doesn't help much: b2sum saves ~0.7 s when warm and nothing when cold, since the disk I/O dominates.

Solution

This PR adds an opt-in METADATA_ONLY parameter to checksum:compute and checksum:validate. When set to "true", the checksum is computed over the metadata of the include paths instead of their contents:

TZ=UTC LC_ALL=C tar --create --file /dev/null --full-time --verbose --verbose ... | md5sum

Why this works:

• When the archive is /dev/null, GNU tar skips reading file contents entirely (an optimization intended for archive size estimation). It only walks the directories and stats each entry. strace confirms this: on the Rust toolchain, it issues 19 read calls (21 KB, all from shared libraries and locale files), compared with ~405k read calls for the content-based checksum.
• With --verbose --verbose --full-time, tar still prints a long listing of every entry it would have archived, in the same --sort name order and subject to the same include/exclude patterns. Each line contains the entry's type, mode, owner/group (normalized to 0/0 by the existing --owner/--group flags), size, mtime with nanosecond precision, path, and symlink/hard link target. Hashing this listing detects added, removed, and renamed files, permission changes, and any modification that updates the size or mtime.
• TZ=UTC LC_ALL=C makes the listing (and therefore the checksum) independent of the local time zone and locale, since tar renders timestamps in local time.
• --mtime "UTC 1970-01-01" is dropped in this mode, since the real mtime is the main change signal.

Trade-offs:

• A content change that preserves both the file size and mtime is not detected. This is acceptable for large, tool-managed outputs (e.g., toolchains and virtual environments) whose contents only change through their installers.
• Conversely, a change to only a file's mtime (e.g., touch) now invalidates the checksum, whereas it doesn't in content mode.

The default remains content-based, so existing callers are unaffected. Like content mode, metadata mode relies on GNU tar (gtar on macOS).

Performance

Measured in Spider on the Rust toolchain (~3.07 GB, ~132k files), median of 3 runs. "Cold" means the toolchain's file contents were evicted from the page cache before each run.

	Content-based (before)	Metadata-only (after)
Checksum validation, warm page cache	~3.6 s	~1.0 s
Checksum validation, cold page cache	~22 s	~1.0 s
task lint:check-rust-format end to end, warm page cache	4.71 s	1.81 s
task lint:check-rust-format end to end, cold page cache	22.97 s	1.96 s

The metadata-only time no longer depends on the page cache, since no file contents are read.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Added unit tests:
  • checksum-test-metadata-only-skip: An unchanged directory is skipped in metadata-only mode.
  • checksum-test-metadata-only-rerun-on-mtime-change: Changing only a file's mtime triggers a rerun in metadata-only mode.
  • checksum-test-skip-on-mtime-change: Changing only a file's mtime is still skipped in content mode (the default).
• task tests:checksum and task tests:remote pass.
• task lint:check-yaml passes.
• Verified that, with METADATA_ONLY unset, the checksum is identical to the one computed before this change.
• Applied this change locally in Spider (METADATA_ONLY: "true" for the Rust toolchain checksum):
  • Measured the performance numbers above through task lint:check-rust-format.
  • Verified with strace that no file contents in the toolchain are read.
  • Verified that touching a file in the toolchain triggers a reinstallation, and that the next run skips it again.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 83eb7ae9-573c-48b0-9ccc-b8dd471b1871

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sitaowang1998]

The code lgtm. I think we need to get an approval from Kirk for high level ides.

[kirkrodrigues]

The code lgtm. I think we need to get an approval from Kirk for high level ides.

Idea makes sense to me.