Skip to content

perf(unpack): poll for the build artifact under a deadline - #115

Merged
deanq merged 1 commit into
mainfrom
deanq/sls-812-unpack-poll
Oct 19, 2026
Merged

deanq merged 1 commit into
mainfrom
deanq/sls-812-unpack-poll

Conversation

@deanq

@deanq deanq commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

When the Flash build artifact was not on the volume at container start, maybe_unpack() slept a fixed 30s (DEFAULT_TARBALL_UNPACK_INTERVAL) before retrying, even if the artifact landed a moment later.

  • Retry every 0.5s until a 60s deadline. The worst case is unchanged: 3 attempts with 30s between them was 60s.
  • Extraction errors are retried too, since a partially written tarball fails to extract.
  • Logging: one warning when the wait starts, one info line with the wait time on recovery, and one error at the deadline. Before, every attempt logged an error with a traceback; logging every poll would mean up to 120 lines.
  • DEFAULT_TARBALL_UNPACK_ATTEMPTS and DEFAULT_TARBALL_UNPACK_INTERVAL are replaced by DEFAULT_TARBALL_UNPACK_TIMEOUT and DEFAULT_TARBALL_UNPACK_POLL_INTERVAL. Nothing else referenced the old names.

Impact

Measured cold start on an RTX 4090 (SLS-811), with container start at t=0:

Event t
first attempt: flash build artifact not found 1.3s
second attempt succeeds 33.4s
result posted 38.5s

About 32s of the ~38s in-container start was this sleep. With this change, a late artifact is picked up within 0.5s of landing.

Test plan

  • New: an artifact missing on the first attempt and present on the second is unpacked after ≤1s of waiting (fails on main: waits 30s)
  • New: a missing artifact is retried until the 60s deadline, then raises with the last error chained
  • Existing failure tests moved to a fake clock; the logging test asserts one warning and one error
  • make quality-check: 295 passed, 78.49% coverage, 14/14 local handler tests
  • Confirm on a real worker after the image is published: the log shows artifact unpacked after Xs of retries instead of a 30s gap

Linear: SLS-812 (related: SLS-811)

When the artifact was not on the volume at container start, maybe_unpack
slept a fixed 30s before retrying, even if the artifact landed a moment
later. On a measured RTX 4090 cold start this was ~32s of a ~38s
in-container start.

- Retry every 0.5s until a 60s deadline (same worst case as 3 x 30s)
- Retry extraction errors too; a partially written tarball fails to extract
- Log the wait once and the failure once, instead of an error per attempt

Linear: SLS-812

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused retry change and updated tests have no identified blocking issues.

No open issues 🎉

What changed in this PR

Reduces worker startup delays when build artifacts arrive late on the mounted volume.

Changes:

  • Replaces fixed retries with 0.5-second polling under a 60-second deadline.
  • Consolidates retry logging.
  • Adds fake-clock tests for timeout handling and prompt recovery.
File Description
tests/​unit/​test_unpack_volume.py Tests deadline handling, late artifacts, and logging.
src/​unpack_volume.py Implements deadline-based polling and consolidated logging.
src/​constants.py Defines the retry timeout and polling interval.

Review effort: Balanced

@deanq
deanq merged commit ef3f8ea into main Oct 19, 2026
26 checks passed
@deanq
deanq deleted the deanq/sls-812-unpack-poll branch October 19, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants