[codeandsolder]

Fixes #2819.

Summary

Multi-level cache writes and backfills can outlive the request that started them. If those tasks are detached without an explicit shutdown boundary, the runtime can cancel accepted cache work before slower cache levels have finished.

This change:

• tracks detached multi-level cache work and drains it before the runtime exits;
• limits concurrent background operations targeting L1+ with slow_write_concurrency (default 4, minimum 1);
• validates that limit consistently for file and environment configuration;
• adds a no-op-by-default Storage::drain_background() hook so shutdown does not need to know the concrete storage type;
• keeps the shutdown RPC alive while active clients and background cache work drain;
• acknowledges shutdown only after that drain completes, then lets the shutdown response finish before the runtime is dropped.

Background-task accounting uses a constant-time atomic pending counter plus Notify, rather than retaining and repeatedly scanning a growing list of JoinHandles. This keeps bookkeeping bounded even when many writes are waiting on the slow-level semaphore.

Panicking background tasks still release their pending-task accounting, so shutdown cannot wait forever on a task that has already terminated.

As a result, --stop-server does not return before accepted background cache work reaches its shutdown durability boundary.

The existing client-request shutdown timeout and the cache-task drain remain separate phases. A stalled background cache operation is still bounded by the daemon's shutdown timeout.

Testing

• cargo +1.99.0 fmt --all -- --check
• focused Rust 1.99 regressions for:
  • slow-level write concurrency and drain;
  • backfill tracking through shutdown;
  • closed slow-level semaphore behavior;
  • background-task panic accounting;
• Mozilla's Rust 1.91 CI-style Clippy command:
  • cargo +1.91.0 clippy --locked --all-targets -- -D warnings -A unknown-lints -A clippy::type_complexity -A clippy::new-without-default
• full Mozilla CI matrix green on the final head.

AI Disclosure: AI developed, human directed and reviewed

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.63548% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.43%. Comparing base (d72ff0b) to head (a1d593a).

Files with missing lines	Patch %	Lines
src/server.rs	96.87%	4 Missing ⚠️
src/config.rs	94.33%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2898      +/-   ##
==========================================
+ Coverage   78.13%   78.43%   +0.29%     
==========================================
  Files          74       74              
  Lines       43570    44020     +450     
==========================================
+ Hits        34043    34525     +482     
+ Misses       9527     9495      -32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.