You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(replica): wait for the latest checkpoint's verification on shutdown - #12094
On SIGTERM, the replica's main returns without waiting for anything. The sandbox launcher, in the same process group and cgroup, gets the same SIGTERM and dies, which makes the replica abort. A checkpoint still being verified is abandoned. On restart, the replica archives it and state-syncs instead, re-reading and rehashing its whole local state: ~1 TB in ~4 minutes on a large mainnet subnet. Upgrades and reboots stop the replica this way. Upgrades lose this race regularly: the upgrade CUP needs only 2f+1 shares, so a slower node can get it while still verifying its checkpoint at that height. An estimated 26–36% of upgrades of large subnets lose it on at least one node.
With this change:
Replica: after SIGTERM or SIGINT, it waits until its latest checkpoint is verified, checking the file system every 100 ms, then shuts down as before. A watchdog exits with status 1 after 70 s. It doesn't wait for a checkpoint that came from state sync, which may stay unverified until superseded.
Launcher, canister and compiler sandboxes: they ignore SIGTERM and SIGINT, which the replica handles, and exit when their socket to the replica reaches EOF, i.e. once the replica has exited. The compiler sandbox first finishes any compilation in progress. As this is in the shared sandbox entry points, sandboxes started by PocketIC, ic-replay, ic-recovery and the fuzzers also ignore both; they still exit on EOF. PocketIC handles Ctrl-C by deleting its instances, so that shutdown now runs with live sandboxes, like the replica's.
Orchestrator: it waits 75 s instead of 30 s after SIGTERM before sending SIGKILL, longer than the replica's 70 s. Both are below the 90 s systemd allows at reboot.
Testnet check: a single-node app subnet with ~5,000 canisters and 1.2 TB of state, at a typical load, stopped with systemctl stop ic-replica (the testnet's own orchestrator, with its 30 s grace period):
Right after checkpoint 7500 was created, before it was verified: the replica waited 1.0 s for its verification, shut down and exited with status 0 after 17 s. No launcher or sandbox process was left, and on restart nothing was archived.
With no checkpoint pending: no wait, exit status 0 after 1.6 s.
The watchdog's 70 s cover the wait and the shutdown together: the shutdown took 16 s in the check above, and validating a checkpoint took 35–39 s on the same testnet with 200k canisters, under load. If both together take longer, the replica exits with status 1 and the checkpoint is archived on restart, as without this change.
Tests: the new latest_checkpoint_status (five cases); and the launcher and both sandboxes, started for real, surviving SIGTERM and SIGINT and exiting 0 on EOF (Linux only; with the default SIGTERM action, all three fail).
On SIGTERM, the replica returned at once and the sandbox launcher, getting the
same SIGTERM, died, aborting the replica and abandoning the verification of
its latest checkpoint, which the next start then archived and state-synced.
The replica now waits until its latest checkpoint is verified, with a 30 s
watchdog, before shutting down. The launcher and sandboxes ignore SIGTERM and
exit once the replica's socket closes. The orchestrator waits 45 s instead of
30 s before escalating to SIGKILL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Process-group SIGINT still kills child processes and triggers a replica-aborting panic that bypasses the checkpoint-verification wait, while the unguarded /proc-based test fails on macOS.
This PR reduces checkpoint loss during replica shutdown by waiting for the latest local checkpoint to finish verification. It complements the orchestrator’s upgrade-time verification gate in #12093.
Changes:
Adds checkpoint-status lookup and shutdown polling, bounded by a 30-second watchdog.
Makes sandbox processes ignore SIGTERM and rely on socket closure for shutdown; extends the orchestrator’s grace period to 45 seconds.
Adds checkpoint-status and child-process shutdown tests.
File
Description
rs/state_layout/src/state_layout/tests.rs
Tests latest-checkpoint selection and status detection.
rs/state_layout/src/state_layout.rs
Adds filesystem-based latest-checkpoint status lookup.
rs/replica/Cargo.toml
Adds the state-layout dependency.
rs/replica/BUILD.bazel
Adds the matching Bazel dependency.
rs/replica/bin/replica/main.rs
Adds verification waiting and a shutdown watchdog.
…tor's SIGKILL
Verifying a checkpoint takes ~31 s with 200k canisters, and shutting down took
16 s on a 1.2 TB testnet state, so 30 s could cut the wait short. Both limits
stay below the 90 s systemd allows at reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Process-group SIGINT can still abort the replica before checkpoint verification finishes, and the new procfs-dependent test fails on macOS without a Linux gate or portable readiness check.
…inux only
The replica handles SIGINT like SIGTERM, but a process-group SIGINT (e.g.
Ctrl-C) still killed the launcher and sandboxes, making the replica abort. The
test reads `/proc`, so it now runs only on Linux, like this module's other
`/proc`-based tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The watchdog’s synchronous error logging can block on output or a shared drain mutex before reaching process::exit(1), defeating the 55-second deadline.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Avoid blocking logging before watchdog process exit
rs/replica/bin/replica/main.rs:92
The watchdog can block before reaching process::exit(1): Error-level logging writes synchronously through a shared drain mutex (rs/monitoring/logger/src/lib.rs:62–91). If the output destination blocks or a stuck log write holds that mutex, this diagnostic defeats the 55-second deadline. Remove the synchronous diagnostic and its unused import/logger argument, or use a genuinely nonblocking diagnostic path.
…tor's SIGKILL
Validating a checkpoint took 35–39 s on a 1.2 TB testnet state with 200k
canisters, under load, and shutting down took 16 s, so 55 s could still cut the
wait short. Both limits stay below the 90 s systemd allows at reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because synchronous watchdog logging can block the forced exit, undermining the replica’s 70-second shutdown deadline.
No open issues 🎉
Review effort: Balanced
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On SIGTERM, the replica's
mainreturns without waiting for anything. The sandbox launcher, in the same process group and cgroup, gets the same SIGTERM and dies, which makes the replica abort. A checkpoint still being verified is abandoned. On restart, the replica archives it and state-syncs instead, re-reading and rehashing its whole local state: ~1 TB in ~4 minutes on a large mainnet subnet. Upgrades and reboots stop the replica this way. Upgrades lose this race regularly: the upgrade CUP needs only 2f+1 shares, so a slower node can get it while still verifying its checkpoint at that height. An estimated 26–36% of upgrades of large subnets lose it on at least one node.With this change:
ic-replay,ic-recoveryand the fuzzers also ignore both; they still exit on EOF. PocketIC handles Ctrl-C by deleting its instances, so that shutdown now runs with live sandboxes, like the replica's.Testnet check: a single-node app subnet with ~5,000 canisters and 1.2 TB of state, at a typical load, stopped with
systemctl stop ic-replica(the testnet's own orchestrator, with its 30 s grace period):The watchdog's 70 s cover the wait and the shutdown together: the shutdown took 16 s in the check above, and validating a checkpoint took 35–39 s on the same testnet with 200k canisters, under load. If both together take longer, the replica exits with status 1 and the checkpoint is archived on restart, as without this change.
Tests: the new
latest_checkpoint_status(five cases); and the launcher and both sandboxes, started for real, surviving SIGTERM and SIGINT and exiting 0 on EOF (Linux only; with the default SIGTERM action, all three fail).🤖 Generated with Claude Code