Repository navigation
feat(orchestrator): delay upgrades until the local checkpoint at the CUP height is verified - #12093
Closed
alin-at-dfinity wants to merge 1 commit into
Closed
alin-at-dfinity wants to merge 1 commit into
alin-at-dfinity wants to merge 1 commit into
Conversation
…CUP height is verified An upgrade can stop the replica before it has verified its checkpoint at the upgrade CUP's height; the restarted replica then archives it and state-syncs, re-reading and rehashing its whole local state. Before executing an upgrade, the orchestrator now waits for the local checkpoint at the CUP height to be verified, for at most 1 minute, except for unsigned CUPs (genesis, recovery) and when the replica isn't running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block approval of the checkpoint-verification upgrade delay.
No open issues 🎉
What changed in this PR
This PR reduces unnecessary state sync after replica upgrades by giving the local checkpoint at the upgrade CUP height time to finish verification.
Changes:
- Adds a checkpoint verification gate for application and NNS upgrades, with a one-minute timeout and exemptions for unsigned CUPs or a stopped replica.
- Adds a read-only checkpoint status helper and records delayed upgrades in the existing metric.
- Tests missing, unverified, and verified checkpoints, timeout behavior, and exemptions.
| File | Description |
|---|---|
rs/state_layout/src/state_layout/tests.rs |
Tests checkpoint verification status, including state-sync markers. |
rs/state_layout/src/state_layout.rs |
Adds a filesystem-only checkpoint verification helper. |
rs/orchestrator/src/upgrade.rs |
Implements and tests the bounded upgrade delay. |
rs/orchestrator/src/processes.rs |
Makes replica running status available outside tests. |
rs/orchestrator/Cargo.toml |
Adds the state-layout dependency for Cargo. |
rs/orchestrator/BUILD.bazel |
Adds the state-layout dependency for Bazel. |
Cargo.lock |
Records the orchestrator’s new dependency. |
Review effort: Balanced
Contributor
Author
|
Closing in favour of #12094, which makes the replica itself wait for its checkpoint's verification on any stop, upgrades included. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An upgrade can stop the replica before it has verified its checkpoint at the upgrade CUP's height. The CUP needs only 2f+1 shares, so a slower node can get it from a peer or the registry mid-verification. On restart, the replica then archives the unverified checkpoint and state-syncs instead, re-reading and rehashing its whole local state: ~1 TB in ~4 minutes on a large mainnet subnet, where an estimated 26–36% of upgrades lose this race on at least one node.
Before
execute_upgrade, the orchestrator now waits until the local checkpoint at the CUP height is verified (itsunverified_checkpoint_markeris gone), for at most 1 minute from the first delay at that height, then upgrades anyway. The replica is already halted at the CUP height, so waiting costs nothing.replica_version_upgrade_prevented{reason="checkpoint_not_verified"}.ic_state_layout::is_checkpoint_verified), with the state root from the replica config.Tests: the new state layout function; and the orchestrator delaying for a missing or an unverified checkpoint and then upgrading once it is verified or after the timeout (NNS and app subnet), plus both exemptions.
Complements #12094, which makes the replica itself wait for verification when stopped.
🤖 Generated with Claude Code