Skip to content

feat(orchestrator): delay upgrades until the local checkpoint at the CUP height is verified - #12093

Closed
alin-at-dfinity wants to merge 1 commit into
masterfrom
alin/orchestrator-wait-checkpoint-verification
Closed

alin-at-dfinity wants to merge 1 commit into
masterfrom
alin/orchestrator-wait-checkpoint-verification

Conversation

@alin-at-dfinity

@alin-at-dfinity alin-at-dfinity commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

An upgrade can stop the replica before it has verified its checkpoint at the upgrade CUP's height. The CUP needs only 2f+1 shares, so a slower node can get it from a peer or the registry mid-verification. On restart, the replica then archives the unverified checkpoint and state-syncs instead, re-reading and rehashing its whole local state: ~1 TB in ~4 minutes on a large mainnet subnet, where an estimated 26–36% of upgrades lose this race on at least one node.

Before execute_upgrade, the orchestrator now waits until the local checkpoint at the CUP height is verified (its unverified_checkpoint_marker is gone), for at most 1 minute from the first delay at that height, then upgrades anyway. The replica is already halted at the CUP height, so waiting costs nothing.

  • The existing 10 s upgrade loop retries it, and each delay is counted in replica_version_upgrade_prevented{reason="checkpoint_not_verified"}.
  • Unlike the other upgrade gates, it also applies to the NNS.
  • It doesn't apply to unsigned CUPs (genesis, recovery) or when the replica isn't running (e.g. a node joining the subnet), as no local checkpoint at that height is coming then.
  • The check only reads the file system (ic_state_layout::is_checkpoint_verified), with the state root from the replica config.

Tests: the new state layout function; and the orchestrator delaying for a missing or an unverified checkpoint and then upgrading once it is verified or after the timeout (NNS and app subnet), plus both exemptions.

Complements #12094, which makes the replica itself wait for verification when stopped.

🤖 Generated with Claude Code

…CUP height is verified

An upgrade can stop the replica before it has verified its checkpoint at the
upgrade CUP's height; the restarted replica then archives it and state-syncs,
re-reading and rehashing its whole local state.

Before executing an upgrade, the orchestrator now waits for the local
checkpoint at the CUP height to be verified, for at most 1 minute, except for
unsigned CUPs (genesis, recovery) and when the replica isn't running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the checkpoint-verification upgrade delay.

No open issues 🎉

What changed in this PR

This PR reduces unnecessary state sync after replica upgrades by giving the local checkpoint at the upgrade CUP height time to finish verification.

Changes:

  • Adds a checkpoint verification gate for application and NNS upgrades, with a one-minute timeout and exemptions for unsigned CUPs or a stopped replica.
  • Adds a read-only checkpoint status helper and records delayed upgrades in the existing metric.
  • Tests missing, unverified, and verified checkpoints, timeout behavior, and exemptions.
File Description
rs/​state_layout/​src/​state_layout/​tests.rs Tests checkpoint verification status, including state-sync markers.
rs/​state_layout/​src/​state_layout.rs Adds a filesystem-only checkpoint verification helper.
rs/​orchestrator/​src/​upgrade.rs Implements and tests the bounded upgrade delay.
rs/​orchestrator/​src/​processes.rs Makes replica running status available outside tests.
rs/​orchestrator/​Cargo.toml Adds the state-layout dependency for Cargo.
rs/​orchestrator/​BUILD.bazel Adds the state-layout dependency for Bazel.
Cargo.lock Records the orchestrator’s new dependency.

Review effort: Balanced

@alin-at-dfinity

Copy link
Copy Markdown
Contributor Author

Closing in favour of #12094, which makes the replica itself wait for its checkpoint's verification on any stop, upgrades included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants