Skip to content

fix(registry): Maintain routing table shard boundaries - #12098

Open
mraszyk wants to merge 3 commits into
masterfrom
fix/routing-table-shard-boundaries
Open

mraszyk wants to merge 3 commits into
masterfrom
fix/routing-table-shard-boundaries

Conversation

@mraszyk

@mraszyk mraszyk commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The routing table is stored in the registry as shards under the canister_ranges_* keys. Each shard is keyed by a canister ID, and its entries are expected to lie between its own shard key (inclusive) and the next shard key (exclusive). get_subnet_for_canister relies on this: it picks the shard with the largest key that is not larger than the canister ID and only looks there.

When computing the shards for an updated routing table, the existing shard keys were preserved unconditionally. If the update produced a canister ID range spanning an existing shard key, the whole range was stored in the shard before that key, while the key itself was kept. For example, with shards

  • 0: [0, 99] -> A
  • 100: [100, 100] -> B, [101, 199] -> A, [200, 299] -> C

migrating canister 100 back to subnet A merges the first three ranges into [0, 199] -> A, which resulted in

  • 0: [0, 199] -> A
  • 100: [200, 299] -> C

so that get_subnet_for_canister reported no subnet for the canister IDs 100 to 199. The same can happen when canister ID ranges are rerouted. Readers of the full routing table (i.e. all shards concatenated) are not affected.

Changes

  • When computing the new shards, drop every preserved shard key satisfying range.start < shard_key <= range.end for a range in the new routing table. Shard keys that coincide with the start of a range or lie in a gap between ranges are kept, so the existing shard structure is otherwise left untouched. The shard builder can also reconstruct a consistent layout from existing inconsistent shard boundaries, but this PR does not add an upgrade-time repair migration.
  • Extend the routing table invariant check: the shard keys must be strictly increasing, each shard must be a valid routing table, and the entries of each shard must lie between its own shard key and the next shard key. Together, these imply that the concatenation of all shards is a valid routing table, so the check is a single pass over the shards.
  • Add an entry to the registry's unreleased changelog.

Upgrade note

The registry checks its invariants in post_upgrade, so the upgrade to this version fails (and is rolled back) if the stored shards do not satisfy the new invariant. An inconsistent deployed layout therefore needs a separate repair before upgrading; a later routing table update cannot repair it because the upgrade would already have failed. The mainnet shards at registry version 65677 (14 shards, 111 ranges) satisfy it. This should be re-checked right before submitting the upgrade proposal.

Tests and benchmarks

  • A migration that merges a range across a shard key drops that shard key.
  • A dropped shard key whose entries overflow the (full) previous shard forces a re-split of that shard.
  • The invariant check rejects a shard with an entry starting before its shard key and a shard with an entry reaching the next shard key.
  • canbench_results.yml is updated. Compared with the PR base, the invariant check benchmark uses about 3.05% more instructions, with heap growth reduced from 138 to 32 Wasm pages. Migration benchmarks use 1.78–1.96% more instructions, and upgrade benchmarks use 0.08–0.16% more instructions. Lookup and snapshot benchmarks change by less than 0.31%.
  • Local validation: RUST_MIN_STACK=16777216 cargo test -p registry-canister --lib --locked passed with 509 tests passing and 2 ignored. The larger stack is needed by an existing large-array test that overflows the default stack.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the routing-table shard-boundary changes.

No open issues 🎉

What changed in this PR

This PR preserves registry routing-table shard boundaries when canister migrations merge ranges across existing shard keys.

Changes:

  • Remove preserved shard keys that fall strictly inside rebuilt canister ranges.
  • Validate each shard’s ranges against its own key and the next shard’s key.
  • Add regression tests for cross-boundary range merging and invalid shard layouts.
File Description
rs/​registry/​canister/​src/​mutations/​routing_table.rs Remove obsolete shard boundaries and test migration-driven range merging.
rs/​registry/​canister/​src/​invariants/​routing_table.rs Enforce shard boundary constraints and test invalid layouts.

Review effort: Balanced

The shard invariant check converted every routing table entry twice (once
per shard and once more for the concatenation of all shards) and cloned
every shard, which regressed the canbench benchmarks. The per-shard checks
already imply that the concatenation is valid, provided that the shard keys
are strictly increasing, so check that instead and drop the second
conversion and the clones.

Also document the invariants that are actually checked, update the canbench
results, and add a test for a dropped shard key whose entries overflow the
previous shard and force a re-split.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human validation of the deployed registry is needed because stricter post-upgrade invariants may reject boundary-spanning shards left by the old merge bug, and no repair migration is included.

No open issues 🎉

Review effort: Balanced

Also trim the doc comment of the routing table invariant check.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The stricter post_upgrade shard invariant can roll back the registry upgrade, requiring human revalidation of the live shard layout immediately before proposal submission rather than reliance on the version 65677 snapshot.

No open issues 🎉

Review effort: Balanced

@mraszyk
mraszyk marked this pull request as ready for review November 13, 2026 12:59
@mraszyk
mraszyk requested a review from a team as a code owner November 13, 2026 12:59

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):

  1. Update unreleased_changelog.md (if there are behavior changes, even if they are
    non-breaking).

  2. Are there BREAKING changes?

  3. Is a data migration needed?

  4. Security review?

How to Satisfy This Automatic Review

  1. Go to the bottom of the pull request page.

  2. Look for where it says this bot is requesting changes.

  3. Click the three dots to the right.

  4. Select "Dismiss review".

  5. In the text entry box, respond to each of the numbered items in the previous
    section, declare one of the following:

  • Done.

  • $REASON_WHY_NO_NEED. E.g. for unreleased_changelog.md, "No
    canister behavior changes.", or for item 2, "Existing APIs
    behave as before.".

Brief Guide to "Externally Visible" Changes

"Externally visible behavior change" is very often due to some NEW canister API.

Changes to EXISTING APIs are more likely to be "breaking".

If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.

If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.

Reference(s)

For a more comprehensive checklist, see here.

GOVERNANCE_CHECKLIST_REMINDER_DEDUP

@zeropath-ai

zeropath-ai Bot commented Nov 13, 2026

Copy link
Copy Markdown

✅ No security or compliance issues detected. Reviewed everything up to 306ad24.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/registry/canister/canbench/canbench_results.yml
    Improve instruction counts and resource usage metrics across multiple benches
► rs/registry/canister/src/invariants/routing_table.rs
    Check routing table shard invariants and handle shard key boundaries
► rs/registry/canister/src/mutations/routing_table.rs
    Add optimization to prune new shards to align with shard keys and adjust mutations logic
Other relevant changes ► rs/registry/canister/unreleased_changelog.md
    Document routing table shard consistency improvements and invariant checks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants