Skip to content

fix: deflake //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_local - #12102

Draft
basvandijk wants to merge 1 commit into
masterfrom
ai/deflake-tecdsa_add_nodes_test-2026-11-13
Draft

basvandijk wants to merge 1 commit into
masterfrom
ai/deflake-tecdsa_add_nodes_test-2026-11-13

Conversation

@basvandijk

Copy link
Copy Markdown
Collaborator

//rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_local flaked on master on 2026-11-12 (run, commit 931cce5). Over the last month, that was the only flake in ~1000 runs across all variants of this test (1 in 224 runs of _local). The test task itself passed. The post-test metrics check failed:

Task assert_no_metrics_errors FAILED -- The metric `critical_errors{error="master_key_transcript_missing"}` on node kvchs-... exceeded the maximum allowed value: 4 > 0.

Root cause

The test adds 3 unassigned nodes to a 4-node subnet that holds 4 IDKG keys.

How the critical error arises

  • The first summary block whose validation context uses the new registry version starts resharing the IDKG keys from the 4 old nodes to all 7 nodes.
  • Each reshare dealing needs verification_threshold = 2f+1 = 5 support shares from the 7 receivers. There are only 4 old nodes, so at least one new node must be caught up and supporting.
  • If the reshare isn't done when the next summary block is made, that block's maker raises master_key_transcript_missing once per key (create_summary_payload_helper in rs/consensus/idkg/src/payload_builder.rs) and the reshare carries over to the next interval.
  • With the shared DKG_INTERVAL of 19, the reshare gets only 20 blocks, ~8 s on this subnet.

What delays the new nodes

  • The reshare can start as soon as the new registry version is 10 s old in the block maker's registry client (the stable registry version age).
  • A new node first has to notice its assignment (the orchestrator checks every 10 s), start its replica, get connected (failed dials are retried on a 10 s cycle) and follow the chain past the reshare params.

Timeline of the failed attempt (old nodes 6dfqk, ln4rn, g54hf, kvchs; new nodes k2bxb, say7m, ymoxn):

Time Event
13:10:17.4 AddNodesToSubnet executed (registry version 5). kvchs's orchestrator stores v5 at 13:10:18.2 and the new nodes' at 13:10:19.4.
13:10:27.4–27.8 The new nodes' orchestrators detect the assignment, ~8 s after storing v5. Their replicas start at 13:10:27.7–28.2.
13:10:28.3 Summary block @180 (made by kvchs) is the first to use v5: Noticed subnet membership or mega encryption key change ... will start key_transcript_creation: height = 180 for all 4 IDKG keys. Block 181 holds the ReshareOfUnmasked params (4 old dealers, 7 receivers).
13:10:30.6–31.1 The old nodes' connections to the new nodes come up (their first dials at 13:10:20.6 timed out).
13:10:34.6 The new nodes reach CUP @180.
13:10:36.3 Summary block @200 (made by kvchs): the reshare hasn't completed, so kvchs logs master_key_transcript_missing 4 times (one per IDKG key) and its metric reaches 4.
13:10:43.6 The reshare completes in block 201 (made by the new node k2bxb). The rest of the test passes.

In the passing retry, the new replicas started ~10 s before the summary block that began the reshare, and the reshare completed at height 184, inside its interval.

Fix

Give this test its own DKG_INTERVAL of 49 instead of the shared 19. 50 blocks last ~19–25 s at the block rates seen in the logs (0.38–0.49 s/block). That covers the new nodes' worst-case join delay, and the strict critical_errors check and the 4+3 scenario stay as they are.

This follows earlier fixes:

Why 49 and not something else

  • A timing model of the race reproduces the observed flake rate at 19. At 49 it shows no misses under realistic stress (fastest observed blocks, delayed registry polls).
  • 29 and 39 leave too little margin.
  • 59 and 79 add another ~17 s and ~50 s per run for no measurable gain.

Cost

  • Two steps of the test scale with the interval: initial key creation (~1.5 intervals) and await_subnet_earliest_topology_version (~2.5 intervals).
  • That adds roughly 45–60 s per run. Measured below.

Side effect: the new nodes are fully caught up before the 7-node committee takes over, so the ~7 s notarization stall at the start of that interval also goes away.

Alternatives considered

  • Tolerate only critical_errors{error="master_key_transcript_missing"} in this test. It costs no runtime, but:

    • It would accept a critical error in the only system test that reshares IDKG keys to newly added nodes.
    • It needs a change to the shared test driver. The default check is the prefix critical_errors, entries must be prefix-free, and there is no replica-side update_*_metrics_to_check.

    Note that the hint printed by the driver, remove_metrics_to_check("critical_errors{error=...}"), is a no-op in this case. That may be worth a separate follow-up.

  • Add 2 nodes instead of 3. With 6 receivers (f=1) the 4 old nodes meet every IDKG threshold by themselves, so the race disappears at no cost. But the test would then no longer require the new nodes to take part in IDKG, and f would no longer grow from 1 to 2.

Verification

  • Checks: cargo check, cargo fmt and ./ci/scripts/rust-lint.sh (exit 0) all pass, and bazel build //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_bin succeeds.
  • Ran bazel test --runs_per_test=//rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_farm@3 --jobs=4 //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_farm //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_head_nns_farm.
    • All 4 runs passed every task, including assert_no_metrics_errors.
    • The test task took 203–212 s; the whole test took 285–292 s.
    • Successful _farm runs on CI over the past week had a median of ~3:54, so the fix adds ~55 s per run, as estimated.
    • These runs used a revision that differed only in the wording of two comment lines.
  • The _local variants were not run locally because the environment I used lacks KVM access. CI runs them on this PR.

This PR was created following the steps in .claude/skills/fix-flaky-tests/SKILL.md.

🤖 Generated with Claude Code

Adding 3 nodes to the 4-node subnet makes it reshare its IDKG keys to
all 7 nodes. Each reshare dealing needs 2f+1 = 5 support shares, so at
least one new node must be caught up before the next summary block.
With the shared DKG_INTERVAL of 19 that leaves only ~8 s, which the new
nodes occasionally miss (orchestrator polling, connection retries,
catching up). The maker of the next summary block then raises the
`master_key_transcript_missing` critical error, which fails the test's
post-test metrics check.

Give the test its own DKG_INTERVAL of 49 (~19-25 s per interval).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-specific DKG interval increase intended to reduce resharing timing failures.

No open issues 🎉

What changed in this PR

This PR reduces timing-related failures in the threshold-signature add-nodes system test by giving newly assigned nodes more time to participate in key resharing.

Changes:

  • Replaces the shared DKG interval of 19 with a test-specific value of 49.
  • Documents the joining and resharing delays behind the longer interval.
  • Preserves the four-to-seven-node scenario and strict critical-error checks.
File Description
rs/​tests/​consensus/​tecdsa/​tecdsa_add_nodes_test.rs Introduces the longer local DKG interval and explains its timing and runtime tradeoffs.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants