Repository navigation
fix: deflake //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_local - #12102
Draft
basvandijk wants to merge 1 commit into
Draft
basvandijk wants to merge 1 commit into
basvandijk wants to merge 1 commit into
Conversation
Adding 3 nodes to the 4-node subnet makes it reshare its IDKG keys to all 7 nodes. Each reshare dealing needs 2f+1 = 5 support shares, so at least one new node must be caught up before the next summary block. With the shared DKG_INTERVAL of 19 that leaves only ~8 s, which the new nodes occasionally miss (orchestrator polling, connection retries, catching up). The maker of the next summary block then raises the `master_key_transcript_missing` critical error, which fails the test's post-test metrics check. Give the test its own DKG_INTERVAL of 49 (~19-25 s per interval). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the test-specific DKG interval increase intended to reduce resharing timing failures.
No open issues 🎉
What changed in this PR
This PR reduces timing-related failures in the threshold-signature add-nodes system test by giving newly assigned nodes more time to participate in key resharing.
Changes:
- Replaces the shared DKG interval of 19 with a test-specific value of 49.
- Documents the joining and resharing delays behind the longer interval.
- Preserves the four-to-seven-node scenario and strict critical-error checks.
| File | Description |
|---|---|
rs/tests/consensus/tecdsa/tecdsa_add_nodes_test.rs |
Introduces the longer local DKG interval and explains its timing and runtime tradeoffs. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
//rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_localflaked on master on 2026-11-12 (run, commit 931cce5). Over the last month, that was the only flake in ~1000 runs across all variants of this test (1 in 224 runs of_local). Thetesttask itself passed. The post-test metrics check failed:Root cause
The test adds 3 unassigned nodes to a 4-node subnet that holds 4 IDKG keys.
How the critical error arises
verification_threshold= 2f+1 = 5 support shares from the 7 receivers. There are only 4 old nodes, so at least one new node must be caught up and supporting.master_key_transcript_missingonce per key (create_summary_payload_helperinrs/consensus/idkg/src/payload_builder.rs) and the reshare carries over to the next interval.DKG_INTERVALof 19, the reshare gets only 20 blocks, ~8 s on this subnet.What delays the new nodes
Timeline of the failed attempt (old nodes 6dfqk, ln4rn, g54hf, kvchs; new nodes k2bxb, say7m, ymoxn):
AddNodesToSubnetexecuted (registry version 5). kvchs's orchestrator stores v5 at 13:10:18.2 and the new nodes' at 13:10:19.4.Noticed subnet membership or mega encryption key change ... will start key_transcript_creation: height = 180for all 4 IDKG keys. Block 181 holds theReshareOfUnmaskedparams (4 old dealers, 7 receivers).master_key_transcript_missing4 times (one per IDKG key) and its metric reaches 4.In the passing retry, the new replicas started ~10 s before the summary block that began the reshare, and the reshare completed at height 184, inside its interval.
Fix
Give this test its own
DKG_INTERVALof 49 instead of the shared 19. 50 blocks last ~19–25 s at the block rates seen in the logs (0.38–0.49 s/block). That covers the new nodes' worst-case join delay, and the strictcritical_errorscheck and the 4+3 scenario stay as they are.This follows earlier fixes:
Why 49 and not something else
Cost
await_subnet_earliest_topology_version(~2.5 intervals).Side effect: the new nodes are fully caught up before the 7-node committee takes over, so the ~7 s notarization stall at the start of that interval also goes away.
Alternatives considered
Tolerate only
critical_errors{error="master_key_transcript_missing"}in this test. It costs no runtime, but:critical_errors, entries must be prefix-free, and there is no replica-sideupdate_*_metrics_to_check.Note that the hint printed by the driver,
remove_metrics_to_check("critical_errors{error=...}"), is a no-op in this case. That may be worth a separate follow-up.Add 2 nodes instead of 3. With 6 receivers (f=1) the 4 old nodes meet every IDKG threshold by themselves, so the race disappears at no cost. But the test would then no longer require the new nodes to take part in IDKG, and f would no longer grow from 1 to 2.
Verification
cargo check,cargo fmtand./ci/scripts/rust-lint.sh(exit 0) all pass, andbazel build //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_binsucceeds.bazel test --runs_per_test=//rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_farm@3 --jobs=4 //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_farm //rs/tests/consensus/tecdsa:tecdsa_add_nodes_test_head_nns_farm.assert_no_metrics_errors.testtask took 203–212 s; the whole test took 285–292 s._farmruns on CI over the past week had a median of ~3:54, so the fix adds ~55 s per run, as estimated._localvariants were not run locally because the environment I used lacks KVM access. CI runs them on this PR.This PR was created following the steps in
.claude/skills/fix-flaky-tests/SKILL.md.🤖 Generated with Claude Code