Skip to content

test(core): e2e finalize-resume round-trip after a post-payment outage (#144) - #226

Merged
Nic-dorman merged 1 commit into
mainfrom
test/finalize-resume-round-trip
Oct 27, 2026
Merged

Nic-dorman merged 1 commit into
mainfrom
test/finalize-resume-round-trip

Conversation

@Nic-dorman

@Nic-dorman Nic-dorman commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Fixes #144.

Before this PR, nothing on main exercised #172's resumable external-signer finalize against a live network: no e2e test called finalize_upload_resumable, finalize_upload_merkle_multi_resumable or finalize_resume (the Tier 1 test #144 cites belonged to #143, which #172 replaced). Only the pure assemble_* unit tests covered the retain/hand-off contract.

Approach: a reversible outage, not node restarts

#144 listed three ways to get "fails after payment, then recovers": restart nodes in MiniTestnet, add per-node stop/start to LocalDevnet (ant-node 0.21's Devnet has neither), or a reversible network block. It called the last one least flaky, but Linux-only via iptables.

The harness already had the mechanism: PointerSilence makes a node read and drop pointer requests while staying in the network. This generalizes it to Silence { pointer_gets, pointer_puts, chunk_puts } and adds MiniTestnet::silence_chunk_puts(bool).

  • During the outage nodes are up but never answer a chunk PUT.
  • Lifting it restores them with the same PeerIds and addresses. Every paid wave chunk is bound to the peers that quoted it, so that's the requirement.
  • No port reuse, no DHT re-convergence, and it's cross-platform.

e2e_pointer.rs changes only for the rename (pointer_silence.gets → silence.pointer_gets, etc.).

Tests

Both run in the existing CI jobs (e2e_file in E2E Tests, e2e_merkle in Merkle E2E). Each one:

  1. Prepares a public upload and pays as an external signer with the testnet wallet.
  2. Silences chunk PUTs on every node. Nothing answers a silenced PUT, so every attempt waits out its timeout:
    • Wave uses the fixed 10s STORE_RESPONSE_TIMEOUT, which adds up to about 90s.
    • Merkle uses merkle_store_timeout_secs. The test lowers it from 270s to 2s for the outage only. At 270s this phase alone took 92 minutes.
  3. Finalizes, and asserts FinalizeOutcome::Partial with 0 stored, all failed, and the data_map_address carried through.
  4. Lifts the silence, restores the merkle timeout, and runs finalize_resume in a loop bounded to 3 attempts, as the API documents. Asserts Complete with everything stored and the same data_map_address.
  5. Asserts the wallet token balance is unchanged across the failed finalize and the resume (no second payment).
  6. Runs data_map_fetch(public address) → file_download and checks the bytes equal the original.
  • e2e_file::test_finalize_resume_after_post_payment_outage_wave_batch: wave-batch, 4 KB file, the default 14-node testnet.
  • e2e_merkle::test_external_merkle_finalize_resume_after_post_payment_outage: merkle multi-batch (cap 3, so ≥2 sub-batches), 500 KB, 35 nodes.

Linear issue

Closes V2-1519

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Test-only: harness and e2e tests, no library code.

Compatibility

  • Wire: none
  • Storage: none
  • API: none (test harness only)

Semver impact

  • breaking
  • feature
  • fix

Test evidence

Rebased 2026-10-23 onto main at 50bc653 (0.3.12 promoted). tests/support/mod.rs conflicted with #218's ChunkGetCounts and sentinel-barrier changes, which also thread through spawn_node. The resolution keeps both sides: main's chunk_gets plumbing, plus this PR's PointerSilence → Silence generalisation with chunk_puts and silence_chunk_puts(). This PR's added and removed lines are otherwise identical to before the rebase. Re-run on Linux (dev2, stable):

  • cargo fmt --all -- --check and cargo clippy --all-targets --all-features -- -D warnings: pass.
  • test_finalize_resume_after_post_payment_outage_wave_batch: passed (144 s).
  • test_external_merkle_finalize_resume_after_post_payment_outage: passed (593 s).
  • e2e_pointer, which exercises the renamed silence switches: 17 passed.

Original evidence follows.
Local runs on Windows 11, against the in-process MiniTestnet and Anvil:

  • e2e_file::test_finalize_resume_after_post_payment_outage_wave_batch: passed in 159s. The finalize during the outage returned Partial after 91s, and the resume completed on its first attempt.
  • e2e_merkle::test_external_merkle_finalize_resume_after_post_payment_outage: passed in 596s. The outage finalize returned after 140s (5505s before the test lowered merkle_store_timeout_secs), and the resume completed on its first attempt.
  • e2e_pointer, all 14 tests: passed after the Silence rename. CI doesn't run this file.
  • cargo fmt --all -- --check: clean.
  • cargo clippy --all-targets --all-features -- -D warnings: didn't finish locally because the machine ran short of memory. CI's Clippy job is the lint check here.

CI runs both new tests on ubuntu-latest and macos-latest (E2E Tests, Merkle E2E). Merkle E2E has a 90-minute limit and has recently taken 48–54 minutes; this test adds about 10.

New dependency

none

ADR

n/a

Mitigation / rollback

Revert the commit. It only touches ant-core/tests/.

🤖 Generated with Claude Code

@Nic-dorman
Nic-dorman force-pushed the test/finalize-resume-round-trip branch from 55092e8 to 323f523 Compare October 23, 2026 10:13
#144)

Nothing on main exercised #172's resumable external-signer finalize
against a live network: no e2e test called finalize_upload_resumable,
finalize_upload_merkle_multi_resumable or finalize_resume, so only the
assemble_* unit tests covered the retain/hand-off contract.

The harness's PointerSilence becomes Silence { pointer_gets,
pointer_puts, chunk_puts }, with MiniTestnet::silence_chunk_puts. A
silenced node stays in the network but reads chunk PUTs off the wire
and drops them; lifting the silence brings every node back with the
same PeerId and address, which paid wave chunks need (each is bound to
the peers that quoted it) and a restart would not give.

Two tests, wave-batch in e2e_file and merkle multi-batch in e2e_merkle,
each pay as an external signer, silence chunk PUTs, assert the
finalize comes back Partial with nothing stored, lift the silence,
resume (bounded to three attempts), and assert Complete, an unchanged
token balance (no second payment), and a byte-exact download through
the public DataMap address.

The merkle test lowers merkle_store_timeout_secs to 2 s for the outage
only: at the 270 s default the doomed finalize alone took 92 minutes.
The wave path's PUT timeout is the fixed 10 s STORE_RESPONSE_TIMEOUT,
so its outage phase takes about 90 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Nic-dorman
Nic-dorman force-pushed the test/finalize-resume-round-trip branch from 323f523 to 56cc7c4 Compare October 23, 2026 18:52

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE

Reviewed head 56cc7c4eb04ccad205bf8e16cd930428cde62ec3 against base ccd4106dd8fd60545bbd31c40d46cc041f85fb8d. No actionable defects found. The request filter drops chunk PUTs before handling; the tests verify partial finalize, resume against the original payment, unchanged token balance, and byte-exact download.

2/2 seats reviewed (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3); both seats agree (two model families). All 18 CI checks passed for this head. git diff --check passed.

Non-blocking clarity note: the wave test says “a fresh retriever” but reuses its client. The address-based fetch still exercises the network chunk-get path, but not a separately created client.

@Nic-dorman
Nic-dorman merged commit 5e01699 into main Oct 27, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Devnet e2e: full finalize → post-payment failure → resume → success round-trip (#140 follow-up)

2 participants