Skip to content

feat(payment)!: enforce the close-group price floor on every node - #257

Open
grumbach wants to merge 3 commits into
WithAutonomi:mainfrom
grumbach:price-floor-live-enforcement
Open

grumbach wants to merge 3 commits into
WithAutonomi:mainfrom
grumbach:price-floor-live-enforcement

Conversation

@grumbach

@grumbach grumbach commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

The ADR-0006 close-group price floor has run in shadow mode since v0.17.1. This turns it on for every node at 30%, with nothing to configure. It also fixes what made enforcement unsafe: the tolerance, griefing by neighbours, a bypass through paid-list repair, and, within a measured margin, trust penalties for refusing a write under the floor.

A week of shadow telemetry (ant-prod-01, 2026-10-14 to 2026-10-20, 699 node instances on 72 hosts, node 0.22.0 and 0.23.0) and a review of the enforcement path found four problems:

  • At its 65% default, the floor would have refused 2.5% of admissions (47 of 1,898). 2.1% of chunks would have been refused by every receiver that evaluated them. An earlier window, 2026-09-10 to 2026-09-16, showed 3.4% and 1.5%.
  • Neighbours gossiping inflated commitments could make every receiver refuse a chunk.
  • One accepting node could still replicate a below-floor write, through PaidNotify and paid-list repair.
  • Four existing checks would turn a price refusal into a trust penalty, three on the refuser and one on the node that accepted the write. Shadow mode never refused, so none of them has fired:
    • the storing node's possession check, at weight 5 for a chunk (about a 48% cut in trust) and weight 1 for a pointer;
    • the responsible audit, which challenges a peer on any key it was hinted as a replica, at weight 5;
    • the prune audit, which penalises an absent close-group peer at weight 5;
    • and peers whose quorum check fails for a key only one node hinted penalise that node, at weight 1.

What changes:

  • The reference is capped at the receiver's own price: min(group median, calculate_price(own committed count)). A node never demands more than the tolerance of what it would itself quote. Neighbours therefore cannot make it refuse a payment that clears its own price, however many of them lie. While gossip is honest, receivers above the median price exactly as before.

  • Tolerance 65% → 30%, a deliberate starting point. No measured admission is refused at 30%, or at 40%, in either production window: 1,898 admissions in October (1,801 of them matched for the capped replay), and 1,103 evaluations in September. The worst October admission clears 30% by 1.78x but 40% by only 1.34x, on a light week of 48 upload bursts. So the floor starts at 30%. A follow-up raises it toward 40%, by a later ADR amendment, once enforced traffic shows that 40% refuses no honest payment. A baseline quote (a near-empty or sybil node) is still refused by 91% of receivers, against 98% at 40%.

  • Paid-list admission is floored too. PaidNotify announces a client's write, with the proof the client presented. Repair and fetch never call the verifier. So this reprices no historical receipt, and one accepting node can no longer give a below-floor write the paid-list votes that repair needs.

  • No peer is judged on a contested write. When a node's verifier first admits the payment for a write (the PUT, a fresh offer or a PaidNotify), it prices the proof against its current view; a later admission of the same write from the payment cache reuses that decision. The write is contested when the node cannot price it (no own commitment, or too thin a view), or when its paid price is below 60% of the uncapped group median, twice the floor. An admitted contested write, one the node cannot price or one paid between the floor and that threshold, is still stored and offered, but:

    • the fan-out (chunk or pointer) schedules no possession check;
    • for a chunk, the key's paid-list entry gets a one-byte mark, which survives restarts. The mark follows the decision the node took when it verified the payment: the verifier remembers which admitted chunk payments the floor contested, in a memory of its own, so a write admitted later from the payment cache is recorded by that decision, and the proof attached to it, which anyone can attach to a PaidNotify or fresh offer and nobody checked, can neither set nor veto the mark. The first paid-list record of a key decides its mark, and a fan-out never schedules a possession check for a write that is marked, by the verified decision or by the record that won, which it reads after recording its own;
    • a marked key goes out in paid hints only, never as a replica hint, so no repair proof is recorded for it, no responsible audit challenges a peer on it, and no peer can penalise this node as its sole hinter. Repair still works through paid hints wherever the group accepted the key. The paid list also keeps its marked keys in memory, updated in commit order and cleared by any write that leaves a key unmarked, so building the hints for a peer reads nothing on disk for them. If the marks cannot be read when the node starts, it still starts and hints no key as a replica until a restart can read them;
    • the responsible audit never samples a marked key; a prune audit still needs its proofs before deleting one but scores no absent peer; paid-list pruning keeps a marked entry while the record is held; and a mark that cannot be read counts as set, with one warning per run of failed reads.

    Merkle proofs, proofs that do not parse, and nodes with no floor wired are judged as before. The 2x margin covers the largest spread measured between receivers' views of one chunk (1.91x). Replayed over all 5,536 pairs of measured receivers, at payments from 0.2x to 1x the median in 0.02x steps, no refusal would be judged at 2x, against 4 pairs at 1.5x. The cost is that these checks are given up on 1.4% of the measured admissions, plus the 2.6% the floor cannot evaluate. It is a margin, not a guarantee. A guarantee needs the refuser to tell the sender, and a node that can say "refused on price" can also dodge ADR-0003's full-node detection that way, so that is a separate decision. The threshold is its own constant, so a later raise takes two releases: the threshold first, then the tolerance once nearly every node runs it.

  • Enforced on every node, nothing to configure. PriceFloorConfig is gone: the tolerance is one public constant, PRICE_FLOOR_TOLERANCE_PERCENT, and the contest threshold a private one, PRICE_FLOOR_CONTESTED_PERCENT (60). ANT_PRICE_FLOOR_ENFORCE and ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, without a warning. The floor is the same everywhere, because a node set stricter than its peers would be penalised by their checks. Backing it out takes a release. A node whose replication engine fails to start runs without one, as before, and its floor skips every evaluation (skip_reason=no_commitment_cache), as does ant-devnet's.

  • Telemetry adds target, group_median_price, own_price, paid_quote_age_secs and contested to the existing ant_node::payment::price_floor line. enforce is now always true, which, like target, tells upgraded nodes apart from shadow-mode ones during the rollout. Each line carries what a replay at any tolerance needs, and the follow-up uses that to judge 40%.

  • Tests.

    • tests/poc_price_floor_live.rs had stopped exercising the close-group floor. Its rejection test already failed on main, and CI does not run it. It is repaired here.
    • Unit tests pin the 30% tolerance and the 60% contest threshold to the atto.
    • Four new e2e tests drive the real schedulers: the chunk possession check, mark and hints; the pointer possession check; the prune audit; and the three record paths (PUT, fresh offer, PaidNotify), each of which must mark a key exactly when its own verified decision was contested, never from the forged proof attached.
    • An env-gated A/B workload test, tests/e2e/price_floor_ab.rs, returns at once unless PRICE_FLOOR_AB=1, so CI only compiles it.
    • The group sample's self-filter is removed as dead code: the gossip cache cannot hold this node, since ingest accepts only senders in the routing table, which never holds self.

Linear issue

Closes V2-870

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: none. No message changes. Upgraded nodes refuse below-floor payments that older nodes admit, and both interoperate, except that an older node penalises upgraded peers that refuse a below-floor write it fanned out (see Rollout skew).
  • Storage: the paid list's value byte, always empty until now, marks price-contested keys. Releases before this one only test presence, so they read a marked key as an ordinary paid key, and a rollback leaves the marks unread.
  • API: removes PriceFloorConfig (with its from_env, enforce and tolerance_percent), PaymentVerifierConfig::price_floor, PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV; adds PRICE_FLOOR_TOLERANCE_PERCENT and four PaidList methods for the mark (insert_price_contested, is_price_contested, counts_as_price_contested and retain_unmarked). neighbor_sync::build_replica_hints_for_peer and audit::audit_tick_with_repair_proofs now take the paid list, and ReplicationEngine::replicate_fresh, which nothing in production calls, is now test-only (test-utils). The two ANT_PRICE_FLOOR_* variables are no longer read. ant-client's test support imports PriceFloorConfig and sets price_floor: PriceFloorConfig::default(); both go when it moves to this release.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

Production replay, October. The source is shadow telemetry from ant-prod-01, 2026-10-14 to 2026-10-20: 2,117 floor lines from 699 instances on 72 hosts. Replaying the logged formula reproduces all 53 of the nodes' own below_floor=true lines (47 distinct admissions). That leaves 1,898 admissions (each a distinct chunk and receiver) over 536 chunks, of which 1,801, over 521 chunks, are matched to the receiver's own committed count. Refused admissions out of those 1,801:

tolerance median only (today) capped (this PR)
30% (new) 0 0
40% 0 0
50% 1 0
55% 11 1
65% (old default) 47 11
  • The lowest settled-to-reference ratio is 0.534 with the cap (0.473 without). That clears 30% by 1.78x and 40% by 1.34x.

  • With zero refusals observed, the 95% upper bounds are 0.17% per admission, 0.58% per chunk (521 chunks hold a matched admission) and 6.4% per upload burst (47 bursts hold one).

  • Share of receivers refusing a 3x payment to a quoter, with the cap:

    quoter's count, as a share of the group median 30% 40%
    0 (a baseline quote) 91.4% 98.0%
    a quarter 75.4% 94.6%
    40% 27.9% 75.2%
    half 0% 39.1%

    Every observed receiver refuses a baseline quote on 84.5% of chunks at 30% (95.2% at 40%). Not caught at 30%: a quoter 40% as full as its group is admitted by 72% of receivers and one half as full by all of them, nor a client shopping for the cheapest member of the close group (no lower than about 0.7x the median).

  • If every neighbour gossiped the protocol maximum, all 1,801 admissions would still clear at 30%. Today's rule would refuse all of them.

  • Replaying the contest margin over 5,536 ordered pairs of receivers of the same chunk, at payments from 0.2x to 1x the median in 0.02x steps: 0 pairs would see a refusal judged at 2x, against 4 at 1.5x. Moving the tolerance to 40% without first lifting the threshold would leave 3.

Second production window, September. This is an independent window a month earlier, 2026-09-10 to 2026-09-16: 1,134 floor lines from 557 instances on 71 hosts, with 1,103 evaluated over 271 chunks. It was measured under today's median-only rule, which refuses at least everything the capped rule does. At 50% that rule refused nothing. At 55% it refused 0.93% of admissions, and at 65% 3.43%, with 1.48% of chunks refused by every observed receiver. September's worst admission therefore sat at 0.50x or above under the stricter rule. October's sat at 0.473x under that rule and 0.534x capped. Neither window has a refusal at 30% or 40%. Own commitment counts were not joined for that window, and its raw dump was not kept, so these figures are from the analysis done at the time.

Live A/B, tests/e2e/price_floor_ab.rs. The same workload ran on main (shadow mode, 2026-10-20) and on this change at 30% (on 375d62f, this PR's code before its rebase onto main). The main run used an earlier revision of the driver, which asserted the shadow outcome and checked trust only on the peers that refused a direct PUT; it was not re-run with this driver. Setup: 24 nodes seeded to median committed counts of 3,949 for this change (range 3,809–5,785) and 4,536 for main (range 3,841–5,818), with real payments on Anvil and a 20–30 s possession delay.

main (shadow) this change (30%)
12 honest uploads 84/84 stores, 0 floor refusals 84/84 stores, 0 floor refusals in 387 floor evaluations
8 near-empty-quoter uploads (paid at baseline) 56/56 mature storers accept 56/56 mature storers refuse on the floor; 48 fresh offers and 152 PaidNotify also refused
near-empty write after the tail 8–9 holders, 17–20 paid-list votes 1 holder (the quoter), 1 vote
possession penalties 0 0: all 8 fan-outs were contested and scheduled no check, and none of the 10 peers that refused or that the quoter fanned the writes out to took a penalty-sized trust drop in the quoter's view (lowest ratio 0.768, against about 0.52 after a possession penalty)

Tests.

  • cargo test --lib --features test-utils: 1,213 passed on the head.
  • The floor tests pin, at the default tolerance:
    • the 30% tolerance and the 60% contest threshold, to the atto, and that the contest reads the uncapped median when the cap binds;
    • the cap;
    • the griefing and disarming bounds. Seven of fifteen overstating neighbours flip nothing and eight flip the verdict. Six understating neighbours leave the floor standing against a baseline quote and seven disarm it, against eight at 40%;
    • all 15 neighbours forged to the maximum still unable to lift a receiver above its own price;
    • the cheapest group member admitted, and a near-empty quoter refused;
    • paid-list and pointer payments floored;
    • the contest decision, including a view that changes after admission;
    • a node with no floor wired contesting nothing;
    • the mark: the verifier remembers a contested payment it admitted, in a memory a small payment cache cannot shrink, and a later cache hit reads that decision, a refused payment is never remembered, the first paid-list record of a key decides, and a marked key is kept across a restart, left out of replica hints and responsible-audit sampling, and kept through paid pruning while its record is held;
    • the in-memory copy of the marks: filled on opening, changed in the order writes take (a marked insert and a batch removal wait for the write order), cleared for a key by a later plain insert or removal of it when a failed commit left its mark in memory alone, used to build replica hints while the paid list cannot be read, and, when the marks cannot be read on opening (a test points the database root past the file), the paid list still opens and hints no replica;
    • a mark that cannot be read counts as set in the responsible audit, paid pruning and a write's fan-out, and a run of such reads logs one warning and a recovery line (the tests make the read fail by holding the thread's one LMDB reader slot).
  • e2e, each with a control: possession_check_skips_a_write_the_floor_cannot_price (a PUT through the handler schedules no possession check when the storer's verified decision was contested, when only the proof it forwards is, or when a marked record of the key won first; a marked key gets a paid hint only), a_fresh_pointer_write_the_floor_cannot_price_is_not_used_to_judge_peers, prune_audit_scores_no_peer_for_a_price_contested_key, and every_record_path_marks_a_key_from_its_own_verified_decision. They settle nothing on chain, so where a test needs a verified contested admission it stands one in; unit tests pin that a real verification records it.
  • Mutations: 54 mutations of the floor, the contest decision, the verifier's memory, the mark, its in-memory copy and every check that reads the mark, run on 375d62f against the unit and e2e tests above: 53 fail at least one test. The one that survives makes prune-audit scoring count a mark that cannot be read as unset. Only the e2e prune test reaches that check, and it cannot make a node's paid-list read fail.
  • cargo test --test poc_price_floor_live --features test-utils: 2/2 against a real Anvil vault. On main it was 2/3.
  • cargo test --test e2e --features test-utils -- --test-threads=1: 124 passed, 3 ignored, on 375d62f.

Gates. All clean on the head, 668f573, which is the same three commits rebased onto main at 8acc8c1, with the PR's diff unchanged:

  • cargo fmt --check;
  • cargo clippy --all-targets --all-features -- -D warnings, on 1.98 and on 1.99;
  • cargo clippy -- -D warnings;
  • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps;
  • RUSTFLAGS="-D warnings" cargo check --no-default-features;
  • scripts/adr-governance.py against main;
  • cargo audit: no vulnerabilities, only the 6 allowed warnings.

On the head, the e2e tests named above passed again (in a targeted run of 9), and so did the live PoC (2/2).

Hosted CI: all 21 checks pass on 668f573, Security Audit included. #258 moved main to rustls-webpki 0.103.16 and webpki-root-certs 1.0.10, which fixes RUSTSEC-2026-0382 and RUSTSEC-2026-0383. This PR leaves Cargo.lock unchanged.

The dev-testnet run that T3 asks for has not happened yet.

Residual risks, also recorded in the ADR.

  • Rollout skew. Until the fleet upgrades, an older node admits below-floor writes and still runs possession checks on them. Each one penalises every upgraded peer that refused it: weight 5 for a chunk, which takes a neutral peer from 0.5 to about 0.26 in that node's view, below the 0.35 swap threshold, for about 9.5 hours (weight 1 for a pointer). The score is local (for peers in that node's persisted close group it survives a restart), and below 0.35 a peer only becomes replaceable when that node's routing bucket is full. An attacker can buy such writes for little more than gas by paying its own near-empty node within the 20 closest to the key, and aim them at old nodes. Keys an older node stored this way stay on it unmarked, before and after it upgrades, so its replica hints and responsible and prune audits keep judging the refusers on them while it holds them, and the refusers can penalise it at weight 1. On the v0.17.0 rollout the managed fleet went from 24 of 73 hosts in the first hour to 65 in six and 71 within about 17; community nodes follow on their own schedule.
  • The contest margin is empirical. It reads the uncapped median, so neighbours that overstate their commitments can widen the set of contested writes in their group, and turn checks off there. A minority lifts the median only as far as the dearest honest members, and overstating needs fake leaves that fail the commitment spot-checks (ADR-0004). Seven of fifteen understating neighbours at the storer narrow it instead, to about 0.33x of the honest median, the same minority that disarms the floor.
  • Contested keys lose coverage. The node that marked a key does not audit peers on it, and the key repairs through paid hints only. That is 1.4% of measured admissions plus the 2.6% the floor cannot price, and the mark stays while the node holds the key, so it also covers every key a node records while it cannot price: its first hour before a commitment, or while its gossip view refills after a restart. The other holders of those keys still check them, unless a quoter front-ran the mark (below).
  • Gaps on nodes running this release. A key repaired from a majority's paid-list votes arrives unmarked, since repair carries no proof to price; a refuser whose own quorum check fails while its neighbours' passes can be judged on it. On the PUT path the record is stored a moment before its paid-list entry is written, so a sync in that moment can hint the key once and a refuser can charge the storer weight 1 once, and a crash in that moment leaves the key unrecorded, so unmarked unless a later verified PaidNotify records it. A PUT served from the cache with no proof attached stores the record without any paid-list entry, as before. A paid-list insert that fails before its commit leaves the key out of the paid list, so unmarked. When a marked insert's commit is what failed, the in-memory copy keeps the mark, since the commit may still have reached the disk, so this node does not hint the key as a replica while the mark stays. A later plain insert that adds the key, or a removal of it, drops the mark once that write commits, and a restart rebuilds the copy from the database. The verifier's memory of contested admissions loses a decision before its key is recorded only if 100,000 newer contested admissions arrive in between. A record admitted between paid pruning's last check and its delete would lose its mark, which needs the node's routing view to change at that moment.
  • The first paid-list record decides a key's mark. A quoter in the 20 closest can pay itself for an honest key, for gas only, and announce it before the client's write lands. At a price between the floor and the contest threshold, or at any price on nodes that cannot price, the nodes it reaches first leave that key unchecked, though they still store and repair it; those among the client's storers skip their possession check too, while storers it did not reach first still run theirs. Paying above the threshold first does the opposite: those nodes record the key unmarked and can judge peers that later refuse the client's cheaper write, until those peers repair it from the votes the front-run created.
  • No per-node switch. Backing the floor out takes a release, so the release train's canary is where it is stopped.
  • One-quote clients. An honest client whose only reachable quoter is a fresh node is refused. No case was observed.
  • Proof reuse. A proof reused for up to a day is covered only against the uncapped median, where the worst measured payment has 1.58x of headroom.
  • Retry load is new. Refusals are not cached. A refused proof is re-verified, on-chain read included, by every peer a client tries: up to 20 targets over 4 rounds.
  • Merkle batches bypass the floor. Under the default client mode those are uploads of 64 chunks or more. The floor covers single-node payments only.

New dependency

none

ADR

https://github.com/grumbach/ant-node/blob/668f57301c987705c8aca97ab9f11a79b000615f/docs/adr/ADR-0006-receiver-side-revenue-floor.md#amendment-2-2026-10-20-enforced-on-every-node (amends ADR-0006, which is still Proposed, and updates ADR-0008's status row)

Mitigation / rollback

There is no per-node switch: reverting this PR in a release restores shadow mode. Older releases read the paid-list marks as ordinary paid keys, so a rolled-back node judges peers on those keys again, as it did before.

@grumbach grumbach changed the title feat(payment)!: enforce the close-group price floor by default feat(payment)!: enforce the close-group price floor on every node Oct 20, 2026
@grumbach
grumbach force-pushed the price-floor-live-enforcement branch 9 times, most recently from f17cb86 to bc714ae Compare October 21, 2026 06:38

@dirvine dirvine left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded: original blockers withdrawn

My original REQUEST_CHANGES on bc714aebfae92f6b68f1bb8847f8328e052ea2df overstated the two blocking findings and claimed invalid panel participation. See the corrected, provisional APPROVE review for the reconciled verdict, verified tests, actual panel coverage and remaining human release/strategy decisions. This is a correction of my review, not a change to the code. The original receipt is preserved locally; this older review's historical GitHub state is not its current recommendation.


Corrected code-review verdict: APPROVE (provisional coverage)

This supersedes my earlier REQUEST_CHANGES review on the same code. The late source-review batch prompted a reconciliation. I overstated two findings as blockers. Baseline comparison, a component timing probe, and bounded adversarial reconciliation do not support retaining those blockers. No branch changes were needed for this correction.

Repository: WithAutonomi/ant-node. Reviewed head: bc714aebfae92f6b68f1bb8847f8328e052ea2df. True merge-base: a6e3388e79d0907ef678bcb6ec0fcaa12822e148, not current main.

Findings corrected

  1. Extra synchronous scan: performance follow-up, not a demonstrated blocking DoS. src/replication/neighbor_sync.rs:93-100 really adds price_contested_keys() on the async executor. However, the baseline already synchronously scans the full paid-list via all_keys() on the same sync-request path. A scratch-only probe linked the exact frozen library, preseeded LMDB, and verified both key counts. At 1,000,000 entries the additional scan's median was 12.156 ms with no marks / 71.591 ms with every key marked; the existing scan was 14.940 / 15.510 ms. These are three warm-cache runs per case with the debug-linked library, not release-build, fleet-load, starvation or attack results. The added cost is real and should be reduced or offloaded, but I did not establish the blocking severity claimed earlier.
  2. Audit read-error telemetry: operational follow-up. src/replication/audit.rs:103-112 intentionally treats an unreadable mark as set and skips judgement. The missing local log/metric is real; the fail-safe direction is correct. No paid-list failure-injection experiment was run. Missing observability alone does not justify the blocking correctness claim I made.
  3. First-record-wins is deliberate, not a missing upgrade. ADR-0006:587-597 records the front-running trade-off. Unconditionally marking an existing unmarked key on a later cheap payment would allow audit suppression for gas. Retain the disclosed residual; do not present that upgrade as a safe fix.

Participation, dissent and coverage limits

  • Original full-production first pass: GLM 5.3 requested changes; Qwen 3.8 27B approved. DS4's first response truncated; its sole correctness retry was labelled with a malformed 39-character OID, so it is not a valid exact-revision correctness receipt. My original public body also contained that malformed OID, although GitHub attached the review to the correct commit shown above.
  • Adversarial reconciliation of the disputed claims: GLM 5.3, Qwen 3.8 27B and DS4 Flash all returned substantive APPROVE responses. GLM withdrew its blocking interpretation after the baseline/measurement comparison. GLM's first reconciliation request timed out and DS4's first reconciliation response truncated; each received one narrowed same-model retry. These are clarifications/retries, not extra votes.
  • Qwen's adversarial pass included the full production delta; the successful GLM and DS4 reconciliation retries were focused packets. There are five valid model-role responses: three full-delta responses and two focused reconciliations, not six complete full-scope roles. Full-production first-pass participation is 2/3; whole-PR coverage remains provisional. No 6/6 or full whole-PR consensus is claimed. DS4's missing full correctness coverage is not silently filled by its focused answer.
  • The two ordinary delegated source-tracing reviews found no concrete implementation blocker. They are not additional named-panel votes, and neither independently ran the tests.

Verification actually executed at this head

  • cargo fmt --check: exit 0.
  • cargo test --locked --lib --features test-utils -- --test-threads=4: 1,202 passed, no failures.
  • cargo test --locked --test e2e --features test-utils -- --test-threads=1: 124 passed, 3 ignored, no failures.
  • cargo test --locked --test poc_price_floor_live --features test-utils -- --test-threads=1: 2 passed, no failures. These use real local Anvil settlements, but injected routing/gossip; they are not mixed-version production tests.
  • The opt-in production-wired price_floor_ab workload and the author's telemetry/mutation campaign were not independently reproduced.
  • CI at this exact head has a failed Security Audit for the inherited rustls-webpki advisory; Cargo.lock is unchanged from the merge-base. Other reported checks succeeded, apart from the skipped claude job. I did not enumerate branch-protection requirements, so the earlier claim that “all required checks” passed was unsupported.

Human release/strategy decisions remain open

The 30% constant is an empirical policy, not proof that all honest traffic clears the floor. My earlier extrapolation from a different shadow sample was too broad: that sample does not establish admission safety across changed views, lone fresh-node quotes, or aged proofs.

ADR-0006:598-619 explicitly describes old binaries penalising upgraded refusers, including marks that survive upgrade; new binaries' contested filtering does not protect old auditors. The operator override is removed by design. Disclosure in a Proposed ADR is not human acceptance of this release strategy. That risk and the unmeasured one-quote liveness residual need the proper human sign-off. This provisional implementation approval does not authorise acceptance of the ADR, a strategy decision, merge, release or deployment.

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected code-review verdict: APPROVE (provisional coverage)

This supersedes my earlier REQUEST_CHANGES review on the same code. The late source-review batch prompted a reconciliation. I overstated two findings as blockers. Baseline comparison, a component timing probe, and bounded adversarial reconciliation do not support retaining those blockers. No branch changes were needed for this correction.

Repository: WithAutonomi/ant-node. Reviewed head: bc714aebfae92f6b68f1bb8847f8328e052ea2df. True merge-base: a6e3388e79d0907ef678bcb6ec0fcaa12822e148, not current main.

Findings corrected

  1. Extra synchronous scan: performance follow-up, not a demonstrated blocking DoS. src/replication/neighbor_sync.rs:93-100 really adds price_contested_keys() on the async executor. However, the baseline already synchronously scans the full paid-list via all_keys() on the same sync-request path. A scratch-only probe linked the exact frozen library, preseeded LMDB, and verified both key counts. At 1,000,000 entries the additional scan's median was 12.156 ms with no marks / 71.591 ms with every key marked; the existing scan was 14.940 / 15.510 ms. These are three warm-cache runs per case with the debug-linked library, not release-build, fleet-load, starvation or attack results. The added cost is real and should be reduced or offloaded, but I did not establish the blocking severity claimed earlier.
  2. Audit read-error telemetry: operational follow-up. src/replication/audit.rs:103-112 intentionally treats an unreadable mark as set and skips judgement. The missing local log/metric is real; the fail-safe direction is correct. No paid-list failure-injection experiment was run. Missing observability alone does not justify the blocking correctness claim I made.
  3. First-record-wins is deliberate, not a missing upgrade. ADR-0006:587-597 records the front-running trade-off. Unconditionally marking an existing unmarked key on a later cheap payment would allow audit suppression for gas. Retain the disclosed residual; do not present that upgrade as a safe fix.

Participation, dissent and coverage limits

  • Original full-production first pass: GLM 5.3 requested changes; Qwen 3.8 27B approved. DS4's first response truncated; its sole correctness retry was labelled with a malformed 39-character OID, so it is not a valid exact-revision correctness receipt. My original public body also contained that malformed OID, although GitHub attached the review to the correct commit shown above.
  • Adversarial reconciliation of the disputed claims: GLM 5.3, Qwen 3.8 27B and DS4 Flash all returned substantive APPROVE responses. GLM withdrew its blocking interpretation after the baseline/measurement comparison. GLM's first reconciliation request timed out and DS4's first reconciliation response truncated; each received one narrowed same-model retry. These are clarifications/retries, not extra votes.
  • Qwen's adversarial pass included the full production delta; the successful GLM and DS4 reconciliation retries were focused packets. There are five valid model-role responses: three full-delta responses and two focused reconciliations, not six complete full-scope roles. Full-production first-pass participation is 2/3; whole-PR coverage remains provisional. No 6/6 or full whole-PR consensus is claimed. DS4's missing full correctness coverage is not silently filled by its focused answer.
  • The two ordinary delegated source-tracing reviews found no concrete implementation blocker. They are not additional named-panel votes, and neither independently ran the tests.

Verification actually executed at this head

  • cargo fmt --check: exit 0.
  • cargo test --locked --lib --features test-utils -- --test-threads=4: 1,202 passed, no failures.
  • cargo test --locked --test e2e --features test-utils -- --test-threads=1: 124 passed, 3 ignored, no failures.
  • cargo test --locked --test poc_price_floor_live --features test-utils -- --test-threads=1: 2 passed, no failures. These use real local Anvil settlements, but injected routing/gossip; they are not mixed-version production tests.
  • The opt-in production-wired price_floor_ab workload and the author's telemetry/mutation campaign were not independently reproduced.
  • CI at this exact head has a failed Security Audit for the inherited rustls-webpki advisory; Cargo.lock is unchanged from the merge-base. Other reported checks succeeded, apart from the skipped claude job. I did not enumerate branch-protection requirements, so the earlier claim that “all required checks” passed was unsupported.

Human release/strategy decisions remain open

The 30% constant is an empirical policy, not proof that all honest traffic clears the floor. My earlier extrapolation from a different shadow sample was too broad: that sample does not establish admission safety across changed views, lone fresh-node quotes, or aged proofs.

ADR-0006:598-619 explicitly describes old binaries penalising upgraded refusers, including marks that survive upgrade; new binaries' contested filtering does not protect old auditors. The operator override is removed by design. Disclosure in a Proposed ADR is not human acceptance of this release strategy. That risk and the unmeasured one-quote liveness residual need the proper human sign-off. This provisional implementation approval does not authorise acceptance of the ADR, a strategy decision, merge, release or deployment.

@grumbach
grumbach force-pushed the price-floor-live-enforcement branch 2 times, most recently from 297f587 to 9c2d279 Compare October 21, 2026 11:36

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — re-review of 9c2d2798c2c78d9af5c756efcc877a55b7a83629

Coverage: 1/1 seat reviewed (sol-1, gpt-6.1-sol, xhigh). This is a focused re-review of the delta since bc714aebfae92f6b68f1bb8847f8328e052ea2df; the previous review did not cover this commit. The seat approved but flagged cache-only marks after failed commits. Source tracing confirms that a later successful unmarked insert can make this a persistent cache/database inconsistency.

P2 — Reconcile cache-only marks after successful unmarked writes

src/replication/paid_list.rs:268-274 adds a key to the in-memory mark set before committing a marked insert, then deliberately retains it if the commit returns an error. If that commit did not persist, a later plain insert(key) sees no LMDB row and successfully writes an empty value, but this branch does not remove the cached mark. retain_unmarked then suppresses the now-unmarked key from replica hints, while is_price_contested and the audit paths read the empty LMDB value and treat it as unmarked. That breaks the first-record-wins/cache-consistency invariant and can persist until restart.

After a successful plain insert commit, remove that key from the cached set while still holding write_order; keep the conservative cached mark when the commit fails. Also reconcile cached-only keys on successful batch removal: remove_batch currently removes from the set only keys LMDB reports as deleted (src/replication/paid_list.rs:628-634). Add regression tests for a cache-only mark followed by a successful plain insert and by a successful batch removal of an absent key.

The hosted Security Audit failure is not attributed to this delta: Cargo.lock is unchanged from the merge-base. This code review does not accept the Proposed ADR or authorise merge, release or deployment.

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — validation gates remain

Coverage: 1/1 seat reviewed (sol-1, gpt-6.1-sol, xhigh). The focused re-review covers head 0425e1e07f1a2b93228532df0dd1e5fa659f0fc1. The earlier seat response was uncertain; its single narrowed retry approved the code change. Source inspection confirms the reported stale-mark paths are addressed. I found no new code defect in this delta.

This request is for outstanding required validation, not a code finding:

  1. T3 dev-testnet evidence is still missing. The PR marks this as T3 and says the required dev-testnet run remains outstanding. The successful hosted “WebRTC Direct five-node devnet” check is not that run. Please attach the required dev-testnet result.
  2. Security Audit is failing on this head. The log reports RUSTSEC-2026-0382 (rustls-webpki 0.103.15, upgrade to at least 0.103.16) and RUSTSEC-2026-0383 (webpki-root-certs 1.0.9, upgrade to at least 1.0.10). Please resolve through the dependency/lockfile update and rerun this check, or provide the repository-approved disposition for this inherited advisory gate.

The targeted local paid-list test did not finish: compilation exceeded the available run window and was terminated before test output. I do not claim it passed. The code opinion is based on the reviewed source and the seat receipt.

The ADR-0006 floor has run in shadow mode since v0.17.1. A week of
production telemetry (2026-10-14..20, 699 node instances on 72 hosts)
showed that its 65% default would have refused 2.5% of admissions, and
2.1% of chunks at every receiver that saw them. Neighbours gossiping
inflated commitments could refuse a chunk everywhere. One accepting node
could still replicate a below-floor write through PaidNotify and
paid-list repair. And a peer that refused a write on price would be
penalised for it by possession checks, responsible audits and prune
audits. This makes enforcement safe and turns it on for every node, with
nothing to configure.

- Cap the reference at the receiver's own price: min(group median,
  calculate_price(own committed count)). A node never demands more than
  the tolerance of what it would itself quote, so lying neighbours
  cannot make it refuse a payment that clears its own price.
- Lower the tolerance from 65% to 30%, a deliberate starting point. No
  measured admission is refused at 30% or at 40%, but the worst clears
  30% by 1.78x and 40% by only 1.34x, on a week of 48 upload bursts. A
  baseline quote is still refused by 91% of receivers (98% at 40%).
- Floor paid-list admission too. PaidNotify only announces a client's
  write, with the proof the client presented, so this reprices no
  historical receipt. It stops one accepting node from giving a
  below-floor write the paid-list votes that repair needs.
- Judge no peer on a contested write. A payment is contested when the
  node cannot price it, or its paid price is below 60% of the uncapped
  group median (twice the floor). The verifier remembers each contested
  chunk payment it admits, in a memory of its own, and every record path
  (PUT, fresh offer, PaidNotify) marks the key's paid-list entry from
  that decision, never from the unchecked proof a cache hit carries; the
  first paid-list record of a key decides. A marked key goes out in paid
  hints only, so no repair proof is recorded, no responsible audit
  samples it and no peer penalises this node as its sole hinter; the
  paid list keeps its marked keys in memory too, so building hints
  scans nothing on disk for them. A prune audit still needs its proofs
  but scores no absent peer; paid-list pruning keeps the marked entry
  while the record is held; and a mark that cannot be read counts as
  set, and is logged. A chunk fan-out schedules no
  possession check when the write is marked (by the verified decision
  or by the record that won, read after its own is written) or the
  proof it forwards is contested, and a pointer fan-out none when its
  proof is contested. Releases before this one only test paid-list
  presence, so they read a marked key as paid. The 2x margin covers the
  largest spread measured between receivers' views (1.91x); it is a
  margin, not a guarantee.
- One floor for every node. PriceFloorConfig and
  PaymentVerifierConfig::price_floor are removed: the tolerance is the
  constant PRICE_FLOOR_TOLERANCE_PERCENT and the contest threshold a
  private one. ANT_PRICE_FLOOR_ENFORCE and
  ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, and backing the
  floor out takes a release.
- The telemetry line adds target, group_median_price, own_price,
  paid_quote_age_secs and contested, and keeps enforce=true constant.
- The group sample drops its self-filter: the gossip cache cannot hold
  this node, since ingest accepts only senders in the routing table.
- Repair tests/poc_price_floor_live.rs, which no longer exercised the
  close-group floor and failed on main. Pin the chunk and pointer
  possession schedulers, the mark and hints, the prune audit and all
  three record paths with e2e tests, each with a control. Add an
  env-gated A/B workload driver, tests/e2e/price_floor_ab.rs.

ADR-0006 gains Amendment 2, recording the measurement, these decisions
and the residuals. ADR-0008's status row is updated.

BREAKING CHANGE: PriceFloorConfig (with from_env, enforce and
tolerance_percent), PaymentVerifierConfig::price_floor,
PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV are removed, and
PRICE_FLOOR_TOLERANCE_PERCENT is added. The floor is enforced at 30% on
every node. neighbor_sync::build_replica_hints_for_peer and
audit::audit_tick_with_repair_proofs take the paid list, and
ReplicationEngine::replicate_fresh is only built with test-utils.
…clears

The paid list keeps its price-contested keys in memory, so building
replica hints reads nothing on disk. A marked insert puts its key in that
copy just before its commit and keeps it if the commit fails, since the
commit may still have reached the disk. When it had not, the copy held a
mark the database lacked, and nothing removed it: a later plain insert of
the key wrote an unmarked entry, and a batch removal cleared only the keys
the database reported deleted. Replica hints then left out a key that
every point read, the audit included, treated as unmarked, until a
restart.

A plain insert that adds the key now drops any in-memory mark for it once
its commit succeeds, still under the write order, and a removal, single or
batched, drops the mark of every key it names, held or not. A failed
commit still leaves the copy as it was, so the copy stays a superset of
the committed marks, and equals them for every key a successful write
touched.

Tests mark a key in memory only, as such a failed commit leaves it, and
check that a plain insert, a batch removal and a single removal of the
key each clear it, while a key whose entry carries the mark keeps it.
The mutation run on the price-floor code now covers the in-memory mark
reconciliation too: 54 mutations, of which 53 fail at least one unit or
e2e test. The one survivor is unchanged: prune-audit scoring reading a
mark that cannot be read as unset, which only the e2e prune test reaches,
and that test cannot make a paid-list read fail.
@grumbach
grumbach force-pushed the price-floor-live-enforcement branch from 0425e1e to 668f573 Compare October 22, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants