Repository navigation
sparrow: silent payment subscribe scans the full silent index regardless of start #1044
Description
blockchain.silentpayments.subscribe scans every row of the silent index on each subscription, independent of the start parameter. The cost of a subscription is therefore proportional to the size of the index, not to the requested range.
Code
src/protocols/sparrow/protocol_sparrow.cpp (e027ad4):
do_silent_subscribesetssub.cursor = zero(L235) and scans[0, get_silent_frontier(zero))(L240).start(height or time) is applied only to the matches after the scan, inconfirm_silent→is_cut(L387, L413).- A resubscription of the same address also resets the cursor to zero and rescans the full index.
Observed
- Index from
node.silent_start_height = 954255to tip ~970860 (~16.6k blocks plus pooled txs), 16 threads. subscribe(start=970758)(~100 blocks): ~30 s at ~1400% CPU, empty history.subscribe(start=954158): ~30 s at ~1400% CPU, the one match at 954258.- Both take the same time. Time grows with the index:
silent_start_height = 709632(taproot activation) would be ~16x the rows. - Sparrow subscribes at least twice when it opens a wallet (once near the tip, then from the wallet birthday), so opening a wallet costs at least two full scans.
Constraint on deriving a start row
Rows are appended in write order, not height order: on block validation (set_silent / set_silents from chaser_validate_parallel, chaser_validate_batch) and on pool admission (chaser_transaction). As noted in query/batch/silent.ipp, txs archived before their confirming block (pooled) have lower links than that block's coinbase. So a binary search over the correlate column (tx links) for the coinbase of the start block would skip confirmed txs that were pooled earlier.
Possible directions
- A height → first-row mapping recorded at block indexation, plus the pooled rows (or a lower bound covering them).
- A start row from the coinbase link of the start block, with confirmed txs below it (pooled before confirmation) handled separately.
- For time-based
start, resolve the time to a height first (header timestamps / MTP), then apply 1 or 2.
Activity
evoskuil commentedon Oct 11, 2026
The "possible directions" assume blocks are stored as whole blocks, which is not the case (see compact blocks).
That decision has already been made. silent_start_height doesn't limit the scan, it limits what is indexed during IBD and limits the return from the scan. If you want a fast scan, then limit the index. OTOH if you are running a server, GPU scanning should be very fast (as is CPU scanning with AVX512-IFMA).
The alternative is to write the index during confirmation, not during validation. And note that validation means block or tx. With compact blocks the txs are validated and indexed as they arrive at the tx pool, not when the compact block arrives or is confirmed. Ordering by block implies moving tx writes to block confirm, and therefore reading the full block in both IBD and compact block scenarios. That's a very significant performance hit in both cases. We could just not do it with compact blocks, and rely on them all being written after any valid start height. But writing block records in order would still dramatically impact IBD.
evoskuil commentedon Oct 11, 2026
We might mitigate this by reliance on the concurrency window. During IBD the window sets a ceiling on block disorder by height. So at a 50k window, blocks, and therefore their IBD txs sets, are never more than 50,000 apart in height - unless there is a disk full condition during IBD. The disk full will result in tx duplicates, and partial txs - some orphaned inputs/outputs/etc. However, because IBD (prior to pooling latched) does not absorb previously-written txs, the ones associated to the block will remain in this order, and that order also dictates silent table ordering. Silent batching does rewrite order within a batch, due to parallel scan table writes of the batch results, however a batch never spans a concurrency window boundary (so that validation can close out the window, and SP completion is a requirement for validation completion).
As a result, we could create a map of block ranges, with everything subsequent to the pooling latch being perpetually in the last range. Because windows move during restarts, this would require a full scan of block headers and txs associations to produce the coinbase link of every block with height, and from that derive a set of SP scanning thresholds.
evoskuil commentedon Oct 11, 2026
Regarding the double scan shown in the OP:
Opening a wallet subscribes only once. The second subscribe comes from Sparrow wiping the wallet’s history and rescanning, and the only automatic path that does that is its “all history changed” check. This is from the code alone; I haven’t run anything, and there’s no local Sparrow log to confirm it.
Opening subscribes once. AppController.addWalletSubTab calls walletForm.refreshHistory(...) (AppController.java:1908). For an SP wallet that goes to refreshHistorySP, which subscribes with getNeededScanStart(). With a stored height that’s stored − 100 = 970758, the first subscribe. Every other refresh on the same WalletForm either waits behind it (spPendingRefresh) or reuses the held cache.
The second start means the stored height was reset. getNeededScanStart() falls back to birth − 100 (954158) only when storedBlockHeight is 0 or null. The stored height only ever moves forward, except in Wallet.clearHistory(), which sets it to 0. So clearHistory() ran between the two subscribes.
What calls clearHistory():
The user’s Refresh Wallet menu (AppController.refreshWallet).
The birth-date dialog (CoinTreeTable).
A settings change (saveAndRefresh).
The “Refresh Wallet” button on the proofs dialog.
Automatically: WalletForm.handleHistoryFailed when it catches AllHistoryChangedException.
Only the last one runs without a click.
If every used node differs, it logs “All used nodes on a non-empty wallet have changed history. Triggering a full wallet refresh.” and throws.
An SP wallet usually has very few used nodes; with one, a single mismatch counts as “all”.
The handler then clears the history, which resets the stored height to 0, drops the SP hold and refreshes. That refresh subscribes from birth − 100, and the server does a second full scan.
Two ways to settle it:
The reporter’s sparrow.log. If it contains that “All used nodes … changed history” line at open, this is confirmed. If it doesn’t, the second subscribe came from a user action.
Our status hash. The format in protocol_electrum::write_status (txhash:height:) matches Sparrow’s getScriptHashStatus. A mismatch would then come from ordering, from unconfirmed heights (we send 0 or −1 via to_signed), or from the history contents. The check is to compare our blockchain.scripthash.subscribe status for the wallet’s SP output script with Sparrow’s calculation from the same history. If ours differs, every open of that wallet in Sparrow costs two full scans, and the fix is on our side.
The dumb part on Sparrow’s side: a disagreement on scripthash status, a check designed for HD wallets, triggers a full silent payment rescan from the birth height. A scripthash disagreement says nothing about whether the SP scan missed anything.
Caveat: my Sparrow clone is from 2026-09-09, so the reporter’s build may differ.
evoskuil commentedon Oct 11, 2026
Sparrow is attempting to deepen the original scan by scanning again. Both of course take the same time, since they are the same scan for us, the only difference is we discard results based on confirmed height.
evoskuil commentedon Oct 11, 2026
I'm doing this, which comes close to the above suggestions, but it:
The last item means that the cost floor will continue to increase over time (until the node is resynced).
evoskuil commentedon Oct 12, 2026
Implemented:
libbitcoin/libbitcoin-database#1022
libbitcoin/libbitcoin-node#1209
#1054
An arraymap index head has been added to the SP scan table, mapping each confirmed block to the scan row at which its range starts. The index is populated once the node is current, which with relay enabled is also when it latches pooling. Pooling is permanent and marks the transition to blocks and compact blocks filling from existing txs, after which block txs are no longer positionally coherent in the scan table. Before that point the index is populated by a single sweep of the table, giving the scan start of any block that has been validated and is currently confirmed. This satisfies client-server scan cursoring by height. Cursoring by time is handled by resolving the time to a confirmed block floor through a binary search of confirmed median time past.
A current node that is not pooling continues to add to the index block by block. Reorgs do not invalidate the index, apart from a rebuild when the node is pooling and reorganizes to a branch archived before the latch. Once pooling is latched the index no longer grows, so the minimum scan depth grows from the latch forward. That can be reset by resyncing the node, but is probably not material even over years, given the CPU and GPU scanning optimizations already in place.