Skip to content

perf(station): resolve each listed coin's market once per catalog revision - #1143

Merged
guyverino merged 1 commit into
mainfrom
perf/catchup-coin-cache
Oct 29, 2026
Merged

guyverino merged 1 commit into
mainfrom
perf/catchup-coin-cache

Conversation

@guyverino

Copy link
Copy Markdown
Collaborator

What & why

The tape catch-up listing (SessionManager::list_tape_catchup) walks every closed trade of a core and called MarketDataSource::resolve_market for each row, which searches the provider's whole catalog. A 70-core station report (#1081) counted 427k searches over ~10k distinct coins per pass: the same answer about forty times. On our station the hourly pass held the machine at 100 % for 20–27 s every hour (cpu_watch).

The listing now keeps, per listing, the market each coin resolved to (session/lifecycle/catchup_markets.rs), and drops all of it whenever any other input of the search moves: the core's catalog revision, its account currency, its exchange. The tape catch-up … listed line gains N market searches, X s resolving, so a station reports both figures itself.

Measured on a local station against our 27 cores (read-only, the same database copy restored before each run, one binary with the cache switched off for "before"), first listing of every core:

before after
rows listed 216,122 216,117
market searches 216,120 21,100
time resolving 8.97 s 0.69 s
listing wall time, all cores 13.9 s 4.2 s
trades judged by retention 199,894 199,894

(The row count moves by a few between runs: the replica receives new closed trades from the live cores.) In a pair of runs with every row's answer dumped, each row looked up in both runs resolved to the same market or the same "not found": zero differences.

Live on our station with this change: the hourly listing at 11:15 UTC (27 listings, 138,764 rows, 3,759 searches) raised no cpu_watch episode; the release's hourly pass reached 100 % of the machine for 20–27 s.

Notable decisions

  • A "not found" is never cached. The catalog revision moves when the feed thread drains the market-list event, a moment after the client applied the list; a "not found" kept in that moment would latch the trade as unknown until the process restarts. Refused coins are few (441 coins over 11k of 210k rows on our cores), so searching them again costs little.
  • A market found in that same moment can lag the new list for the rest of one listing. Closing that window fully needs a catalog version that moves atomically with the list in moonproto.
  • The cache lives for one listing only; nothing carries over to the next pass.

Known limitations

  • Unresolved rows still call tradable_coins per row, as before this change.

Issues

Refs #1081 — CPU of the hourly listing; the out-of-memory cause in #1081 is still open.

How to verify

cargo build -p moon-ui-gpui --bin moonterminal --target x86_64-pc-windows-msvc --all-targets
cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked --target x86_64-pc-windows-msvc -- -D warnings
cargo test --workspace --target x86_64-pc-windows-msvc

fmt exit 0, build exit 0, clippy zero warnings, 8559 tests green (three new in catchup_markets/tests.rs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant