Repository navigation
perf: keep Telegram, chart, detects and report paths at milliseconds under load - #1148
Merged
Merged
Conversation
Moonbot-lines tested each of up to 1000 history rows against every retained closed order linearly, in both the archive and the trade source: 1.52 ms per refresh at 1000 x 500 on the UI thread. Order captions checked every displaced caption against the whole accepted prefix, so a pile of captions at the book edge grew quadratically. - Sort finite close times once per refresh and answer each history row with one binary search over the 15 second window (0.10 ms). - Merge competing captions into runs that a displaced caption reads instead of the prefix: 400/800/1600 captions take 0.50/1.02/2.13 ms. Measured and left as they are: trace ranking 0.08 ms, trace stamps 0.48 ms, sell-line notionals 0.09 ms, the price-alert order scan 0.25 ms, order geometry 0.77 ms per chart at ten cores. Geometry and captions feed DX11, wgpu and Metal from the same CPU layout. Parity tests compare both changes against frozen copies of the old code.
Taking a Detects card in ran detect_snapshot on the UI thread: a copy and fold of the provider trade ring plus a kline cache read that can block for its 250 ms timeout. A panel just built, or the add-to-chart replay, took that snapshot for every eligible detection before the 48-card trim - 401 ms at 20000 candidates. Every Detects slider step also wrote detects_view.toml synchronously (11 ms median). - Select the surviving cards first, so a pass requests at most the card budget (20000 -> 48 snapshots, selection 1.2 ms). - Capture a cheap request on the UI thread and assemble the history on the background executor; a result reaches only the card generation that asked for it, so replaced or pruned cards drop it. - Read retained prints with a time-filtered cursor scan instead of a capacity-sized reservation; the row set and order are unchanged. - Route display preference saves through the existing persistence coordinator, coalesced and flushed on quit. Selection, ties, replay skipping, the 48-card budget and crowd seats are pinned by parity tests against frozen copies of the old code.
Every published Report result, the automatic 5-second reads included, cleared the natural-width cache, and the next render formatted and measured up to 500 rows x every visible column on the UI thread (14 ms). Strategy discovery ran two SELECT DISTINCT arms over the whole scope every minute, sliced totals tested every row against every slice, and Profit Monitor read its deals twice. - Keep widths across unchanged publications and measure only unseen row identities; identities are prepared by the background reader (14 -> 0.17 ms per 500-row page). - Discover strategy identities through covering indexes (27.5 -> 0.12 ms at 100000 rows). - Route sliced totals through an ordered per-core slice index (84 -> 70 ms at 100000 rows x 32 slices). - Fold Profit Monitor's latest-trade pass into its grouped read (81 -> 65 ms at 100000 rows). - Order Summary by true UTC only and reuse group keys; the former tie order is restored within equal-time runs. Widths, rows, totals and their order are pinned by parity tests against frozen copies of the old SQL and Rust.
The feed-to-UI channel is an unbounded FIFO, and a stalled UI drain lets full order snapshots from every core pile up. Replacing an undrained snapshot with a newer one looked like the fix, but it loses history that only snapshots carry: terminal order states, omitted-row closures and per-snapshot line revisions. Keep the FIFO and add: - an ignored ten-core probe of a 2-second stall at 20 Hz x 200 orders (400 snapshots, about 366 MB of owned row payload); - a queue/session regression proving the drained history keeps every snapshot-only terminal row in order. The snapshot converter measured 0.24 ms median per core at 200 orders and stays as it is.
The notification loop runs every 100 ms on the station and the desktop. Each tick cloned every enabled chat's settings and full 72-hour ledger under the store lock before deciding whether a read was due (4.6 ms at 10 chats x 50000 ledger entries). Every delivery ack cloned and saved the whole settings + ledgers + outbox document, and the sender cloned the remaining outbox after each send and found rows linearly, so draining 10000 queued cards took 160 ms of CPU. - Check the busy, interval and due gates before capturing anything; capture only what the read uses (4.6 -> 0.00005 ms). - Keep the outbox as an indexed FIFO with per-chat heads and photo ownership, serialized as the same JSON array (drain 160 -> 3.1 ms, sender snapshot 2.7 -> 0.003 ms). No card is capped or dropped. - Persist each ack as a small fsynced receipt in a sidecar journal, replayed at load and retired after the next snapshot, instead of rewriting the whole document; the crash window between Bot API acceptance and the durable ack is the same as before. - Cache disclosure purges by grant and queue revision. - Move the pure trade decisions into the optimized moon-core backend. Per-chat order, pacing, retries and revocation are pinned by parity tests against the previous FIFO and acknowledgement behaviour.
Core Status rebuilds its cache once per second on the UI thread, and every rebuild queried the persisted warning episodes. The query filtered with an OR across cores and servers and sorted every matching episode before LIMIT, so it grew with uptime: 58.6 ms at 100000 retained episodes, once a second, in every view. - Add covering indexes and read each core and server through them, expanding ties at the cutoff and merging candidates in rowid order so the rows and tie order match the old query. - Read only when the store's persistence revision or the scope changed, on the background executor; an older result that lands after a newer one is dropped (8.7 ms off the UI thread). The list shown is unchanged; parity tests run the old query on the old schema as the oracle, ties and boundaries included.
The single-pass Profit Monitor read picked the latest core name with a custom aggregate that accepted only NULL, integer and real close dates. A legacy or untyped replica row with a text close date failed the whole read, where the old two-pass SQL had followed SQLite's MAX order and returned results. A REAL maximum close date also failed the decode, as it did before the rewrite. - Order close dates as SQLite's MAX does across every storage class (NULL < numbers < text < blob) and pick the name exactly as the old statement did, ties included. - Decode a non-integer latest close date without failing the read. - Seek strategy discovery directly for windows of a day or less; the identity probe stays for longer windows (100000 deals, one day: 4.0 -> 3.6 ms, one month: 80.9 -> 38.7 ms on the replica). Parity tests run the frozen old SQL over every close-date storage class, mixed precision and all-NULL groups.
The receipt journal added for delivery acks could replay stale receipts. An older binary ignores the sidecar, resends journal-only acked cards and drops ack_sequence on its next save; after upgrading again the leftover journal replayed in full and stale "replaced" receipts overwrote report slots with older message ids. One corrupt complete line also kept the notify store from opening at all. - Checkpoint the store when TelegramService stops, on both the station shutdown and desktop quit paths, so a normal binary swap leaves an empty journal. - Discard a journal that does not continue the snapshot's cursor; replay the valid prefix of a corrupt journal and open. - Fsync the journal's directory when the file is first created. - Take the store lock once per delivered-id pruning pass. - Remove a chart photo whose row a replayed receipt removed. The sidecar is documented as part of the notification state: copy or restore both files together.
Opening the warning store ran index DDL on the UI thread inside the startup initializer. The two read indexes added for Core Status were built there over the existing history on the first launch (35 ms at 100000 episodes, 899 ms at 1000000), and an older migration dropped and recreated the series index on every launch (70 ms at 100000 series rows). The store also never enabled WAL, so the new background reader could hold back an episode insert on the UI thread. - Build read indexes over existing history on a background connection; a fresh empty store still builds them inline. Store open now returns in 1.3-2.4 ms. - Rebuild the series index only when its stored definition differs, and queue captures, episodes and pruning behind that one-time maintenance instead of waiting on the UI thread. An explicit Storage forget waits for it to keep its synchronous count (104 ms worst case measured). - Enable WAL with synchronous NORMAL through moon-core's shared helper, so reads and inserts no longer block each other. Core Status rows and their order are unchanged; parity tests run with and without the new indexes.
The Telegram Mini App polls the station's owner thread: Orders and Strategies every 2 s per open client. Each Orders poll cloned every order on every visible core and rescanned a snapshot map per row; each Strategies poll reindexed all orders and sorted per-strategy coin lists. A burst of polling clients could fill the 64-entry owner work queue, and requests past it were answered 503. - Index each core's active orders by strategy and quote/market once per order-table revision, and keep an open-panic market index. - Answer an Orders or Strategies poll whose exact inputs did not change from its cached response; rows are shared, the JSON is the same. - Coalesce identical in-flight reads onto one work item, with write barriers so a read never jumps an earlier write; replies wake within 1 ms. The coin-chart scan measured 0.079 ms and stays as it is. Rows, order, expiry and wire JSON are pinned by parity tests against frozen full scans.
A "Today" report or a report button could take seconds to answer. The status command fetched the latest GitHub release on the serial bot loop with a 10-second bound, so every update behind it waited. The station owner only sampled incoming work every 100 ms, and a finished report waited for the next sample again; the report read also ran a second full totals pass for the exchange buttons. - Wake the station and desktop owners when bot work or a report completion is published instead of on the next 100 ms tick. - Serve status from a release result refreshed in the background at startup and on the existing release-check cadence; a stale result starts a refresh without waiting. No network call runs on the bot loop. - Read the exchange-button activity in the same totals batch. - Share only an explicit 429 retry_after deadline between the reply and notification clients; replies keep their own client and never wait behind notification pacing. With 100000 deals in Today across ten cores, a 50-card backlog and a concurrent chart render, the bot issues the reply in 74 ms p95 (301 ms with tick polling). Report text and keyboards are pinned by parity tests.
The chart's price-volume profile refolded its whole window under the shared VolumeBook lock on the UI thread on every recheck (3.5 ms at 20000 mini-candles plus 100000 prints), the first side-series seed walked the whole retained history there too, and every frozen-replay history read re-filtered and copied up to 98000 ticks, camera moves included. - Fold the profile forward from a cursor and expire the retained prefix incrementally; refold fully only when the window, row width, provider, client epoch, catalog or archive moves (3.47 -> 0.045 ms). - Seed side series on a background worker; the UI submits the job in 0.005 ms and the lock is not held across the history walk. - Answer replay reads from the prepared sorted slice and fit prices once (full read 0.82 -> 0.23 ms, 1-second window 0.14 -> 0.003 ms). Retained price-line scans and the kline recorder's first pass measured under 0.25 ms and stay as they are. Bins, rows, order and revisions are pinned by parity tests against the frozen full fold.
The Mini App read coalescing and the causal owner wakes landed side by side and did not compile together: MiniAppPolls::relay still took a bare SyncSender<Work> while owner work now publishes through WorkSender, which wakes the owner after admission. The receipt shutdown test fixture also lacked the new work_waker field. Publish relayed Mini App requests through WorkSender, so a coalesced read wakes the owner at once; write-generation advance, write barriers and fanout are unchanged. Give the shutdown fixture its waker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
With many deals and busy Telegram chats, several hot paths ran whole-table
passes or blocked the UI / bot thread. This set keeps them at milliseconds.
Each commit carries its own symptom, cause, fix and numbers.
Also: Mini App polls answered from cached reads, notification receipts
checkpointed at shutdown and stale ones rejected, and a feed test that pins
the stalled-order backlog.
How to verify
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningsmake test(nextest: 8644 passed, 136 skipped; doc tests green)Not covered