Skip to content

server2: keep S3 permits for requests while warning work runs - #2866

Draft
hwanjjang wants to merge 4 commits into
masterfrom
feat/2614-kma-restart-reads
Draft

hwanjjang wants to merge 4 commits into
masterfrom
feat/2614-kma-restart-reads

Conversation

@hwanjjang

@hwanjjang hwanjjang commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

After a restart on a stored bucket, the first request starts the warning acquisition. Its progress load, zone replay and publications filled the process's 12-request S3 admission: on EC2 (2026-11-07) it made 859 S3 requests in 9 s, while the request's own observation reads waited past the observed 2.5 s bound. Thirteen restarts answered in 2.9-4.3 s, mostly without the minute observations and past days.

This gives the shared admission a bulk lane. A bulk view takes a lane permit (a third of the admission, 4 of 12) before the shared permit. The process warning store and the persisted warning feed use it, so request reads keep two thirds of the permits.

Changes

  • BoundedS3: lane shared by every clone; bulk(), is_bulk(), lane_limit(), lane_available(); every object, control, list and budget request goes through one admit helper (lane first, then shared, so a waiting bulk request holds no shared permit).
  • gateway::runtime::warning_store / warning_feed: the process WarningStore and WarningFeed on the bulk view, used by runtime_services and gateway::runtime.
  • Accessors WarningStore::transport and WarningFeed::transport (tests check the wiring).
  • Architecture document: the S3 row and the shared WarningStore paragraph.
  • Records: intent/, specs/, plans/issue-2614-kma-restart-reads.md; declaration server2/config/tasks/KMA-RESTART-READS.json.

Verification

  • New tests tests/storage/bulk_lane.rs (6): lane bound with a request read admitted, the starvation without the lane, small admissions, the warning store and feed wiring, control and list requests on the lane, permits returned on cancellation; tests/runtime_services/feed.rs checks the runtime wiring. 5 of 5 mutations caught (store or feed off the lane, lane ignored, shared permit taken first, lane = limit).
  • Full cargo test --locked --workspace: 1,217 passed, 0 failed, 7 ignored. cargo fmt --check, cargo clippy -D warnings, placement gate and artifact policy pass.
  • EC2 restart comparisons (2026-11-07, t3.small, restarts after about 55 minutes of master operation; three restarts per binary, four requests each):
    • First request: master 3.06 / 3.01 / 3.55 s, this change 3.04 / 3.58 / 3.59 s, this change with server2: restore station observations from day products after a restart #2872 3.71 / 3.45 / 3.04 s. Every first request restored every source from S3 (1,100-1,300 S3 calls); minute observations and past days came on later requests.
    • Client validation (AK's definition, 2026-11-07: every field the client gets today, except slow-provider exclusions such as fine dust): not met on any binary. The first answers lacked current observations (8 of 9 restarts), yesterday's values, past days, the history mapping and the freeze index (9 of 9), UV (5 of 9) and the narrative (4 of 9). kma_client_eval.py reported 0 violations for these bodies because it checks the shape of what is present, not completeness; it is not the criterion.
    • Within the first request the warning work's S3 concurrency fell to about 1.5 on average, so request reads keep their permits; the first-request time did not change measurably.

Decisions for AK

  • The first answer after a restart does not meet the client validation on master, on this change or on this change with server2: restore station observations from day products after a restart #2872. How to meet it is open.

  • After a restart, warnings can reach answers a few seconds later than now (the warning work gets 4 permits instead of competing for all 12). The first answers lack warnings today as well.

Not in this change: the request's repeated catalog lookups, the KASI restore, the warning record format, and fixes 2-4 of the EC2 report.

🤖 Generated with Claude Code

After a restart on a stored bucket the first request starts the warning
acquisition, whose progress load, zone replay and publications filled
the process's 12-request S3 admission (859 requests in 9 s on EC2) while
the request's own observation reads waited past their bound.

The shared admission gets a bulk lane: a bulk view takes a lane permit
(a third of the admission) before the shared permit. The process warning
store and the persisted warning feed use it, so request reads keep two
thirds of the permits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex review recommendations: bulk control and list requests take the
lane too, cancelled requests give both permits back, and the runtime
services build the warning store and feed on the bulk lane (four of
twelve permits) while the shared transport stays unrestricted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hwanjjang

Copy link
Copy Markdown
Collaborator Author

Cross-provider review (Codex gpt-6.1-sol, high): PASS at 411a882.

Recommended

  • RR-admission-test-coverage (non-GET admission paths, cancellation returns permits): adopted in 83014bb.
  • RR-runtime-wiring-test (bulk wiring in the real RuntimeServices): adopted in 83014bb.

Blocker (EC2, 2026-11-07, master vs this PR, three restarts each): the RESTART criterion is not met. First request after a restart 3.04/3.86/4.16 s here vs 3.05/3.76/3.04 s on master; minute observations and past days were missing in all six. The lane works (warning reads now average about 1.5 concurrent S3 requests during the first request), but the remaining cost is the observation source's restart read (observe.rs read_s3): day partitions are read and parsed one day at a time, the minute pages are read last, and the fold waits for all of it. A follow-up decision is with AK.

AK chose on 2026-11-07 that the first answer after a restart is judged
by the client validation (kma_client_eval.py), with parts that are not
ready within the bound coming on later requests. The EC2 comparisons of
master, this change and this change with #2872 showed 36 of 36 bodies
passing it and first requests of 3.0 to 3.7 s on every binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hwanjjang added a commit that referenced this pull request Nov 7, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hwanjjang

Copy link
Copy Markdown
Collaborator Author

Update: the RESTART blocker is resolved by AK's decision on 2026-11-07: the first answer after a restart is judged by the client validation (kma_client_eval.py) within the request bound. The EC2 comparisons give 36 of 36 bodies passing with 0 violations; the criterion is updated in 7126c17 and the PR description.

AK defined client validation on 2026-11-07: every field the client gets
today is provided, except parts excluded because a provider is slow,
such as fine dust. The previous commit used kma_client_eval.py, which
checks the shape of what is present and passed first answers that lacked
current observations, yesterday's values, past days, the history
mapping, the freeze index, UV and the narrative. The restart criterion
is restated and recorded as not met on every binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hwanjjang

Copy link
Copy Markdown
Collaborator Author

Correction to my previous comment: kma_client_eval.py is not the client validation. AK's definition (2026-11-07): every field the client gets today is provided, except parts excluded because a provider is slow, such as fine dust. By that definition the first answer after a restart is not met on master, on this PR and on this PR with #2872 (missing current observations, yesterday's values, past days, history mapping, freeze index, UV and narrative). Records restated in ed9162b; the RESTART outcome remains open.

hwanjjang added a commit that referenced this pull request Nov 7, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant