[hwanjjang]

Summary

Stacked on #2866 (base feat/2614-kma-restart-reads) so CI builds both restart fixes together for one EC2 comparison. AK chose this design on 2026-11-07 ("A로 진행").

The EC2 restart comparison of #2866 showed that the first weather request after a restart misses the minute observations and the past days, and that S3 admission was not the remaining cause. The observation source's restart read (observe.rs read_s3) restored every stored AWS hourly and city page of the 8-day window one day after another and parsed each page again; the minute pages came last, and every minute page call read the descriptor of every group in the day's minute catalog. Measured parse cost: about 10 ms per AWS page on an M4, so roughly 1.0-1.4 s per day and 12-17 s for 8 days on a t3.small, plus 4-5 s of sequential GETs, against a 2.5 s request wait.

Changes

• Minute pages (storage/catalog.rs lookup_members, records.rs): a restart lists the day's minute catalog once and restores only the newest acquisition of each minute within the readers' 40 minutes; the newest four before the first answer, the rest after it.
• Day product (observe_product.rs, kind station-pages, one per KST day): the parsed inputs of the hourly fold (the newest AWS hourly and city page of each hour, with their page identities), written by any process that folds, at most once per ten minutes per day and only when what the day is made of changed. No request waits for it. Excluded from the disk cache like the image forecast product.
• Restart read: each day's newest product is read and compared with the day catalogs (S3 is truth); only pages the product lacks are restored and parsed. Today and yesterday are compared before the first answer, the other days after it (LoadRest). An unreadable product, an older schema or parse revision, or a grown city set falls back to the day's raw pages and the product is stored again.
• S16: page identities, legacy answers and the response-first durability set are unchanged. A day stays partial when its product's read had an S06-excluded group (excluded carried in the product) or a newer group is not complete.
• Documents: server2.md (S15b-2 restart read, S16 note), KMA data design (+ regenerated Archify sequence), infrastructure. Records: intent/, specs/, plans/issue-2614-kma-restart-observe.md; declaration server2/config/tasks/KMA-RESTART-OBSERVE.json.

Verification

• New tests: targeted_members_read_one_catalog_and_only_their_groups, a_restart_restores_only_the_minute_pages_within_reach (200 stored minute groups: at most 20 descriptor and 20 raw GETs), and six observe_product tests (restart answers 168 past hours and minute rows with providers unreachable and no raw GET for covered pages; a later page restored alone and equal to a cold read; fallbacks; two writers; unsettled while LoadRest runs; an excluded group stays partial; products not cached). Each failed before the change; mutations caught: LoadRest dropped from settled, excluded not carried, made-of check ignored.
• Full cargo test --locked --workspace on the stack at 9c16049: 1,236 passed, 0 failed, 7 ignored; CI passes. cargo fmt --check, cargo clippy -D warnings, placement gate and artifact policy pass. Archify: validate, deliver, strict check and browser check pass.
• EC2 restart comparison (2026-11-07, t3.small, bucket after about 55 minutes of master operation, so one day partition only; master / server2: keep S3 permits for requests while warning work runs #2866 / this stack at 9c16049, three restarts each, first request to Seoul):
  • First request: master 3.06 / 3.01 / 3.55 s, server2: keep S3 permits for requests while warning work runs #2866 3.04 / 3.58 / 3.59 s, stack 3.71 / 3.45 / 3.04 s. Past days were missing in all nine first answers; minute observations were present only in the third stack restart.
  • The observation Load ended at 2.17-2.30 s with products (stack r2, r3) against 3.31-4.54 s for master's whole restart read; the remaining days continued in LoadRest.
  • Client validation (AK's definition: every field the client gets today, except slow-provider exclusions such as fine dust): not met by any first answer after a restart, on any binary (missing current observations, yesterday's values, past days, history mapping, freeze index, UV and narrative). kma_client_eval.py reported 0 violations for these bodies; it checks shape, not completeness.
  • Every first request made 1,100-1,300 S3 calls across all sources (forecasts, air, warnings, KASI, observations), so the first answer stays bound by the general restart restore, not by the observation read alone. A bucket with only one day of pages cannot show the 8-day saving this change targets.

Decisions for AK

• The first answer after a restart does not meet the client validation (AK's definition) with or without this change; this change shortens the observation restart read only. How to meet the criterion is open.

• Products are stored under kma-web / station-pages and never deleted: estimated 300-400 KB gzipped per day product, about 25 MB a day per writing process at the ten-minute limit (not measured).

• The first answer after a start never records daily summaries (S16): the days past yesterday are compared after it, so a later request records them.

• A group older than a day product's pages, published after the product and excluded by S06, is not noticed on a restart from the product (a raw read would mark the day partial).

• Right after the first deployment, the oldest day of the window has no product and is read from its raw pages after the first answer.

Not in this change: a product for the minute pages, the ASOS station state, the collector's restart probing.

🤖 Generated with Claude Code

[hwanjjang]

Cross-provider review (Codex gpt-6.1-sol, high): PASS at 9c16049 (third round).

Required (all fixed, each with a regression test seen failing first)

• RO-city-growth: a station joining the city set after a product left reduced AWS rows; fixed by a full re-read (Expand, 78886d2) that keeps answers unsettled until done and retries independently (9c16049).
• RO-minute-fallback: an incomplete newest minute acquisition dropped the minute; up to three candidates per minute now (78886d2).
• RO-sticky-excluded: a repaired excluded group stayed counted; products record knownIds/excludedIds and re-check them (78886d2).
• RO-expand-progress: a slow full re-read discarded completed batches; batches now apply as they complete (9c16049).

Recommended

• RO-record-key-overflow: adopted (9c16049).
• RO-shared-budget (whether this PR slows the mid forecast reads seen failing on CI for 78886d2): not adopted as a code change. The same failure reproduced under CPU load on the stack base 83014bb without this PR, the S3 request counts for that request are identical, and CI passes at 9c16049.

EC2 restart comparison: see the PR description. The first answer after a restart is still bound by the general restart restore (1,100-1,300 S3 calls across all sources).

[hwanjjang]

Cross-provider re-review (Codex gpt-6.1-sol, high), round 4 on the commits after the round-3 PASS (b34c001, 52a270c, 06018c2, e708d12): CHANGES_REQUESTED.

Required

• RO2-ready-join-race: a response joining after the writer counted zero waiters and before Ready can still be released early as NoWaiters. Same as RFW-zero-count-join on server2: count response-first waiters before a batch is ready #2903.

Recommended

• RO2-dormancy-test: the 2,000-acquisition test can hit the normal DormancyCap on a slow runner. Selected.

Both are being fixed once and applied identically here and on #2903.

[hwanjjang]

Cross-provider re-review (Codex gpt-6.1-sol, high), round 5: PASS at 76add9e. RO2-ready-join-race and RO2-dormancy-test are resolved. The writer and its tests are byte-identical to #2903.

[hwanjjang]

CI passes 14 of 14 at d840e9a, which merges master through #2866. Two earlier attempts were cancelled by the 45-minute job limit of the foundation job; passing runs take about 44 minutes. The cross-provider re-review passed (round 5). The restart race (#2900) builds on this PR; its EC2 results are on #2900.