Repository navigation
Conversation
KMA-RESTART-RACE (AK 2026-11-07, plan A). resolver::race holds the 700 ms head start: a restore not done by then is reported (one operation.action, restore/timeout/fallback) and left running for the caller, who starts the providers of what memory lacks beside it. A head start no earlier than the end of the wait is none, so a source without one keeps its waits. IndexListing gains a bulk view for reads and writes no request waits for: the bulk lane of a shared BoundedS3 admission, otherwise the same transport. CatalogStore exposes its limits so a second store on that view uses the same ones. The task declaration lists every path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stored read of a forecast office now runs as owned work with its own 30 s deadline. A waiter that stops at its deadline neither cancels the read nor sets the one-second error gate; the next request joins the running read or answers from its outcome (KMA-RESTART-RACE, decision 6). Only a completed fault of the owned read sets the gate. A reader may prefetch an office for an answer that attaches the narrative later: the service starts the read and, when it has no answer at the given head start (still running or failed), fetches and publishes that office once, single flight and at most once per office and ten minutes. A newer published narrative stays the office's answer when an older stored read ends after it. A read that ended with no stored narrative fetches nothing. The cache test that expected the gate after a waiter's timeout now expects the next request to join the read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The observed source computes the 700 ms head start beside its wait; the legacy source has none. A restore not done by then is raced by the owned acquisitions of what memory lacks, as on an empty S3 (AK 2026-11-07, plan A), and a restore done by then leaves them to what S3 lacks. Station pages: the load is waited for only until the head start. A late load is raced by the minute page, the latest hourly and city page, the previous hourly page while a fetch now is within its ongoing reach, and a fill of the hours of today and yesterday the day catalogs do not list (each once per six hours; older days come from S3 alone). Whatever arrives first is folded while the rest is waited for. Provider pages are held newest-wins, so a late restore never replaces them, and a product page that loses stays accounted for S16. ASOS history starts with the request, its station from the current fold or the default station list; the recovery run starts at the head start, a run before the read creates the station entry, and the read loads both kinds together. Life: a process's first read restores today's partitions before the rest of the window; flights are decided at the head start; a request uses an acquisition's published records at once over the stored fold (LifeOutcome::Published carries them), within its cut. The backend prefetches the request's narrative office after prepare. The request reads the warning zone state at its start through a request-lane view of the process warning store and takes the first state, the stored one or the one the cycle offers. The rest of the restart read, full reads of reduced pages, day product writes, the scheduled image production and the KASI capture pass use the bulk lane. Tests cover each family's race, the fast path, the fill, the default station list, the needed-first life read, the narrative read beside the snapshot, the zone-state read beside a cycle that never ends, the lanes (also with a full bulk lane) and the whole first answer. The test transport holds reads of stored keys for a while, tags the bulk lane and can hold it full. A cycle that never ends no longer holds the answer until the wait ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The architecture document gains the restart race section: the 700 ms head start, what each family's restore is and what races it, the fill of today's and yesterday's absent hours, the lanes, the provider calls a raced restart adds and the failure paths. The restart read, the warning offer, the shared warning store, the narrative supplement and cache contract, and the S15b-2 acquisition table point to it. The KMA data design describes restores with a head start. The task intent, plan and spec (with the test of each criterion) are added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart note now says that stored folds restore first and that providers fill what is still missing at 700 ms, with the other reads on the bulk lane. Regenerated through Archify (validate, deliver, check and browser-check pass; the note was inspected in a browser). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart-race tests slowed stored reads until 2.6 s after their setup while the request waits 2.5 s from its own start: 100 ms of real time decided whether the restore ended after the answer. With 200 ms of setup before the request, life_s3::the_life_read_restores_todays_slots_first failed (the older days were in the first answer). Counted::hold now returns a gate: the reads of stored keys under its prefixes wait until the test opens it, which it does right after the answer the restore must follow (inside the 3 s S06 lookup deadline), and the test checks that a read was held. The nine tests use it; with a 1 s setup delay the life test still passes. The station gates hold the station pages, products and seed only: the town RSS is read by every request and is not raced. The 700 ms head start stays checked on a paused clock (tests/resolver/race.rs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stations_by joined the minute, hourly and city acquisitions as one part, so pages that arrived early waited, stored but unfolded, for the slowest provider. With the restore later than the bound and the city provider slow, that part ended at the bound, the last fold had no time left and the first answer used the old fold or the seed alone: the new observe_product::pages_that_arrive_before_a_slow_provider_reach_the_first_answer failed 3/3 with "Fail to find stn Weather info of stnId=108". Each owned acquisition is now a part of its own, so the fold runs when any of them ends while the others are waited for. The test passes 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tive-regression) The office race installed its published narrative unconditionally, so an acquisition that ended after the stored read had answered with a newer publication replaced it (publish_once is per publication, so both stay stored). The new full_response::a_late_raced_acquisition_keeps_the_newer_stored_narrative failed 3/3: the hour-old "acquired" narrative answered over the current "stored" one. Both the stored read and the race now install through OfficeCache::install, which keeps an unexpired answer with a later announcement, as the read already did. The test passes 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ight) The request race limited only races (cache.raced) and the three-minute refresh only its cycle, so both could fetch and publish the same office at once. The new full_response::the_race_joins_the_refresh_fetch_of_the_office holds the refresh's fetch of office 109 and starts a request's race: the race fetched the office again (3/3). One OwnedFlights fetch-and-publish per office now serves both; the one that comes second joins the running fetch (or its result for 3 s), and both install the publication with the newest-wins rule. The refresh used to clear the office's answer instead, so a stored read that started before its publication could answer the older narrative for 30 s after it; it now installs what it published. The test passes 10/10, and the narrative, cache and runtime-services suites pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…budget-accounting) The restart-race section added logical acquisitions of five APIs into one "at most 11 units of a key's 10,000" figure. Units are charged per key and API (each with its own daily allowance), per page and per attempt, and a page can take a funded second attempt, so 11 was neither the unit count nor a bound. The section now lists, per quota owner (AsosHourly, AsosDaily, LivingWthrIdx, HealthWthrIdx, VilageFcstMsg), the acquisitions a raced restart adds, their pages, the normal units and the bound with failed pages, from DataGoKrApi::daily_units, the 999/1,000-row pages and their 8-page limit, FundedExecutor's two attempts per page and the narrative's key loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Cross-provider review (Codex gpt-6.1-sol, high). Round 1 (at 479ceb3, before this PR was opened): CHANGES_REQUESTED.
Round 2 (at 2651e22): CHANGES_REQUESTED. RR2-page-progress, RR2-narrative-singleflight, RR2-ci-clock and RR2-budget-accounting are resolved.
|
…-regression) OfficeCache::install compared an incoming narrative only with an unexpired answer, so the protection ended with the 30 s TTL: a race started at 0.7 s whose fetch answers an older announcement at 29 s and whose publication takes 2 s ends at 31 s, after the newer stored answer (installed at 0.8 s) expired, and installed the older one with a fresh TTL. The new paused-clock test cache::an_older_announcement_never_replaces_a_newer_one_after_the_ttl (in-memory S3, tests/narrative/cache.rs) failed 3/3: "acquired" (older) answered instead of "stored". The office now keeps the latest announcement it answered apart from the answer's TTL. An older narrative is not installed and the answer is left as it is (an expired one is not extended), so the next request reads S3, which holds the newer one. The test passes 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Cross-provider review (Codex gpt-6.1-sol, high), round 3: PASS at ab0aba1. All five findings are resolved: RR2-page-progress, RR2-narrative-regression, RR2-narrative-singleflight, RR2-ci-clock and RR2-budget-accounting. Readiness waits for CI and for the response-first test failure seen on #2872's CI, which is being diagnosed. |
The first life read listed the partitions of UV, oak, pine and weeds one family after another before it read today's slots. On EC2 (race-r2, 2026-11-08) those eight LISTs took 0.25-0.68 s before UV's restore began, and UV missed the first answer in every race run. The four listings now run together (one per family, the same reads); today's slots are still read and folded first, and the read stays one owned flight. The new paused-clock life_s3::the_needed_first_read_lists_the_life_families_together (an in-memory S3 whose LISTs take 300 ms) failed 3/3 with the sequential listing (a UV acquisition started: today's UV was not folded by the 700 ms head start) and passes 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Cross-provider review (Codex gpt-6.1-sol, high), round 4: PASS at 80bc152. This covers the merges of #2872's writer fixes and 80bc152, which lists the four life families together. EC2 restart comparison, 2026-11-08 02:38–02:42 KST (t3.small, debug logging, binaries master 5533931 and this PR at 17e6a41). First Seoul answer after a restart vs the same process 5 s later:
|
…RST) The first life read restored each needed catalog one S3 read at a time: on EC2 after the restarts, today's UV partition (three 4-page acquisitions) took 16 reads in a row and missed the life section. The needed partitions of every family (today and the latest before) are now restored together, with at most four S3 reads in flight shared among them (a third of the process's S3 admission, so the station restore keeps the rest); single flight, today first and the rest of the window after are unchanged. Storage (opt-in per call): CatalogStore::lookup_wide reads a catalog's group descriptors and records ahead, up to its width at a time, then restores over them with the same checks and result as lookup. It waits for one store I/O admission as lookup does and takes the others only if they are free at once, so no other reader's lookup queues behind it. Only the life read's needed partitions use it (Reader::acquisitions_wide); every other reader still uses lookup. Red: the_needed_first_read_restores_todays_records_together (paused clock, 170 ms per GET) had no UV on the first answer: 16 reads end at 2.72 s, past the 2.5 s life section. Green: today's UV, at most 4 reads in flight, the first answer at 850 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a_slow_life_read_starts_uv_at_the_head_start failed now and then (3/30 on 0d98504, 2/30 on 80bc152) with no UV or weeds on the first answer. The process before the restart keeps its pooled connections until the S3 test peer closes them 2 s after they opened, so the restarted process's inputs (8 permits) could fill the peer's 16 connections (sampled at 14-16): the quota reservation of the raced acquisition was refused at the head start (connection_reset) and the acquisition failed. A restarted process's inputs now admit 4 requests at once (restarted_transport, all seven restarted processes in life_s3). With it the peer peaked at 12-14 connections and the test passed 30/30. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Cross-provider review (Codex gpt-6.1-sol, high), round 5: CHANGES_REQUESTED at 7f35848. 0d98504 makes the life read restore today's records with 4 reads in flight; 7f35848 fixes the test server's connection limit. Required
Both are being fixed test-first. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plan A for the first answer after a restart, which AK chose on 2026-11-07 ("권고대로"). The EC2 restart comparison found every first answer without yesterday values, past days and the history mapping; most also lacked current observations, and some lacked UV or the forecast narrative. The restores ran behind each other and behind CPU work. AK ruled that a part S3 holds but a restarted process has not restored yet is not an allowed omission.
src/resolver/race.rs): a source waits for its stored restore at most 700 ms into the KMA request. It then starts the owned acquisitions for what memory lacks, as on an empty S3, and the answer uses whatever is ready by the request bound. A restore that finishes in time makes no extra provider call.Stacked on #2872 (which stacks on #2866). Grid forecasts are not raced; they are already read per request and were complete in every restart.
Checks
KMA-RESTART-RACE.json) and the artifact policy pass.server2.md(including data.go.kr units per quota owner) andserver2-kma-data-design.md. The data design diagram was regenerated through Archify, and its note was checked in a browser.Provider calls (only when a restore misses the head start)
Not in this PR
🤖 Generated with Claude Code