Repository navigation
Conversation
… nightly (#144) * test(engine): re-record the I9 request bound for lifetime source pins, and alert on a red nightly A provenance pin is checked against the pinned source's lifetime rows, so a pinned resubmit whose window holds none of that source's points answers no_data/no_observations, where the recorded bound held refused/ facet_not_available. The covering array carries 22 such doors, which were re-recorded with that change; the 378 cells outside it failed I9 in every nightly full grid. tests/fixtures/answer_invariants_requests_uncompared.json is re-recorded with tests.engine._answer_invariants_record measure and write. full-grids.yml now opens the issue "Nightly full grids: failing", or comments on it, when a scheduled run fails. docs/development.md describes the grids, the nightly and how to re-record the bound. * test(engine): check I9 on the covering array only I9 holds each door to a recorded request bound, which a pull request that changes an answer re-records for the cells it runs: the covering array. The nightly ran I9 on every cell, so an intended answer change outside the array failed every nightly until someone re-recorded the full product. checks_for now runs every invariant but I9 on cells outside the covering array, and test_i9_checks_exactly_the_committed_cells pins that split. measure records the covering array, and the uncompared fixture is dev's.
…headline year as its answer does (#149)
#110) * fix(search): judge stated-word coverage on the members the read served Coverage was rated at binding repair on a prediction of the members the answer would serve (`standard.members_served`). The prediction read premises materialisation settles later: the direction, the breakdown, the role class, an unbinding and availability. On "refugee assistance Kenya" the direction was unresolved at bind, so the words were rated against both directions' members, and the part refused with qualifier_not_bound after the read had settled on "Refugee assistance". `phase_election.judge_coverage` now judges coverage once per pass, after the read, and `served_member_names` reads the members the read served (`mat_result.sv_dcids`). Before any read, clearing falls back to the words every named member holds. Binding repair and `_unbind` record only the bound records, which `unrealisable._asked_terms` reads. A refusal before the read keeps its own reason. `members_served`, the served-member memo and `member_has_data_where_asked` are removed. Co-authored-by: Claude <noreply@anthropic.com> * fix(search): rate words on a lone served series behind another label A read serving one member whose name differs from the elected shape's label, as "Total population" behind "Refugee population", was rated on the label: the request carried served names only when there were two or more, and Jev kept its bind-time rating, made on the menu label. `stated_words_read.serves_beyond_label` holds when the served names are anything other than the shape's own label. Gemini's prompt and Jev's `describe` name the served series exactly when it holds ("the series 'X'" for one name), and Jev's `restated` asks every rated word again. A lone served name equal to the label leaves the prompt byte-identical. One Gemini stated-words fixture is recorded. Co-authored-by: Claude <noreply@anthropic.com> * fix(search): rank a held breakdown recall missed by its in-window data On a family with no total, the breakdown cap ranked each signature by the data its recalled members hold in the requested window (`_breakdown_rank`). No read covers the family members recall did not return, so a breakdown whose only in-window data sat in such a member scored nothing and was cut, and "under-five mortality" for 2020 could answer no_data under a cap of two. `standard.Candidates` carries the members of the admitted family behind each signature, recalled or not. When the cap cuts a signature and the family holds members outside the shape's coverage, `_settle_breakdown` reads them at the shape's availability places in one batched facet read (`_breakdown_coverage`). A failed read is noted as a lost read and ranks on what the part holds. `availability.read_coverage` is shared with the menu's availability read. Co-authored-by: Claude <noreply@anthropic.com> * fix(search): judge coverage before an unbinding refuses the part A part that no table serves once an unbinding at materialise drops a bound value refused through `_unserved_refusal`, which sets the judgement before the pass's coverage judgement runs, so the refusal named no uncovered word, value requirement or withheld term. `_settle_served_dataset` now judges coverage on that refusal path first, over the elected shape's named members, as no read has served. A served member with no name sends no served names, so the binder is never told its dcid is a series and keeps the label's rating, as the lexical readers do. The breakdown read is tested where the cap cuts nothing and where the read fails. On a part naming no place the read does not run, and the cap ranks on the recalled members alone. Co-authored-by: Claude <noreply@anthropic.com> * docs(search): tighten the docstrings of the coverage fixes Co-authored-by: Claude <noreply@anthropic.com> * fix(search): judge each candidate's coverage on the members it serves Coverage was judged once per pass, on the elected read. Candidates the part returns beside it were judged on the elected answer's words, so with data in both directions "refugee assistance Kenya" returned the generic "Assistance received" beside "Refugee assistance given" with no query_qualifier_dropped warning. `phase_election.judge_answer_coverage` judges each admitted candidate in `assemble_candidates`, before grounding, by the same rule on the candidate's shape, bindings and read. A candidate whose words the lexical readers clear asks no binder; the others are rated concurrently, one call each. A coverage record is unmet only on an answer whose own records hold its phrase (`requirements.judge`), and each candidate carries the phrases it leaves unread (`_Candidate.uncovered`). One Gemini stated-words fixture is recorded, and bind-only test doubles answer the candidates' rating with `NO_RATING`. Co-authored-by: Claude <noreply@anthropic.com> * docs(search): state what a lone survivor's refusal names Co-authored-by: Claude <noreply@anthropic.com> * fix(search): rate words on the label when the read serves the labelled series When the read serves the one member the elected shape holds under the final bindings, the label names that series in full, so the stated-words rating reads the label and names no served series (_serves_the_labelled_series in phase_election). Naming the series again dropped what the label states: "DAC2b interest received on lending to Kenya in 2015" rated "lending" 0.0 on the series' short name and refused, where the label "net disbursements (interest received on OOF and export credits, non-grant)" covers it. Co-authored-by: Claude <noreply@anthropic.com> * fix(search): judge each candidate on its own records and bound the breakdown read judge_answer_coverage cut each candidate's coverage records to bare phrases, and the ledger keeps one requirement per phrase, so a second candidate's trace named the first candidate's code ("maternal ODA" on DAC1 named CRSTypeOfFlow10). Each candidate now carries its own ResolvedQualifier records through _Candidate.uncovered, answer_terms and judge, so each trace names its own codes. _breakdown_coverage read every unrecalled family member at every named place in one facet request. availability.read_coverage now reads in standard retrieval's chunks of at most 25,000 pairs, and the breakdown read takes at most region_max_members members, one from each breakdown in turn. Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
#150) * test(engine): record I9's bound for the covering array cells it checks measure records the covering array and, for each preview-on cell, the preview-off doors i9_latency runs beside it. uncompared lists only doors of committed cells, _dump writes null at a position the base record bounds, and growth judges a same-answer door with I9's own _beyond and grants. measure then write runs clean and idempotent on an unchanged tree, and the held list holds 167 covering-array doors. * test(engine): keep a held I9 record while its door passes it * test(engine): keep a held I9 door the preview-off door now bounds, and refuse one the grid no longer measures
…er's words (#119) Two extracted names written alike in the query ("aid from Congo to Congo") stay two choices. An entity's identity is its dcid, else its name, else its text (ParsedEntity.identity). text holds the request's words and feeds the resolver; the optional name gives an unpinned row an identity apart from its words. A name beside a dcid refuses request_invalid, as does an unpinned row whose name or text is a dcid another row pins. Binding, roles, pins, members and options key on the identity, so a chosen row posts its words and dcid and two pins are two entities, while a self-flow pinned twice stays one. The raw-text door records each name's grounded span in RegionResult.entity_surfaces, which resolution reads, and both doors pass the request's words to the leg as entity_words, so a pinned row's words are masked in the variable phrase. surface_resolution.is_surface and the parsed door compare words through qualifier_match._normalize. entity_checks.part_entities holds the parsed door's entity checks, and an option the door would refuse is not offered. Every user-facing quote of an entity reads its words (words_of): warnings, departures, role-not-admitted and unresolved terms. Repeated words in an entity_ambiguous message name each group by direction or ordinal, and two untold places written alike carry their graph labels. NoDataOption and SentenceToken carry entity_choice, the index of the group an option settles and the slot asks, and Part.entity_refs has one entry per restatement row. Conformance clauses C12, C50c, C50e and C141a describe the rules. Co-authored-by: Claude <noreply@anthropic.com>
…quest, and require provenance on the mint (#152) * fix(search): serve a top facet with no provenance to a native unit request, and require provenance on the mint * test(engine): report a scope count beside a response that states no Spec * test(engine): record I9's bound for the covering array cells it checks measure records the covering array and, for each preview-on cell, the preview-off doors i9_latency runs beside it. uncompared lists only doors of committed cells, _dump writes null at a position the base record bounds, and growth judges a same-answer door with I9's own _beyond and grants. measure then write runs clean and idempotent on an unchanged tree, and the held list holds 167 covering-array doors. * test(engine): keep a held I9 record while its door passes it * test(engine): keep a held I9 door the preview-off door now bounds, and refuse one the grid no longer measures
The client contract at 98e53ee serves dfo-apis 99e6203: entity_choice on NoDataOption and SentenceToken, ParsedEntity.name, and one Part.entity_refs entry per restatement row, with its scenarios recorded at that commit. All six spec-kit pins move together. Co-authored-by: Claude <noreply@anthropic.com>
…mint elects the facet the answer serves (#157) * fix(materialise): read facet dates in a range mint's probe, as the answer does * fix(search): read facet dates for option coverage under a range, and test the date forwards * fix(search): read dates on the options door only for coverage * test: drop dates from an undated read in the base graph doubles, and guard the no-place probe
…#156) * fix: answer SDK users' gaps in warnings, refusals, resolve and counts coverage_below_scope says which years were asked and which the data covers, in its message and as requested_years and served_years. A name correction made by extraction becomes a name_corrected assumption that carries the typed words in asked. Graph resolve keeps a shared name resolved and lists the other nodes in near_matches, and a kinds filter that removes every hit answers with the same-name nodes of the asked kind. The answer sentence and role_scope_defaulted count providers and recipients without party totals, and the defaulted-time note reads for people. One window rule (src/search/scope_window.py) admits a role read in Search and a mint, sidecar or options definition in Datasets, refusing an over-long window with request_invalid before anything is written. One role-kind rule (src/search/role_kinds.py) derives the kinds each role takes from the code index's parties and refuses a wrong-kind entity with invalid_role_ref, naming its kind and the role's kinds in the series' own wording. Facet keys are checked one by one with alternatives, and role refusals offer the spec's role keys first. CORS allows X-DFO-Client, and every request log line records the cleaned client header beside the trace id. * fix: narrow every code candidate by code_system and install resolvekit with Datasets Graph resolve narrowed only code-index hits by code_system, so a node sharing a hit's name, a resolvekit hit or a fallback node classed code could come from another system: "European Union" under CRSPurposeCodeEnum returned DAC1AidType2102. One _Admission rule in entity_resolution/resolver.py now decides kinds and code system on every candidate path, reading CodeIndex.system_of_code, which CodeIndexCodes also uses. A code_system request on a resolver without a code index raises ValueError. Role-kind checks build the membership policy with resolvekit at runtime, so a Datasets-only install failed every role-bearing mint with ModuleNotFoundError. The datasets extra now carries dfo-data-tools's resolve extra, import datasets_api still loads no resolvekit, and CI's datasets_only_install job syncs the datasets extra alone and runs tests/datasets_api/test_independence.py, which checks a recipient role for Kenya in that install. The held-record re-record helper and i9_latency share _held_grants(door, ...), which folds in the resubmit's role-kind read. * test: pin code_system on the resolvekit-hit and fallback paths test_a_code_system_narrows_the_codes_every_path_finds covers the shared-name, resolvekit-hit and fallback paths; under CRSPurposeCodeEnum each rejects DAC1AidType2102, and each rejection fails when the admission rule ignores code_system.
The client contract at 26f733c is recorded against dfo-apis d80df9f: party totals left out by default, datasets presented in reading order, and the client-identity, base-URL and role-keys rules, with Assumption.asked, the WarningModel year ranges and the name_corrected kind served. The registry maps the Presentation write refusals at the write routes' statuses, and a no_data part keeps its reason's gap kind under a timed-out answer. All six spec-kit pins move together.
…deadlines (#143) * test(search): drive five engine deadline tests from events, not the wall clock Five tests bounded an answer on elapsed time and could fail on a stalled CI runner, as test_option_coverage did before #134. The fan-out soft deadline test now jumps a patched clock past the deadline once its slow leg is in its phase. The two withheld-read tests cut the bounded await when the stalled read reports and assert the read had not finished when the answer returned. The JSON and SSE route tests assert the soft and hard deadlines the route binds rather than elapsed seconds. * test(search): assert the SSE stream arms its own timer at the hard cap The SSE hard-cap test checked the budget and the worker's deadline but not the stream's timeout, so a stream expiring at the soft deadline passed. The test now records the deadline the stream's asyncio.timeout is armed with and asserts it is the hard deadline.
… does (#160) Coverage's open-role breadth dimension counted every member of the scope, group totals included, while the heading's RoleAxisFact.member_count and the role_scope_defaulted warning count parties only (party_count). An answer could say 92 providers in its heading and 108 in its coverage. Both coverage paths now count through party_count, which moves from spec_build to materialise.enrichment beside party_kind_of so coverage can import it without a cycle through retrieval.download.
… by the OECD's conventions (#163) * feat(search): add the scope-choice record and its compile step Search can now report how each part was counted. ResolvedSpec gains an optional scope_choices record: the part's base identity, the explicit and carried entries, one choice per settled axis with its options and their availability, and the compiled identity. A pure compile derives every default after the stated entries, never moves an explicit or carried entry, and refuses a selection it cannot serve. The dataset_resubmit door accepts scope_entries on an identity entry, compiles them over the posted identity and returns the compiled spec, refusing with requested_option_unavailable or conflicting_requested_options (422). The sector-grouping and measure-basis declaration tables join the declarations hash. No axis is live yet, so every response is unchanged. The spec-kit pins move to the commit that registers the two codes. * feat(search): read stated scope words and carried choices before binding Parsed requests may carry the user's own message and the conversation's carried scope entries. A deterministic, negation-aware scanner turns stated provider and measure-basis words into explicit entries and notes "total" intent; a single-part request reads the message, a multi-part request reads each part's own text. Entries reach every part before the binders run and apply only to an axis the part settled (a sector entry only to its concept). The raw-text cache key includes the stated scope, so "bilateral" and plain queries never share an entry. * feat(search): count a summed region with its own residual buckets A part that sums over a recipient region binds the region's members plus the regional and unspecified buckets the served dataset publishes, and reports a region scope choice (members and residuals, or members only). Listings, rankings and "African countries" bind members only. Residuals are left out of every member count, ranking and unavailable entity by party kind, and the read-as-members departure names what the data allocates to no single country. A switch through dataset_resubmit round-trips to the first spec_id. * feat(search): bind "health" as CRS 120 + 130 and report the sector choice A declared sector grouping turns its concept words into the grouping's code set with declared_grouping provenance: "health" binds CRS 120 + 130 and "humanitarian aid" binds 720, 730 and 740. A code the text names keeps its binding and states an explicit choice. The sector rule reads the base identity alone, so search and dataset_resubmit offer the same options: the grouping and each member code, with option_not_covered marking an option no covered member declares. * feat(search): settle sector and region together and never refuse a fresh search's scope Both live axes register in rules.AXIS_RULES. build_part_scope applies the entry a binder read (AxisRecord.entry) by one rule: it joins the posted entries when the part's table on its axis holds the option the record settled, which covers an axis the answer does not reach and a base that no longer holds the binder's option. ScopeFacts keeps one shape, with the part's operations, groups and datasets in PartFacts. Carried entries reach read_sector and region_selection through ScopeState.entries_on; parsed-door tests pin a carried sector:120 on a "health aid" part and a carried region:members_only on a summed region. On the parsed and raw_text doors build_part_scope runs with refuse false: an entry compile refuses is dropped into dropped_entries (carried_choice_not_applied for a carried entry, the new stated_choice_not_applied for a stated one, with the refusal's cause), and the defaults derive without it. The 422s stay on dataset_resubmit scope_entries, so the request pointers ScopeIntake carried for them are gone. Option labels are standalone noun phrases: "health incl. population and reproductive health (CRS 120 + 130)", "health, CRS 120 alone", "Africa incl. regional and unspecified amounts", "Africa's countries only". The templates sit in code_index.scope_conventions.OPTION_LABELS and feed the rule version; the region is named from the part's group, else its total. The capability pins in the first-answer snapshot and the render fixtures are re-recorded. * feat(search): elect the measure basis from the declared convention table The basis follows the question's shape and window: grant equivalents for a donor-only total from 2018 (the OECD headline, flows 10 and 60), net flows before 2018 and for a window crossing it, net disbursements for a recipient's all-sector total (DAC2a receipts), and gross disbursements once a sector, instrument or modality is bound. A stated or carried basis wins and drives dataset election, so a stated grant equivalent for a recipient reads the CRS. Each part reports a basis scope choice, and alternative_measures no longer lists siblings of another basis, so a settled basis raises no count question. * fix(search): state only served scope, keep one entry per axis and key the parsed cache on stated scope A fresh search compiles only the entries its binders applied and reports the rest as not applied, so the record never claims an option the served answer does not hold; a basis served off the convention records its own rule. Same-shape merges state the merged scope. A switch replaces the entry on its axis, and a posted entry without a concept takes its axis's concept. Grant equivalents read flows 10 and 60 however the basis was elected, so switches round-trip. The parsed cache keys on the stated scope, not the raw message. "commitment" in the pledge sense states no basis, region labels come from a declared name table, and the axis rules scan each family once per call. * feat(datasets): mint a composite spec as one frame from its declared components A declared composite provenance names how its components derive from the composite identity: bilateral CRS plus imputed multilateral stacked over the years both cover, or the OECD headline spliced from DAC1 net before 2018 and grant equivalents from 2018. The mint materialises each component through the single-provenance path under its own pins, keeps only the composite window, writes a zero row where the imputed table leaves a covered cell empty, and stamps the composite on every row. Citations and a re-render from the version record name the component tables. The admission estimate counts the zero rows, a frame mixing raw provenances is still refused, and every reader of facets.provenance either expands the composite or refuses it. * fix(search): count donor totals on the convention basis, report unruled stated words and label prices as served Recall that finds only a dataset's amount family now carries the sibling families that measure a basis the conventions count it on, so a donor-only "ODA" from 2018 is counted in grant equivalents like "aid". A stated word on an axis no rule serves, such as "bilateral", is reported as not applied. Basis labels state the prices and currency the part is served in. A word naming a resolved entity is read as that entity and not rated as a qualifier, and judges read a declared grouping by its label. * feat(search): read a summed region as the OECD counts it on DAC data A declared table maps graph regions to the DAC regions the OECD publishes totals for: sub-Saharan Africa, Asia, Europe, the Middle East, the African subregions and South America. On a part summing over such a region on DAC data, Search binds the DAC members and the residual amounts the served dataset publishes, so figures match the OECD; the region choice states the composition and offers the graph's member countries as the other option. Rankings, listings and every other dataset keep the graph's place. * fix(search): exempt an entity's name word only when no vocabulary reads it A word from a resolved entity's name, such as "Gates", is read as the entity only when no qualifier or measure vocabulary reads it, so "women" beside UN Women or "malaria" beside the Global Fund is still rated against the series. Merged parts and resubmits report the basis rule the search records, and region totals skip the reading match for a single member. * feat(search): offer and default the inclusive and OECD headline readings A CRS part gains a provider choice: bilateral flows, or bilateral plus imputed multilateral over the years the imputed table covers, offered shortened with its window when it covers only part of the request. A total from a named donor defaults to the inclusive reading when it covers the request. A donor-only DAC1 window crossing 2018 is counted as the OECD headline, net to 2017 and grant equivalents from 2018. Both answer as one composite spec with its components, switch and switch back to the same spec_id, and read through the component tables wherever Search reads a provenance. * docs(search): state the scope choice rules as they hold * fix(search): restate a composite answer's headline and report a stated provider the part lacks A composite answer states its own headline year and series, and counts the StatVars of every component. A typed provider word on a part with no provider axis, such as a DAC1 donor total, is reported as not applied. * fix(search,datasets): default a composite only when available and keep its latest year out of the base A headline or inclusive reading becomes the default only when every component is available; otherwise the served shape stands with the rule it was served under. An inclusive latest window keeps the base on latest and elects the latest year both tables cover after compile, so a switch to bilateral answers bilateral's own latest year. The worker estimates the composed frame, and a composite is labelled from its declaration. * fix(search): state a composite answer's window from the years it serves The restated composite takes its date range, its when slot and the data years its departures read from the years the components serve, so a latest-year answer names the elected year. * fix(datasets,search): leave unserved members out of a composite and cite both of its tables A composite mint passes each component's unserved members through to its coverage warning and writes no zero row for a member its facet group left out. A composite answer cites each component table and names its measure from the declaration. * refactor(code-index): name the conventions' datasets and codes in the declarations The scope conventions read the dataset provenances and the CRS codes their groupings and basis tables name from code_index.declarations, the one module where dataset literals live. * docs(search,datasets): state the composite rules as they hold * fix(code-index): read the composite declarations' datasets through the public names * fix(search,materialise): offer the inclusive scope only where it adds, over a year both tables hold The inclusive provider scope is offered for a part whose named providers the imputed table carries, and for a pool only when none of them is a multilateral whose own outflows the table attributes to its funders (REDISTRIBUTED_PROVIDERS), so a pool of the US and IDA, or a part naming no provider, keeps the bilateral count. A latest read elects its year among the years both tables hold. A composite zero row states the unit its cell states, so a member absent from one component takes its own currency. * fix(search): apply carried scope by one rule, keep basis routes and region names, and build scope off the loop A carried sector entry applies to a part when the grouping offers it and it was made for the concept, or names codes the part's words matched. A part keeps alternative measures on other bases unless its measure-basis choice offers them. A region switch restates the recipient group for the members it reads, so the group name and departure survive. Every text origin indexes the part's normalised scope text, scope choices build in a worker thread, one elector holds the convention-dataset gate, and the declarations hash covers every scope table. * fix(search): apply a basis convention only where its own dataset serves the part A convention row is a dataset and a basis: it applies when the elected shape reads that dataset, and otherwise the part takes the row of the dataset it reads, so a CRS-only recipient part binds gross. A convention basis with no series at the places falls back to the shape's default and records the departure. On grant equivalents ODA binds flows 10 and 60 and its label names both, and an echoed assumption holds only when it names the declared default the slot states. * fix(search,materialise): serve composite years from table coverage and compose each frame once A stacked composite serves the years every component table covers, so a year the imputed table covers keeps its bilateral figure with imputed zeros. Every path that restates a composite elects its latest window, explicit entries attach only where their concept holds, a composite provenance names its component datasets to unit conversion and skills, and the ceiling counts only the rows a composite keeps. The mint builds the composed frame once and folds the digest, statistics and output profile in that pass, cutting a 43,680-row composite mint from 7.4 s to 3.8 s with byte-identical output. * test(dataset-store): stand in a plain stored definition for the worker's composite check The worker asks the stored definition whether it binds a composite before estimating, so the live over-ceiling tests stand in a definition whose identity binds none, and their estimate stand-in takes the composite it is given. * fix(search): elect no basis convention for a place with no direction, and pin spec-kit 455e958 A place named without a direction has no part shape until its direction settles, so basis election leaves it to the house policy. Test fixtures model the net members DAC2a publishes, the pass-end test holds only its listing on the held pool, model registries list the new fields, and every dfo-spec-kit pin moves to 455e958 on spec-kit dev.
…data requests (#165) * perf(materialise): keep the mint fetch pool full and hedge slow SDMX data requests stream_rows submits a new chunk whenever any chunk completes and reads results in chunk order, so one slow mixer response no longer idles the other workers. The two mint SDMX clients re-send a data request still pending after 0.5 s, on threads of their own, and take the first answer. A composite's whole (a part's dataset with its sector unbound) is pinned by tests on the existing resubmit path. * fix(dc_sdmx): take a hedged read's first success and always release its slot A duplicate that fails at once no longer beats a primary that answers late; the read raises only when both sends fail. The loser's slot is released in a finally, so a raising close() cannot turn hedging off. A door-level composite mint, pooled with one hedged slow chunk against serial, gives the same content version and CSV. * docs: tighten the comments and docstrings this branch added * test(datasets_api): give the recording SDMX client hedge_after and pin which clients hedge * fix(dc_sdmx,materialise): let only a successful response win a hedge and bound the fetch look-ahead A hedged read takes the first 2xx response; an error from one send waits for the other, and the read raises only when both fail. The duplicate checks the read's deadline before it goes out, every send counts in the client's request lifecycle, a failed thread start releases its slot, and no duplicate is sent while four losing sends still run. The group fetch dispatches only within twice the worker count of the oldest unconsumed chunk, so buffered results stay bounded. Search's whole-request test and README paragraph move to the Search provider change. * fix(dc_sdmx): end a hedged read's wait when it is stopped, expires or gets a lasting client error Every hedged path waits through one guarded race. A 2xx or a 4xx other than 408 and 429 decides it; a 408, 429, 5xx or transport error waits for the other send. A duplicate reserves its loser place under the hedge lock, after checking the primary has not answered. The generation docstring states the true memory bound. * fix(dc_sdmx): hold a concurrency slot for every send until it settles The total-concurrency slot is a reference-counted lease that each first send holds until it settles, and a duplicate holds a loser place, so running sends never exceed total_concurrency plus the loser cap, whether the read wins, fails, expires or is stopped. One finally settles every send. A 2xx wins only with an SDMX-CSV content type. Hedges fired and won are counted as sdmx_hedges and sdmx_hedge_wins in Server-Timing. A stopped read that takes its own slot releases nothing. * fix(dc_sdmx): wake guarded waits on a stop and make each slot lease per holder A guard settles an ended future on abort or expiry, and every hedged wait, including the one before a duplicate, waits on it with the deadline remainder as its timeout. Each holder takes its own slot lease, so a second release does nothing. A duplicate whose thread fails to start leaves the read waiting on its request, a primary whose thread fails to start is sent on the caller's thread, no duplicate goes out with less than hedge_after left, and sdmx_hedge_wins counts only a 2xx win. Undecided races are one int under the hedge lock. * fix(dc_sdmx): free a losing first send's slot once its race is decided A first send still running when its duplicate wins releases its total-concurrency slot and takes the winner's loser place, so the total limit governs live reads and the loser cap governs stragglers. 401 and 403 from one send wait for the other, as 408 and 429 do. The look-ahead factor is _DISPATCH_LOOK_AHEAD_POOLS, and sdmx_requests is documented to include hedge duplicates. * fix(materialise): collect finished fetches before each dispatch pass stream_rows learnt of finished and failed fetches only inside its blocking wait, which it skips while the head chunk is ready. A failed fetch then went unseen and later chunks were still sent, and finished fetches kept their places. Each pass now collects them without blocking first. The README lists sdmx_hedge_wins. * fix(dc_sdmx): refuse a hedge delay that is not positive SdmxClient raises ValueError for a hedge_after of zero or less, which would duplicate every data request. The DATA_HEDGE_AFTER_SECONDS docstring states the race rule the class follows, and the docstrings this branch edited are rewrapped to their neighbours' width. * fix(dc_sdmx): close a losing send as soon as the race is decided Losers took their settle callbacks only when the hedged send exited, after the caller had read the winner's body, so a loser held its connection and loser place for the whole read. They now take them when the race is decided. The dispatch target docstring states the look-ahead bound. * fix(dc_sdmx): free a winning duplicate's loser place A duplicate that won after its request had ended kept its loser place for the whole body read, so live reads counted against the loser cap. It frees the place as the race is decided. Closing a settled send also suppresses RuntimeError and ValueError, and the docstrings say the wait-for-the-other-send rule holds once a duplicate is in flight. * docs(dc_sdmx): state the hedge rule once, in the SdmxClient docstring
* feat(search): state the provider scope of a DAC1 total ODA part
A DAC1 total counts core contributions to multilaterals as the donor's ODA, so it now carries a provider choice whose one available option is provider:contributions ("bilateral plus core contributions to multilaterals"). Bilateral and inclusive are unavailable with reason counts_core_contributions, and DAC1's own bilateral line is never offered as bilateral. Spec_ids do not change; the scope rule tables' hash moves the capability version.
* docs: tighten the comments and docstrings this branch added
* refactor(code_index): declare DAC1's total ODA aid type beside the other dataset codes
The provider convention reads DAC1_TOTAL_ODA from declarations.py, so no election module names a declared dataset's codes.
* fix(search): declare the provider scope of each DAC1 aid type for bilateral donors
PUBLISHED_PROVIDER_SCOPES maps DAC1 total ODA to contributions and DAC1 bilateral ODA to bilateral, for donors outside the declared multilateral providers. A stated inclusive provider on a DAC1 total drops with contributions_counted, since the total already counts core contributions. ProviderScope is declared once, contributions is an explicit no-op edit, and the settled record names the rule that set its option. Search's whole-request test and README paragraph live here.
* fix(search): name the rule that set a provider record's held option
A convention whose scope the part does not hold no longer names the record; it reads bilateral_flows, so a fresh search and a resubmit agree.
* fix(search): decide DAC1 provider records from donor structure and keep each record consistent
A DAC1 total or bilateral ODA part gets a provider record only for a country, the EU institutions, or a declared group of countries, and never for a pool of an EU member with the EU institutions. On a DAC1 total, bilateral binds DAC1 bilateral ODA and is available to a switch; inclusive is unavailable with contributions_counted. Each option names its own rule, so a fresh search and a resubmit record the same rule, and the record never holds an option it marks unavailable. The provider axis has one option builder, and the published table is read once per identity.
* fix(search): check sibling DAC1 options against coverage and name rules a request matched
A DAC1 total's bilateral option and bilateral ODA's contributions option are available only where a series covers the part's donor, basis and window. A provider record falls back to an available option on every table. A record names a convention's rule only when the request matched it in this call; otherwise each option names its own scope rule from PROVIDER_SCOPE_RULES, bilateral_flows or bilateral_plus_imputed. Availability objects are per option, the convention lookup is memoised, and AxisTable checks its held rule.
* fix(search): report the years a sibling DAC1 option serves and keep slot options to published tables
A sibling DAC1 option serves the years every donor in the part holds, read per basis segment on the headline, and is partial or unavailable otherwise. When a convention matched, a record falls back to the held option before bilateral. Slot-bound provider options are allowed only on published tables, published tables skip the CRS convention lookup, and the pass and the build share one composite predicate.
* fix(search): make the DAC1 provider record describe what the answer serves
A recorded bilateral on a base holding the inclusive composite now
strips the composite, and a fallback names the served option's rule. A
partial sibling total binds its covered run of years as the window
(SlotBinding.window), a latest window counts a donor holding any year,
and sibling and held years come from one coverage scan per family. A
pool loses its record only when it pairs the EU institutions with an
EU member, read from declared membership. AxisTable.rule derives from
rules.
* fix(search): measure a DAC1 sibling total against every year the pool holds
A pool's sibling option compared its coverage with the years every
donor held, so one donor holding none made the option fully available
and a switch understated the pool. It now compares with the union of
the donors' years, and a partial sibling states its covered run in its
label.
* fix(search): judge a DAC1 sibling total per donor and pin its latest year
A year counts as covered when every donor holding the part's own total
that year holds the sibling too, so a donor with no own total places
no demand. On a latest window the sibling elects the year its pool
shares and pins it: available when it matches the total's year,
partial with that year otherwise, unavailable when the donors share
none. donor_years reads only the window's years.
* fix(search): guard the DAC1 sibling total against empty coverage
A latest sibling whose elected year no donor demands is unavailable
rather than raising, and a part whose own total holds no year judges
the sibling per donor. The EU-institutions double-count guard reads
membership over the part's window (policy_members_between), so a pool
with the United Kingdom before 2020 gets no record. The
SlotBinding.window description covers the latest case.
* test(search): drop an assertion on the whole test's own helper
* docs(search): state the sibling-option rules as the code applies them
* fix(search): judge each requested option on the identity the other requested edits produce
A resubmit switching measure basis and provider together judged the DAC1
sibling on the old basis and refused it as option_not_covered, though the
same switches one after the other compiled. Each explicit entry is now
judged on the axis table read with the other explicit entries' edits
applied; options that conflict with another requested option make no edit,
so the conflict rule still decides that pair.
…nd DAC countries' total (#167)
…callers (#169) A POST /api/datasets carrying X-DFO-Mint-Admission: background is admitted without waiting and only while it leaves a work slot, a materialise permit and a place in the operation backlog for foreground callers. Where a foreground mint would wait or be refused, it answers 503 service_unavailable with constraint_id mint_background_declined: at the work executor, the materialise permit, a replaced release, a rebuild of the subject already running, and a full backlog. Once admitted it runs as a foreground mint, fold permit included, so a foreground caller joins it and never fetches twice. A head hit and an existing operation are still served. Requests without the header are unchanged, and the OpenAPI document does not move.
* fix(assembler): list a per-period selection's own rows in a table A table over a frame that keeps some series in some periods by design (a per-period top-N rank, a value filter) padded every series to the whole window with status "missing", so a top-donor-per-year table for Kenya showed ten rows, five of them "Missing" for donors that did not lead that year. selects_per_period reads the frame's recorded steps; for such a frame _build_table lists the frame's rows and the MAX_TABLE_ROWS check counts the rows the table will hold. Tables over fetched frames and every chart keep their padding. * fix(assembler): trim a per-period top-N table in the pipeline Any filter step enabled row suppression, so a no-op filter hid a genuine missing year, and the table's description still counted the series the trim removed. Only a per-period rank with a limit selects now. _build_candidate trims right after bind_all: a padded point drops only where the frame holds a row for another series that period, and a series left with no point drops, so the mint, selection_facts, the description and the MAX_TABLE_ROWS check all read the same series. _build_table is back to its base form. * fix(assembler): read a selection off the output chain and its latest year selects_per_period matched a per-period rank anywhere in the frame's history, so a share whose denominator was a top one trimmed genuine gaps in its numerator. It now walks back from the last step through single-input steps only. On a selection frame a relative window's latest is the newest period any selected series holds: latest_anchor took the year half the series hold, so a top-donor table with three donors ended in 2018 though the frame ran to 2021. * fix(assembler): follow the input a step's rows come from selects_per_period stopped at every two-input step, so a share or quotient whose numerator is a per-period top one, or a join with a ranked side, padded each unchosen donor-year again. The walk now follows the roles a step's rows come from (numerator for share and quotient, both sides of a join) and stops at sum and compare. On a selection frame latest is the newest period any frame row holds, the same held period the trim reads. * fix(assembler): end a selection's window at its own donors' latest year Anchoring latest on every frame row refused a USA stat card and an IDA-plus-USA table at latest..latest whenever an unselected donor led the newest year. Latest on a selection frame is again the newest period any of the card's selected series holds. The walk also follows the denominator of a share or quotient keyed by denominator_by, since each key then keeps only its denominator group's periods. * docs(assembler): wrap the selects_per_period docstring
…181) * perf(search): compute table years from per-pair masks built at load table_years needed only the years any series of a family holds, yet series_by_year copied the whole mask matrix for each of 128 years to count one bit: 583 allocations and 8,301 reductions, 73% of a dataset_resubmit compile's CPU and 8.7 s and 13 s on deployed dev. CoverageIndex now ORs each (variable, facet) pair's year masks once when it is built (24,267 pairs in 9 ms on the dev release), and years_held ORs the chosen pairs under the request's source scope. series_by_year finds the held years the same way and counts only those. _served_entry and _as_composite are timed as resolve steps. * docs(search): name the served_entry and as_composite resolve steps * test(search): a memo hit books the scope compile that names its key
#180) * perf(datasets): keep a head until the import it was built from changes A head was rebuilt once it was older than 24 h, so a mint of total US ODA rebuilt in 46.6 s although its CRS and imputed sources had not changed. A head whose provenances are all DFO's (data_commons DFO in the bound origin catalogue) now records Manifest.imported_data, a digest of the release's import_execution_id and its platform parts, and is served while the bound release carries that import, up to a 7-day backstop (DATASETS_HEAD_IMPORTED_DATA_MAX_AGE_SECONDS). A new import rebuilds it with rebuild_reason imported_data. Builder, serving and DFO-layer parts stay out of the key, since deploys move them without changing observations. Every other head keeps the 24 h rule. * fix(datasets): key a head on everything its mint read A composite dropped its components' import key, a run over a StatVar base Data Commons also holds was keyed though a base update could win the election without an import, the rate-table cache reused a table a same-day re-import replaced, and the key left out the code index and role axes the mint reads. The composite carries the key its components share; a run is keyed only when the catalogue lists DFO alone for every member StatVar; the rate-table cache keys on the bound import key; and Release.imported_data hashes the import id, the platform parts, the code index part and the role axes and build year from declarations. * fix(datasets): recheck the catalogue and composites before serving a keyed head A keyed head could serve old rows for up to seven days when the catalogue later listed base Data Commons for a member, when a deploy changed a composite's declaration, or when the mint's default scope read fell back unsettled. The judge reruns the DFO-alone check against the bound catalogue; a run is keyed only when every candidate its election ranked is DFO's and its default scope settled; and the composite fingerprint carries a digest of the deployed declaration and the basis measures it reads, recomputed on refresh. * docs(datasets): bump DEPENDENCIES_SCHEMA when the materialiser's output changes A deploy moves no head key, so a keyed head serves the old frame until the next import or its maximum age unless the schema is bumped.
#182) * perf(datasets): coalesce a composite's members and fetch on 16 workers The cold total-ODA mint took 37.6 s on dev. Its composite pass built one frame per member, 615 frames of about 76 rows, paying about 4 ms of fixed Arrow work on each, and its fetch waited on four lanes of mixer requests whose server time is about 0.09 s whatever their size. Composed members now join into chunks of up to 100,000 rows, with identical rows and content version, and a mint fetches on 16 workers over 16 client slots. The request split keeps its floor of four, so request counts and sync estimates are unchanged. * fix(datasets): widen composed members to one unit layout before joining A member whose unit cells carry a conversion recipe has a wider unit struct, so pa.concat_tables raised ArrowInvalid on a chunk mixing it with a plain member. Each member table is widened to the recipe layout first; build_frame_from_arrow narrows it back when no cell carries a recipe, so frames without recipes are unchanged.
) * perf(datasets): send no request for members with no dispatch facet _plan_collection_chunks packed eight consecutive members and skipped a chunk only when none had a dispatch facet, so 753 of the 1,974 members the total-ODA mint fetched held no facet and rode for nothing. Idle members now plan into chunks of their own that take the skip path and send no request, sync_estimate counts only members with coverage rows under the bound roles with lanes per component, and the batch target rises from 8 to 16. Total ODA sends 77 requests where it sent 247, with the same content version, rows and artefacts. * fix(datasets): never plan fewer requests for more members stat_var_chunk_positions planned 3 requests for 5 members and 4 for 4, so when the election idled a covered member the estimate could fall a request short of dispatch. Each group now splits into min(lanes, n) consecutive runs whose sizes differ by at most one, so the count never falls as members are added and the estimate can only overshoot.
…rices (#191) * fix(search): let named values a family group lacks drop out of its collapse A named value no partition of a group carries no longer blocks the others, so commitments steers the collapse when the unit text also names RealValue. * fix(search): elect the partition carrying the most named values, unconvertible ones first A price basis can still be served by conversion after election and any other named value cannot, so the collapse ranks partitions by the values they carry that unit conversion cannot reach, then by those it can.
* fix(search): merge parts that bind the differing slot to a set _build_merged_slots reads a value as a set of one and merges the union, refusing code sets that share a code and sets on the where axis. * fix(search): state one provenance for a merged slot and keep a whole member set's ref A merged slot filter reads text_recognized when any member named the property in text, a value without a ref refuses the merge, and a merged set equal to one member's set keeps that member's set_ref.
…e answers by component (#186) #163 made Search answer by the OECD's conventions: a recipient's total in net flows, a donor's total from 2018 in grant equivalents, a sector, instrument or modality part in gross disbursements, and "health" as CRS 120 and 130. The corpus still expected the earlier answers, so 67 cases regressed against main. Each now expects the convention's answer, with a `why` that names the rule, and `dac1.disbursement_net` joins the DAC families. The materialisation check counted no observations for an answer served by the OECD headline composite, because no graph series carries a composite's provenance. It now derives the components with `component_identities` and sums each component's count under its own table's provenance and inside its own years.
…190) * fix(datasets): store series conversions in the list form prod reads manifest_facts writes per-series records through encode_series_conversions; the decoder still reads the packed form dev already stored. * perf(datasets): decode series conversions only for a read and share each distinct record The observations door decodes conversions inside reads(), the decoder and encoder build each distinct record once, and the door test covers the packed form dev already stored.
…onclusive (#192) * fix(eval): report a check with only degraded base samples as base inconclusive A degraded base sample says nothing about the base code, so the gate judges only undegraded base samples and exits 3 when none is left. * fix(eval): treat only a degraded base pass as inconclusive A degraded base failure still shows the base fails, so it counts as before; a degraded pass cannot support a regression, and the three-pass bar stays three undegraded passes.
* fix(search): keep convention bases off the bind menu Recall marks the rows it adds for a measure basis and their shapes leave the menu; the house-policy step admits a sibling only beside a pool shape it is another basis of and only when it holds the pick's drafts. * fix(search): read a basis sibling like any shape and withhold it only from the binder A basis sibling carries its source family from recall and goes through every menu stage; seating keeps it out of the binder's prompt while its source is ranked. The house-policy step admits it beside a pooled source with one policy path. Stated-words requests say whether the shape was on the bind menu, so Jev rates a moved sibling's words afresh. The election helpers live in binders/election.py. * fix(search): hold withheld basis siblings in their own list Seating puts basis siblings in state.basis_siblings, never in capped_shapes, so menu readers cannot see them. A sibling joins only beside a source the binder reports holding, the move is checked against the drafts the policies judged, and only the moved seat is read as off the bind menu. * fix(search): bind a slot value the query names when the bind leaves it open Settlement binds the one value of an unbound slot that the query's words name, by label or by its dcid in words, as text_recognized, so a named gas no longer depends on the bind model; measure axes, measure-name words and values every member holds are left alone.
Contributor
Author
|
Replaced by the release PR from the cut branch release/2026-10-29 at 35e1406. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases
devatfd0965b05to production: 24 pull requests. Dev servesfd0965b05.Search answers: #119 keeps each extracted name its own choice (
entity_choice), #149 words a nearest-real suggestion as its answer does, #110 judges stated-word coverage on the members the read served, #152 serves a top facet with no provenance to a native-unit request, #157 and #162 make the answer, its bulk read and a mint elect the same facet (with themember_unavailabledeparture andunserved_membersoncoverage_below_scope), #156 answers SDK users' gaps in warnings, refusals, resolve and counts, #160 counts an open role's parties, #163 reports the scope choices each answer settled (resolved.scope_choices), #166 and #167 state DAC1 provider scopes and bind DAC countries' total, and #179 lists a per-period selection's rows in a table.Performance: #165 keeps the mint fetch pool full and hedges slow SDMX requests, #168 caches arc answers per snapshot and folds mints in Arrow, #169 admits a mint sent with
X-DFO-Mint-Admission: backgroundonly while foreground callers keep room, #180 keeps a head until its import changes, #181 builds table-year masks at load, and #182 and #184 coalesce composite members and skip members with no dispatch facet. Tests: #143, #144 and #150. #154 and #159 move the spec-kit pin.The deploy runs no migration. It reads four new settings, all defaulted (
groupby_fold_max_rows,groupby_row_fold_max_rows,groupby_fold_max_concurrency,head_imported_data_max_age_seconds), andSEARCH_SCOPE_MIN_YEARandSEARCH_SCOPE_MAX_WINDOW_YEARSkeep their names undersearch.scope_window. Every variable dev's tfvars set is also set ininfra/dfo-prod/apis. Thedatasetsextra addsresolve(resolvekit). The spec-kit pin moves from0d19172to455e958, so spec-kit's release merges first.deploy-prod.ymlis unchanged;ci.ymlandfull-grids.ymlchange.The search wire gains
entity_choice,resolved.scope_choices, themember_unavailabledeparture andunserved_members. dfo-ai's release readsresolved.scope_choicesand sends background mints, and the website's vendored client readsentity_choice, so both release after this deploys.Testing: CI and the eval gate passed on each pull request into dev. CI and Deploy dev passed on
fd0965b05; Eval nightly and Full grids passed one828b9c7b. The production deploy waits for approval.