Repository navigation
feat(search,datasets): count a donor total with its imputed multilateral spending and the OECD headline - #164
Merged
Merged
Conversation
…components A declared composite provenance names how its components derive from the composite identity: bilateral CRS plus imputed multilateral stacked over the years both cover, or the OECD headline spliced from DAC1 net before 2018 and grant equivalents from 2018. The mint materialises each component through the single-provenance path under its own pins, keeps only the composite window, writes a zero row where the imputed table leaves a covered cell empty, and stamps the composite on every row. Citations and a re-render from the version record name the component tables. The admission estimate counts the zero rows, a frame mixing raw provenances is still refused, and every reader of facets.provenance either expands the composite or refuses it.
A CRS part gains a provider choice: bilateral flows, or bilateral plus imputed multilateral over the years the imputed table covers, offered shortened with its window when it covers only part of the request. A total from a named donor defaults to the inclusive reading when it covers the request. A donor-only DAC1 window crossing 2018 is counted as the OECD headline, net to 2017 and grant equivalents from 2018. Both answer as one composite spec with its components, switch and switch back to the same spec_id, and read through the component tables wherever Search reads a provenance.
# Conflicts: # packages/dfo-data-tools/src/code_index/scope_conventions.py # packages/dfo-data-tools/src/graph_release/declarations.py # src/search/scope/basis.py
…d provider the part lacks A composite answer states its own headline year and series, and counts the StatVars of every component. A typed provider word on a part with no provider axis, such as a DAC1 donor total, is reported as not applied.
# Conflicts: # src/search/README.md # src/search/engine/region/pipeline.py # src/search/models/scope.py # src/search/scope/basis.py # src/search/scope/compile.py
…p its latest year out of the base A headline or inclusive reading becomes the default only when every component is available; otherwise the served shape stands with the rule it was served under. An inclusive latest window keeps the base on latest and elects the latest year both tables cover after compile, so a switch to bilateral answers bilateral's own latest year. The worker estimates the composed frame, and a composite is labelled from its declaration.
# Conflicts: # packages/dfo-data-tools/pyproject.toml # pyproject.toml # uv.lock
The restated composite takes its date range, its when slot and the data years its departures read from the years the components serve, so a latest-year answer names the elected year.
…ite both of its tables A composite mint passes each component's unserved members through to its coverage warning and writes no zero row for a member its facet group left out. A composite answer cites each component table and names its measure from the declaration.
…, over a year both tables hold The inclusive provider scope is offered for a part whose named providers the imputed table carries, and for a pool only when none of them is a multilateral whose own outflows the table attributes to its funders (REDISTRIBUTED_PROVIDERS), so a pool of the US and IDA, or a part naming no provider, keeps the bilateral count. A latest read elects its year among the years both tables hold. A composite zero row states the unit its cell states, so a member absent from one component takes its own currency.
# Conflicts: # packages/dfo-data-tools/src/graph_release/declarations.py # src/search/engine/assemble.py # src/search/engine/entry/dataset_resubmit.py # src/search/engine/region/phase_election.py # src/search/scope/state.py
…d compose each frame once A stacked composite serves the years every component table covers, so a year the imputed table covers keeps its bilateral figure with imputed zeros. Every path that restates a composite elects its latest window, explicit entries attach only where their concept holds, a composite provenance names its component datasets to unit conversion and skills, and the ceiling counts only the rows a composite keeps. The mint builds the composed frame once and folds the digest, statistics and output profile in that pass, cutting a 43,680-row composite mint from 7.4 s to 3.8 s with byte-identical output.
# Conflicts: # src/search/engine/region/phase_election.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A total from a named donor now counts bilateral flows plus the donor's imputed multilateral spending, and a donor total whose window crosses 2018 is counted as the OECD headline: net flows to 2017 and grant equivalents from 2018. Both are composite specs. A declared provenance (
COMPOSITESincode_index/declarations.py) names how each component derives from the composite identity, andcode_index/composites.component_identitiesis the one derivation Search, resubmit and the mint share.The mint (
datasets_api/serve.prewrite_definition,materialise/composite.compose_components) materialises each component through the existing single-provenance path under its own pins, keeps the years every stacked component covers, writes a zero row where the imputed table leaves a covered cell empty (and none for a member its facet group left out), and stamps the composite provenance on every row. No row is a total, so a sum over the frame counts each figure once. Citations and a re-render from the version record name the component tables, the admission estimate counts the zero rows, a frame mixing raw provenances is still refused, and every reader offacets.provenanceeither expands a composite or refuses it.Search gains a provider choice on CRS parts (
search/scope/provider.py): bilateral, or bilateral plus imputed over the years the imputed table covers, offered shortened with its window when it covers only part of the request. The convention table defaults a named-donor total to the inclusive reading when every component is available, and alatestwindow elects the latest year both tables cover.measure_basis:oecd_headlinejoins the basis choice on DAC1 donor parts. Switches round-trip to the same spec_id, and a typed "bilateral" applies.Stacked on the scope-choices PR (#163); merge that first. Reviewers should challenge the zero-row rule in
compose_components, the base identity of an inclusive default inscope/build._with_recorded_composites, andtable_identity.The composite figure on real data is checked on dev after merge: "total health aid from the US to Africa in 2024" should mint about $7,906.6m, equal to the bilateral $5,813.5m plus the imputed component in every year. A tagged dev revision confirmed the search side and the bilateral mint; its worker could not take tasks addressed to the tag URL.
Testing: targeted suites (
tests/scope,tests/code_index, graph release, composite mint and composer, resubmit, first-answer scenarios,test_corpus_offline) pass locally; the full suite runs on CI. On a tagged dev revision: the inclusive default (spec_153b4e19), its switch round trip, the bilateral mint at $5,813,494,557, the 2011 shortened window, the headline election for 2015 to 2020 and the latest-year election.