diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cbfe334..59d4925 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -128,7 +128,7 @@ jobs: run: uv run tools/validate_unit_registry.py - name: Test the registry validator - run: uv run --with jsonschema --with pytest pytest tools/tests -q + run: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q - name: Validate identity vectors run: uv run tools/validate_vectors.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..5468804 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing + +Issues and pull requests are welcome. The ONE Data team maintains this repository. For a large change, such as a new contract, a schema change, or a new package, open an issue first so the design can be discussed before you write it. + +Please follow the [Code of Conduct](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/CODE_OF_CONDUCT.md). Report security problems through GitHub private vulnerability reporting (the "Report a vulnerability" button under the Security tab, see the [security policy](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SECURITY.md)), and keep them out of public issues. For questions, see [SUPPORT.md](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SUPPORT.md). + +## Setup + +Everything runs locally and needs no access to ONE's Google Cloud project. Install [uv](https://docs.astral.sh/uv/) and clone the repository. Each package under `python/` is its own uv project with a committed `uv.lock`, so work from inside the package directory: + +```bash +cd python/spec-kit +uv sync +``` + +The six packages are `spec-kit`, `frame-grammar`, `dfo-spec-kit-components`, `provenance-docs`, `descriptions`, and `client-contract`. Python 3.12 or later is required, and CI also runs 3.13. + +## Checks before a pull request + +Run these in each package you changed. They are the commands CI runs. + +```bash +uv run pytest +uv run ruff check . +uv run ruff format --check . +uv run ty check src/ +``` + +`` is `dfo_spec_kit`, `dfo_frame_grammar`, `dfo_spec_kit_components`, `dfo_provenance_docs`, `dfo_descriptions`, or `dfo_client_contract`. `uv run ruff format .` rewrites files to the formatter's style. Each package also has a `.pre-commit-config.yaml`. `pre-commit install` runs the same linters, plus codespell, actionlint, and mdformat, on each commit. + +### Conformance vectors and registries + +The validators under `tools/` check the registries and vector suites without importing the packages, so they test the fixtures independently of the implementations. Run them from the repository root. Each is a self-contained script and uv installs its dependencies. + +```bash +uv run tools/validate_registry.py +uv run tools/validate_unit_registry.py +uv run tools/validate_vectors.py +uv run tools/validate_frame_vectors.py +uv run tools/validate_unit_vectors.py +uv run tools/validate_provenance_docs_vectors.py +uv run tools/validate_descriptions_vectors.py +uv run tools/validate_client_contract.py +uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q +``` + +The last command tests the validators themselves. [tools/README.md](tools/README.md) says what each validator checks. Each package's tests drive its implementation against the same vectors or fixtures, and `uv run pytest` runs them. + +A change to `registries/codes.json` or `registries/units.json` also needs the packaged mirror under `python/spec-kit/src/dfo_spec_kit/` to match byte for byte. The validators fail if the two differ. Edit the file under `registries/` and copy it to the mirror. [registries/README.md](registries/README.md) has the format. + +## Branches and commits + +Open pull requests against `dev`, the default branch. CI runs on pull requests and pushes to `dev` and `main`, and skips draft pull requests until they are marked ready for review. Commit messages follow Conventional Commits, for example `ci: run spec-kit CI on dev and main`. + +## What a pull request needs + +- Passing CI, meaning the lint, test, and validator jobs in `.github/workflows/ci.yml`. +- Tests for new behaviour. A change to a contract updates its vectors in the same pull request. +- Updated docs. A change to a package's public behaviour adds an entry under `Unreleased` in that package's `CHANGELOG.md`. +- Compliance with [policies/](policies/README.md), which says which change to which artifact needs which version bump. + +Maintainer notes, such as how the packages depend on one another and how releases are checked, are in [docs/development.md](docs/development.md). diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..4b7a716 --- /dev/null +++ b/LICENSE @@ -0,0 +1,22 @@ + +MIT License + +Copyright (c) 2026 ONE Campaign + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index d290c0e..670e102 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,54 @@ -# dfo-spec-kit +# DFO spec kit -The executable form of the platform's shared contracts. It exists so the consumers of these contracts, the Search API in `apis`, the harness and Analysis MCP in `ai`, the website, and the docs, share one source of truth instead of hand-maintained mirrors. This repo is not the spec registry nor the op-spec schema. +The spec kit holds the shared data contracts of the Development Finance Observatory as JSON schemas, registries, conformance vectors, and Python packages that implement them. Services that must agree on the same bytes, ids, and document formats import them from here instead of keeping their own copies. -The target design (repository shape, artifact families, versioning, governance, and delivery sequence) lives in the kit design, maintained outside this repo. This README only orients. That document is authoritative. +It is part of the [Development Finance Observatory](https://github.com/DevelopmentFinanceObservatory), an initiative of [ONE Data](https://data.one.org) built on [Google Data Commons](https://datacommons.org). The org profile has the map of all repositories. The services that consume the kit are `apis` (the Search API), `ai` (the harness and Analysis MCP), `website`, and `docs`. -## Two schema families +This repository is under active development. Interfaces, schemas, and data may change without notice. -The kit carries two schema families. +## What the kit is for -The **dataset-spec family** (spec models, identity projection, JCS canonicalisation, SHA-256 hashing) is Python-first. The behaviour lives in Python, and JSON Schema is exported from the pydantic models. +Each contract answers one question that two or more services must answer identically. -The **component-spec family** (the presentation catalogue, composition schema, operation union) is TypeScript-first. Zod v4 is the author and validator, and generated artifacts (TS prop types, Pydantic models, tool metadata) are frozen and checked for drift. Each family is authored in the language where its behaviour lives. Committed JSON Schema is the neutral frozen centre between them, held together by cross-language conformance fixtures. +- Which dataset is this request asking for? A dataset spec is reduced to a canonical form and hashed into a `spec_id`, so every service mints the same id for the same request. +- Has this table of observations changed? A materialised frame is serialised to canonical bytes and hashed into a `content_version`. +- How do units compare? A unit ref reads as a typed value (money with a currency and price basis, a percent, a count), and refuses comparisons that would mix price bases. +- What does a card or chart look like on the wire? A catalogue of JSON schemas describes each presentation component. +- How is a result documented? Step records and descriptions say what was done to a result, and provenance documents carry transformations, changelogs, and methodology. +- What do the DFO clients expect from the API? The client contract holds the served OpenAPI document, the response members the clients act on, the next step for each refusal code, and recorded scenarios the clients replay. The API's CI checks its app against the same files. -## Status +Every contract ships with conformance vectors, language-neutral fixtures that any implementation, in any language, can run against. Where prose and vectors disagree, the vectors win. -Six Python distributions exist so far. `python/spec-kit/` (`dfo-spec-kit`, import `dfo_spec_kit`): `identity/` implements the spec1 identity scheme (role-slot projection, JCS canonicalisation, SHA-256 `spec_id` minting), conformance-tested against the vector suite. `units/` reads a unit ref into a typed value (`Money`, `UnpricedMoney`, `Percent`, `Count` or `Other`) from `registries/units.json`, conformance-tested against `vectors/units/`. Money whose ref leaves prices unstated cannot be compared until `complete_prices` applies the StatVar's measurement qualifier, and constant-price money with no base year cannot be compared at all. `models/` has no code yet; model extraction comes next. `python/frame-grammar/` (`dfo-spec-kit-frame-grammar`, import `dfo_frame_grammar`): the frame grammar's executable form -- canonical frame bytes and `content_version` minting for a materialised observation table -- conformance-tested against its own vector suite, and depending on `dfo-spec-kit` for the identity canonicalisation primitives it reuses. `python/dfo-spec-kit-components/` (`dfo-spec-kit-components`, import `dfo_spec_kit_components`): the presentation contract bundle as package data, holding the envelope schema, the per-member component schemas, the write-side intent grammar, and their conformance fixtures. Generated Pydantic models for the catalogue land in this same package once the catalogue freezes. `python/provenance-docs/` (`dfo-spec-kit-provenance-docs`, import `dfo_provenance_docs`): the provenance documentation grammar, which decodes the escaped Markdown documents a Provenance node carries and parses the transformations, changelog, methodology and notes templates, conformance-tested against its own vector suite. `python/descriptions/` (`dfo-spec-kit-descriptions`, import `dfo_descriptions`): the step records and descriptions contract, holding the step schema, the operation list, the description schema and a check of the structure rules a step list must meet beyond its schema, conformance-tested against its own vector suite. `python/client-contract/` (`dfo-spec-kit-client-contract`, import `dfo_client_contract`): the contract the DFO Python and TypeScript clients are built to, as package data: the served OpenAPI document of the composed dfo-apis app, the response members the clients act on, the caller's next step for each refusal code, the code of each error a client raises itself, the scenario and `DatasetRef` formats with their records, and the client rules with their vectors. dfo-apis CI checks its app against the same files. +## Packages -The Python distributions live at `python/spec-kit/`, `python/frame-grammar/`, `python/dfo-spec-kit-components/`, `python/provenance-docs/`, `python/descriptions/`, and `python/client-contract/`, each its own installable unit with its own version and dependency surface. A TypeScript package (`ts/`) opens only when the component catalogue freeze lands. +Six Python distributions live under `python/`. Each has its own version, dependencies, tests, and README. All need Python 3.12 or later. -`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit` as an intra-repo path source rather than a git dependency, so the two packages develop together in this repo: a change to identity canonicalisation and a change here that depends on it land in the same commit. `dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions` as an intra-repo path source, because the envelope's `description` references `description.schema.json` by its `$id`. A consumer pins both distributions at one revision. Its catalogue bundle is static data plus one accessor function, `bundle_dir()`. `dfo-spec-kit-provenance-docs` also stands alone, depending on `markdown-it-py` only. `dfo-spec-kit-descriptions` depends on `rfc8785` only. `dfo-spec-kit-client-contract` depends on the standard library only. +| Distribution | Import name | What it does | +| ------------------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `dfo-spec-kit` | `dfo_spec_kit` | Dataset-spec identity (canonical form, `spec_id`) and typed units read from `registries/units.json`. | +| `dfo-spec-kit-frame-grammar` | `dfo_frame_grammar` | Canonical frame bytes and `content_version` for a materialised observation table. | +| `dfo-spec-kit-components` | `dfo_spec_kit_components` | The presentation catalogue: envelope and component schemas, the write-side intent grammar, and fixtures, as package data. | +| `dfo-spec-kit-provenance-docs` | `dfo_provenance_docs` | Decoder and parsers for the Markdown documents a Provenance node carries. | +| `dfo-spec-kit-descriptions` | `dfo_descriptions` | Step and description schemas, the operation list, and a check of the rules a step list must meet. | +| `dfo-spec-kit-client-contract` | `dfo_client_contract` | The contract the DFO clients are built to: served OpenAPI document, strict fields, refusal actions, scenarios, rules. | -Publishing is undecided: no registry workflow exists for any package. Until that's resolved, install a Python package as a git dependency pinned to one commit, e.g. +`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit`. `dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions`. The other four have no dependency inside the kit. + +## What's inside + +```text +docs/ normative specifications: identity-v1.md, frame-grammar-v1.md, and maintainer notes +policies/ compatibility policy per artifact family +python/ the six packages +registries/ codes.json (warning and refusal codes) and units.json (unit refs) +schemas/ JSON Schema for the registries and the vector suites +tools/ validators for the registries, vector suites, and client contract +vectors/ cross-language conformance vectors for identity and units +``` + +## Getting started + +No package is published to PyPI. Install from git, pinned to one commit. Replace `` with a commit SHA from this repository. ```bash uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/spec-kit" @@ -31,7 +59,7 @@ uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subd uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/client-contract" ``` -A package's metadata names its intra-repo dependency by distribution name alone, and no index serves these distributions. uv reads the path source in `[tool.uv.sources]` and resolves the dependency from the same checkout, so `uv add` of `frame-grammar` or `dfo-spec-kit-components` alone installs `dfo-spec-kit` or `dfo-spec-kit-descriptions` at that commit. Every other installer reads the bare name and fails, so name both distributions at the same commit: +A package declares its kit dependency by name only. uv resolves it from the same commit through a path source in the package's `[tool.uv.sources]`, so `uv add` of `frame-grammar` also installs `dfo-spec-kit`. Other installers read the bare name and fail, so with pip name both distributions at the same commit: ```bash pip install \ @@ -39,7 +67,7 @@ pip install \ "dfo-spec-kit-descriptions @ git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/descriptions" ``` -A uv project pins each kit distribution it imports in its own `[tool.uv.sources]`, every entry at the same `rev`, as `apis` pins `dfo-spec-kit-components` and `dfo-spec-kit-frame-grammar`: +A uv project that imports more than one kit distribution pins each in its own `[tool.uv.sources]`, every entry at the same `rev`: ```toml [tool.uv.sources] @@ -47,30 +75,20 @@ dfo-spec-kit-components = { git = "https://github.com/DevelopmentFinanceObservat dfo-spec-kit-descriptions = { git = "https://github.com/DevelopmentFinanceObservatory/spec-kit.git", rev = "", subdirectory = "python/descriptions" } ``` -## Layout +The kit needs no access to ONE's Google Cloud project. Everything here installs, runs, and tests locally. -``` -dfo-spec-kit/ -├── docs/frame-grammar-v1.md # the normative frame grammar / content_version specification -├── docs/identity-v1.md # the normative identity binding-state denotation specification -├── schemas/registry/ # JSON Schema for the code registry -├── schemas/spec/ # frozen JSON Schema exported from the pydantic models -├── schemas/vectors/ # record schemas + docs for the identity and unit vector suites -├── registries/codes.json # the seeded warning/refusal code registry -├── registries/units.json # the unit refs dfo_spec_kit.units types, its currency prefixes and codes -├── vectors/identity/ # cross-language identity conformance fixtures -├── vectors/units/ # cross-language typed-unit conformance fixtures -├── python/spec-kit/ # the dfo-spec-kit package (import dfo_spec_kit) -├── python/frame-grammar/ # the dfo-spec-kit-frame-grammar package (import dfo_frame_grammar), -│ # its own frame grammar vector suite shipped as package data -├── python/dfo-spec-kit-components/ # the dfo-spec-kit-components package (import dfo_spec_kit_components), -│ # the presentation catalogue bundle shipped as package data -├── python/provenance-docs/ # the dfo-spec-kit-provenance-docs package (import dfo_provenance_docs), -│ # the provenance documentation grammar and its vector suite as package data -├── python/descriptions/ # the dfo-spec-kit-descriptions package (import dfo_descriptions), -│ # the step and description schemas, the operation list and their vector suite as package data -├── python/client-contract/ # the dfo-spec-kit-client-contract package (import dfo_client_contract), -│ # the served OpenAPI document, strict fields, caller actions, scenarios, ref format and client rules as package data -├── policies/ # compatibility policy per artifact family -└── tools/ # codegen + drift-check scripts, including the registry and vector validators -``` +## Documentation + +- [docs/identity-v1.md](docs/identity-v1.md) specifies what a spec identity's bindings denote. +- [docs/frame-grammar-v1.md](docs/frame-grammar-v1.md) specifies the frame byte grammar and `content_version`. +- Each package README under `python/` documents that package's API and vector suite. +- [policies/](policies/README.md) states which change to which artifact needs which version bump. +- [docs/development.md](docs/development.md) holds maintainer notes. + +## Contributing + +Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). + +## Licence + +MIT. See [LICENSE](LICENSE). diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..5f2ba4c --- /dev/null +++ b/docs/development.md @@ -0,0 +1,31 @@ +# Maintainer notes + +Background on how the repository fits together. Contributors do not need it to open a pull request (see [CONTRIBUTING.md](../CONTRIBUTING.md)). + +## Two schema families + +The kit carries two schema families, each authored in the language where its behaviour lives. Committed JSON Schema is the neutral centre between them, held together by cross-language conformance vectors. + +The dataset-spec family (spec models, identity projection, JCS canonicalisation, SHA-256 hashing) is Python-first. The behaviour lives in `python/spec-kit`, and `schemas/spec/` is the place for JSON Schema exported from its pydantic models. It holds only a README today, because `dfo_spec_kit.models` has no code yet. + +The component-spec family (the presentation catalogue, composition schema, operation union) is TypeScript-first: Zod v4 authors and validates, and generated artifacts (TypeScript prop types, Pydantic models, tool metadata) are checked for drift. The family's artifact in this repository is the catalogue bundle in `python/dfo-spec-kit-components/`. The repository has no TypeScript package. + +[policies/dataset-spec.toml](../policies/dataset-spec.toml) and [policies/component-spec.toml](../policies/component-spec.toml) state the compatibility rules for each family. + +## Dependencies between packages + +`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit` through a path source (`../spec-kit`, editable) in its `[tool.uv.sources]`. The two packages develop together here, so a change to identity canonicalisation and a change in the frame grammar that depends on it land in the same commit. + +`dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions` the same way, because the envelope's `description` references `description.schema.json` by its `$id`. The catalogue bundle is static data plus one accessor function, `bundle_dir()`. + +`dfo-spec-kit-provenance-docs` depends on `markdown-it-py` only. `dfo-spec-kit-descriptions` depends on `rfc8785` only. `dfo-spec-kit-client-contract` depends on the standard library only. + +A package's published metadata names its kit dependency by distribution name alone, and no index serves these distributions. uv reads the path source and resolves the dependency from the same checkout. Every other installer reads the bare name and fails, which is why the README shows pip installs that name both distributions at one commit. A consumer pins both at one revision. + +## Releases + +No registry workflow exists for any package, so releases are git revisions. The compatibility policies define "released" as reachable from the most recent git tag matching `v*`. `tools/validate_registry.py` diffs the working registry against that tag to enforce that published code entries do not change, apart from the deletion exception named in the script and growth of a published entry's `mappings`. This repository has no tag reachable from `dev`, so the check is skipped and the `registry` CI job fetches full history so that it can run once one exists. + +## Ownership + +`CODEOWNERS` lists a maintainer for `/schemas/`, `/registries/`, `/vectors/`, and `/python/`. diff --git a/python/client-contract/CHANGELOG.md b/python/client-contract/CHANGELOG.md index ce4270a..ea1159e 100644 --- a/python/client-contract/CHANGELOG.md +++ b/python/client-contract/CHANGELOG.md @@ -7,20 +7,37 @@ All notable changes to the `dfo-spec-kit-client-contract` Python package will be ### Added - Package scaffolding (uv, ruff, ty, pytest). -- `served/openapi.json`, the composed dfo-apis app's OpenAPI document, and `served/SOURCE.json`, naming dfo-apis commit `0ba344270784ae9929a369da8acd00b399dfced8`. The document admits `timed_out` as a no_data reason and the optional `entity_refs` on each answer part, a list of `EntityRefs`, the graph nodes each restatement entity text binds. +- `served/openapi.json`, the composed dfo-apis app's OpenAPI document, and `served/SOURCE.json`, naming dfo-apis commit `d80df9f5fd598b7345255470932d202ee5d49942`. The document admits `timed_out` as a no_data reason and the optional `entity_refs` on each answer part, a list of `EntityRefs` with one entry per restatement entity row, the graph nodes that row binds, and a stored manifest warning states its counts and the refs it names (`CountedRefsModel`, `CountedEntityYearsModel`, `EntityYearsModel`). `NoDataOption` and `SentenceToken` carry `entity_choice`, the index into `NoData.ambiguous_entities` of the group an entity_ambiguous option settles or a slot token asks, so a client groups a place menu by it when two wordings share one text. - `strict-fields.json` and its schema: 29 members that stay required and non-nullable, each one a client reads, and the answer kinds, part statuses and operation statuses the clients branch on. -- `actions.json` and its schema: the caller's next step for each of the 102 active refusal codes on `problem_details` or `no_data_reason`, with the gap kind of each no_data reason. +- `actions.json` and its schema: the caller's next step for each of the 104 active refusal codes on `problem_details` or `no_data_reason`, with the gap kind of each no_data reason. - The status fallback for a refusal whose code has no row: 400 and 422 to `InvalidRequestError`, 401 to `AuthenticationError`, 403 to `PermissionDeniedError`, 404 and 410 to `NotFoundError`, 408 to `DfoTimeoutError`, 429 to `RateLimitError`, and any other 4xx or 5xx to `InvalidRequestError` or `ServerError`. -- `scenario.schema.json`, the scenario record, and 120 scenarios: 31 recorded against the deployed dfo-apis dev app at `0ba344270`, and 89 synthetic, each derived from a recorded one by a one-line edit, with 19 binary bodies totalling 1,124,502 bytes and records and bodies together 13,161,129 bytes. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the `SearchWarning` rule, the stacked dataset's `part` column, the frame layout, a strict fetch's `DataWarning`, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 `capability_unavailable`, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, `x-dfo-warning` messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose `retry_after_ms` changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no `base_content_version`, the newest plain build, so a derived version listed before it is skipped. +- `scenario.schema.json`, the scenario record, and 126 scenarios: 34 recorded against the deployed dfo-apis dev app at `d80df9f5f`, and 92 synthetic, each derived from a recorded one by a one-line edit, with 21 binary bodies totalling 1,134,603 bytes and records and bodies together 13,695,441 bytes. `search-place-ambiguous-twice` pins two entity_ambiguous groups of one text, each option and slot token naming its group by `entity_choice`, and `choose-refused-label-in-two-groups` pins that a label both groups offer matches two options, so a choice there names its option by index. The scenarios cover the several-part choice entries and the assumptions each identity entry echoes, a picked reading whose assumption Search states in its answer, the place choice through a no_data part's options, the choice refusals, a part without variable_text, an unknown status without a restatement, a shared label, a one-part option whose data_query holds two parts, the separate and operations echo, an unsupported part's placeholder, the timed_out, unsupported and unknown gaps, an unknown answer kind, the `SearchWarning` rule, the stacked dataset's `part` column, the frame layout, a strict fetch's `DataWarning`, a spec fetch that rebinds one of two entity roles and keeps the other, one that drops a role bound to null, a spec whose destination is a named group fetched with a window or with its origin rebound, whose sentence lookup posts the groups of the roles the build left bound, a twice cut-off build, a cut-off build retried after its Retry-After, a retryable refusal whose Retry-After passes the 60-second cap, a build refused 504 then 503 and one refused 503 then 504, a refused build, a build refused 501 `capability_unavailable`, a digest mismatch, a signed URL refused on both rounds, a relative files-route Location, an artifact url on a foreign origin, on the storage host and absolute on the API's origin, a manifest definition without identity or schema_version, a Search spec without identity or resolved, nested specs no rule reads, an empty parts list, a Search body that is not an object, a codeless 503 problem document that is not retryable, a 401, an identity resubmit answering another spec, a part whose status is not a string, a Search spec whose definition names another spec_id, a Search response with no JSON body, an option with no data_query, empty entities and facets, a Sunset header issued once, `x-dfo-warning` messages and a Deprecation and Sunset pair, an unreadable versions page, a mint and a spec fetch whose build answers 202 and is polled to done, a resumed operation, a rate-limited poll, 503 polls retried within the budget an answered poll restores, a 202 that reads running, polls whose `retry_after_ms` changes from one to the next, a 404 poll that raises, a failed operation posted again, a failed operation with no error, and a ref whose stored file expired, read through the file selector, rebuilt at the same version, at a newer one, or refused with the newest version read from the versions route: the first version it lists with no `base_content_version`, the newest plain build, so a derived version listed before it is skipped. - `client-codes.json` and its schema: the 23 codes a client sets on the errors it raises itself, each with its error class, retryable flag and condition, disjoint from `actions.json` and `registries/codes.json`, and the `client_codes()` and `client_codes_schema()` accessors. Every scenario error the client raises itself carries its code, and `refusal_without_code` names a refusal whose body carries no code. -- `proxy-codes.json` and its schema, the website proxy's `rate_limited` and `transient_unavailable` codes with their statuses, which the proxy sends in problem documents of the shape every refusal has, both retryable, with `Retry-After` on the 429, so a client reads one refusal shape, and `warning-codes.json` and its schema, the 6 codes a client sets on the warnings it issues itself, with the `proxy_codes()` and `warning_codes()` accessors. +- `proxy-codes.json` and its schema, the website proxy's `rate_limited` and `transient_unavailable` codes with their statuses, which the proxy sends in problem documents of the shape every refusal has, both retryable, with `Retry-After` on the 429, so a client reads one refusal shape, and `warning-codes.json` and its schema, the 7 codes a client sets on the warnings it issues itself, with the `proxy_codes()` and `warning_codes()` accessors. - `ref_digest_changed` in `warning-codes.json`, a `DfoWarning` for a ref member whose `content_version` serves a file with another digest than the ref's, because the server rendered the same data again. The client returns the data, with no error under strict. - `timed_out` in `actions.json` and `registries/codes.json`, the no_data reason Search gives a part it left unresolved at the request's time limit, with action `ask_data_question` and gap kind `timed_out`, which `actions.schema.json` now admits. The client-side `timeout` code, a transport timeout, is unchanged. +- A no_data part's gap kind comes from its own reason, whatever the answer's `diagnostics.timed_out` says. `answer-timed-out` gives its part the reason `timed_out`, and `answer-timed-out-keeps-reasons` pins a part that keeps `variable_not_resolved` as a `not_understood` gap in a timed-out answer. - Error outcomes give the error's `trace_id`, the body's, else the `x-dfo-trace-id` header's, else null. Scenario outcomes give `retry_waits`, the waits before each retry, and `warnings`, the warning codes a step's responses issued. Scenario steps take no `timeout`. - `dataset-ref.schema.json`, the frozen `DatasetRef` format, with accepted and rejected examples. -- `without_party_totals` in the ref's artifact rendering, `true` for the version rendered with party totals left out and absent for a plain rendering, with `ref-without-party-totals` and `ref-without-party-totals-false-rejected` as its examples. -- `tools/capture_client_contract.py`, which captures `served/openapi.json` from a dfo-apis checkout in process and records the scenarios against a base URL, dev by default, or with `record --offline` derives their outcomes again from the records with no request. A local app's signed URLs are written as the storage URLs a deployment signs. +- `without_party_totals` in the ref's artifact rendering, `true` for a member fetched with party totals left out and absent for a plain rendering, with `ref-without-party-totals`, a default fetch's cut rendering, `ref-one-member`, a fetch with party totals and its plain rendering, and `ref-without-party-totals-false-rejected` as its examples. Each ref example's rendering is the one the build that answered its file posted, and the validator checks it. +- `tools/capture_client_contract.py`, which captures `served/openapi.json` from a dfo-apis checkout in process and records the scenarios against a base URL, dev by default, or with `record --offline` derives their outcomes again from the records with no request. A local app's signed URLs are written as the storage URLs a deployment signs. `record` reads the rule tables from `--package-dir` before any request, and looks for each long build's spec from the year after the one its record built, stopping at the first start year whose estimate is within the sync budget. - `tools/validate_client_contract.py`, which checks the contract against its schemas, the served document and `registries/codes.json`, checks `client-codes.json` against them and against the reasons the key-lookup rule raises, and checks each scenario's `expect` against the outcomes its exchanges, `actions.json` and `client-codes.json` give, each Search request against the input its steps derive, a sentence lookup's `entity_groups` included, each build's `input.scope` against the narrowing its step names over the spec's bound roles, each build's options against its layout, each Search request's whole body, each download against its artifact's digest, each recorded scenario's commit against `served/SOURCE.json`, each problem document's code against `registries/codes.json` or, for the website proxy's, `proxy-codes.json`, each proxy problem document's content type, members and `retryable: true`, each 202 against an operation that reads `done`, and each absolute URL against the storage host, each ref example's members against the dataset members and downloaded files of the scenario it names, and the scenarios directory against a 16,000,000-byte budget for records and bodies together. A record that ends a run of retries while a retry remains does not fit. The outcomes include each long build's operation id, poll waits and resubmissions, and a `VersionUnavailableError`'s newest version. -- The client rules and their vectors: the credentials path and file format, the key lookup order, with an empty `api_key` refused as `key_empty` and an empty profile key as `profile_without_key`, the key format check, and the cache directory and layout. +- The contract README states that a POST for a build identical to one pending or running returns that operation, so a client that stops waiting posts the same build to join it, and that `fetch` waits for a build with no whole-call deadline by default, with a download that delivers no byte for 30 s a retryable `network_error`. +- A fetch of an answer or a spec leaves out party totals by default: its build posts `without_party_totals: true` in `options`, so the server drops the group totals of the identity's open party roles, and a role the caller binds keeps what it binds. Fetch steps take `party_totals: true` for the client option, `party_totals` in Python and `partyTotals` in TypeScript, false by default, which posts no `without_party_totals`. The wire flag and the ref format are unchanged, and a ref records `without_party_totals: true` for a member fetched with the default. `fetch-spec-party-totals-left-out` and `fetch-spec-party-totals-kept` pin both on DAC2a ODA to Kenya in 2022 with the origin open. A mint leaves them out by default too, and its step takes `party_totals`, so its `result` answers the table a fetch of the spec answers; `mint-async-build` keeps them. A resumed operation's `result` takes the rendering its build posted from the done manifest's artifact. +- The `reading-order` rule and its vectors, which carry rows: a client presents a dataset's columns with the reading columns first, the time period, provider, recipient, subject, purpose, value, unit, flow type and provenance, each slot filled by the long layout's label or the frame layout's code, the flow type by CRS `crsTypeOfFlow` or else DAC2a `dac2aAidType`, then the columns whose values differ between rows, then those that hold one value, each group in the server's order, with `stat_var` last. It sorts each member's rows by the filled slots without `value` and the time period, then the varying columns, then the time period, ascending with nulls last and ties in served order, so each series' years read together. Two cells hold the same value when they are equal as JSON values: numbers by value, so `1` and `1.0` are one value, booleans apart from numbers, NaN the same as NaN, lists item by item in order and structs field by field in any order. A column of lists or structs is no sort key. A vector's rows hold one cell per column and each column one JSON type, and `integer-and-float-are-one-value` and `nested-cells-compare-by-value` pin the comparison. Scenario outcomes give `columns` in that order, and the validator checks it against the served files, list and struct columns included, and reports a malformed vector. +- The `client-identity` rule and its vectors: every request a client sends to the API carries `X-DFO-Client: / ()`, such as `dfo-python/0.1.0 (CPython 3.14)` or `dfo-ts/0.1.0 (Node 22.18)`, `browser` as the runtime in a browser page or browser worker, and `other` in any other runtime, such as Cloudflare Workers, with `pattern` matching every well-formed value. A request to another origin, a signed storage URL included, carries none. A caller's headers or header provider naming `X-DFO-Client` in any letter case raises `InvalidRequestError` `header_invalid`, whose condition in `client-codes.json` names that case, and `User-Agent` stays as each client sends it. A scenario replay checks the header on each `path` and `origin_path` request and its absence on each `url` request. +- The `role-keys` rule and its vectors: a caller narrows a spec with `entities=` keyed by `place`, `origin` and `destination`, and a client posts `place` as the wire role `observationAbout`. `Spec.roles`, `specRoles(spec)` and the clients' errors name the caller keys, and a flow spec takes both directions. A key outside the spec's roles raises `InvalidRequestError` `invalid_argument` naming the key and the spec's roles, so `observationAbout` on GDP is refused naming `place` and `financingRecipient` on a flow spec naming `origin` and `destination`. A spec that names no role posts `place` as `observationAbout` and other keys as typed, and refuses `observationAbout` naming `place`. +- The client rules and their vectors: the credentials path and file format, the key lookup order, with an empty `api_key` refused as `key_empty` and an empty profile key as `profile_without_key`, the key format check, the base URL order, `base_url_invalid` for an invalid value, and the cache directory and layout. +- The `base-url` rule states the whole validity rule, and seven new vectors pin it: no query, fragment, `.` or `..` segment, character outside visible ASCII, empty userinfo or non-decimal IPv4 host; trailing slashes dropped; a relative URL valid only in a TypeScript client inside a browser page. +- The `base-url` rule also refuses a `.` or `..` segment written with `%2e`, a backslash, and a port outside 1 to 65535 or written other than in decimal digits, an empty one included, so a client on Python's `urlsplit` and one on the WHATWG URL parser accept the same values. Ten vectors pin them. +- The `file-selector` rule and its vectors: the file selector's query holds the `format`, `bundle` and `layout` of a ref member's rendering, the keys `query_keys` lists, in that order, and leaves out `without_party_totals`, which the selector refuses with 400 `request_invalid` because the version records its own cut. `fetch-ref-expired-file-selector` and `fetch-ref-artifact-unlisted` ask that query, and the validator and the capture build it from `query_keys`. +- Ref format 2 in `dataset-ref.schema.json`: `format_version: 2` and one or more derived members, each `{parts, sentence, spec: {spec_id, recipe: {normal_form_hash, replay_key}}, content_version, output_digest, roots, execution_identity, resolved_periods, artifact}`, with an `ops_` id, an `ocv_` version, `rnf_` and `rpk_` recipe hashes, an `out_` output digest, the root dataset versions with the renderings they were read with, and each period stated relative to the latest with the year it resolved to. Format 1 is unchanged and holds dataset members only, and format 2 holds no dataset member. A derived member's `parts` is empty. A scenario's fetch-by-ref step admits a format 1 ref only, and a format 2 ref fails there on its `format_version`. Nine new ref examples, two accepted and seven rejected, each derived identity computed by the dfo-apis analysis runner. The validator checks a derived member's roots against the scenario it names, each root's rendering as the posted one in the frame layout, and that the derived members of the ref examples and the `output-comparison` vectors agree wherever one identity fixes another. +- A derived member's root records its rendering in `root_rendering`, the artifact selector's members and `groupby`, the sum DFO read the root with as `options.groupby` states it: `by`, one or more distinct column names, `op` `sum`, and `levels`, a depth from 0 to 99 for each kept hierarchical axis, absent when none is set. A breakdown or a sum over StatVars reads a root that way. The artifact's `rendering` admits no `groupby`. `ref-derived-grouped-root` is accepted, its `by` sorted as DFO records it and its identities computed by the dfo-apis analysis runner over the scenario's rows summed to their purpose's sector code, and `ref-derived-root-groupby-op-rejected` and `ref-artifact-groupby-rejected` are rejected. The validator checks a summed root's rendering against its scenario's build without its `groupby`, which no recorded build posts. +- The analysis and publication rules DFO applies, each with vectors and a reference implementation in `tests/test_analysis_rules.py`: `output-digest`, the `out2` digest of a derived frame's values with the `stat_var`, members, period and unit each belongs to, without `measurementMethod`, `provenance`, `status` or a unit's `recipe`, reading the variable `compare`, `sum_of`, `share_of` or `quotient_of` writes under an `mv_` name hashed from its verb and its inputs' variables, 35 characters at any depth, so an engine release or upstream rebuild that moves no number keeps the digest, with 22 canonicalisation vectors, each minted name given with its name document and one naming a `quotient_of` over a `sum_of` by the inner step's name; `output-comparison`, how `==` and `same_data` compare two refs, with 12 vectors; `operation-outputs`, `share_of` in per cent under `Percent`, `join` in long format by `stat_var`, `compare` as data minus other, `sum_of` as data plus other, and `growth` as a fraction in `value` with no unit, leaving out a member with no value at an end of its window, a relative change over a zero or sign-crossing base and a change that is not finite, and refusing two rows of one member in one period with `role_ambiguous`, with 10 vectors; `rolling-composition`, one latest period per recipe tree, refusing rolling steps over different populations with `rolling_populations_differ` even when they settle on one year, with 6 vectors; and `publication`, a share's numbered slots that keep their URLs across a live replace, a snapshot that refuses `share_not_replaceable`, and the write token checks, with 8 vectors. +- `actions.json` rows for `allowance_exhausted` (`wait_for_reset`), `rolling_populations_differ`, `several_members`, `share_not_replaceable` and `write_token_required` (`fix_request`), and `write_token_invalid` (`stop`). +- The `stop` action, class `PermissionDeniedError`, for a refusal no permission or credential the caller can obtain clears: `write_token_invalid`, since a share takes only the write token issued when it was created and a share created without one takes none. +- The retry rules: a refusal whose code's action is `wait_for_reset` is never retried and raises `QuotaExceededError` at once, whatever its document and status say, and a refusal whose `Retry-After` is longer than 60 seconds (`RETRY_AFTER_MAX_S`) raises at once with that wait, a refused poll included. `search-retry-after-past-cap` pins the cap. +- `series_left_out` in `warning-codes.json`, class `SeriesLeftOutWarning`, and the `series-left-out` rule with 8 vectors: a fetch of derived members issues it once when any member's derived manifest lists left-out series in `left_out` or says `left_out_complete` false, a null `left_out` issues nothing, and a strict fetch raises it in place of its data. `tools/validate_client_contract.py` takes the code from the rule. +- `actions.json` rows (`fix_request`) for the 34 refusal codes the analysis engine serves on `datasets.analysis`, including `compare_no_key_matched` and `sum_of_no_key_matched`. [unreleased]: https://github.com/DevelopmentFinanceObservatory/spec-kit/commits/main/python diff --git a/python/client-contract/README.md b/python/client-contract/README.md index b0e93d9..0a0aa87 100644 --- a/python/client-contract/README.md +++ b/python/client-contract/README.md @@ -48,9 +48,9 @@ uv run ty check src/dfo_client_contract | `warning-codes.schema.json` | The shape of `warning-codes.json` | | `scenario.schema.json` | One scenario record | | `scenarios/` | The scenario records, and the binary response bodies they name | -| `dataset-ref.schema.json` | The frozen `DatasetRef` format | +| `dataset-ref.schema.json` | The frozen `DatasetRef` formats, 1 and 2 | | `refs/` | Accepted and rejected `DatasetRef` examples | -| `rules/` | The client rules, one vector file each, with `rules.schema.json` and the rules' text | +| `rules/` | The client, analysis and publication rules, a vector file each, with schema and text | | Function | Returns | | ----------------- | ----------------------------------------------------------- | @@ -73,7 +73,7 @@ uv run ty check src/dfo_client_contract ## Strict fields -`strict-fields.json` lists the members a client cannot act without. Each pointer names one property of one component schema, `/components/schemas//properties/`. A `required` entry stays in its schema's `required` list and admits no null, so a client raises `ContractError` when one is missing or null. Every entry is a member some client rule acts on, and a client checks a body only along them: from the body's root it goes down through strict members alone, so a member no client reads goes unchecked, however deep. A `Spec` where no rule reads it, in `resolved.equivalent_readings`, `no_data.flagged_match` or `no_data.alternate_match`, may lack `resolved` or its definition's `identity` and raises nothing, which `answer-nested-specs-unchecked` pins at all three. A strict member that holds a JSON type its schema does not admit raises `ContractError` `contract_violation` at its pointer, as a missing one does, which `answer-candidate-specs-not-a-list` pins with `candidates.specs` as an object. An integer is a JSON number with no fractional part that is not a boolean, as JSON Schema's `integer` reads it, since a JavaScript client cannot tell `1.0` from `1`: a strict member such as a manifest definition's `schema_version`, and a ref's `definition.schema_version`, admit `1.0` and refuse `1.5` and `true`. Each item of a strict list, `parts`, `candidates.specs` or `artifacts`, must be an object, and an item of another type raises `contract_violation` at its index pointer, such as `/parts/0/candidates/specs/0`. A member a client reads that no strict pointer names, and that holds a JSON type other than the list, object or string its rule reads, reads as absent, and so does every member of a holder that is not an object; an item of another type in such a list reads as absent too. So `no_data.options` as an object or null offers no option, `mentions` as a string mentions nothing, and a `diagnostics` that is not an object holds no `timed_out` and no warnings. `answer-operations-not-a-list` pins a definite part's `operations` as an object. An item of an `assumptions` or `reading_assumptions` list reads only when it is an object whose `kind`, `binding_ref` and `text` are strings, and an item of an `operations` list only when it is an object whose `verb` and `phrase` are strings; any other item reads as absent. A read item holds its schema's members alone: an assumption's `refs` as its string items when it is a list, and an operation's `over` when it is a string or null and its `target_year` when it is null or a number other than a boolean with no fractional part, read as that integer, so `2020.0` reads and echoes as `2020` and `2020.5` reads as absent. An optional member of another type reads as absent, and a client reads no member the schema does not name. An `enums` entry keeps every listed value, the values a client branches on: the answer kinds, the part statuses and the operation statuses. A client still accepts a value no entry lists. +`strict-fields.json` lists the members a client cannot act without. Each pointer names one property of one component schema, `/components/schemas//properties/`. A `required` entry stays in its schema's `required` list and admits no null, so a client raises `ContractError` when one is missing or null. Every entry is a member some client rule acts on, and a client checks a body only along them: from the body's root it goes down through strict members alone, so a member no client reads goes unchecked, however deep. A `Spec` where no rule reads it, in `resolved.equivalent_readings`, `no_data.flagged_match` or `no_data.alternate_match`, may lack `resolved` or its definition's `identity` and raises nothing, which `answer-nested-specs-unchecked` pins at all three. A strict member that holds a JSON type its schema does not admit raises `ContractError` `contract_violation` at its pointer, as a missing one does, which `answer-candidate-specs-not-a-list` pins with `candidates.specs` as an object. An integer is a JSON number with no fractional part that is not a boolean, as JSON Schema's `integer` reads it, since a JavaScript client cannot tell `1.0` from `1`: a strict member such as a manifest definition's `schema_version`, and a ref's `definition.schema_version`, admit `1.0` and refuse `1.5` and `true`. Each item of a strict list, `parts`, `candidates.specs` or `artifacts`, must be an object, and an item of another type raises `contract_violation` at its index pointer, such as `/parts/0/candidates/specs/0`. A member a client reads that no strict pointer names, and that holds a JSON type other than the list, object or string its rule reads, reads as absent, and so does every member of a holder that is not an object; an item of another type in such a list reads as absent too. So `no_data.options` as an object or null offers no option, `mentions` as a string mentions nothing, and a `diagnostics` that is not an object holds no warnings. `answer-operations-not-a-list` pins a definite part's `operations` as an object. An item of an `assumptions` or `reading_assumptions` list reads only when it is an object whose `kind`, `binding_ref` and `text` are strings, and an item of an `operations` list only when it is an object whose `verb` and `phrase` are strings; any other item reads as absent. A read item holds its schema's members alone: an assumption's `refs` as its string items when it is a list, and an operation's `over` when it is a string or null and its `target_year` when it is null or a number other than a boolean with no fractional part, read as that integer, so `2020.0` reads and echoes as `2020` and `2020.5` reads as absent. An optional member of another type reads as absent, and a client reads no member the schema does not name. An `enums` entry keeps every listed value, the values a client branches on: the answer kinds, the part statuses and the operation statuses. A client still accepts a value no entry lists. The answer's `kind` and each part's `status` are the discriminators of their unions in the served document. A search response with no `kind` raises `ContractError`, as a part with no `status` does, although the served document gives `kind` a default and leaves it out of `required`. A part `status` a client does not know raises nothing and makes the part an `unknown` gap. An answer `kind` a client does not know raises `ContractError`, because the client cannot read the answer. A Search response body that is not an object, a `resolution` answer whose `parts` is empty, at pointer `/parts`, since the served schema gives it at least one item, a part whose `status` is not a string, at pointer `/parts//status`, and a definite or candidate spec whose `definition.spec_id` differs from its `spec_id` raise `ContractError` `contract_violation`, in a Search response as in a ref; `search-body-not-an-object`, `answer-empty-parts`, `answer-part-status-not-a-string` and `answer-definition-spec-id-mismatch` pin the four. @@ -92,12 +92,13 @@ The required members are each part's `status` and its payload (`spec`, `candidat | `take_newest_version` | `VersionUnavailableError` | Takes the newest version | | `log_in` | `AuthenticationError` | Logs in | | `request_scope` | `PermissionDeniedError` | Asks for the permission the call needs | +| `stop` | `PermissionDeniedError` | Stops, as nothing it can obtain allows the call | | `wait` | `RateLimitError` | Waits, then retries | | `wait_for_reset` | `QuotaExceededError` | Waits for the quota to reset | | `report` | `ServerError` | Reports the trace id, after the client's retries | | `upgrade` | `ContractError` | Upgrades the client | -A code that maps to `no_data_reason` also carries `gap_kind`, the kind of gap a no_data part with that reason becomes: `choice`, `no_data`, `not_understood`, `unsupported` or `timed_out`. `timed_out` is a part Search left unresolved at its time limit. Sending the question again, or a narrower one, may answer it. `choose` goes with `choice` and only with it. `choose` and `ask_data_question` name a next step inside an answer, so a code that also reaches a problem document takes another action. The website proxy answers its own refusals with problem documents of the same shape, with `type`, `title`, `status`, `detail`, `code`, `retryable` and `trace_id`, the trace id repeated in the `x-dfo-trace-id` header. `proxy-codes.json` lists their codes with their status, `rate_limited` at 429, which also sends `Retry-After`, and `transient_unavailable` at 502, both retryable. The validator refuses a problem document whose code is neither an active registry code nor a proxy code at that status. No proxy code has a row here, so a client raises a proxy refusal with the class its status maps to: `RateLimitError` for `rate_limited`, which `search-proxy-rate-limited` pins, and `ServerError` for `transient_unavailable`, which `search-proxy-unavailable` pins. `capability_unavailable` is a well-formed request for a format, reshape or conversion class the deployment does not enable, answered 501. Its action is `fix_request`, since the caller asks for another one, and `fetch-spec-capability-unavailable` pins it. +A code that maps to `no_data_reason` also carries `gap_kind`, the kind of gap a no_data part with that reason becomes: `choice`, `no_data`, `not_understood`, `unsupported` or `timed_out`. `timed_out` is a part Search left unresolved at its time limit. Sending the question again, or a narrower one, may answer it. `choose` goes with `choice` and only with it. `choose` and `ask_data_question` name a next step inside an answer, so a code that also reaches a problem document takes another action. The website proxy answers its own refusals with problem documents of the same shape, with `type`, `title`, `status`, `detail`, `code`, `retryable` and `trace_id`, the trace id repeated in the `x-dfo-trace-id` header. `proxy-codes.json` lists their codes with their status, `rate_limited` at 429, which also sends `Retry-After`, and `transient_unavailable` at 502, both retryable. The validator refuses a problem document whose code is neither an active registry code nor a proxy code at that status. No proxy code has a row here, so a client raises a proxy refusal with the class its status maps to: `RateLimitError` for `rate_limited`, which `search-proxy-rate-limited` pins, and `ServerError` for `transient_unavailable`, which `search-proxy-unavailable` pins. `capability_unavailable` is a well-formed request for a format, reshape or conversion class the deployment does not enable, answered 501. Its action is `fix_request`, since the caller asks for another one, and `fetch-spec-capability-unavailable` pins it. `write_token_invalid` takes `stop`: a share takes only the write token issued when it was created, and a share created without one takes none, so no scope or new token lets the caller replace or delete it. `allowance_exhausted` takes `wait_for_reset`, and a client raises it at once as `QuotaExceededError` with its `Retry-After`, the time until the allowance resets, as the retry rules below state. A client maps a refusal whose code has no row, or that carries no code, on its HTTP status: @@ -114,11 +115,11 @@ A client maps a refusal whose code has no row, or that carries no code, on its H A 408 is a timeout the caller retries, and a 410 names a resource that is gone, so the caller stops as it does on a 404. A client reads a refusal's body as a problem document when its content type is `application/problem+json` or `application/json`; a body of any other type reads as no body, so the refusal carries no code. A no_data reason with no row becomes a `no_data` gap. -An answer's gaps follow from its parts. A candidates part is a `choice` gap, its options the candidates. A no_data part is a gap of its reason's `gap_kind`, its options the no_data `options`, and of kind `timed_out` instead when the answer's `diagnostics.timed_out` is true. An unsupported part is an `unsupported` gap. A part whose status a client does not know is an `unknown` gap. +An answer's gaps follow from its parts. A candidates part is a `choice` gap, its options the candidates. A no_data part is a gap of its reason's `gap_kind`, its options the no_data `options`, whatever `diagnostics.timed_out` says. Search marks a part it left unresolved at its time limit with the reason `timed_out`. An unsupported part is an `unsupported` gap. A part whose status a client does not know is an `unknown` gap. ## Client codes -`client-codes.json` lists, sorted by code, each code a client sets on an error it raises when no problem document carries one: the `code`, the error `class` it raises, `retryable`, and a one-line `condition`. `retryable` is the flag both clients generate onto the error: `deadline_exceeded`, `download_failed` and `network_error` are retryable, `digest_mismatch` and the rest are not, and `refusal_without_code` is retryable as its status is, which its row gives as `by_status`. Both clients set an error's `code` from this list, so every error a client raises carries a code. A refusal keeps its problem document's `code`, the website proxy's included. A refusal whose body carries no code raises `refusal_without_code` with the class its HTTP status maps to, which its row gives as `by_status`. No other row does. A problem document without a code still gives the error its `retryable` and `trace_id`, so a 503 whose body says `retryable: false` raises at once, which `search-refused-without-code` pins. Every refusal's trace id is the response's trace id, as the trace-id rule under Scenarios gives it. `search-refused-without-code` pins the body's winning over the header, and `search-unauthorized` the header's for a refusal with no body. No client code is a code in `actions.json` or `registries/codes.json`, and each class is one the scenario schema names. Each reason the key-lookup rule raises, `key_empty`, `profile_not_found`, `profile_without_key` and `credentials_file_invalid`, is a row of class `AuthenticationError`, as are `key_invalid`, a key outside visible ASCII, and `credentials_file_unreadable`, a credentials file that exists but cannot be read. `header_invalid`, a header value with a control character, `insecure_transport`, a key over plain http to a non-loopback host, and `base_url_invalid`, a base URL that is empty, not absolute http or https, or carries userinfo, are `InvalidRequestError` rows. `response_too_large`, a decoded body past its cap, is a `ServerError` row. None of these six is retryable. The list holds no warning. `warning-codes.json` lists, sorted by code, each code a client sets on a warning it issues itself, with its class and condition: `ref_digest_changed`, a `DfoWarning`, for a ref member whose `content_version` serves a file with another digest than the ref's, because the server rendered the same data again, where the client returns the data; `route_deprecated`, a `DfoDeprecationWarning`, for a `Deprecation` or `Sunset` header; `server_warning`, a `DfoWarning`, for an `x-dfo-warning` header; and `key_checksum_mismatch`, `key_not_base62` and `key_unknown_prefix`, each a `DfoWarning`, for a key that fails the key-format rule with that reason. Both clients generate their warning codes from it and issue no other. No warning code is a code in another table or in `registries/codes.json`. A `SearchWarning` carries the code of the one way the readings differ from the question, a departure's or a warning's code or the verb of an operation Search did not run, and a null code when they differ in several ways. A `DataWarning` carries the manifest warning's code. +`client-codes.json` lists, sorted by code, each code a client sets on an error it raises when no problem document carries one: the `code`, the error `class` it raises, `retryable`, and a one-line `condition`. `retryable` is the flag both clients generate onto the error: `deadline_exceeded`, `download_failed` and `network_error` are retryable, `digest_mismatch` and the rest are not, and `refusal_without_code` is retryable as its status is, which its row gives as `by_status`. Both clients set an error's `code` from this list, so every error a client raises carries a code. A refusal keeps its problem document's `code`, the website proxy's included. A refusal whose body carries no code raises `refusal_without_code` with the class its HTTP status maps to, which its row gives as `by_status`. No other row does. A problem document without a code still gives the error its `retryable` and `trace_id`, so a 503 whose body says `retryable: false` raises at once, which `search-refused-without-code` pins. Every refusal's trace id is the response's trace id, as the trace-id rule under Scenarios gives it. `search-refused-without-code` pins the body's winning over the header, and `search-unauthorized` the header's for a refusal with no body. No client code is a code in `actions.json` or `registries/codes.json`, and each class is one the scenario schema names. Each reason the key-lookup rule raises, `key_empty`, `profile_not_found`, `profile_without_key` and `credentials_file_invalid`, is a row of class `AuthenticationError`, as are `key_invalid`, a key outside visible ASCII, and `credentials_file_unreadable`, a credentials file that exists but cannot be read. `header_invalid`, a header value with a control character or a caller's header named `X-DFO-Client`, `insecure_transport`, a key over plain http to a non-loopback host, and `base_url_invalid`, a base URL that is empty, not absolute http or https, or carries userinfo, are `InvalidRequestError` rows. `response_too_large`, a decoded body past its cap, is a `ServerError` row. None of these six is retryable. The list holds no warning. `warning-codes.json` lists, sorted by code, each code a client sets on a warning it issues itself, with its class and condition: `ref_digest_changed`, a `DfoWarning`, for a ref member whose `content_version` serves a file with another digest than the ref's, because the server rendered the same data again, where the client returns the data; `route_deprecated`, a `DfoDeprecationWarning`, for a `Deprecation` or `Sunset` header; `server_warning`, a `DfoWarning`, for an `x-dfo-warning` header; `series_left_out`, a `SeriesLeftOutWarning`, for a derived frame that left series out for missing data, where the client returns the frame; and `key_checksum_mismatch`, `key_not_base62` and `key_unknown_prefix`, each a `DfoWarning`, for a key that fails the key-format rule with that reason. Both clients generate their warning codes from it and issue no other. No warning code is a code in another table or in `registries/codes.json`. A `SearchWarning` carries the code of the one way the readings differ from the question, a departure's or a warning's code or the verb of an operation Search did not run, and a null code when they differ in several ways. A `DataWarning` carries the manifest warning's code. ## Scenarios @@ -134,19 +135,20 @@ A scenario is one client-level call, the HTTP exchanges it makes, and the outcom | `exchanges` | The requests the call makes, in order, each with the response to give | | `expect` | One outcome per step: an `answer`, a `dataset`, an `operation`, a `newest_version` or an `error` | -A step that acts on an earlier result follows one: `choose` follows `search` or `choose`, a `fetch` with no target follows `search`, `choose` or another `fetch` with no target and acts on the last answer, and `result` follows `mint` or `operation`. A `fetch` of an answer or a spec, `mint` and `operation` take a `layout`, `long` by default or `frame`, and a `fetch` of an answer or a spec takes `strict`. A `spec` in a step holds `spec_id`, the server's unmodified `definition`, the `sentence` when the client knows it, and the server's `resolved.entity_groups` as `entity_groups` when it names any. +A step that acts on an earlier result follows one: `choose` follows `search` or `choose`, a `fetch` with no target follows `search`, `choose` or another `fetch` with no target and acts on the last answer, and `result` follows `mint` or `operation`. A `fetch` of an answer or a spec, `mint` and `operation` take a `layout`, `long` by default or `frame`, a `fetch` of an answer or a spec takes `strict`, and that `fetch` and `mint` take `party_totals`. A `spec` in a step holds `spec_id`, the server's unmodified `definition`, the `sentence` when the client knows it, and the server's `resolved.entity_groups` as `entity_groups` when it names any. An exchange's request names its target in one of three ways. `path` is a path the client composes under the base URL, such as `/search/resolve`. `origin_path` is a root-relative URL the server returned, such as an artifact's `url`, or the path and query of an absolute one on the API's origin, which resolves against the base URL's origin, whatever path that URL has. `url` is an absolute URL outside the API, such as the signed storage URL a files route redirects to, which the client fetches with a bare request carrying no client header and no key. `json` holds the request body members that must match; a recorded scenario lists every member the client sent. The response gives `status`, the `headers` a client reads, and a `json` body or a `body_file`, a binary body in `scenarios/`. A scenario's own body files are named `.` and `-.`; a synthetic scenario may instead name a body file its parent names. -A `newest_version` outcome gives the `content_version` the step found, or null. An answer outcome gives `kind`, `status`, `sentence`, `complete`, the distinct `specs` of its definite parts, and its `gaps`, each with the `part` index, the gap `kind` and the number of `options`. A dataset outcome gives `rows`, `columns`, `complete`, its `members`, each with `parts`, `spec_id`, `content_version` and `sentence`, and the `missing` gaps. For one member the columns are the artifact's columns; a dataset of several members adds a `part` column and gives `part_rows`, one entry per member in member order with the `part` value of that member's rows and their number. A fetch of an answer also gives `search_warning`, whether the fetch issued a `SearchWarning`. An error outcome gives the error `class` and its `code`: a refusal's code, a warning's code, which for a `SearchWarning` may be null, or for any other error the client raises itself its code in `client-codes.json`. Every error outcome gives its `trace_id`: a refusal's, as the refusal rules below give it; a failed operation's, its error document's `trace_id`, else the `x-dfo-trace-id` header of the response that carried the failed resource; a `VersionUnavailableError`'s, its first refusal's, or null when nothing refused. An error the client raises itself follows one rule. An error that describes a response carries that response's trace id, and an error that describes the caller's input carries null. The trace id is the one a user quotes to find the fault in the server's logs. The errors that describe a response are `unexpected_response`, `contract_violation` on a body, `answer_declined`, `answer_incomplete`, and a strict fetch's `SearchWarning` or `DataWarning`. A response's trace id is the top-level `trace_id` member of its body when the body is a JSON object holding one as a non-empty string, a problem document or any other body, else its `x-dfo-trace-id` header, else null. A body `trace_id` that is empty or not a string reads as absent, so the header applies. No scenario holds one, and the validator's tests pin it. A Search answer's `diagnostics.trace_id` is not read, and `answer-timed-out` pins the header's winning over a `diagnostics.trace_id` that differs from it. `answer_declined`, `answer_incomplete`, a `SearchWarning`, and a `contract_violation` a choice raises on an option describe the held answer, so they carry the trace id of the Search response that answered it. The errors that describe the caller's input, `choice_invalid`, `invalid_argument` and the key-lookup reasons, carry null, and so do `digest_mismatch` and `download_failed`, which describe a storage-host fetch, and `deadline_exceeded`, `network_error` and `offline`, which describe no response. `search-no-json-body`, `fetch-spec-strict-warning`, `answer-timed-out`, `answer-empty-parts`, `fetch-spec-files-route-no-content` and `choose-option-two-part-query` pin a trace id on an error that describes a response, `operation-failed-without-error` the header of the response that carried a failed resource with no error, and `choose-refused-shared-label` a null one on a choice whose answer's response carries a trace id. A `VersionUnavailableError` also gives `newest_version`, the version the lookup found, or null. A dataset or error outcome of a step that made a long build gives `long_builds`, one entry per build the server ran on, in order: the `operation_id` the client polled last, the `poll_waits` in seconds before each poll, and the number of `resubmissions`. Any outcome of a step that retried a refused request gives `retry_waits`, the seconds before each retry in order, and any outcome of a step whose API responses issued warnings gives `warnings`, their codes from `warning-codes.json` in order. +A `newest_version` outcome gives the `content_version` the step found, or null. An answer outcome gives `kind`, `status`, `sentence`, `complete`, the distinct `specs` of its definite parts, and its `gaps`, each with the `part` index, the gap `kind` and the number of `options`. A dataset outcome gives `rows`, `columns`, `complete`, its `members`, each with `parts`, `spec_id`, `content_version` and `sentence`, and the `missing` gaps. For one member the columns are the artifact's columns in the order the reading-order rule gives; a dataset of several members adds a `part` column and gives `part_rows`, one entry per member in member order with the `part` value of that member's rows and their number. A fetch of an answer also gives `search_warning`, whether the fetch issued a `SearchWarning`. An error outcome gives the error `class` and its `code`: a refusal's code, a warning's code, which for a `SearchWarning` may be null, or for any other error the client raises itself its code in `client-codes.json`. Every error outcome gives its `trace_id`: a refusal's, as the refusal rules below give it; a failed operation's, its error document's `trace_id`, else the `x-dfo-trace-id` header of the response that carried the failed resource; a `VersionUnavailableError`'s, its first refusal's, or null when nothing refused. An error the client raises itself follows one rule. An error that describes a response carries that response's trace id, and an error that describes the caller's input carries null. The trace id is the one a user quotes to find the fault in the server's logs. The errors that describe a response are `unexpected_response`, `contract_violation` on a body, `answer_declined`, `answer_incomplete`, and a strict fetch's `SearchWarning` or `DataWarning`. A response's trace id is the top-level `trace_id` member of its body when the body is a JSON object holding one as a non-empty string, a problem document or any other body, else its `x-dfo-trace-id` header, else null. A body `trace_id` that is empty or not a string reads as absent, so the header applies. No scenario holds one, and the validator's tests pin it. A Search answer's `diagnostics.trace_id` is not read, and `answer-timed-out` pins the header's winning over a `diagnostics.trace_id` that differs from it. `answer_declined`, `answer_incomplete`, a `SearchWarning`, and a `contract_violation` a choice raises on an option describe the held answer, so they carry the trace id of the Search response that answered it. The errors that describe the caller's input, `choice_invalid`, `invalid_argument` and the key-lookup reasons, carry null, and so do `digest_mismatch` and `download_failed`, which describe a storage-host fetch, and `deadline_exceeded`, `network_error` and `offline`, which describe no response. `search-no-json-body`, `fetch-spec-strict-warning`, `answer-timed-out`, `answer-empty-parts`, `fetch-spec-files-route-no-content` and `choose-option-two-part-query` pin a trace id on an error that describes a response, `operation-failed-without-error` the header of the response that carried a failed resource with no error, and `choose-refused-shared-label` a null one on a choice whose answer's response carries a trace id. A `VersionUnavailableError` also gives `newest_version`, the version the lookup found, or null. A dataset or error outcome of a step that made a long build gives `long_builds`, one entry per build the server ran on, in order: the `operation_id` the client polled last, the `poll_waits` in seconds before each poll, and the number of `resubmissions`. Any outcome of a step that retried a refused request gives `retry_waits`, the seconds before each retry in order, and any outcome of a step whose API responses issued warnings gives `warnings`, their codes from `warning-codes.json` in order. -Each outcome follows from the exchanges and `actions.json`, and `tools/validate_client_contract.py` fails a record whose `expect` says otherwise. A record whose outcomes it cannot derive, because its exchanges do not fit its steps or a malformed member raises while it reads them, is reported as that record's problem, and the other records are checked. It runs the steps over the exchanges by the rules below: a refusal followed by the same request is a retry where the retry rules allow one, except on a poll, where the long-build rules count retries, and a record that ends a run of retries while the rules allow another does not fit, since the client makes it, a refusal's class is its code's action or else its status, the class of an error the client raises itself is its code's row in `client-codes.json`, a gap's kind is its reason's `gap_kind`, option and candidate counts are those the bodies hold, a dataset's members, `complete`, `missing` and `search_warning` follow from the answer and the manifests, each POST carries its body, each build posts the body the rules give for the spec it builds, so the builds' order and spec_ids are checked, each Search request posts the whole body `{"schema_version": 1, "input": ...}` with the input the rules give: the question, a choice's input, or a member's identity, each poll names the operation the client holds, each files route is the one the artifact's `url` names, each signed download is the URL the files route's `location` names, each retry waits the wait the retry rules give, and each response's warning headers issue the warnings the warning rules give. A dataset's `rows`, `columns` and `part_rows` row counts are the Parquet bodies' and are not derived, except that `part` comes first and `part_rows` is given exactly when the dataset has several members, each entry's `part` is its member's value and their rows sum to `rows`. +Each outcome follows from the exchanges and `actions.json`, and `tools/validate_client_contract.py` fails a record whose `expect` says otherwise. A record whose outcomes it cannot derive, because its exchanges do not fit its steps or a malformed member raises while it reads them, is reported as that record's problem, and the other records are checked. It runs the steps over the exchanges by the rules below: a refusal followed by the same request is a retry where the retry rules allow one, except on a poll, where the long-build rules count retries, and a record that ends a run of retries while the rules allow another does not fit, since the client makes it, a refusal's class is its code's action or else its status, the class of an error the client raises itself is its code's row in `client-codes.json`, a gap's kind is its reason's `gap_kind`, option and candidate counts are those the bodies hold, a dataset's members, `complete`, `missing` and `search_warning` follow from the answer and the manifests, each POST carries its body, each build posts the body the rules give for the spec it builds, so the builds' order and spec_ids are checked, each Search request posts the whole body `{"schema_version": 1, "input": ...}` with the input the rules give: the question, a choice's input, or a member's identity, each poll names the operation the client holds, each files route is the one the artifact's `url` names, each signed download is the URL the files route's `location` names, each retry waits the wait the retry rules give, and each response's warning headers issue the warnings the warning rules give. A dataset's `columns` are derived: the validator reads each downloaded Parquet file with pyarrow, its columns in the order the server wrote them and which of them vary between rows, and orders them as the reading-order rule gives. The outcome holds no rows, so the rule's vectors pin the row order. A dataset's `rows` and `part_rows` row counts are the Parquet bodies' and are not derived, except that `part_rows` is given exactly when the dataset has several members, each entry's `part` is its member's value and their rows sum to `rows`. A recorded scenario's request and response bodies validate against their route's schemas in `served/openapi.json`. A required member stays required when its schema admits null: the body carries it, as null or a value. A member the schema leaves optional may be absent or null, and a client reads both as null. A client replays a scenario by serving the exchanges in order: - A request matches the next exchange on its method and target, and on each listed `json` member, compared as parsed JSON values. +- Each `path` and `origin_path` request carries the `X-DFO-Client` header the client-identity rule gives, and each `url` request carries none. Records keep no request headers, so the replay checks the header on every request it serves. - A request with no exchange left, or one that matches none, fails the scenario, and so does an exchange the call never makes. - `path` resolves against the base URL the replay gives the client, and `origin_path` against that base URL's origin, so a replay can run under any base URL form. - A replay substitutes its own base URL's origin for the origin of the record's `origin.base_url`, a synthetic record's parent's, wherever a response names it, in its JSON body and its headers. `fetch-spec-same-origin-artifact-url` names the recorded origin in its manifest's artifact `url`s, and the validator replays every scenario under its recorded base URL and again under `https://replay.example.org` with the same path, so a record that holds only under its recorded host fails. @@ -159,15 +161,16 @@ The records also fix these client rules: - A fetch builds each distinct `spec_id` once, in part order, and finishes one member (build, the sentence when the client lacks it, files, download) before the next. - A fetch of a spec posts its narrowing as the build's `input.scope`, and posts no `scope` when it narrows nothing. A mint takes a layout and no narrowing, so it posts no `scope`. `window` posts `{"kind": "range", "start_year", "end_year"}` for a pair of years, or `{"kind": "all"}` or `{"kind": "latest"}`, and `facets` posts each axis as `{"ref": ...}`, and `facets: {}` names no axis and posts no `facets`. The server's `scope.entities` replaces the spec's whole entity binding, so `entities` posts a whole map: the roles the spec's `definition.identity.entities` binds, with each role the caller names replaced by its refs and each role the caller binds to null dropped. Naming `origin` alone on a spec bound to `origin` and `destination` keeps `destination`. A fetch that names no role, `entities: {}` included, posts no `entities`, so a fetch whose narrowing is all empty posts no `scope`; `fetch-spec-empty-narrowing` pins it. - A built `spec_id` that differs from the posted one, or a spec with no sentence, takes its sentence from Search: the client posts the built definition's `identity` as one `dataset_resubmit` identity entry, with the member's first part as `variable_text`, or `"dataset"` when no answer part names it, and checks the answer's `spec_id` equals the built one. The entry's `entity_groups` holds the spec's `resolved.entity_groups` for each role the built identity binds to the same entities as the spec's definition, so a window or facets narrowing keeps every group and a role the narrowing rebound or dropped loses its group. The entry leaves `entity_groups` out when it keeps no group, as for a resumed operation, whose spec the client does not hold. The server reads a role posted without its group as its members, so a narrowed spec of flows to the least developed countries would read "44 recipients" in place of "the 44 Least Developed Countries". -- A dataset of several members puts the `part` column first, then each member's columns in member order, each column once. A column that members hold in different types widens as pyarrow's permissive promotion widens it, and a pair that cannot widen raises `ContractError` `contract_violation`. A member file that already holds a `part` column raises `contract_violation` when members stack. Rows and columns are not derived, so no scenario pins these. The `part` value on a member's rows is the member's `parts` joined by `"; "`, so a member that answers two folded parts reads `"health ODA disbursements; education ODA disbursements"`, or the member's sentence when `parts` is empty. A definite part whose `variable_text` is absent or empty adds nothing to its member's `parts`, which `answer-folded-parts-unnamed` pins. -- A choice names its part by index, by variable text, or by a word that one gap's options mention, and its option, a candidate among them, by index or label. A boolean part or option is neither an index nor a label. A candidate's label is the `text` of its spec's `resolved.sentence_plan` tokens in the `heading` segment, a token whose `segment` is not a string counted in it, joined and stripped of surrounding white space, or else its `resolved.rendered_sentence`. A candidate with neither a heading nor a sentence has a null label; no scenario pins it, and the validator's tests do. A no_data option's label is matched among the options that mention the word the part was named by. With no part named, exactly one gap must await a choice: a `choice` gap, or any gap with options. +- A dataset of several members puts the `part` column first, then the members' columns, gathered in member order with each column once and ordered by the reading-order rule, which also sorts each member's rows. A column that members hold in different types widens as pyarrow's permissive promotion widens it, and a pair that cannot widen raises `ContractError` `contract_violation`. A member file that already holds a `part` column raises `contract_violation` when members stack. Rows and columns are not derived, so no scenario pins these. The `part` value on a member's rows is the member's `parts` joined by `"; "`, so a member that answers two folded parts reads `"health ODA disbursements; education ODA disbursements"`, or the member's sentence when `parts` is empty. A definite part whose `variable_text` is absent or empty adds nothing to its member's `parts`, which `answer-folded-parts-unnamed` pins. +- A choice names its part by index, by variable text, or by a word that one gap's options mention, and its option, a candidate among them, by index or label. A boolean part or option is neither an index nor a label. A candidate's label is the `text` of its spec's `resolved.sentence_plan` tokens in the `heading` segment, a token whose `segment` is not a string counted in it, joined and stripped of surrounding white space, or else its `resolved.rendered_sentence`. A candidate with neither a heading nor a sentence has a null label; no scenario pins it, and the validator's tests do. A no_data option's label is matched among the options that mention the word the part was named by. An `entity_ambiguous` option carries `entity_choice`, the index into `no_data.ambiguous_entities` of the group it settles, and a menu groups the options by it. When one place name is ambiguous at two positions, as in "aid from Congo to Congo", each group offers the same labels, so a label matches an option in each group and the choice names its option by index, which `choose-refused-label-in-two-groups` pins. With no part named, exactly one gap must await a choice: a `choice` gap, or any gap with options. - A choice raises `InvalidRequestError` `choice_invalid` and makes no request when no gap, or more than one, awaits it, when the part is named by a boolean, is unknown, names more than one part or already has a reading, when the gap offers no options, or when the option is a boolean, is out of range or matches no label or several. No scenario can hold a boolean part or option, and the validator's tests pin both. It raises the same when an entry it would post cannot be built: a part with no `variable_text`, or a part of a status the client does not know that has no `restatement`. A chosen option whose `data_query` holds other than one part raises `ContractError`, and so does a chosen option with no `data_query`, in a one-part answer as in a several-part one, before any request. `choose-refused-shared-label`, `choose-refused-unknown-candidate-label`, `choose-refused-shared-candidate-label`, `choose-refused-part-without-variable-text`, `choose-refused-unknown-status`, `choose-option-two-part-query` and `choose-option-without-data-query` pin these, and `choose-candidate-by-label` pins a candidate chosen by its label. - Every pick of a candidate asks Search for the picked answer, and no client builds it locally. In a one-part answer the pick posts `dataset_resubmit`, without `raw_query`, with one identity entry: the candidate's identity and `entity_groups`, and the part's `variable_text`, `separate` and `operations` as below. The picked answer, its sentence with any "Assumed: …" clause, and its assumptions are Search's response. A choice of a no_data option in a one-part answer posts the option's `data_query` as the input. - In a several-part answer a choice posts `dataset_resubmit`, without `raw_query`, with one entry per part in part order. The chosen candidate posts an identity entry with its identity and `entity_groups`, and the chosen option its `data_query`'s one part as a parsed entry. A definite part posts an identity entry with its spec's identity and `entity_groups`. A candidates part left alone posts a `candidates` entry: its candidate set in the server's order, each with its identity and `entity_groups`, and its `ordering` and `truncated` echoed. The server cannot recompute that order, because it rests on the first answer's bind election and availability reads, and a parsed entry would cost an LLM bind. Any other part posts its `restatement` as a parsed entry, or a placeholder entry with its no_data `reason` or unsupported `code` when it has none. Every entry but a parsed one carries the part's `variable_text`, `separate` when true and `operations` when any of the part's operations reads. `entity_groups` is left out when the spec has none. `choose-echoes-separate-and-operations` pins the echo on a definite part's entry, and `choose-posts-unsupported-placeholder` an unsupported part's placeholder. - Each identity a choice posts echoes its reading's assumptions as `assumptions`: the chosen candidate's entry carries its part's `reading_assumptions` under the candidate's `spec_id`, a definite part's entry carries the part's `assumptions`, and each candidate of a `candidates` entry carries its own `reading_assumptions` entry. Search then states the echoed items that still hold, so the picked answer states what that reading assumed. `[]` states none. An identity with no `assumptions` reads as a fresh request, so a client that holds no list for a reading, from a part that carries no `assumptions` or `reading_assumptions`, posts none. The identity entry that asks for a built sentence posts no `assumptions`, since it reads only the spec's own `rendered_sentence`. An echoed `operations` or `assumptions` list posts exactly the items a client reads under Strict fields, in order, each built from the members it read: an optional member that read as absent is left out, and a `refs` list posts its string items. An `assumptions` list with no item read posts `[]`, and an `operations` list with none is left out. The item rule checks JSON types alone, so the echo posts what a client read, and the server still checks it against its own value constraints: the `verb` enum, the lengths of `phrase` and `text`, and at most one item in `refs`. `choose-echo-drops-malformed-items` pins both lists. - A strict fetch raises the manifest's first warning above `info` severity as a `DataWarning` with that warning's code, before the sentence lookup and the download. - An identity resubmit whose answer is not one definite part of the built `spec_id`, with a sentence, raises `ContractError`. -- The `frame` layout posts `{"format": "parquet", "bundle": false, "layout": "frame"}` as the build's `options` and downloads the artifact whose rendering's `layout` is `frame`; the `long` layout posts `{"format": "parquet", "bundle": false}` and downloads the bare Parquet artifact with any other layout. A fetch of an answer with several members in the `frame` layout raises `InvalidRequestError` before any build. +- The `frame` layout posts `{"format": "parquet", "bundle": false, "layout": "frame"}` as the build's `options` and downloads the artifact whose rendering's `layout` is `frame`; the `long` layout posts `{"format": "parquet", "bundle": false}` and downloads the bare Parquet artifact with any other layout. A fetch of an answer or a spec, and a mint, add `"without_party_totals": true` to either, as the next rule gives. A client matches an artifact on its rendering's format, bundle and frame layout only, and `without_party_totals` plays no part in the match. A fetch of an answer with several members in the `frame` layout raises `InvalidRequestError` before any build. +- A fetch of an answer or a spec, and a mint, leave out party totals by default. The build posts `"without_party_totals": true` in `options`, and the server drops the rows whose party, on a flow party role the identity binds no entity on, the code index declares a group total, before any reshape. A role the caller binds keeps what it binds, so a total named by its dcid still comes back. The client option is `party_totals`, false by default, in Python, and `partyTotals` in TypeScript, and a fetch or mint step's `party_totals: true` stands for it: the build posts no `without_party_totals` and keeps every total. The wire is the served rendering flag as it was; only the client's default changes. A cut that leaves rows out answers a version of its own, listed by the versions route with the plain build as its `base_content_version`, and a cut that leaves nothing out answers the plain version and its file. The manifest's artifact rendering carries `without_party_totals: true` in both cases. A mint posts the options a fetch of its spec posts, so its `result` answers the table that fetch answers, which `mint-async-build` pins with `party_totals: true`. A resumed operation posts no build, and its `result` answers the rendering the operation's build posted: the layout's rendering, with `without_party_totals: true` when the done manifest's artifact for that layout carries it. `fetch-spec-party-totals-left-out` pins the default on a DAC2a spec whose origin the fetch leaves open, where the cut drops the donor group totals, and `fetch-spec-party-totals-kept` the option. - A download asks the route that redirects to the signed URL, an artifact's `url` or the file selector, and fetches the URL its `location` names once. A files route answers 302 with the `location` the client fetches, or 200 with the file as its body, which the client hashes with no further request, which `fetch-spec-files-route-serves-file` pins. A refusal reads by the refusal rules, and any other status, another 2xx or another 3xx included, raises `ContractError` `unexpected_response` with that response's trace id, which `fetch-spec-files-route-no-content` pins with a 204 and `fetch-spec-files-route-temporary-redirect` with a 307. The route ask is an API request and follows the retry rules below, so a retryable refusal is retried within the budget and any other refusal, `artifact_expired` among them, raises at once. A failed fetch gets exactly one more round. A fetch that answers anything but 200, which `fetch-spec-signed-fetch-partial` pins with a 206, or bytes that do not hash to the artifact's digest ask the route again, under the same retry rules, and fetch again, with no wait between rounds. A second failure raises `ServerError` with `download_failed`, or `digest_mismatch` when the bytes are wrong. A network error or timeout on the fetch is retryable, so it starts the second round too, and when the second round fails the same way the client raises that failure as itself, `network_error` or `deadline_exceeded`. The replay holds no network failure, so no scenario pins that case. - A manifest artifact's `url` names its files route on the API's origin. A root-relative `url` resolves against the base URL's origin, and an absolute one on that origin is asked as its path and query, so both make the same `origin_path` request, which `fetch-spec-same-origin-artifact-url` pins. An artifact `url` on any other origin, the storage host's included, raises `ContractError` `contract_violation` before the sentence lookup and any request for the file, which `fetch-spec-foreign-artifact-url` and `fetch-spec-storage-artifact-url` pin. - A relative `location` resolves against the files route's URL, and the client fetches it on the API's origin as an `origin_path` request. A `location` on an origin other than the API's or `https://storage.googleapis.com` raises `ContractError` `contract_violation` before any fetch. A fetch on the storage host is bare and carries no API key. `fetch-spec-relative-location` pins the relative case. @@ -175,16 +178,17 @@ The records also fix these client rules: - A declined or catalogue answer is not complete, and fetching it raises `DeclinedError`. - Fetching an answer with a gap raises `IncompleteAnswerError` before any request, unless the fetch is partial and some part has a reading. - A fetch of an answer issues one `SearchWarning` when a definite part has an operation Search did not run, a departure, or a warning at severity `warn` or `degraded` with a `message`, or the answer's `diagnostics.warnings` holds such a warning. An `info` warning, a warning with no `message`, and an operation that does not read under Strict fields issue nothing. Each way is one line a client shows: `Not done: ()` coded by the verb, the phrase quoted as a JSON string that keeps non-ASCII characters as they are, the departure's `text` coded by its code, or `Note: ` coded by the warning's code, in part order with the diagnostics warnings last, and ways that show the same line count once, under the first one's code. A strict fetch raises the `SearchWarning` before any build, with the code of its one line, or a null code when there are several lines. A strict fetch raises it before the `frame` layout's several-member check. -- A ref's version is found in this order: the snapshot route and the files route at the stored digest; the file selector, `GET /datasets/{id}/{content_version}/file` with the ref's rendering as the query; and a build of the ref's spec in the ref's rendering, which must answer the ref's `spec_id` and the same `content_version`. Both recovery steps keep the ref's identity, so a recovered member carries the ref's `spec_id` and `content_version`: a snapshot manifest whose `definition.spec_id` or `content_version` is not the ref's, or a rebuild manifest whose `definition.spec_id` is not the ref's, raises `ContractError` `contract_violation` before any download, which `fetch-ref-expired-rebuild-other-spec` pins for the rebuild. A snapshot refusal other than `artifact_expired` ends the lookup, so `snapshot_not_found` raises `NotFoundError` with no further request. A files route or selector refusal of `artifact_expired`, `artifact_not_found` or `not_found` takes the next step, and a selector file of another digest is not the stored file. A snapshot manifest whose artifacts list no artifact of the ref member's digest sends the client to the selector in the ref member's own rendering, and a file that hashes to that digest gives the dataset, which `fetch-ref-artifact-unlisted` pins. A rebuild that answers another version, or that raises `InvalidRequestError`, `NotFoundError` or `VersionUnavailableError`, ends in `VersionUnavailableError` with the first refusal's code, or `version_unavailable` when nothing refused. Its `newest_version` is the version the rebuild answered, or else the newest stored version, as a `newest_version` step finds it: the first version `GET /datasets/{id}/versions?limit=100` lists with a null `base_content_version`, the newest plain build, read page by page through `&cursor=` with each page's `next_cursor` until one is listed. The route lists versions newest first and needs no key. A version with a `base_content_version` is derived from a plain build, a groupby collapse or a party-totals cut, and is skipped, which `fetch-ref-newest-version` pins. The newest version is null when the list ends without a plain version or the route refuses with `NotFoundError`. Any other refusal of the versions route raises, and a page whose `versions` is not a list of versions each with a `content_version` and a `base_content_version` that is a string or null, or whose `next_cursor` is neither a string nor null, raises `ContractError` `contract_violation`, which `newest-version-unreadable-page` pins. -- A build answers 200 with the manifest, or 202 with an operation the client polls at `/datasets/operations/{operation_id}`, the id of the operation it holds. The 202 reads `pending` or `running`, because a submission starts a build or joins one under way, and a `done` operation left by an earlier submission of a rejected head is built again. A deterministic failure answers a 202 that reads `failed`. `fetch-spec-async-build` pins `pending` and `fetch-spec-accepted-running` pins `running`. A mint's build answers 202. A record whose mint answers 200 fits no mint step. Before each poll it waits the `Retry-After` of the refusal that answered the last poll, whatever its status, as every retry of a refused request waits; else the last operation resource's `retry_after_ms`, else the `Retry-After` of the last answered poll, the response that carried that resource, else a backoff of 1, 2, 4 and then 8 seconds that advances one step each time it is used, a poll after a refusal included. `fetch-spec-poll-pacing-changes` pins a `retry_after_ms` that changes from one poll to the next, so a client waits each poll's own value. +- A ref's version is found in this order: the snapshot route and the files route at the stored digest; the file selector, `GET /datasets/{id}/{content_version}/file` with the query the file-selector rule in `rules/README.md` builds from the ref's rendering, which leaves out `without_party_totals`; and a build of the ref's spec in the ref's rendering, which must answer the ref's `spec_id` and the same `content_version`. Both recovery steps keep the ref's identity, so a recovered member carries the ref's `spec_id` and `content_version`: a snapshot manifest whose `definition.spec_id` or `content_version` is not the ref's, or a rebuild manifest whose `definition.spec_id` is not the ref's, raises `ContractError` `contract_violation` before any download, which `fetch-ref-expired-rebuild-other-spec` pins for the rebuild. A snapshot refusal other than `artifact_expired` ends the lookup, so `snapshot_not_found` raises `NotFoundError` with no further request. A files route or selector refusal of `artifact_expired`, `artifact_not_found` or `not_found` takes the next step, and a selector file of another digest is not the stored file. A snapshot manifest whose artifacts list no artifact of the ref member's digest sends the client to the selector for the ref member's own rendering, and a file that hashes to that digest gives the dataset, which `fetch-ref-artifact-unlisted` pins. A rebuild that answers another version, or that raises `InvalidRequestError`, `NotFoundError` or `VersionUnavailableError`, ends in `VersionUnavailableError` with the first refusal's code, or `version_unavailable` when nothing refused. Its `newest_version` is the version the rebuild answered, or else the newest stored version, as a `newest_version` step finds it: the first version `GET /datasets/{id}/versions?limit=100` lists with a null `base_content_version`, the newest plain build, read page by page through `&cursor=` with each page's `next_cursor` until one is listed. The route lists versions newest first and needs no key. A version with a `base_content_version` is derived from a plain build, a groupby collapse or a party-totals cut, and is skipped, which `fetch-ref-newest-version` pins. The newest version is null when the list ends without a plain version or the route refuses with `NotFoundError`. It names the newest plain build, which a fetch with the default party totals rule answers only when the cut leaves nothing out; when the cut leaves rows out, the fetch answers a version whose `base_content_version` is that build. So a client does not present `newest_version` as comparable to the `content_version` of a cut ref or a cut fetch: two different versions there need not mean newer data. Any other refusal of the versions route raises, and a page whose `versions` is not a list of versions each with a `content_version` and a `base_content_version` that is a string or null, or whose `next_cursor` is neither a string nor null, raises `ContractError` `contract_violation`, which `newest-version-unreadable-page` pins. +- A build answers 200 with the manifest, or 202 with an operation the client polls at `/datasets/operations/{operation_id}`, the id of the operation it holds. The 202 reads `pending` or `running`, because a submission starts a build or joins one under way, and a `done` operation left by an earlier submission of a rejected head is built again. A deterministic failure answers a 202 that reads `failed`. `fetch-spec-async-build` pins `pending` and `fetch-spec-accepted-running` pins `running`. A POST for a build identical to one that is pending or running returns that same operation, because the server derives the operation id from the request: dfo-apis `src/datasets_api/operation_store.py` `compute_operation_id` hashes the request identity, the origin catalogue build id and the rendering. A client that stops waiting for a build can post the same build again to join it. A mint's build answers 202. A record whose mint answers 200 fits no mint step. Before each poll it waits the `Retry-After` of the refusal that answered the last poll, whatever its status, as every retry of a refused request waits; else the last operation resource's `retry_after_ms`, else the `Retry-After` of the last answered poll, the response that carried that resource, else a backoff of 1, 2, 4 and then 8 seconds that advances one step each time it is used, a poll after a refusal included. `fetch-spec-poll-pacing-changes` pins a `retry_after_ms` that changes from one poll to the next, so a client waits each poll's own value. - A retryable 429 poll spends no retry, and a 429 whose body says it is not retryable raises at once, as a refusal's `retryable` outranks its status, which `fetch-spec-poll-rate-limited-not-retryable` pins. Any other retryable refused poll spends one of the two retries, which a poll that answers restores, and a refusal that is not retryable raises. `fetch-spec-poll-refused` pins two 503s polled again after an answered poll, `fetch-spec-poll-refused-backoff` a 503 with no `Retry-After` among polls no response paces, after which the backoff advances, `fetch-spec-poll-refused-retry-after` a 503 whose `Retry-After` paces the next poll, and `fetch-spec-poll-not-found` a 404 that raises at once. A poll that answers a failed operation whose error is retryable posts the same build once more and polls the operation it starts; a second failure, an error that is not retryable, and a resumed operation, whose build request the client does not hold, raise the operation's error. A refused poll whose code is `operation_failed` is a failed operation with that error. A failed operation whose resource carries no `error` raises `ServerError` with `operation_failed`, retryable by status 500 as a refusal without a document is, which `operation-failed-without-error` pins. A 202 or resumed operation that already reads `failed` raises its error before any poll. -- Any other refusal is retried when it is retryable: as its problem document's `retryable` says, else when its status is 408, 429, 500, 502, 503 or 504. A client makes two retries by default, and the scenarios assume that default. A build's POST shares those two retries with every other refusal of it, and once a 504 cuts the build off the client posts it once more only, whatever that post answers: a build refused 504 then 503 raises the 503, a build refused 503 then 504 is posted once more after the 504, and a second 504 raises. `fetch-spec-504-then-503`, `fetch-spec-503-then-504` and `fetch-spec-504-twice` pin the three. A refusal that is not retryable, or one past the retries, raises its error. -- Each retry waits the refusal's `Retry-After` when it carries one, and otherwise the client's backoff for that attempt: 0.5 seconds before the first retry, doubling to at most 8. The build's 504 cut-off retry follows the same rule. `retry_waits` pins these waits, and `fetch-spec-504-retry-after` pins a cut-off that carries `Retry-After`. +- Any other refusal is retried when it is retryable: never when its code's action is `wait_for_reset`, whatever its document and status say, else as its problem document's `retryable` says, else when its status is 408, 429, 500, 502, 503 or 504. A client makes two retries by default, and the scenarios assume that default. A build's POST shares those two retries with every other refusal of it, and once a 504 cuts the build off the client posts it once more only, whatever that post answers: a build refused 504 then 503 raises the 503, a build refused 503 then 504 is posted once more after the 504, and a second 504 raises. `fetch-spec-504-then-503`, `fetch-spec-503-then-504` and `fetch-spec-504-twice` pin the three. A refusal that is not retryable, or one past the retries, raises its error. +- Each retry waits the refusal's `Retry-After` when it carries one, and otherwise the client's backoff for that attempt: 0.5 seconds before the first retry, doubling to at most 8. The build's 504 cut-off retry follows the same rule. A client waits a `Retry-After` of at most 60 seconds (`RETRY_AFTER_MAX_S`). A refusal whose `Retry-After` is longer raises at once with that wait on the error, a refused poll included, so no call sleeps for minutes unasked. `retry_waits` pins these waits, `fetch-spec-504-retry-after` pins a cut-off that carries `Retry-After`, and `search-retry-after-past-cap` a retryable 503 whose `Retry-After` of 61 raises with no retry. - A 401 resolves the key again by the key-lookup rule. When the key changed, the client sends the request once more at once, with no wait and spending no retry. When it did not, or the second answer is a 401 too, the 401 raises `AuthenticationError`. A replay's key never changes, so `search-unauthorized` pins the second case and no scenario the first. - A `request(method, path)` whose `path` is not a path on the base URL, because it starts with `//` or names a scheme, raises `InvalidRequestError` `invalid_argument` before any request. No scenario step makes such a request. - A 2xx response of a JSON route with no JSON body raises `ContractError` `unexpected_response`. `search-no-json-body` pins it for Search. - A `Deprecation` or `Sunset` header on an API response, either alone, issues `route_deprecated` once per process per distinct pair of their values, and an `x-dfo-warning` header issues `server_warning` once per process per message. A signed fetch's headers issue nothing. `search-sunset-warning` pins a Sunset header alone, issued once over two responses that carry the same value, and `search-server-warning` pins a repeated `x-dfo-warning` message issued once, a new message issued again, and a Deprecation beside the same Sunset issued as a new pair. -- Python synchronous platform name resolution can exceed the following timeout and deadline limits. Asynchronous name resolution is cancellable. Timeouts follow the clients' design and are not replayed, since the replay advances the clock only by the waits the client asks for: the client's `timeout`, 30 s by default, bounds one request attempt; `search(timeout=60)` bounds the whole search, each attempt reading until that deadline, and None means no limit; `fetch(timeout=1800)` bounds the whole fetch, each download getting the time left on the fetch's deadline; a synchronous build POST, and mint's POST, read for up to 150 s within any outer deadline; `result(timeout=1800)` bounds the wait, and None means no limit. A poll attempt that hangs past the per-attempt timeout spends a retry, and a deadline that passes during a body read raises `DfoTimeoutError` `deadline_exceeded`. In TypeScript an infinite remaining time arms no timer, and a finite one above 2^31-1 ms is clamped. No scenario step takes a timeout. +- A fetch of derived members issues `series_left_out`, a `SeriesLeftOutWarning`, once when the recipe anchor of any member lists left-out series (`Recipe.anchor.left_out` non-empty in the served document): series the rolling anchor dropped because they hold no row at the anchor period. The client returns the frame. A strict fetch raises the warning in place of its data. A recipe with no rolling period has a null anchor and issues nothing. No scenario serves a derived member yet, so the `series-left-out` rule's vectors in `rules/` pin it. +- Python synchronous platform name resolution can exceed the following timeout and deadline limits. Asynchronous name resolution is cancellable. Timeouts follow the clients' design and are not replayed, since the replay advances the clock only by the waits the client asks for: the client's `timeout`, 30 s by default, bounds one request attempt; `search(timeout=60)` bounds the whole search, each attempt reading until that deadline, and None means no limit; `fetch` waits for a build with no whole-call deadline by default, and `fetch(timeout=60)` bounds the whole fetch, each download getting the time left on the fetch's deadline; a download that delivers no byte for 30 s raises the retryable `network_error`; a synchronous build POST, and mint's POST, read for up to 150 s within any outer deadline; `result(timeout=1800)` bounds the wait, and None means no limit. A poll attempt that hangs past the per-attempt timeout spends a retry, and a deadline that passes during a body read raises `DfoTimeoutError` `deadline_exceeded`. In TypeScript an infinite remaining time arms no timer, and a finite one above 2^31-1 ms is clamped. No scenario step takes a timeout. ## Recording @@ -207,15 +211,15 @@ Each record names the commit in the responses' `x-dfo-commit` header, and the ru uv run tools/capture_client_contract.py record --base-url http://127.0.0.1:8083/api ``` -A record's `origin.base_url` is the base URL the run was given, a local app's included. A local app's signed URL, which its launcher serves at `/__e2e__/blob/{bucket}/{object}` on the API's origin, is written as `https://storage.googleapis.com/{bucket}/{object}` with its query redacted, as a deployment signs it, because a production download is cross-origin and the client sends no key there; the run, like the validator, refuses a record holding any absolute URL other than such a storage URL, or a `location` under the base URL, so a local app's host under another name or port is refused too. A server answers a version it has built inline, so `mint-async-build` and `operation-resume` each need a spec just above the 100,000-row sync budget that the server has not built: ODA disbursements from the 32 DAC countries to the 133 G77 countries from 1963 to 2024 for the mint, and from the 38 OECD countries to the G77 countries from 1966 to 2024 for the resumed operation, each one variable over some 3,700 donor and recipient pairs, so the done manifest stays small. A local recording therefore starts on a fresh datasets database with `DATASETS_SYNC_BUDGET_ROWS=100000`. `--scenarios-dir` writes records outside the package, unvalidated. +A record's `origin.base_url` is the base URL the run was given, a local app's included. A local app's signed URL, which its launcher serves at `/__e2e__/blob/{bucket}/{object}` on the API's origin, is written as `https://storage.googleapis.com/{bucket}/{object}` with its query redacted, as a deployment signs it, because a production download is cross-origin and the client sends no key there; the run, like the validator, refuses a record holding any absolute URL other than such a storage URL, or a `location` under the base URL, so a local app's host under another name or port is refused too. A server answers a version it has built inline, so `mint-async-build` and `operation-resume` each need a spec just above the 100,000-row sync budget that the server has not built: ODA disbursements from the 32 DAC countries to the 133 G77 countries to 2024 for the mint, and from the 38 OECD countries to the G77 countries to 2024 for the resumed operation, each one variable over some 3,700 donor and recipient pairs, so the done manifest stays small. The tool chooses each start year, the earliest whose spec is in that row band and has no version on the versions route. Each record builds both specs, so the next record takes later start years, and the run fails when no year in the tool's range is left. A local recording runs with `DATASETS_SYNC_BUDGET_ROWS=100000`. `--scenarios-dir` writes records outside the package, unvalidated. ## DatasetRef -`dataset-ref.schema.json` is the frozen format a client writes for `data.ref` and reads back in any language: `format_version: 1` and one or more `members`, each `{parts, sentence, spec: {spec_id, definition}, content_version, artifact: {digest, rendering}}`. `parts` holds the variable text of each answer part the member answers, in query order, and is empty for a spec fetched outside an answer. `definition` is the server's definition unmodified, so it holds the `schema_version`, `spec_id` and `identity` the served `CanonicalDefinition` requires and no other member, and a rebuild posts it as it is. The format takes any integer `schema_version` and leaves the version to the server, which checks it when a rebuild posts the definition. `rendering` is the selector a client matches an artifact on, `{format, bundle}` with `layout: "frame"` for the frame layout and `without_party_totals: true` for the version rendered with party totals left out, never the server's rendering object. The `without_party_totals` key is written only when true, so a plain rendering omits it and `false` is not a valid value. Beyond the schema, each member's `definition.spec_id` equals its `spec.spec_id`, and members hold distinct `spec_id`s. `ref-bare-definition-rejected`, `ref-definition-without-identity-rejected` and `ref-definition-without-schema-version-rejected` pin the definition's required members, and `ref-definition-extra-member-rejected` a member it does not admit. `ref-without-party-totals` is the accepted cut rendering and `ref-without-party-totals-false-rejected` pins `false`. `refs/` holds accepted examples and rejected ones, each `{format_version, name, description, scenario, ref}` with `expect: "accept"` or `error`, `{by: "schema"}` or `{by: "rule", reason}`, the reason `definition_spec_id_mismatch` or `repeated_spec_id`. `scenario` names the recorded scenario that serves the ref's members, and is left out of a ref with no members. Each member's `spec_id`, `content_version` and `sentence` are those of a dataset member of that scenario, and its artifact digest is the digest of a file the scenario downloads, so a re-recording that changes a member also fails the refs that still name the old one. +`dataset-ref.schema.json` is the frozen format a client writes for `data.ref` and reads back in any language. Format 1 is `format_version: 1` and one or more dataset `members`, each `{parts, sentence, spec: {spec_id, definition}, content_version, artifact: {digest, rendering}}`. `parts` holds the variable text of each answer part the member answers, in query order, and is empty for a spec fetched outside an answer. `definition` is the server's definition unmodified, so it holds the `schema_version`, `spec_id` and `identity` the served `CanonicalDefinition` requires and no other member, and a rebuild posts it as it is. The format takes any integer `schema_version` and leaves the version to the server, which checks it when a rebuild posts the definition. `rendering` is the selector a client matches an artifact on, `{format, bundle}` with `layout: "frame"` for the frame layout and `without_party_totals: true` for a fetch that left party totals out, never the server's rendering object. A ref records the rendering its fetch posted, so a member fetched with the default records `without_party_totals: true` whether or not the cut left out a row, and a member fetched with party totals records no such key. The `without_party_totals` key is written only when true, so a plain rendering omits it and `false` is not a valid value. Beyond the schema, each member's `definition.spec_id` equals its `spec.spec_id`, and members hold distinct `spec_id`s. `ref-bare-definition-rejected`, `ref-definition-without-identity-rejected` and `ref-definition-without-schema-version-rejected` pin the definition's required members, and `ref-definition-extra-member-rejected` a member it does not admit. `ref-without-party-totals` is the cut rendering a default fetch records, `ref-one-member` the plain rendering a fetch with party totals records, and `ref-without-party-totals-false-rejected` pins `false`. `refs/` holds accepted examples and rejected ones, each `{format_version, name, description, scenario, ref}` with `expect: "accept"` or `error`, `{by: "schema"}` or `{by: "rule", reason}`, the reason `definition_spec_id_mismatch` or `repeated_spec_id`. `scenario` names the recorded scenario that serves the ref's members, and is left out of a ref with no members. Each member's `spec_id`, `parts`, `content_version` and `sentence` are those of a dataset member of that scenario, and its artifact digest is the digest of a file the scenario downloads, so a re-recording that changes a member also fails the refs that still name the old one. Each member of an example the schema accepts records the rendering the scenario's build of its spec posted as `options`. Format 2 is `format_version: 2` and one or more derived `members`, each the frame an analysis recipe computed, and holds no dataset member, so a ref of dataset members has one spelling, format 1. A derived member is `{parts, sentence, spec: {spec_id, recipe: {normal_form_hash, replay_key}}, content_version, output_digest, roots, execution_identity, resolved_periods, artifact: {digest, rendering}}`. `spec_id` is the recipe's `ops_` id and `content_version` its `ocv_` version. `normal_form_hash` (`rnf_`) names the computation whatever data it ran on, and `replay_key` (`rpk_`) names it on these roots under this execution identity. `output_digest` (`out_`) is the digest of the frame's output that `rules/README.md` defines. `roots` lists each dataset version the recipe ran on, `{spec_id, content_version, rendering}` with the rendering the recipe read, a `spec_` id with a `cv_` version and never a derived frame, since a recipe's normal form replaces each derived input by its own recipe. A root's rendering holds the selector's members and, for a root read summed, such as by a breakdown or a sum over StatVars, `groupby`, the `options.groupby` DFO read it with: `{by, op: "sum"}`, `by` one or more distinct column names, with `levels` mapping each kept hierarchical axis it rolls up to a depth from 0 to 99. An artifact's `rendering` admits no `groupby`. `execution_identity` names the engine, frame grammar and reduction policy versions that ran it, joined by `|`, such as `0.9.0|1.1.0|0.2.0`. `resolved_periods` lists each period the recipe states relative to the latest period, `{anchor: "latest", offset, rule: "half1", period}`, with the year it resolved to, and is empty for a recipe that states none. DFO computes every one of these members, and a client copies them from the server's answer and computes none. `parts` is empty for a derived member, which no answer part names. A client that reads format 1 refuses format 2, so clients that read format 2 ship before any service writes it. `==` and `same_data` compare refs as `rules/README.md` states under output comparison. `ref-derived-member`, `ref-derived-member-fixed-window` and `ref-derived-grouped-root` are accepted derived refs, the first with two resolved periods, the second with none, and the third with a root read summed, and `ref-derived-in-format-1-rejected`, `ref-dataset-member-in-format-2-rejected`, `ref-derived-without-output-digest-rejected`, `ref-derived-root-without-rendering-rejected`, `ref-derived-root-groupby-op-rejected`, `ref-artifact-groupby-rejected` and `ref-derived-period-after-latest-rejected` pin what the schema refuses. `repeated_spec_id` holds across derived members too, which `ref-derived-repeated-spec-id-rejected` pins, and `definition_spec_id_mismatch` concerns dataset members alone. A derived example's `scenario` serves its roots: each root's `spec_id` and `content_version` are a dataset member's of that scenario, and its rendering the one the scenario's build posted, apart from a summed root's `groupby`, which no recorded build posts. No scenario serves a derived member itself, so its own identifiers and artifact digest are checked by the schema alone. ## Rules -`rules/README.md` states the shared rules for credentials, key lookup and format, and disk caching. A rule applies where the client and runtime implement its capability. Python implements disk caching; TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Credentials-file rules apply to runtimes with local credentials support. Each rule's vectors are in `rules/.json`. +`rules/README.md` states the shared rules for credentials, key lookup and format, the base URL, disk caching, the order a client presents a dataset's columns and rows in, the keys a caller narrows a spec's roles by, the file selector's query, the `X-DFO-Client` header that identifies the client to the API, and when a fetch of derived members warns that series were left out. It also states the rules DFO applies and a client reads the results of: the output digest of a derived frame, how `==` and `same_data` compare refs, the units and shapes of the operations' outputs, how the rolling steps of one recipe share one latest period, and how a share's cards are published, replaced and taken down. A rule applies where the client and runtime implement its capability. Python implements disk caching; TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Credentials-file rules apply to runtimes with local credentials support. Each rule's vectors are in `rules/.json`. ## Validation @@ -225,5 +229,5 @@ A record's `origin.base_url` is the base URL the run was given, a local app's in uv run ../../tools/validate_client_contract.py # Replay the rule vectors through a reference implementation of each rule -uv run pytest tests/test_rules.py +uv run pytest tests/test_rules.py tests/test_analysis_rules.py ``` diff --git a/python/client-contract/pyproject.toml b/python/client-contract/pyproject.toml index d613df2..a301906 100644 --- a/python/client-contract/pyproject.toml +++ b/python/client-contract/pyproject.toml @@ -45,6 +45,7 @@ dev = [ "pre-commit>=4.6.0", "jsonschema>=4.23", "referencing>=0.36", + "rfc8785>=0.1.4", ] [tool.ruff] diff --git a/python/client-contract/src/dfo_client_contract/actions.json b/python/client-contract/src/dfo_client_contract/actions.json index 34b2b09..cb28717 100644 --- a/python/client-contract/src/dfo_client_contract/actions.json +++ b/python/client-contract/src/dfo_client_contract/actions.json @@ -1,6 +1,10 @@ { "format_version": 1, "codes": [ + { + "code": "allowance_exhausted", + "action": "wait_for_reset" + }, { "code": "artifact_expired", "action": "take_newest_version" @@ -46,10 +50,42 @@ "code": "column_unknown", "action": "fix_request" }, + { + "code": "compare_no_key_matched", + "action": "fix_request" + }, + { + "code": "comparison_periods_disjoint", + "action": "fix_request" + }, + { + "code": "composed_unit_sum_unsupported", + "action": "fix_request" + }, + { + "code": "conflicting_requested_options", + "action": "fix_request" + }, { "code": "constraint_violation", "action": "fix_request" }, + { + "code": "coverage_insufficient", + "action": "fix_request" + }, + { + "code": "cross_frame_facet_mismatch", + "action": "fix_request" + }, + { + "code": "cross_frame_unit_mismatch", + "action": "fix_request" + }, + { + "code": "cross_frame_unit_unbound", + "action": "fix_request" + }, { "code": "default_facet_undefined", "action": "fix_request" @@ -63,6 +99,14 @@ "action": "ask_data_question", "gap_kind": "no_data" }, + { + "code": "derived_input_ordinal", + "action": "fix_request" + }, + { + "code": "derived_input_unusable", + "action": "fix_request" + }, { "code": "directional_children_unsupported", "action": "fix_request" @@ -119,6 +163,10 @@ "code": "facet_binding_unsupported", "action": "fix_request" }, + { + "code": "facet_heterogeneous", + "action": "fix_request" + }, { "code": "facet_not_available", "action": "fix_request", @@ -128,14 +176,30 @@ "code": "facet_probe_ceiling", "action": "fix_request" }, + { + "code": "filter_predicate_type_mismatch", + "action": "fix_request" + }, + { + "code": "filter_unit_conversion_unsupported", + "action": "fix_request" + }, { "code": "frame_not_refreshable", "action": "fix_request" }, + { + "code": "frame_too_large", + "action": "fix_request" + }, { "code": "frame_unavailable", "action": "check_id" }, + { + "code": "growth_gap_year", + "action": "fix_request" + }, { "code": "identity_collision", "action": "report" @@ -152,6 +216,14 @@ "code": "invalid_role_ref", "action": "fix_request" }, + { + "code": "join_no_common_key_columns", + "action": "fix_request" + }, + { + "code": "join_output_collision", + "action": "fix_request" + }, { "code": "member_not_resolvable", "action": "fix_request" @@ -168,6 +240,10 @@ "code": "mixed_peers_anchor_types", "action": "fix_request" }, + { + "code": "mixed_time_grain", + "action": "fix_request" + }, { "code": "no_matching_slice", "action": "ask_data_question", @@ -182,6 +258,10 @@ "code": "node_not_held", "action": "check_id" }, + { + "code": "non_finite_result", + "action": "fix_request" + }, { "code": "not_found", "action": "check_id" @@ -190,6 +270,10 @@ "code": "observations_unavailable", "action": "fix_request" }, + { + "code": "op_spec_invalid", + "action": "fix_request" + }, { "code": "open_role_unavailable", "action": "fix_request" @@ -202,6 +286,10 @@ "code": "operation_not_found", "action": "check_id" }, + { + "code": "operation_uncovered", + "action": "fix_request" + }, { "code": "option_target_invalid", "action": "fix_request" @@ -215,8 +303,8 @@ "action": "fix_request" }, { - "code": "pin_not_permitted", - "action": "request_scope" + "code": "payload_too_large", + "action": "fix_request" }, { "code": "pivot_not_homogeneous", @@ -234,6 +322,10 @@ "code": "pivot_over_column_cap", "action": "fix_request" }, + { + "code": "place_levels_mixed", + "action": "fix_request" + }, { "code": "population_empty", "action": "fix_request" @@ -259,6 +351,18 @@ "code": "query_too_long", "action": "fix_request" }, + { + "code": "quotient_of_no_key_matched", + "action": "fix_request" + }, + { + "code": "quotient_of_side_unit_unsupported", + "action": "fix_request" + }, + { + "code": "quotient_of_zero_denominator", + "action": "fix_request" + }, { "code": "rate_coverage_gap", "action": "fix_request", @@ -292,6 +396,10 @@ "code": "request_invalid", "action": "fix_request" }, + { + "code": "requested_option_unavailable", + "action": "fix_request" + }, { "code": "resolution_ambiguous", "action": "fix_request" @@ -300,6 +408,10 @@ "code": "resource_exhausted", "action": "fix_request" }, + { + "code": "role_ambiguous", + "action": "fix_request" + }, { "code": "role_incompatible", "action": "fix_request", @@ -315,6 +427,18 @@ "action": "fix_request", "gap_kind": "unsupported" }, + { + "code": "rolling_anchor_unresolved", + "action": "fix_request" + }, + { + "code": "rolling_grain_unsupported", + "action": "fix_request" + }, + { + "code": "rolling_populations_differ", + "action": "fix_request" + }, { "code": "rollup_not_summable", "action": "fix_request" @@ -331,10 +455,34 @@ "code": "service_unavailable", "action": "report" }, + { + "code": "several_members", + "action": "fix_request" + }, { "code": "shape_mismatch", "action": "fix_request" }, + { + "code": "share_not_replaceable", + "action": "fix_request" + }, + { + "code": "share_of_grain_mismatch", + "action": "fix_request" + }, + { + "code": "share_of_key_unknown", + "action": "fix_request" + }, + { + "code": "share_of_key_unmatched", + "action": "fix_request" + }, + { + "code": "share_of_zero_denominator", + "action": "fix_request" + }, { "code": "skill_not_found", "action": "check_id" @@ -388,6 +536,14 @@ "code": "spec_not_found", "action": "check_id" }, + { + "code": "stat_var_unresolved", + "action": "fix_request" + }, + { + "code": "sum_of_no_key_matched", + "action": "fix_request" + }, { "code": "timed_out", "action": "ask_data_question", @@ -397,6 +553,10 @@ "code": "timeout", "action": "report" }, + { + "code": "unit_heterogeneous", + "action": "fix_request" + }, { "code": "unit_not_resolved", "action": "ask_data_question", @@ -430,6 +590,14 @@ { "code": "write_refused", "action": "request_scope" + }, + { + "code": "write_token_invalid", + "action": "stop" + }, + { + "code": "write_token_required", + "action": "fix_request" } ] } diff --git a/python/client-contract/src/dfo_client_contract/actions.schema.json b/python/client-contract/src/dfo_client_contract/actions.schema.json index a102481..884937d 100644 --- a/python/client-contract/src/dfo_client_contract/actions.schema.json +++ b/python/client-contract/src/dfo_client_contract/actions.schema.json @@ -37,6 +37,7 @@ "take_newest_version", "log_in", "request_scope", + "stop", "wait", "wait_for_reset", "report", diff --git a/python/client-contract/src/dfo_client_contract/client-codes.json b/python/client-contract/src/dfo_client_contract/client-codes.json index 025fdf5..a0e8c9f 100644 --- a/python/client-contract/src/dfo_client_contract/client-codes.json +++ b/python/client-contract/src/dfo_client_contract/client-codes.json @@ -65,7 +65,7 @@ "code": "header_invalid", "class": "InvalidRequestError", "retryable": false, - "condition": "A header value holds a control character." + "condition": "A header value holds a control character, or a caller's header names X-DFO-Client." }, { "code": "insecure_transport", diff --git a/python/client-contract/src/dfo_client_contract/contract.py b/python/client-contract/src/dfo_client_contract/contract.py index 9503495..a8df709 100644 --- a/python/client-contract/src/dfo_client_contract/contract.py +++ b/python/client-contract/src/dfo_client_contract/contract.py @@ -7,14 +7,26 @@ from pathlib import Path from typing import Any -# The rules under `rules/`, each a file of vectors named after the rule. +# The rules under `rules/`, each a file of vectors named after the rule: the +# client rules, then the analysis and publication rules DFO applies. RULES = ( "key-format", "credentials-path", "credentials-file", "key-lookup", + "base-url", "cache-dir", "cache-layout", + "reading-order", + "client-identity", + "role-keys", + "file-selector", + "series-left-out", + "output-digest", + "output-comparison", + "operation-outputs", + "rolling-composition", + "publication", ) diff --git a/python/client-contract/src/dfo_client_contract/dataset-ref.schema.json b/python/client-contract/src/dfo_client_contract/dataset-ref.schema.json index 4e59f21..dbf129d 100644 --- a/python/client-contract/src/dfo_client_contract/dataset-ref.schema.json +++ b/python/client-contract/src/dfo_client_contract/dataset-ref.schema.json @@ -2,26 +2,65 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/DevelopmentFinanceObservatory/spec-kit/main/python/client-contract/src/dfo_client_contract/dataset-ref.schema.json", "title": "DatasetRef", - "description": "The frozen reference to the exact versions a dataset holds. Every DFO client writes and reads this format, so a ref saved by one language loads in another.", - "type": "object", - "additionalProperties": false, - "required": [ - "format_version", - "members" - ], - "properties": { - "format_version": { - "const": 1 - }, - "members": { - "type": "array", - "minItems": 1, - "items": { - "$ref": "#/$defs/member" + "description": "The frozen reference to the exact versions a dataset holds. Every DFO client writes and reads this format, so a ref saved by one language loads in another. Format 1 holds dataset members, format 2 derived members, each the result of an analysis recipe.", + "if": { + "type": "object", + "required": [ + "format_version" + ], + "properties": { + "format_version": { + "const": 2 } } }, + "then": { + "$ref": "#/$defs/format_2" + }, + "else": { + "$ref": "#/$defs/format_1" + }, "$defs": { + "format_1": { + "type": "object", + "additionalProperties": false, + "required": [ + "format_version", + "members" + ], + "properties": { + "format_version": { + "const": 1 + }, + "members": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/member" + } + } + } + }, + "format_2": { + "type": "object", + "additionalProperties": false, + "required": [ + "format_version", + "members" + ], + "properties": { + "format_version": { + "const": 2 + }, + "members": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/derived_member" + } + } + } + }, "member": { "type": "object", "additionalProperties": false, @@ -97,40 +136,244 @@ "pattern": "^sha256:[0-9a-f]{64}$" }, "rendering": { - "description": "The selector a client matches an artifact's rendering on. Never the server's rendering object.", + "$ref": "#/$defs/rendering" + } + } + } + } + }, + "derived_member": { + "description": "The frame an analysis recipe computed. DFO computes every member below; a client copies them from the server's answer and computes none of them.", + "type": "object", + "additionalProperties": false, + "required": [ + "parts", + "sentence", + "spec", + "content_version", + "output_digest", + "roots", + "execution_identity", + "resolved_periods", + "artifact" + ], + "properties": { + "parts": { + "description": "Empty, since no answer part names a derived frame.", + "type": "array", + "maxItems": 0 + }, + "sentence": { + "description": "The server's sentence for the derived frame.", + "type": "string", + "minLength": 1 + }, + "spec": { + "type": "object", + "additionalProperties": false, + "required": [ + "spec_id", + "recipe" + ], + "properties": { + "spec_id": { + "$ref": "#/$defs/ops_id" + }, + "recipe": { + "description": "The recipe's two hashes. `normal_form_hash` names the computation whatever data it ran on; `replay_key` names it on these roots under this execution identity.", "type": "object", "additionalProperties": false, "required": [ - "format", - "bundle" + "normal_form_hash", + "replay_key" ], "properties": { - "format": { - "enum": [ - "csv", - "xlsx", - "parquet" - ] - }, - "bundle": { - "type": "boolean" + "normal_form_hash": { + "type": "string", + "pattern": "^rnf_[0-9a-f]{32}$" }, - "layout": { - "const": "frame" - }, - "without_party_totals": { - "description": "Selects the artifact rendered with party totals left out. The key is absent for a plain rendering, so `false` is not a valid value.", - "const": true + "replay_key": { + "type": "string", + "pattern": "^rpk_[0-9a-f]{32}$" } } } } + }, + "content_version": { + "type": "string", + "pattern": "^ocv_[0-9a-f]{32}$" + }, + "output_digest": { + "description": "The digest of the frame's output alone, as rules/README.md defines it. `same_data` compares it.", + "type": "string", + "pattern": "^out_[0-9a-f]{32}$" + }, + "roots": { + "description": "The dataset versions the recipe ran on, each with the rendering it read.", + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "$ref": "#/$defs/root" + } + }, + "execution_identity": { + "description": "The engine, frame grammar and reduction policy versions that ran the recipe, joined by `|`.", + "type": "string", + "pattern": "^[0-9A-Za-z.+-]+\\|[0-9A-Za-z.+-]+\\|[0-9A-Za-z.+-]+$" + }, + "resolved_periods": { + "description": "Each period the recipe states relative to the latest period, with the period it resolved to. Empty for a recipe that states none.", + "type": "array", + "uniqueItems": true, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "anchor", + "offset", + "rule", + "period" + ], + "properties": { + "anchor": { + "const": "latest" + }, + "offset": { + "type": "integer", + "maximum": 0 + }, + "rule": { + "const": "half1" + }, + "period": { + "type": "string", + "pattern": "^[0-9]{4}$" + } + } + } + }, + "artifact": { + "type": "object", + "additionalProperties": false, + "required": [ + "digest", + "rendering" + ], + "properties": { + "digest": { + "type": "string", + "pattern": "^sha256:[0-9a-f]{64}$" + }, + "rendering": { + "$ref": "#/$defs/rendering" + } + } } } }, + "root": { + "type": "object", + "additionalProperties": false, + "required": [ + "spec_id", + "content_version", + "rendering" + ], + "properties": { + "spec_id": { + "$ref": "#/$defs/spec_id" + }, + "content_version": { + "$ref": "#/$defs/member/properties/content_version" + }, + "rendering": { + "$ref": "#/$defs/root_rendering" + } + } + }, + "root_rendering": { + "description": "The rendering an analysis read a root in: the members of the artifact selector, and `groupby` when the analysis read the root summed. `rendering`, the artifact selector, has no `groupby`.", + "$ref": "#/$defs/selector", + "properties": { + "groupby": { + "$ref": "#/$defs/groupby" + } + }, + "unevaluatedProperties": false + }, + "groupby": { + "description": "The sum the root was read with, the mint option `options.groupby` as DFO records it: one row per distinct combination of the columns `by` keeps, every other summable axis summed away. `levels` rolls each kept hierarchical axis it names up to the ancestor at that depth before the sum, 0 being the root codes. The key is absent when no level is set.", + "type": "object", + "additionalProperties": false, + "required": [ + "by", + "op" + ], + "properties": { + "by": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1 + } + }, + "op": { + "const": "sum" + }, + "levels": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "integer", + "minimum": 0, + "maximum": 99 + } + } + } + }, + "selector": { + "description": "The members of the selector a client matches an artifact's rendering on, shared by an artifact's rendering and a derived member's root rendering.", + "type": "object", + "required": [ + "format", + "bundle" + ], + "properties": { + "format": { + "enum": [ + "csv", + "xlsx", + "parquet" + ] + }, + "bundle": { + "type": "boolean" + }, + "layout": { + "const": "frame" + }, + "without_party_totals": { + "description": "Selects the artifact rendered with party totals left out. The key is absent for a plain rendering, so `false` is not a valid value.", + "const": true + } + } + }, + "rendering": { + "description": "The selector a client matches an artifact's rendering on. Never the server's rendering object.", + "$ref": "#/$defs/selector", + "unevaluatedProperties": false + }, "spec_id": { "type": "string", "pattern": "^spec_[0-9a-f]{32}$" + }, + "ops_id": { + "type": "string", + "pattern": "^ops_[0-9a-f]{32}$" } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-answer-member.json b/python/client-contract/src/dfo_client_contract/refs/ref-answer-member.json index 50815d9..c8d9a62 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-answer-member.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-answer-member.json @@ -1,8 +1,8 @@ { "format_version": 1, "name": "ref-answer-member", - "description": "A member built from an answer names the part it answers.", - "scenario": "fetch-spec-known-sentence", + "description": "A member built from an answer names the part it answers, the member of answer-definite-fetch.", + "scenario": "answer-definite-fetch", "ref": { "format_version": 1, "members": [ @@ -10,85 +10,84 @@ "parts": [ "ODA disbursements for basic health care" ], - "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 1995 to 2024. Current US dollars · OECD Creditor Reporting System (CRS).", "spec": { - "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "spec_id": "spec_774d6c001a6250aa67ca676ab067895e", "definition": { "schema_version": 1, - "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "spec_id": "spec_774d6c001a6250aa67ca676ab067895e", "identity": { - "identity_version": "spec1", "shape": { "kind": "statvar_schema", + "stat_type": "measuredValue", "population_type": "FinancialFlow", "measured_property": "amount", - "stat_type": "measuredValue", "measurement_qualifier": "Nominal" }, "slots": [ { - "property": "crsCooperationModality", "binding": { "state": "unbound" - } + }, + "property": "crsCooperationModality" }, { - "property": "crsPurpose", "binding": { - "state": "value", - "ref": "CRSPurposeCode12220" - } + "ref": "CRSPurposeCode12220", + "state": "value" + }, + "property": "crsPurpose" }, { - "property": "crsTypeOfFinance", "binding": { "state": "unbound" - } + }, + "property": "crsTypeOfFinance" }, { - "property": "crsTypeOfFlow", "binding": { - "state": "value", - "ref": "CRSTypeOfFlow10" - } + "ref": "CRSTypeOfFlow10", + "state": "value" + }, + "property": "crsTypeOfFlow" }, { - "property": "financialFlowType", "binding": { - "state": "value", - "ref": "Disbursement" - } + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" }, { - "property": "nettingBasis", "binding": { - "state": "value", - "ref": "Gross" - } + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" } ], "entities": { - "destination": [ - "country/NGA" - ], "origin": [ "country/DEU" + ], + "destination": [ + "country/NGA" ] }, + "identity_version": "spec1", "requested_window": { - "kind": "range", - "start_year": 2020, - "end_year": 2023 + "kind": "all" } } } }, - "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "content_version": "cv_b92d94774702c7f05f2075d0b03dd756", "artifact": { - "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-artifact-groupby-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-artifact-groupby-rejected.json new file mode 100644 index 0000000..c280c1a --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-artifact-groupby-rejected.json @@ -0,0 +1,82 @@ +{ + "format_version": 1, + "name": "ref-artifact-groupby-rejected", + "description": "An artifact's rendering selects a file and names no groupby: only a derived member's root records the sum it was read with.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria by top-level purpose, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_40511987433085c4f2dadc6b7aae2755", + "recipe": { + "normal_form_hash": "rnf_920ac02001d7529fd9c17ebadb452c99", + "replay_key": "rpk_a0a019687e913a1204cbcb089a58492f" + } + }, + "content_version": "ocv_6b4b9814c24f8d0c94624826c0c7773e", + "output_digest": "out_6927b7d353696180a34110d8dde723d6", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true, + "groupby": { + "by": [ + "crsPurpose", + "time_period" + ], + "op": "sum", + "levels": { + "crsPurpose": 0 + } + } + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:2f052058736db19e262bb8496f0da6c90f9deef851cb0224cff42dfb04bd0bd1", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "groupby": { + "by": [ + "crsPurpose", + "time_period" + ], + "op": "sum", + "levels": { + "crsPurpose": 0 + } + } + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-bare-definition-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-bare-definition-rejected.json index 1cdc751..29ed83e 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-bare-definition-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-bare-definition-rejected.json @@ -20,7 +20,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-dataset-member-in-format-2-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-dataset-member-in-format-2-rejected.json new file mode 100644 index 0000000..bda6c56 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-dataset-member-in-format-2-rejected.json @@ -0,0 +1,99 @@ +{ + "format_version": 1, + "name": "ref-dataset-member-in-format-2-rejected", + "description": "Format 2 holds derived members only. A ref of dataset members is written as format 1.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-definition-extra-member-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-definition-extra-member-rejected.json index 2ef647e..fc306c8 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-definition-extra-member-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-definition-extra-member-rejected.json @@ -87,7 +87,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-definition-spec-id-mismatch-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-definition-spec-id-mismatch-rejected.json index ef3d06c..ffd5bb5 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-definition-spec-id-mismatch-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-definition-spec-id-mismatch-rejected.json @@ -84,7 +84,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-identity-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-identity-rejected.json index 65ad764..c9c4202 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-identity-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-identity-rejected.json @@ -21,7 +21,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-schema-version-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-schema-version-rejected.json index 9628b91..78199eb 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-schema-version-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-definition-without-schema-version-rejected.json @@ -85,7 +85,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-grouped-root.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-grouped-root.json new file mode 100644 index 0000000..8b27887 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-grouped-root.json @@ -0,0 +1,70 @@ +{ + "format_version": 1, + "name": "ref-derived-grouped-root", + "description": "A format 2 ref naming a growth over a root of fetch-spec-known-sentence read summed, as a breakdown by purpose reads it: the root's rendering carries the groupby the analysis read it with, which keeps the time period and the purpose rolled up to its root codes and sums every other axis away.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria by top-level purpose, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_40511987433085c4f2dadc6b7aae2755", + "recipe": { + "normal_form_hash": "rnf_920ac02001d7529fd9c17ebadb452c99", + "replay_key": "rpk_a0a019687e913a1204cbcb089a58492f" + } + }, + "content_version": "ocv_6b4b9814c24f8d0c94624826c0c7773e", + "output_digest": "out_6927b7d353696180a34110d8dde723d6", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true, + "groupby": { + "by": [ + "crsPurpose", + "time_period" + ], + "op": "sum", + "levels": { + "crsPurpose": 0 + } + } + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:2f052058736db19e262bb8496f0da6c90f9deef851cb0224cff42dfb04bd0bd1", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "expect": "accept" +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-in-format-1-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-in-format-1-rejected.json new file mode 100644 index 0000000..82dc189 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-in-format-1-rejected.json @@ -0,0 +1,62 @@ +{ + "format_version": 1, + "name": "ref-derived-in-format-1-rejected", + "description": "Format 1 holds dataset members only, so a derived member needs format 2.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-member-fixed-window.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-member-fixed-window.json new file mode 100644 index 0000000..808c33f --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-member-fixed-window.json @@ -0,0 +1,47 @@ +{ + "format_version": 1, + "name": "ref-derived-member-fixed-window", + "description": "A format 2 ref naming a growth over a fixed window, which states no period relative to the latest and so resolves none.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_a41f297b3f33f2a7c2ac6b3cedd4d3ad", + "recipe": { + "normal_form_hash": "rnf_5f5a142beef739694b06899a7f0426a7", + "replay_key": "rpk_4f697e3235da4b9ae1404f112c2c78b6" + } + }, + "content_version": "ocv_62a72043403b8a2497cc184623df87df", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "expect": "accept" +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-member.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-member.json new file mode 100644 index 0000000..87a84f3 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-member.json @@ -0,0 +1,60 @@ +{ + "format_version": 1, + "name": "ref-derived-member", + "description": "A format 2 ref naming a growth over the latest four years of the member of fetch-spec-known-sentence: its ops_ id and recipe hashes, its ocv_ version and output digest, its root with the rendering it read, the execution identity, and the two periods the recipe states relative to the latest, resolved.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "expect": "accept" +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-parts-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-parts-rejected.json new file mode 100644 index 0000000..830d6ea --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-parts-rejected.json @@ -0,0 +1,64 @@ +{ + "format_version": 1, + "name": "ref-derived-parts-rejected", + "description": "A derived member answers no answer part, so its parts are empty.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [ + "ODA from Germany to Nigeria for basic health care" + ], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-period-after-latest-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-period-after-latest-rejected.json new file mode 100644 index 0000000..f02467c --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-period-after-latest-rejected.json @@ -0,0 +1,56 @@ +{ + "format_version": 1, + "name": "ref-derived-period-after-latest-rejected", + "description": "A resolved period counts back from the latest period, so its offset is 0 or below.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": 1, + "rule": "half1", + "period": "2024" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-repeated-spec-id-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-repeated-spec-id-rejected.json new file mode 100644 index 0000000..e42fff8 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-repeated-spec-id-rejected.json @@ -0,0 +1,111 @@ +{ + "format_version": 1, + "name": "ref-derived-repeated-spec-id-rejected", + "description": "Each member holds a distinct spec_id, an ops_ id included.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + }, + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "rule", + "reason": "repeated_spec_id" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-groupby-op-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-groupby-op-rejected.json new file mode 100644 index 0000000..1ed2bb3 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-groupby-op-rejected.json @@ -0,0 +1,72 @@ +{ + "format_version": 1, + "name": "ref-derived-root-groupby-op-rejected", + "description": "A root's groupby sums, so any other op is not a ref.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria by top-level purpose, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_40511987433085c4f2dadc6b7aae2755", + "recipe": { + "normal_form_hash": "rnf_920ac02001d7529fd9c17ebadb452c99", + "replay_key": "rpk_a0a019687e913a1204cbcb089a58492f" + } + }, + "content_version": "ocv_6b4b9814c24f8d0c94624826c0c7773e", + "output_digest": "out_6927b7d353696180a34110d8dde723d6", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true, + "groupby": { + "by": [ + "crsPurpose", + "time_period" + ], + "op": "mean", + "levels": { + "crsPurpose": 0 + } + } + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:2f052058736db19e262bb8496f0da6c90f9deef851cb0224cff42dfb04bd0bd1", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-without-rendering-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-without-rendering-rejected.json new file mode 100644 index 0000000..b038de9 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-root-without-rendering-rejected.json @@ -0,0 +1,56 @@ +{ + "format_version": 1, + "name": "ref-derived-root-without-rendering-rejected", + "description": "Each root records the rendering the recipe read, since a cut with party totals left out and a plain one are two inputs.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4" + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-derived-without-output-digest-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-derived-without-output-digest-rejected.json new file mode 100644 index 0000000..6947fcf --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/refs/ref-derived-without-output-digest-rejected.json @@ -0,0 +1,61 @@ +{ + "format_version": 1, + "name": "ref-derived-without-output-digest-rejected", + "description": "A derived member carries the output digest same_data compares.", + "scenario": "fetch-spec-known-sentence", + "ref": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "error": { + "by": "schema" + } +} diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-frame-layout.json b/python/client-contract/src/dfo_client_contract/refs/ref-frame-layout.json index a8495b7..3cad5c6 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-frame-layout.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-frame-layout.json @@ -85,7 +85,8 @@ "rendering": { "format": "parquet", "bundle": false, - "layout": "frame" + "layout": "frame", + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-one-member.json b/python/client-contract/src/dfo_client_contract/refs/ref-one-member.json index 50e1dd6..e6a2563 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-one-member.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-one-member.json @@ -1,89 +1,73 @@ { "format_version": 1, "name": "ref-one-member", - "description": "A ref to one bare long Parquet version, the member of fetch-spec-known-sentence.", - "scenario": "fetch-spec-known-sentence", + "description": "A ref to one bare long Parquet version fetched with party_totals=True, whose rendering records no without_party_totals, the member of fetch-spec-party-totals-kept.", + "scenario": "fetch-spec-party-totals-kept", "ref": { "format_version": 1, "members": [ { "parts": [], - "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a.", "spec": { - "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", "definition": { "schema_version": 1, - "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", "identity": { - "identity_version": "spec1", "shape": { "kind": "statvar_schema", + "stat_type": "measuredValue", "population_type": "FinancialFlow", "measured_property": "amount", - "stat_type": "measuredValue", "measurement_qualifier": "Nominal" }, "slots": [ { - "property": "crsCooperationModality", "binding": { - "state": "unbound" - } + "ref": "DAC2aAidType206", + "state": "value" + }, + "property": "dac2aAidType" }, { - "property": "crsPurpose", - "binding": { - "state": "value", - "ref": "CRSPurposeCode12220" - } - }, - { - "property": "crsTypeOfFinance", "binding": { "state": "unbound" - } + }, + "property": "dacTypeOfFinance" }, { - "property": "crsTypeOfFlow", "binding": { - "state": "value", - "ref": "CRSTypeOfFlow10" - } + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" }, { - "property": "financialFlowType", "binding": { - "state": "value", - "ref": "Disbursement" - } - }, - { - "property": "nettingBasis", - "binding": { - "state": "value", - "ref": "Gross" - } + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" } ], "entities": { "destination": [ - "country/NGA" - ], - "origin": [ - "country/DEU" + "country/KEN" ] }, + "identity_version": "spec1", "requested_window": { "kind": "range", - "start_year": 2020, - "end_year": 2023 + "end_year": 2022, + "start_year": 2022 } } } }, - "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "content_version": "cv_2aa4cc2fd852cd9b84347c61d07d609a", "artifact": { - "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "digest": "sha256:05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426", "rendering": { "format": "parquet", "bundle": false diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-repeated-spec-id-rejected.json b/python/client-contract/src/dfo_client_contract/refs/ref-repeated-spec-id-rejected.json index 830c942..629887d 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-repeated-spec-id-rejected.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-repeated-spec-id-rejected.json @@ -86,14 +86,13 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, { - "parts": [ - "ODA disbursements for basic health care" - ], + "parts": [], "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", "spec": { "spec_id": "spec_cb5baa4161de718f18583060674dbca6", @@ -172,7 +171,8 @@ "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } diff --git a/python/client-contract/src/dfo_client_contract/refs/ref-without-party-totals.json b/python/client-contract/src/dfo_client_contract/refs/ref-without-party-totals.json index e458ea3..776d5e9 100644 --- a/python/client-contract/src/dfo_client_contract/refs/ref-without-party-totals.json +++ b/python/client-contract/src/dfo_client_contract/refs/ref-without-party-totals.json @@ -1,7 +1,7 @@ { "format_version": 1, "name": "ref-without-party-totals", - "description": "A ref to the version rendered with party totals left out, the member of fetch-spec-known-sentence.", + "description": "A ref a fetch with the default party totals rule writes, its rendering naming without_party_totals, the member of fetch-spec-known-sentence.", "scenario": "fetch-spec-known-sentence", "ref": { "format_version": 1, diff --git a/python/client-contract/src/dfo_client_contract/rules/README.md b/python/client-contract/src/dfo_client_contract/rules/README.md index fdd4a5e..48d3a49 100644 --- a/python/client-contract/src/dfo_client_contract/rules/README.md +++ b/python/client-contract/src/dfo_client_contract/rules/README.md @@ -1,6 +1,6 @@ # Client rules -Clients implement these rules for the capabilities available in their runtime. Python implements every rule. TypeScript implements the key and credentials rules in Node; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. Each rule's vectors are in `.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below. +Clients implement these rules for the capabilities available in their runtime. The first twelve rules, from credentials path to series left out, are client rules, and Python implements all twelve. TypeScript implements the key, credentials and base-URL rules in Node, and the reading-order, client-identity, role-keys, file-selector and series-left-out rules in every runtime; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. The last five rules, from output digest to publication, are DFO's: dfo-apis applies them where it computes, stores and publishes, and a client reads their results. Of these, a client implements only the output comparison, over the digests DFO gives it, as `==` and `same_data` in Python and `equals` and `sameData` in TypeScript. Each rule's vectors are in `.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below. ## Credentials path (`credentials-path`) @@ -34,6 +34,16 @@ A client resolves its key when it is built, and again after a 401. The first sou A missing file goes on to the next source, and so does a missing `default` profile. A file that fails to parse raises `credentials_file_invalid`, a profile named by `DFO_PROFILE` that the file lacks raises `profile_not_found`, and a chosen profile with no `key`, or an empty one, raises `profile_without_key`. The file is read only when no earlier source wins. The argument may also be a callable or a key object, called or unwrapped when the client resolves the key. The vectors pass strings, and the Colab source has no vectors. +## Base URL (`base-url`) + +A client resolves its base URL when it is built. The first source found wins: + +1. the `base_url` argument of the call or constructor, as given; +1. `DFO_BASE_URL`; +1. the public default, `https://developmentfinance.org/api/v1`. + +Only the chosen source is checked. A base URL is valid when it is an absolute `http` or `https` URL with a host and none of these: userinfo, an empty one as in `https://@host/api` included; a query; a fragment; a `.` or `..` path segment, with any of its dots written `%2e` or `%2E` included; a backslash; a port that is not 1 to 65535 in decimal digits, an empty port after a `:` included; a character outside visible ASCII, a control character, a space or a non-ASCII character; or a host written as an IPv4 address in any form but four decimal numbers, such as `127.1`, `0x7f.1` or `2130706433`. An invalid one raises `base_url_invalid`, so an empty `base_url` argument raises it where an empty `DFO_BASE_URL` counts as unset. A relative URL is invalid, except in a TypeScript client inside a browser page, which resolves a relative base URL against the page's address. A valid value is used as given, less its trailing slashes, so the client's base URL reads `https://developmentfinance.org/api/v1` for `https://developmentfinance.org/api/v1/`; the rule fixes the source and the validity, and leaves the rest of the URL's form to the client. A key sent over plain `http` to a host that is not loopback is refused by `insecure_transport`, which this rule does not decide. + ## Key format (`key-format`) A secret key is `dfo_k_`, then a base62 body, then a 6-character checksum; `dfo_p_` marks a browser-safe key. Base62 is `0-9A-Za-z` in that order. The checksum is the CRC32 (ISO-HDLC, as zlib computes it) of the body's ASCII bytes, written in base62 and left-padded with `0` to six characters. Issued keys have a 32-character body, 44 characters in all, but the check ignores length. The last six characters after the prefix are the checksum and the rest is the body. @@ -58,4 +68,85 @@ Inside the cache directory: - a downloaded artifact is `files/sha256/`, keyed by its digest; - a manifest is `manifests///.json`, where `` is the format, then `-bundle` for a bundle, then `-frame` for the frame layout: `parquet`, `parquet-frame`, `csv-bundle`. -A client writes an entry to a temporary name in the same directory and renames it into place, so a reader sees a complete entry. Concurrent writers of one content-addressed artifact write identical bytes. Manifest updates can replace another complete manifest and lose a cache candidate, so cache availability is best-effort. A read re-hashes an artifact, and a mismatch counts as a miss. +The key leaves out a rendering's `without_party_totals`: a cut that leaves rows out answers a version of its own, and one that leaves none out answers the plain version's file. A client writes an entry to a temporary name in the same directory and renames it into place, so a reader sees a complete entry. Concurrent writers of one content-addressed artifact write identical bytes. Manifest updates can replace another complete manifest and lose a cache candidate, so cache availability is best-effort. A read re-hashes an artifact, and a mismatch counts as a miss. + +## Reading order (`reading-order`) + +A client presents a dataset's columns and rows in reading order, in every table it returns. `reading_columns` in `reading-order.json` lists nine slots in order: the time period, the provider, the recipient, the subject, the purpose, the value, the unit, the flow type and the provenance. Each slot names the long layout's label column, then the frame layout's code column, and the flow type slot names the CRS pair, then the DAC2a aid type pair. The first name of a slot that the dataset holds fills the slot, and a slot whose names it holds none of stays empty. + +| Slot | Long layout | Frame layout | +| ----------- | ------------------------------------------------ | ------------------------------------------------------ | +| Time period | `time_period` | `time_period` | +| Provider | `financingProvider_label` | `financingProvider` | +| Recipient | `financingRecipient_label` | `financingRecipient` | +| Subject | `observationAbout_label` | `observationAbout` | +| Purpose | `crsPurpose_label` | `stat_var_crsPurpose` | +| Value | `value` | `value` | +| Unit | `unit_label` | `unit` | +| Flow type | `crsTypeOfFlow_label`, else `dac2aAidType_label` | `stat_var_crsTypeOfFlow`, else `stat_var_dac2aAidType` | +| Provenance | `provenance_label` | `provenance` | + +The columns come in four groups: + +1. the filled slots, in slot order; +1. the other columns whose values differ between rows, in the order the server wrote them; +1. the other columns that hold one value in every row, null included, in the order the server wrote them; +1. `stat_var`, last whether or not it varies. + +A table with no row or one row has no column that varies. Two cells hold the same value when they are equal as JSON values: numbers by value, so `1` and `1.0` are one value, booleans apart from numbers, so `true` and `1` differ, and null the same as null. NaN, which a Parquet double column can hold, is the same as NaN. A list cell is the same as another when their items are the same, in order, and a struct cell when it has the same fields with the same values, in any field order. A dataset of several members puts `part` first and judges the groups over the stacked table, where a member's rows hold null in a column that member lacks. So a CRS series of Germany's basic health ODA to Nigeria, split by co-operation modality, reads `time_period`, `financingProvider_label`, `financingRecipient_label`, `crsPurpose_label`, `value`, `unit_label`, `crsTypeOfFlow_label` and `provenance_label`, then `crsCooperationModality` and `crsCooperationModality_label`, which tell its rows apart, then its 15 constant columns, `stat_var` last. + +The rows of each member are sorted by the filled slots in slot order, leaving out `value`, a measure, and the time period, then by the varying columns in presented order, `stat_var` after them when it varies, then by the time period, so each series' years read together. A column that holds lists or structs is no sort key, since the rule gives them no order. The keys of that series are the provider, recipient, purpose, unit, flow type and provenance labels, then `crsCooperationModality` and `crsCooperationModality_label`, then the time period, so it reads every A02 year, then every C01 year. Each key sorts ascending, numbers by value and strings by Unicode code point, with nulls last. Rows whose keys tie keep the order the server wrote them. The members follow one another in member order. + +A vector gives the dataset's `members`, each with its `part` when there are several, its `columns` and its `rows` as the server wrote them, and the `columns` and `rows` a client presents. Each row holds one cell per column, and the cells of a column, nulls aside, are all strings, all numbers, all booleans, all lists or all structs, since the rule orders no value against one of another type. The order moves columns and rows and changes no value. A client hashes a downloaded file before it orders anything, so the digest and the ref keep the server's file. + +## Client identity (`client-identity`) + +Every request a client sends to the API carries `X-DFO-Client: / ()`. `` is `dfo-python` or `dfo-ts`, and `` is the package version as the package states it. `` names the runtime and its major and minor version. In Python it is the implementation as `platform.python_implementation()` names it, then the interpreter's major and minor version, such as `CPython 3.14`. In TypeScript it is `Node`, `Deno` or `Bun` with the major and minor of that runtime's version, such as `Node 22.18`, `browser` in a browser page or browser worker, and `other` in any other runtime, such as Cloudflare Workers. `pattern` in `client-identity.json` matches every well-formed value. A vector gives the `client`, the `version`, the `runtime` as its `name` and full `version`, or `browser` or `other` with no version, and the `value` the client sends. + +An API request is one to the base URL's origin, a `path` or `origin_path` request in a scenario. A request to any other origin, a signed storage URL included, carries no `X-DFO-Client`. In a browser, the files-route request the TypeScript client sends with `redirect: "follow"` carries the API key alone, because the browser forwards every other header to the storage host it is redirected to. + +A caller cannot replace or remove the header. A client checks the caller's `headers` argument before any request, and the headers a header provider returns each time the client calls it. A name equal to `X-DFO-Client` in any letter case raises `InvalidRequestError` `header_invalid`, whatever its value, the empty string included. A vector gives `via`, `argument` or `provider`, the caller's `headers`, and the `verdict`, `accept` or `refuse` with the `reason`. Every other header is the caller's to set, `User-Agent` included. Python sends `dfo-python/`, TypeScript sends `dfo-ts/` whenever it runs outside a browser page or browser worker (Node, Deno, Bun, or a server runtime such as Cloudflare Workers) and none inside one, where the browser forbids setting it, and a caller's `User-Agent` replaces the client's. + +## Role keys (`role-keys`) + +A caller narrows a spec by its entity roles with `entities=`, keyed by the caller keys `keys` in `role-keys.json` lists: `place`, the place a measure is about, and the flow directions `origin` and `destination`. A client posts `place` as the wire role `observationAbout` and each direction as itself. `Spec.roles` in Python, `specRoles(spec)` in TypeScript, and every error a client raises name the caller key. Each role has one caller spelling, so `observationAbout` is a wire role and no caller key. + +The roles a spec takes are those its `definition.identity` binds in `entities` or lists in `open_roles`, in that order, each under its caller key with the dcids the spec binds it to, or null when it binds none. A flow spec, one that names `origin` or `destination`, takes both, since a direction it neither binds nor opens reads as the total over it. The direction it leaves unnamed comes last. A spec that names no role takes `{}`. + +A client checks each key the caller types before any request. A key outside the spec's roles raises `InvalidRequestError` `invalid_argument`, and the message names the key and the spec's roles. So GDP for Kenya refuses `origin` and `observationAbout` naming `place`, and a flow spec refuses `place` and a column name such as `financingRecipient` naming `origin` and `destination`. A spec that names no role leaves the check to the server, since the client cannot tell which roles it takes. Such a spec posts `place` as `observationAbout` and any other key as typed, and still refuses `observationAbout` naming `place`. + +The server replaces a spec's whole binding with the posted one, so a client posts the spec's own `identity.entities` with each role the caller names replaced by its dcids and each role the caller binds to null dropped. A vector gives the spec's `identity` and its `roles`. A vector that narrows adds the caller's `entities`, then either `posted`, the wire entities the build posts, or the `error` with its `reason` and the keys its message `names`, the typed key first. + +## File selector (`file-selector`) + +The file selector, `GET /datasets/{spec_id}/{content_version}/file`, redirects to a stored version's file in a rendering. A client asks it for a ref member's version when the files route at the stored digest refuses, or when the snapshot manifest lists no artifact of that digest. Its query holds the keys `query_keys` in `file-selector.json` lists, `format`, `bundle` and `layout`, in that order, each one the member's rendering holds, as `key=value` pairs joined by `&`, with booleans written `true` and `false`. The query leaves out `without_party_totals`: the version records the cut it was minted with, and the server lays that over the query. The server refuses any other key with 400 `request_invalid`, so a selector query that carried the cut would end the lookup. A vector gives the `rendering` and the `query` a client sends, without the leading `?`. + +## Series left out (`series-left-out`) + +A derived member's manifest, from `GET /datasets/derived/{ops_id}/{ocv_version}`, lists in `left_out` every series a step of its recipe dropped for a data gap, such as a growth endpoint the series lacks or a key the other side holds no partner for. Each entry names the series' `coordinates`, its `step` as the verb and node index, the `side` it came from, the engine's `reason` and its `rows`. `left_out_complete` is false when the steps dropped more series than `left_out` lists. Both are null for a recipe stored without them. + +A fetch of derived members issues `series_left_out`, as `SeriesLeftOutWarning`, once when any member's manifest lists a left-out series or says `left_out_complete` false, and returns the frame. A strict fetch raises the warning in place of the data. A null `left_out` issues nothing. A vector gives each member's `left_out` and `left_out_complete` in member order, and `strict`, then the `warnings` the fetch issues or the warning it `raises`. + +## Output digest (`output-digest`) + +A derived member's `output_digest` names the output of its frame: its values, with the variable, members, period and unit each belongs to, and nothing about how they were computed. DFO computes it when it stores a derived frame, and a client copies it from the server's answer. + +The digest is taken over a frame as `docs/frame-grammar-v1.md` defines it. Its document is the RFC 8785 (JCS) serialisation of `{"digest_version": "out2", "columns": [...], "rows": [...]}`. `columns` lists `stat_var`, then the frame's role columns in the frame's order, then `time_period`, `value` and `unit`. Each row holds the frame row's cells in those columns, with `recipe` dropped from every unit-request object, in a `unit` cell and in either part of a composed unit. The rows are sorted by the UTF-8 bytes of each row's own JCS serialisation, ascending, and a row that repeats another stays. The digest is `out_` followed by the first 32 lowercase hexadecimal characters of the SHA-256 of the document's UTF-8 bytes. + +The document leaves out `measurementMethod`, `provenance` and `status`, and a unit's `recipe`. These name the source an output came from and how its unit was converted, so they move when a source is imported again or a conversion recipe is revised while no number moves. `stat_var` stays in, so two frames that hold the same values under different variables hold different outputs. A join writes each side's rows under that side's variable, so a join with its sides swapped holds the same rows and the same output, while a frame holding each side's values under the other side's variable holds another. `compare`, `sum_of`, `share_of` and `quotient_of` write rows under a variable of their own (`operation-outputs`), which a frame spells `dcid:` and the step's `ops_` id. That id hashes the versions the step read and, through their `ocv_` versions, the engine that computed them, so the document reads that variable under a name built from what was computed. The name is `mv_` and the first 32 hex digits of the SHA-256 of a name document, the JCS serialisation of an object holding `verb`, the step's verb, and one member per frame slot the step fills, keyed by the slot's name in the recipe tree (`frame_ref`, `right_frame_ref`, `denominator_frame_ref`), each the names of the variables that slot's input carries, sorted by their UTF-8 bytes. A root dataset carries its frame's `stat_var` values, `filter`, `aggregate`, `rank` and `growth` carry their input's variables, `join` the union of its inputs', and a step that writes its own variable carries that variable's `mv_` name. A name document thus holds the names of its inputs and not their documents, so every name is 35 characters however deep the recipe, and a name document grows only with the variables the roots under it hold. The prefix marks the name as computed, apart from a dataset's `dcid:` variable and an operation's `ops_` id, in the style of the `out_` digest. A one-step `quotient_of` of ODA by population has the name document `{"denominator_frame_ref":["dcid:Count_Person"],"frame_ref":["dcid:Amount_ODA"],"verb":"quotient_of"}` and names its variable `mv_d355e59c50ff8cf07e86a4da12b64297`. Equal computations over equal variables thus share a digest across recipes, engine releases and upstream rebuilds that move no number, and a step with its sides swapped holds another. A `filter` that narrows a root holding several variables leaves the name listing all of them, which errs towards two outputs differing. A vector gives the `frame`, the `variables` its operations wrote where it holds any, each name keyed by its `ops_` id, the `name_documents` of those names and of every `mv_` name their documents hold, each keyed by its name, the `document` as text and the `digest`. `quotient-of-a-sum-of` divides a `sum_of` by population, so its name document lists the `sum_of` step's `mv_` name. Vectors whose frames differ only in what the document leaves out, in row order, or in the `ops_` id of a variable named alike, share a digest. + +## Output comparison (`output-comparison`) + +Two refs answer two questions. `==` in Python and `equals` in TypeScript say whether two refs name the same versions, recipes and renderings: they hold as many members, and each member matches the other ref's member at its position on its `spec_id`, its `content_version`, its artifact's `rendering` and, for a derived member, its `recipe`. A rolling recipe and a fixed one therefore stay apart in a cache even when their rows match. `same_data` in Python and `sameData` in TypeScript say whether two refs hold the same output: they hold as many members, and at each position both members are derived members with the same `output_digest`, or both are dataset members with the same `content_version` and the same `without_party_totals`. A dataset version's `content_version` already hashes its frame alone, so a dataset member needs no digest of its own, and its layout and file format hold the same rows and do not enter. A dataset member and a derived member never hold the same output. Both comparisons ignore `format_version`, `parts`, `sentence`, artifact digests, roots, execution identities and resolved periods. So an engine release or an upstream rebuild that moves no number keeps `same_data` true, and a pipeline that rebuilds on `not same_data` rebuilds only when an output moved. This holds for an output under a variable its operation wrote, since the digest names that variable by its verb and its inputs' variables (`output-digest`), so a one-step `quotient_of` and a `filter` that keeps every row followed by the same `quotient_of` hold the same data. A vector gives two refs, `left` and `right`, and the two verdicts, which hold either way round. + +## Operation outputs (`operation-outputs`) + +DFO's operations keep the units and shapes Studio's have. `share_of` gives a share in per cent, 100 times the ratio, under the unit `Percent`. With a second reading, each row of `data` is divided by the row of `other` that `on` pairs it with in the same period. With none, each member of `by` is divided by the total across every member in the same period, after its own rows add up. `join` returns the rows of both readings that `on` pairs, in long format: one row per side for each key and period both hold, told apart by `stat_var`, each keeping its own unit. `compare` gives `data` minus `other`, and `sum_of` gives `data` plus `other`, for each paired member and period both hold, in the unit both share. A relative `growth` is a fraction in the `value` column, 0.25 for 25 per cent, with no unit, and an absolute one is the change in the input's unit. A growth row's period is the window as a span, such as `2018/2022`, and a member without a value at both ends has no row. A relative growth also leaves out a member whose start or end value is zero or whose values change sign (`zero_or_sign_crossing_base`), and either growth leaves out a change that is not finite (`non_finite`). Two rows of one member in one period of the window refuse the step with `role_ambiguous`. An output keeps `data`'s names for the roles `on` pairs. `share_of`, `compare` and `sum_of` write rows under a variable of their own, whose spelling is the operation's; `join` and `growth` keep their inputs' variables. A vector gives the `verb`, its `arguments`, the rows of `data` and of `other`, and either the output `rows`, in any order and leaving `stat_var` out of a row an operation writes under its own variable, or the `error`. + +## Rolling composition (`rolling-composition`) + +A recipe tree has one latest period. Each rolling step of the tree states the populations it counts the latest period over, one for each root dataset it reads with the columns it narrows that dataset's series by. Every rolling step of one tree states the same populations, compared in their one spelling, whatever order they are listed in. A tree whose rolling steps state different populations is refused with `rolling_populations_differ` before anything runs, whether they would settle on different years or on the same one. The latest period is then the newest period at least half of the series of every population hold, the `half1` rule of `dfo_frame_grammar.latest_anchor`. A population no series falls in, or populations no period satisfies, refuse `rolling_anchor_unresolved`. A vector gives every `series` of the roots with the periods it holds, the rolling `steps` with their populations, and the `period` or the `error`. + +## Publication (`publication`) + +A share publishes cards under a stable URL, live with `latest` true or as a snapshot with it false. It holds numbered slots, the first card in slot 1, and each slot's embed and data URLs are named by its number. Creating a share returns a write token. Replacing a share's cards and taking it down each need that token: without one they are refused with `write_token_required`, and with another token, or on a share created without one, with `write_token_invalid`. The token is checked first. Only a live share can be replaced, and a snapshot refuses `share_not_replaceable`, so a cited snapshot URL draws the same figure for good. A replacement puts its cards in slots 1 onwards, so every URL a share has published keeps working: a slot past the replacement's last card holds no card until a later replacement fills it again. A refused step changes nothing. A snapshot is taken down like a live share. A vector gives the `steps`, each naming a share, and one outcome per step: the share's `latest` and slots, `removed`, or the `error`. A step's `write_token` is `held` for the token the share's creation returned and `other` for another one. diff --git a/python/client-contract/src/dfo_client_contract/rules/base-url.json b/python/client-contract/src/dfo_client_contract/rules/base-url.json new file mode 100644 index 0000000..b778e4e --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/base-url.json @@ -0,0 +1,255 @@ +{ + "format_version": 1, + "rule": "base-url", + "vectors": [ + { + "name": "argument-wins", + "base_url": "https://arg.example.org/api/v1", + "env": { + "DFO_BASE_URL": "https://env.example.org/api/v1" + }, + "source": "argument", + "url": "https://arg.example.org/api/v1" + }, + { + "name": "environment-beats-default", + "env": { + "DFO_BASE_URL": "https://env.example.org/api/v1" + }, + "source": "environment", + "url": "https://env.example.org/api/v1" + }, + { + "name": "default", + "env": {}, + "source": "default", + "url": "https://developmentfinance.org/api/v1" + }, + { + "name": "empty-environment-counts-as-unset", + "env": { + "DFO_BASE_URL": "" + }, + "source": "default", + "url": "https://developmentfinance.org/api/v1" + }, + { + "name": "loopback-http-is-valid", + "base_url": "http://127.0.0.1:8083/api", + "env": {}, + "source": "argument", + "url": "http://127.0.0.1:8083/api" + }, + { + "name": "empty-argument-raises", + "base_url": "", + "env": { + "DFO_BASE_URL": "https://env.example.org/api/v1" + }, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "invalid-argument-raises-despite-valid-environment", + "base_url": "not a url", + "env": { + "DFO_BASE_URL": "https://env.example.org/api/v1" + }, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "relative-argument-raises", + "base_url": "/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "schemeless-argument-raises", + "base_url": "developmentfinance.org/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "other-scheme-raises", + "base_url": "ftp://developmentfinance.org/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "hostless-argument-raises", + "base_url": "https:///api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "userinfo-raises", + "base_url": "https://ana:secret@developmentfinance.org/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "invalid-environment-raises", + "env": { + "DFO_BASE_URL": "developmentfinance.org" + }, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "invalid-environment-is-not-read-when-the-argument-wins", + "base_url": "https://arg.example.org/api/v1", + "env": { + "DFO_BASE_URL": "developmentfinance.org" + }, + "source": "argument", + "url": "https://arg.example.org/api/v1" + }, + { + "name": "query-raises", + "base_url": "https://developmentfinance.org/api/v1?key=1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "fragment-raises", + "base_url": "https://developmentfinance.org/api/v1#top", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "dot-segment-raises", + "base_url": "https://developmentfinance.org/api/../v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "control-character-raises", + "base_url": "https://developmentfinance.org/api\n/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "non-decimal-ipv4-raises", + "base_url": "http://0x7f.1:8083/api", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "empty-userinfo-raises", + "base_url": "https://@developmentfinance.org/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "trailing-slash-is-dropped", + "base_url": "https://developmentfinance.org/api/v1/", + "env": {}, + "source": "argument", + "url": "https://developmentfinance.org/api/v1" + }, + { + "name": "encoded-dot-segment-raises", + "base_url": "https://developmentfinance.org/api/%2e%2e/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "mixed-encoded-dot-segment-raises", + "base_url": "https://developmentfinance.org/api/.%2E/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "encoded-single-dot-segment-raises", + "base_url": "https://developmentfinance.org/api/%2E/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "backslash-raises", + "base_url": "https://developmentfinance.org\\api\\v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "port-out-of-range-raises", + "base_url": "https://developmentfinance.org:99999/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "port-zero-raises", + "base_url": "https://developmentfinance.org:0/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "port-not-decimal-raises", + "base_url": "https://developmentfinance.org:abc/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "empty-port-raises", + "base_url": "https://developmentfinance.org:/api/v1", + "env": {}, + "error": { + "reason": "base_url_invalid" + } + }, + { + "name": "highest-port-is-valid", + "base_url": "https://developmentfinance.org:65535/api/v1", + "env": {}, + "source": "argument", + "url": "https://developmentfinance.org:65535/api/v1" + }, + { + "name": "dot-inside-a-segment-is-valid", + "base_url": "https://developmentfinance.org/api/v1.%2e", + "env": {}, + "source": "argument", + "url": "https://developmentfinance.org/api/v1.%2e" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/client-identity.json b/python/client-contract/src/dfo_client_contract/rules/client-identity.json new file mode 100644 index 0000000..480fb20 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/client-identity.json @@ -0,0 +1,183 @@ +{ + "format_version": 1, + "rule": "client-identity", + "header": "X-DFO-Client", + "pattern": "^(dfo-python|dfo-ts)/[0-9A-Za-z.+-]+ \\(([A-Za-z]+ [0-9]+\\.[0-9]+|browser|other)\\)$", + "vectors": [ + { + "name": "python-cpython", + "client": "dfo-python", + "version": "0.1.0", + "runtime": { + "name": "CPython", + "version": "3.14.0" + }, + "value": "dfo-python/0.1.0 (CPython 3.14)" + }, + { + "name": "python-pre-release-interpreter", + "client": "dfo-python", + "version": "0.1.0", + "runtime": { + "name": "CPython", + "version": "3.15.0a1" + }, + "value": "dfo-python/0.1.0 (CPython 3.15)" + }, + { + "name": "python-pypy", + "client": "dfo-python", + "version": "0.1.0", + "runtime": { + "name": "PyPy", + "version": "3.11.13" + }, + "value": "dfo-python/0.1.0 (PyPy 3.11)" + }, + { + "name": "python-pre-release-package", + "client": "dfo-python", + "version": "0.2.0rc1", + "runtime": { + "name": "CPython", + "version": "3.12.11" + }, + "value": "dfo-python/0.2.0rc1 (CPython 3.12)" + }, + { + "name": "python-local-version", + "client": "dfo-python", + "version": "0.2.0.dev3+g41c47f8", + "runtime": { + "name": "CPython", + "version": "3.13.5" + }, + "value": "dfo-python/0.2.0.dev3+g41c47f8 (CPython 3.13)" + }, + { + "name": "ts-node", + "client": "dfo-ts", + "version": "0.1.0", + "runtime": { + "name": "Node", + "version": "22.18.0" + }, + "value": "dfo-ts/0.1.0 (Node 22.18)" + }, + { + "name": "ts-deno", + "client": "dfo-ts", + "version": "0.1.0", + "runtime": { + "name": "Deno", + "version": "2.1.4" + }, + "value": "dfo-ts/0.1.0 (Deno 2.1)" + }, + { + "name": "ts-bun", + "client": "dfo-ts", + "version": "0.1.0", + "runtime": { + "name": "Bun", + "version": "1.2.19" + }, + "value": "dfo-ts/0.1.0 (Bun 1.2)" + }, + { + "name": "ts-browser", + "client": "dfo-ts", + "version": "0.1.0", + "runtime": { + "name": "browser" + }, + "value": "dfo-ts/0.1.0 (browser)" + }, + { + "name": "ts-other-runtime", + "client": "dfo-ts", + "version": "0.1.0", + "runtime": { + "name": "other" + }, + "value": "dfo-ts/0.1.0 (other)" + }, + { + "name": "ts-pre-release-package", + "client": "dfo-ts", + "version": "0.2.0-rc.1", + "runtime": { + "name": "Node", + "version": "24.4.1" + }, + "value": "dfo-ts/0.2.0-rc.1 (Node 24.4)" + }, + { + "name": "header-argument-naming-it-refused", + "via": "argument", + "headers": { + "X-DFO-Client": "my-app/1.0" + }, + "verdict": "refuse", + "reason": "header_invalid" + }, + { + "name": "lower-case-name-refused", + "via": "argument", + "headers": { + "x-dfo-client": "my-app/1.0" + }, + "verdict": "refuse", + "reason": "header_invalid" + }, + { + "name": "empty-value-refused", + "via": "argument", + "headers": { + "X-DFO-CLIENT": "" + }, + "verdict": "refuse", + "reason": "header_invalid" + }, + { + "name": "provider-naming-it-refused", + "via": "provider", + "headers": { + "Accept-Language": "en", + "X-Dfo-Client": "dfo-python/0.1.0 (CPython 3.14)" + }, + "verdict": "refuse", + "reason": "header_invalid" + }, + { + "name": "other-headers-accepted", + "via": "argument", + "headers": { + "X-Request-Id": "f3a9" + }, + "verdict": "accept" + }, + { + "name": "user-agent-accepted", + "via": "argument", + "headers": { + "User-Agent": "my-app/2.0" + }, + "verdict": "accept" + }, + { + "name": "longer-name-accepted", + "via": "provider", + "headers": { + "X-DFO-Client-Id": "notebook-7" + }, + "verdict": "accept" + }, + { + "name": "no-headers-accepted", + "via": "argument", + "headers": {}, + "verdict": "accept" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/file-selector.json b/python/client-contract/src/dfo_client_contract/rules/file-selector.json new file mode 100644 index 0000000..de7d35e --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/file-selector.json @@ -0,0 +1,47 @@ +{ + "format_version": 1, + "rule": "file-selector", + "query_keys": [ + "format", + "bundle", + "layout" + ], + "vectors": [ + { + "name": "long-cut", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + }, + "query": "format=parquet&bundle=false" + }, + { + "name": "long-plain", + "rendering": { + "format": "parquet", + "bundle": false + }, + "query": "format=parquet&bundle=false" + }, + { + "name": "frame-cut", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + }, + "query": "format=parquet&bundle=false&layout=frame" + }, + { + "name": "keys-in-listed-order", + "rendering": { + "layout": "frame", + "bundle": true, + "format": "csv" + }, + "query": "format=csv&bundle=true&layout=frame" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/operation-outputs.json b/python/client-contract/src/dfo_client_contract/rules/operation-outputs.json new file mode 100644 index 0000000..556fc17 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/operation-outputs.json @@ -0,0 +1,684 @@ +{ + "format_version": 1, + "rule": "operation-outputs", + "vectors": [ + { + "name": "share-of-in-percent", + "description": "ODA as a share of GNI comes back in per cent, 100 times the ratio, under the Percent unit, keyed by data's role. Ghana has no GNI row and Nigeria no ODA row, so neither has a share.", + "verb": "share_of", + "arguments": { + "on": { + "destination": "observationAbout" + } + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2023", + "value": 250.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 90.0, + "unit": "USDollar" + } + ], + "other": [ + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/KEN", + "time_period": "2022", + "value": 120000.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/KEN", + "time_period": "2023", + "value": 125000.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/NGA", + "time_period": "2022", + "value": 400000.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "destination": "country/KEN", + "time_period": "2022", + "value": 0.25, + "unit": "Percent" + }, + { + "destination": "country/KEN", + "time_period": "2023", + "value": 0.2, + "unit": "Percent" + } + ] + }, + { + "name": "share-of-across-members", + "description": "With no second reading, each member of by gets its share of the total across every member in the same year: donors giving 40 and 60 get 40 and 60 per cent. A donor's rows to several recipients add up first.", + "verb": "share_of", + "arguments": { + "by": [ + "origin" + ] + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "origin": "country/DEU", + "destination": "country/ETH", + "time_period": "2024", + "value": 40.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "origin": "country/FRA", + "destination": "country/ETH", + "time_period": "2024", + "value": 60.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "origin": "country/DEU", + "destination": "country/ETH", + "time_period": "2023", + "value": 30.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "origin": "country/FRA", + "destination": "country/ETH", + "time_period": "2023", + "value": 10.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "origin": "country/FRA", + "destination": "country/SOM", + "time_period": "2023", + "value": 10.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "origin": "country/DEU", + "time_period": "2024", + "value": 40.0, + "unit": "Percent" + }, + { + "origin": "country/FRA", + "time_period": "2024", + "value": 60.0, + "unit": "Percent" + }, + { + "origin": "country/DEU", + "time_period": "2023", + "value": 60.0, + "unit": "Percent" + }, + { + "origin": "country/FRA", + "time_period": "2023", + "value": 40.0, + "unit": "Percent" + } + ] + }, + { + "name": "join-long-format", + "description": "join returns both readings' matched rows in long format, one row per side, told apart by stat_var, each side keeping its own unit, the key named as data names it.", + "verb": "join", + "arguments": { + "on": { + "destination": "observationAbout" + } + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2023", + "value": 250.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 90.0, + "unit": "USDollar" + } + ], + "other": [ + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/KEN", + "time_period": "2022", + "value": 120000.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/KEN", + "time_period": "2023", + "value": 125000.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "observationAbout": "country/NGA", + "time_period": "2022", + "value": 400000.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "destination": "country/KEN", + "time_period": "2022", + "value": 120000.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2023", + "value": 250.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_EconomicActivity_GrossNationalIncome", + "destination": "country/KEN", + "time_period": "2023", + "value": 125000.0, + "unit": "USDollar" + } + ] + }, + { + "name": "compare-data-minus-other", + "description": "compare gives data minus other for each paired member and period both hold, in the unit both share.", + "verb": "compare", + "arguments": { + "on": { + "destination": "observationAbout" + } + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2023", + "value": 250.0, + "unit": "USDollar" + } + ], + "other": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_NetDisbursement", + "observationAbout": "country/KEN", + "time_period": "2022", + "value": 280.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_NetDisbursement", + "observationAbout": "country/KEN", + "time_period": "2023", + "value": 265.5, + "unit": "USDollar" + } + ], + "rows": [ + { + "destination": "country/KEN", + "time_period": "2022", + "value": 20.0, + "unit": "USDollar" + }, + { + "destination": "country/KEN", + "time_period": "2023", + "value": -15.5, + "unit": "USDollar" + } + ] + }, + { + "name": "sum-of-paired", + "description": "sum_of adds the paired rows of two readings over the periods both hold, in the unit both share.", + "verb": "sum_of", + "arguments": { + "on": { + "destination": "observationAbout" + } + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2023", + "value": 250.0, + "unit": "USDollar" + } + ], + "other": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_NetDisbursement", + "observationAbout": "country/KEN", + "time_period": "2022", + "value": 280.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_NetDisbursement", + "observationAbout": "country/KEN", + "time_period": "2023", + "value": 265.5, + "unit": "USDollar" + } + ], + "rows": [ + { + "destination": "country/KEN", + "time_period": "2022", + "value": 580.0, + "unit": "USDollar" + }, + { + "destination": "country/KEN", + "time_period": "2023", + "value": 515.5, + "unit": "USDollar" + } + ] + }, + { + "name": "growth-as-a-fraction", + "description": "A relative growth is a fraction in the value column, 0.25 for 25 per cent, with no unit, its period the window as a span. Nigeria has no 2018 value, so it has no growth.", + "verb": "growth", + "arguments": { + "by": [ + "destination" + ], + "window": [ + "2018", + "2022" + ] + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018", + "value": 200.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 250.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018", + "value": 80.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 70.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2022", + "value": 50.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018/2022", + "value": 0.25, + "unit": null + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018/2022", + "value": -0.125, + "unit": null + } + ] + }, + { + "name": "growth-as-an-amount", + "description": "An absolute growth is the change in the input's own unit.", + "verb": "growth", + "arguments": { + "by": [ + "destination" + ], + "window": [ + "2018", + "2022" + ], + "change": "absolute" + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018", + "value": 200.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 250.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018", + "value": 80.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 70.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2022", + "value": 50.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018/2022", + "value": 50.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018/2022", + "value": -10.0, + "unit": "USDollar" + } + ] + }, + { + "name": "growth-leaves-out-undefined-bases", + "description": "A relative growth leaves out a member whose start or end value is zero or whose values change sign, as zero_or_sign_crossing_base. Kenya grows by half; Ghana starts at zero, Nigeria ends at zero and Senegal crosses from positive to negative, so none of the three has a row.", + "verb": "growth", + "arguments": { + "by": [ + "destination" + ], + "window": [ + "2018", + "2022" + ] + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018", + "value": 200.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 300.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018", + "value": 0.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 70.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2018", + "value": 50.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2022", + "value": 0.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/SEN", + "time_period": "2018", + "value": 40.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/SEN", + "time_period": "2022", + "value": -10.0, + "unit": "USDollar" + } + ], + "rows": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018/2022", + "value": 0.5, + "unit": null + } + ] + }, + { + "name": "growth-amount-keeps-zero-bases", + "description": "An absolute growth is defined over a zero or sign-crossing base, so Ghana and Senegal keep their rows; a change that overflows to a non-finite value is left out as non_finite, so Nigeria has none.", + "verb": "growth", + "arguments": { + "by": [ + "destination" + ], + "window": [ + "2018", + "2022" + ], + "change": "absolute" + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018", + "value": 0.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2022", + "value": 70.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/SEN", + "time_period": "2018", + "value": 40.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/SEN", + "time_period": "2022", + "value": -10.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2018", + "value": -1e+308, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/NGA", + "time_period": "2022", + "value": 1e+308, + "unit": "USDollar" + } + ], + "rows": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/GHA", + "time_period": "2018/2022", + "value": 70.0, + "unit": "USDollar" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/SEN", + "time_period": "2018/2022", + "value": -50.0, + "unit": "USDollar" + } + ] + }, + { + "name": "growth-refuses-two-rows-of-one-member", + "description": "Two rows of Kenya in 2018, one per provider, leave its start value ambiguous under by destination alone, so growth refuses with role_ambiguous.", + "verb": "growth", + "arguments": { + "by": [ + "destination" + ], + "window": [ + "2018", + "2022" + ] + }, + "data": [ + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018", + "value": 120.0, + "unit": "USDollar", + "provider": "country/GBR" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2018", + "value": 80.0, + "unit": "USDollar", + "provider": "country/FRA" + }, + { + "stat_var": "dcid:Amount_FinancialFlow_ODA_Disbursement", + "destination": "country/KEN", + "time_period": "2022", + "value": 250.0, + "unit": "USDollar", + "provider": "country/GBR" + } + ], + "error": "role_ambiguous" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/output-comparison.json b/python/client-contract/src/dfo_client_contract/rules/output-comparison.json new file mode 100644 index 0000000..93b12b1 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/output-comparison.json @@ -0,0 +1,1618 @@ +{ + "format_version": 1, + "rule": "output-comparison", + "vectors": [ + { + "name": "same-derived-ref", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": true, + "same_data": true + }, + { + "name": "sentence-and-file-differ", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Another sentence for the same frame.", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:fe0f5e3c97d3d02c67422f55b887768edee4670f82971fce48579e3cb326e509", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": true, + "same_data": true + }, + { + "name": "rolling-and-fixed-recipe-same-output", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_a41f297b3f33f2a7c2ac6b3cedd4d3ad", + "recipe": { + "normal_form_hash": "rnf_5f5a142beef739694b06899a7f0426a7", + "replay_key": "rpk_4f697e3235da4b9ae1404f112c2c78b6" + } + }, + "content_version": "ocv_62a72043403b8a2497cc184623df87df", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": true + }, + { + "name": "engine-release-moves-no-number", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_f0b5865b2d90ff6a89f196b5d046bce6" + } + }, + "content_version": "ocv_8f0cd00ac4727431f3a3cbae4abc35e3", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.1|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": true + }, + { + "name": "newer-data-moves-the-output", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2021 to 2024. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_16602b92e22d0aaaf8f7e2cf41bcc925", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1ea0c543030b65a5680e7c653d6865a0" + } + }, + "content_version": "ocv_846c87c1b91ea260e927b34ab0f96775", + "output_digest": "out_a258ff67f0d7e61a18b8105da50821a0", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_06fe2b28793f70d868d047201166c31d", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2021" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2024" + } + ], + "artifact": { + "digest": "sha256:79fef3b30a845b187dfb0363282ffe64546fa77f72d75ca2dec2647da17f9304", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": false + }, + { + "name": "upstream-rebuild-moves-no-number", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_b4b38a6defb8a169442aada645297e5c", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_7ff8c2c2aacbc373e07f60dc51d614bf" + } + }, + "content_version": "ocv_845384eedabdd3f0ff6c637694981faf", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_ddc47df6c1e8526631c61fbe12affc09", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": true + }, + { + "name": "derived-file-format-differs", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:78892fc7549d08ac2af6f4be5362f9b232db89a2482980df97204655ba6ef97a", + "rendering": { + "format": "csv", + "bundle": false + } + } + } + ] + }, + "equal": false, + "same_data": true + }, + { + "name": "same-dataset-ref", + "left": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "right": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "equal": true, + "same_data": true + }, + { + "name": "dataset-layout-differs", + "left": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "right": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + } + ] + }, + "equal": false, + "same_data": true + }, + { + "name": "dataset-party-totals-differ", + "left": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "right": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false + } + } + } + ] + }, + "equal": false, + "same_data": false + }, + { + "name": "dataset-and-derived", + "left": { + "format_version": 1, + "members": [ + { + "parts": [], + "sentence": "Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Current US dollars · OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "definition": { + "schema_version": 1, + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "crsCooperationModality", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsPurpose", + "binding": { + "state": "value", + "ref": "CRSPurposeCode12220" + } + }, + { + "property": "crsTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "crsTypeOfFlow", + "binding": { + "state": "value", + "ref": "CRSTypeOfFlow10" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/NGA" + ], + "origin": [ + "country/DEU" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2020, + "end_year": 2023 + } + } + } + }, + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "artifact": { + "digest": "sha256:c44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170", + "rendering": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": false + }, + { + "name": "member-count-differs", + "left": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "right": { + "format_version": 2, + "members": [ + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_e1edc6acb97615bdd9e813730378ce19", + "recipe": { + "normal_form_hash": "rnf_4e2298354b4323699033912586859aa7", + "replay_key": "rpk_1185a76368dca1b0301c899170c7dff3" + } + }, + "content_version": "ocv_3187efa2294f182bfe7cab296659b74f", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [ + { + "anchor": "latest", + "offset": -3, + "rule": "half1", + "period": "2020" + }, + { + "anchor": "latest", + "offset": 0, + "rule": "half1", + "period": "2023" + } + ], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + }, + { + "parts": [], + "sentence": "Change in Germany's gross ODA disbursements to Nigeria for basic health care, 2020 to 2023. Computed by DFO from OECD Creditor Reporting System (CRS).", + "spec": { + "spec_id": "ops_a41f297b3f33f2a7c2ac6b3cedd4d3ad", + "recipe": { + "normal_form_hash": "rnf_5f5a142beef739694b06899a7f0426a7", + "replay_key": "rpk_4f697e3235da4b9ae1404f112c2c78b6" + } + }, + "content_version": "ocv_62a72043403b8a2497cc184623df87df", + "output_digest": "out_82b81c6278499b468aa3d6a9c5a6aa6b", + "roots": [ + { + "spec_id": "spec_cb5baa4161de718f18583060674dbca6", + "content_version": "cv_bf242138f8d0a892c97da9373d1626f4", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame", + "without_party_totals": true + } + } + ], + "execution_identity": "0.9.0|1.1.0|0.2.0", + "resolved_periods": [], + "artifact": { + "digest": "sha256:4c8fec27543c7bedbe9d2c436f42c5a65f721106c128d0a55d2962c2c67de033", + "rendering": { + "format": "parquet", + "bundle": false, + "layout": "frame" + } + } + } + ] + }, + "equal": false, + "same_data": false + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/output-digest.json b/python/client-contract/src/dfo_client_contract/rules/output-digest.json new file mode 100644 index 0000000..fa57304 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/output-digest.json @@ -0,0 +1,900 @@ +{ + "format_version": 1, + "rule": "output-digest", + "vectors": [ + { + "name": "two-rows", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"USDollar\"]]}", + "digest": "out_fb35c1b4f4a67b6ca2ecd8f334220dfa" + }, + { + "name": "another-stat-var", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "USDollar", + null + ], + [ + "dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"USDollar\"],[\"dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"USDollar\"]]}", + "digest": "out_2753cf795f19e151b39cc9c6d14e698d" + }, + { + "name": "rows-in-another-order", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"USDollar\"]]}", + "digest": "out_fb35c1b4f4a67b6ca2ecd8f334220dfa" + }, + { + "name": "facets-left-out", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "measurementMethod", + "provenance", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "USDollar", + "dcid:OECDReported", + "dcid:OECD_DAC2a", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "USDollar", + "dcid:OECDReported", + "dcid:OECD_DAC2a", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"USDollar\"]]}", + "digest": "out_fb35c1b4f4a67b6ca2ecd8f334220dfa" + }, + { + "name": "another-value", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.5, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.5,\"USDollar\"]]}", + "digest": "out_a2e2612bf1d8ec3e5e94ccb9d601e028" + }, + { + "name": "another-unit", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + "Euro", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "Euro", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,\"Euro\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"Euro\"]]}", + "digest": "out_f79a304d277ad9a080a37d71f4e5af39" + }, + { + "name": "another-period", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2023", + 120.5, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "country/DEU", + "2022", + 98.25, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2023\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"country/DEU\",\"2022\",98.25,\"USDollar\"]]}", + "digest": "out_51e11cc3d1e0228562e557a15a9ed8cf" + }, + { + "name": "another-role-name", + "frame": { + "columns": [ + "stat_var", + "observationAbout", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2022", + 120.5, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"observationAbout\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2022\",120.5,\"USDollar\"]]}", + "digest": "out_1694c016b8cd08bb08e68151d4c2a9bc" + }, + { + "name": "converted-unit", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + { + "currency": "EUR", + "prices": "constant", + "base_year": 2021 + }, + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,{\"base_year\":2021,\"currency\":\"EUR\",\"prices\":\"constant\"}]]}", + "digest": "out_e7f45286159d090b59d8c07efba3311c" + }, + { + "name": "converted-unit-recipe-left-out", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "country/DEU", + "2022", + 120.5, + { + "currency": "EUR", + "prices": "constant", + "base_year": 2021, + "recipe": "fx_oecd_2025_10" + }, + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"country/DEU\",\"2022\",120.5,{\"base_year\":2021,\"currency\":\"EUR\",\"prices\":\"constant\"}]]}", + "digest": "out_e7f45286159d090b59d8c07efba3311c" + }, + { + "name": "composed-unit-recipe-left-out", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c", + "country/KEN", + "2022", + 2.25, + { + "numerator": { + "currency": "EUR", + "prices": "constant", + "base_year": 2021, + "recipe": "fx_oecd_2025_10" + }, + "denominator": { + "stat_var": "dcid:Count_Person" + } + }, + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c\",\"country/KEN\",\"2022\",2.25,{\"denominator\":{\"stat_var\":\"dcid:Count_Person\"},\"numerator\":{\"base_year\":2021,\"currency\":\"EUR\",\"prices\":\"constant\"}}]]}", + "digest": "out_9dcf831992537dbf2dd2cabe816ee483" + }, + { + "name": "composed-unit", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c", + "country/KEN", + "2022", + 2.25, + { + "numerator": { + "currency": "EUR", + "prices": "constant", + "base_year": 2021 + }, + "denominator": { + "stat_var": "dcid:Count_Person" + } + }, + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:ops_4b1e0c9a7d2f3e5b6a8c9d0e1f2a3b4c\",\"country/KEN\",\"2022\",2.25,{\"denominator\":{\"stat_var\":\"dcid:Count_Person\"},\"numerator\":{\"base_year\":2021,\"currency\":\"EUR\",\"prices\":\"constant\"}}]]}", + "digest": "out_9dcf831992537dbf2dd2cabe816ee483" + }, + { + "name": "join-of-two-variables", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2022", + 120.5, + "USDollar", + null + ], + [ + "dcid:Amount_EconomicActivity_GrossNationalIncome", + "country/KEN", + "2022", + 104000.0, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_EconomicActivity_GrossNationalIncome\",\"country/KEN\",\"2022\",104000,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2022\",120.5,\"USDollar\"]]}", + "digest": "out_e010a9208379d7d1acec4994d4218d18" + }, + { + "name": "join-sides-swapped", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_EconomicActivity_GrossNationalIncome", + "country/KEN", + "2022", + 104000.0, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2022", + 120.5, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_EconomicActivity_GrossNationalIncome\",\"country/KEN\",\"2022\",104000,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2022\",120.5,\"USDollar\"]]}", + "digest": "out_e010a9208379d7d1acec4994d4218d18" + }, + { + "name": "join-values-under-swapped-variables", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_EconomicActivity_GrossNationalIncome", + "country/KEN", + "2022", + 120.5, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2022", + 104000.0, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_EconomicActivity_GrossNationalIncome\",\"country/KEN\",\"2022\",120.5,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2022\",104000,\"USDollar\"]]}", + "digest": "out_ef49e17ee52f27ccdccb6709fe12a404" + }, + { + "name": "quotient-of-variable-named", + "frame": { + "columns": [ + "stat_var", + "financingRecipient", + "financingRecipient_party_kind", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/KEN", + "party", + "2021", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/KEN", + "party", + "2022", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/NGA", + "party", + "2021", + 0.1, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/NGA", + "party", + "2022", + 0.14285714285714285, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ] + ] + }, + "variables": { + "ops_f72f69cb463d4fdbeb6751800ba2544a": "mv_d355e59c50ff8cf07e86a4da12b64297" + }, + "name_documents": { + "mv_d355e59c50ff8cf07e86a4da12b64297": "{\"denominator_frame_ref\":[\"dcid:Count_Person\"],\"frame_ref\":[\"dcid:Amount_ODA\"],\"verb\":\"quotient_of\"}" + }, + "document": "{\"columns\":[\"stat_var\",\"financingRecipient\",\"financingRecipient_party_kind\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/KEN\",\"party\",\"2021\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/KEN\",\"party\",\"2022\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/NGA\",\"party\",\"2021\",0.1,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/NGA\",\"party\",\"2022\",0.14285714285714285,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}]]}", + "digest": "out_e64e2b613cf8ceb4e3b0cd93a54cf5ce" + }, + { + "name": "filter-then-quotient-of", + "frame": { + "columns": [ + "stat_var", + "financingRecipient", + "financingRecipient_party_kind", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_664db264b67e0ba2cec6b95c18ea4635", + "country/KEN", + "party", + "2021", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_664db264b67e0ba2cec6b95c18ea4635", + "country/KEN", + "party", + "2022", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_664db264b67e0ba2cec6b95c18ea4635", + "country/NGA", + "party", + "2021", + 0.1, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_664db264b67e0ba2cec6b95c18ea4635", + "country/NGA", + "party", + "2022", + 0.14285714285714285, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ] + ] + }, + "variables": { + "ops_664db264b67e0ba2cec6b95c18ea4635": "mv_d355e59c50ff8cf07e86a4da12b64297" + }, + "name_documents": { + "mv_d355e59c50ff8cf07e86a4da12b64297": "{\"denominator_frame_ref\":[\"dcid:Count_Person\"],\"frame_ref\":[\"dcid:Amount_ODA\"],\"verb\":\"quotient_of\"}" + }, + "document": "{\"columns\":[\"stat_var\",\"financingRecipient\",\"financingRecipient_party_kind\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/KEN\",\"party\",\"2021\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/KEN\",\"party\",\"2022\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/NGA\",\"party\",\"2021\",0.1,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_d355e59c50ff8cf07e86a4da12b64297\",\"country/NGA\",\"party\",\"2022\",0.14285714285714285,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}]]}", + "digest": "out_e64e2b613cf8ceb4e3b0cd93a54cf5ce" + }, + { + "name": "quotient-of-sides-swapped", + "frame": { + "columns": [ + "stat_var", + "financingRecipient", + "financingRecipient_party_kind", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/KEN", + "party", + "2021", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/KEN", + "party", + "2022", + 0.2, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/NGA", + "party", + "2021", + 0.1, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_f72f69cb463d4fdbeb6751800ba2544a", + "country/NGA", + "party", + "2022", + 0.14285714285714285, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ] + ] + }, + "variables": { + "ops_f72f69cb463d4fdbeb6751800ba2544a": "mv_df0088e6b8552d16ee6000c8b7da3814" + }, + "name_documents": { + "mv_df0088e6b8552d16ee6000c8b7da3814": "{\"denominator_frame_ref\":[\"dcid:Amount_ODA\"],\"frame_ref\":[\"dcid:Count_Person\"],\"verb\":\"quotient_of\"}" + }, + "document": "{\"columns\":[\"stat_var\",\"financingRecipient\",\"financingRecipient_party_kind\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"mv_df0088e6b8552d16ee6000c8b7da3814\",\"country/KEN\",\"party\",\"2021\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_df0088e6b8552d16ee6000c8b7da3814\",\"country/KEN\",\"party\",\"2022\",0.2,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_df0088e6b8552d16ee6000c8b7da3814\",\"country/NGA\",\"party\",\"2021\",0.1,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_df0088e6b8552d16ee6000c8b7da3814\",\"country/NGA\",\"party\",\"2022\",0.14285714285714285,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}]]}", + "digest": "out_7b765553f4377c65a8c7799c070be936" + }, + { + "name": "quotient-of-a-sum-of", + "frame": { + "columns": [ + "stat_var", + "financingRecipient", + "financingRecipient_party_kind", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:ops_a3a85219d81e742e831d19c7966e562f", + "country/KEN", + "party", + "2021", + 0.6, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_a3a85219d81e742e831d19c7966e562f", + "country/KEN", + "party", + "2022", + 0.6, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_a3a85219d81e742e831d19c7966e562f", + "country/NGA", + "party", + "2021", + 0.3, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ], + [ + "dcid:ops_a3a85219d81e742e831d19c7966e562f", + "country/NGA", + "party", + "2022", + 0.42857142857142855, + { + "numerator": "USDollar", + "denominator": "Person" + }, + null + ] + ] + }, + "variables": { + "ops_a3a85219d81e742e831d19c7966e562f": "mv_288cc4adaed3cc3195d7e8c6cc4f87aa" + }, + "name_documents": { + "mv_ae90cefe711d4addb439f873f2010dba": "{\"frame_ref\":[\"dcid:Amount_ODA\"],\"right_frame_ref\":[\"dcid:Amount_ODA\"],\"verb\":\"sum_of\"}", + "mv_288cc4adaed3cc3195d7e8c6cc4f87aa": "{\"denominator_frame_ref\":[\"dcid:Count_Person\"],\"frame_ref\":[\"mv_ae90cefe711d4addb439f873f2010dba\"],\"verb\":\"quotient_of\"}" + }, + "document": "{\"columns\":[\"stat_var\",\"financingRecipient\",\"financingRecipient_party_kind\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"mv_288cc4adaed3cc3195d7e8c6cc4f87aa\",\"country/KEN\",\"party\",\"2021\",0.6,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_288cc4adaed3cc3195d7e8c6cc4f87aa\",\"country/KEN\",\"party\",\"2022\",0.6,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_288cc4adaed3cc3195d7e8c6cc4f87aa\",\"country/NGA\",\"party\",\"2021\",0.3,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}],[\"mv_288cc4adaed3cc3195d7e8c6cc4f87aa\",\"country/NGA\",\"party\",\"2022\",0.42857142857142855,{\"denominator\":\"Person\",\"numerator\":\"USDollar\"}]]}", + "digest": "out_61999d3b24a82e4c28621d0fe434959a" + }, + { + "name": "relative-growth", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2018/2022", + 0.25, + null, + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/NGA", + "2018/2022", + -0.125, + null, + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2018/2022\",0.25,null],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/NGA\",\"2018/2022\",-0.125,null]]}", + "digest": "out_f648b4fa7a61f8a10daf34eb6b1e8e01" + }, + { + "name": "number-spellings", + "frame": { + "columns": [ + "stat_var", + "destination", + "time_period", + "value", + "unit", + "status" + ], + "rows": [ + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2020", + 1e+21, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2021", + 1.5e-07, + "USDollar", + null + ], + [ + "dcid:Amount_FinancialFlow_ODA_Disbursement", + "country/KEN", + "2022", + 3.0, + "USDollar", + null + ] + ] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2020\",1e+21,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2021\",1.5e-7,\"USDollar\"],[\"dcid:Amount_FinancialFlow_ODA_Disbursement\",\"country/KEN\",\"2022\",3,\"USDollar\"]]}", + "digest": "out_c02c263d6ff798a41efd17e1872382d5" + }, + { + "name": "empty-frame", + "frame": { + "columns": [ + "stat_var", + "destination", + "origin", + "time_period", + "value", + "unit", + "status" + ], + "rows": [] + }, + "document": "{\"columns\":[\"stat_var\",\"destination\",\"origin\",\"time_period\",\"value\",\"unit\"],\"digest_version\":\"out2\",\"rows\":[]}", + "digest": "out_a8fab64c203e3de84f95af840aa6257c" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/publication.json b/python/client-contract/src/dfo_client_contract/rules/publication.json new file mode 100644 index 0000000..55b36eb --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/publication.json @@ -0,0 +1,444 @@ +{ + "format_version": 1, + "rule": "publication", + "vectors": [ + { + "name": "live-replace-keeps-slots", + "description": "Replacing a live share's cards keeps its slots, so each slot's embed and data URLs draw the new card.", + "steps": [ + { + "op": "share", + "share": "tile", + "latest": true, + "cards": [ + "cmp_a", + "cmp_b" + ] + }, + { + "op": "replace", + "share": "tile", + "cards": [ + "cmp_c", + "cmp_d" + ], + "write_token": "held" + } + ], + "outcomes": [ + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + }, + { + "slot": 2, + "card": "cmp_b" + } + ] + } + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_c" + }, + { + "slot": 2, + "card": "cmp_d" + } + ] + } + } + ] + }, + { + "name": "live-replace-with-fewer-cards", + "description": "A replacement with fewer cards leaves the later slots empty, and a later one with more fills them again, under the same URLs.", + "steps": [ + { + "op": "share", + "share": "tile", + "latest": true, + "cards": [ + "cmp_a", + "cmp_b", + "cmp_c" + ] + }, + { + "op": "replace", + "share": "tile", + "cards": [ + "cmp_d" + ], + "write_token": "held" + }, + { + "op": "replace", + "share": "tile", + "cards": [ + "cmp_a", + "cmp_b", + "cmp_e" + ], + "write_token": "held" + } + ], + "outcomes": [ + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + }, + { + "slot": 2, + "card": "cmp_b" + }, + { + "slot": 3, + "card": "cmp_c" + } + ] + } + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_d" + }, + { + "slot": 2, + "card": null + }, + { + "slot": 3, + "card": null + } + ] + } + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + }, + { + "slot": 2, + "card": "cmp_b" + }, + { + "slot": 3, + "card": "cmp_e" + } + ] + } + } + ] + }, + { + "name": "snapshot-replace-refused", + "description": "A snapshot cannot be replaced, even with its write token, so a cited URL draws the same figure for good.", + "steps": [ + { + "op": "share", + "share": "figure", + "latest": false, + "cards": [ + "cmp_a" + ] + }, + { + "op": "replace", + "share": "figure", + "cards": [ + "cmp_b" + ], + "write_token": "held" + }, + { + "op": "read", + "share": "figure" + } + ], + "outcomes": [ + { + "share": { + "latest": false, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "error": "share_not_replaceable" + }, + { + "share": { + "latest": false, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + } + ] + }, + { + "name": "replace-without-token-refused", + "description": "Replacing a share needs its write token.", + "steps": [ + { + "op": "share", + "share": "tile", + "latest": true, + "cards": [ + "cmp_a" + ] + }, + { + "op": "replace", + "share": "tile", + "cards": [ + "cmp_b" + ] + }, + { + "op": "read", + "share": "tile" + } + ], + "outcomes": [ + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "error": "write_token_required" + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + } + ] + }, + { + "name": "replace-with-another-token-refused", + "description": "A write token other than the share's is refused, and the check comes before the share's mode, so a caller without the token learns nothing else.", + "steps": [ + { + "op": "share", + "share": "figure", + "latest": false, + "cards": [ + "cmp_a" + ] + }, + { + "op": "replace", + "share": "figure", + "cards": [ + "cmp_b" + ], + "write_token": "other" + } + ], + "outcomes": [ + { + "share": { + "latest": false, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "error": "write_token_invalid" + } + ] + }, + { + "name": "tokenless-share-stays-fixed", + "description": "A share created with no write token, as a guest's share migrates, refuses every token, so it stays as it is.", + "steps": [ + { + "op": "share", + "share": "guest", + "latest": true, + "cards": [ + "cmp_a" + ], + "tokenless": true + }, + { + "op": "replace", + "share": "guest", + "cards": [ + "cmp_b" + ], + "write_token": "held" + }, + { + "op": "unshare", + "share": "guest", + "write_token": "held" + }, + { + "op": "read", + "share": "guest" + } + ], + "outcomes": [ + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "error": "write_token_invalid" + }, + { + "error": "write_token_invalid" + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + } + ] + }, + { + "name": "unshare-snapshot", + "description": "A snapshot can be taken down with its write token.", + "steps": [ + { + "op": "share", + "share": "figure", + "latest": false, + "cards": [ + "cmp_a" + ] + }, + { + "op": "unshare", + "share": "figure", + "write_token": "held" + } + ], + "outcomes": [ + { + "share": { + "latest": false, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "removed": true + } + ] + }, + { + "name": "unshare-without-token-refused", + "description": "Taking a share down needs its write token.", + "steps": [ + { + "op": "share", + "share": "tile", + "latest": true, + "cards": [ + "cmp_a" + ] + }, + { + "op": "unshare", + "share": "tile" + }, + { + "op": "read", + "share": "tile" + } + ], + "outcomes": [ + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + }, + { + "error": "write_token_required" + }, + { + "share": { + "latest": true, + "slots": [ + { + "slot": 1, + "card": "cmp_a" + } + ] + } + } + ] + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/reading-order.json b/python/client-contract/src/dfo_client_contract/rules/reading-order.json new file mode 100644 index 0000000..a2a318f --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/reading-order.json @@ -0,0 +1,1527 @@ +{ + "format_version": 1, + "rule": "reading-order", + "reading_columns": [ + [ + "time_period" + ], + [ + "financingProvider_label", + "financingProvider" + ], + [ + "financingRecipient_label", + "financingRecipient" + ], + [ + "observationAbout_label", + "observationAbout" + ], + [ + "crsPurpose_label", + "stat_var_crsPurpose" + ], + [ + "value" + ], + [ + "unit_label", + "unit" + ], + [ + "crsTypeOfFlow_label", + "stat_var_crsTypeOfFlow", + "dac2aAidType_label", + "stat_var_dac2aAidType" + ], + [ + "provenance_label", + "provenance" + ] + ], + "vectors": [ + { + "name": "varying-column-follows-reading-columns", + "members": [ + { + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingRecipient", + "financingRecipient_label", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsPurpose_label", + "crsTypeOfFlow", + "crsTypeOfFlow_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/oda_health_grant", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2021, + 5.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ], + [ + "dc/sv/oda_health_loan", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "C01", + "Project-type interventions", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2020, + 7.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ], + [ + "dc/sv/oda_health_grant", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2020, + 4.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ], + [ + "dc/sv/oda_health_loan", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "C01", + "Project-type interventions", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2021, + 8.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ] + ] + } + ], + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "crsCooperationModality", + "crsCooperationModality_label", + "financingProvider", + "financingRecipient", + "crsPurpose", + "crsTypeOfFlow", + "unit", + "currency", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2020, + "Germany", + "Nigeria", + "Basic health care", + 4.0, + "US dollars", + "ODA", + "OECD CRS", + "A02", + "Sector budget support", + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USDollar", + "USD", + "provenance/CRS", + "dc/sv/oda_health_grant" + ], + [ + 2021, + "Germany", + "Nigeria", + "Basic health care", + 5.0, + "US dollars", + "ODA", + "OECD CRS", + "A02", + "Sector budget support", + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USDollar", + "USD", + "provenance/CRS", + "dc/sv/oda_health_grant" + ], + [ + 2020, + "Germany", + "Nigeria", + "Basic health care", + 7.0, + "US dollars", + "ODA", + "OECD CRS", + "C01", + "Project-type interventions", + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USDollar", + "USD", + "provenance/CRS", + "dc/sv/oda_health_loan" + ], + [ + 2021, + "Germany", + "Nigeria", + "Basic health care", + 8.0, + "US dollars", + "ODA", + "OECD CRS", + "C01", + "Project-type interventions", + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USDollar", + "USD", + "provenance/CRS", + "dc/sv/oda_health_loan" + ] + ] + }, + { + "name": "all-constant-table", + "members": [ + { + "columns": [ + "stat_var", + "stat_var_label", + "observationAbout", + "observationAbout_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "base_year", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2023, + 108.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 113.4, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ] + ] + } + ], + "columns": [ + "time_period", + "observationAbout_label", + "value", + "unit_label", + "provenance_label", + "stat_var_label", + "observationAbout", + "unit", + "currency", + "base_year", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2022, + "Kenya", + 113.4, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + 2023, + "Kenya", + 108.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ] + ] + }, + { + "name": "one-row-table-is-constant", + "members": [ + { + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingRecipient", + "financingRecipient_label", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsPurpose_label", + "crsTypeOfFlow", + "crsTypeOfFlow_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/oda_health_grant", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2021, + 5.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ] + ] + } + ], + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsTypeOfFlow", + "unit", + "currency", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2021, + "Germany", + "Nigeria", + "Basic health care", + 5.0, + "US dollars", + "ODA", + "OECD CRS", + "country/DEU", + "country/NGA", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USDollar", + "USD", + "provenance/CRS", + "dc/sv/oda_health_grant" + ] + ] + }, + { + "name": "empty-table", + "members": [ + { + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingRecipient", + "financingRecipient_label", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsPurpose_label", + "crsTypeOfFlow", + "crsTypeOfFlow_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "provenance", + "provenance_label" + ], + "rows": [] + } + ], + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsTypeOfFlow", + "unit", + "currency", + "provenance", + "stat_var" + ], + "rows": [] + }, + { + "name": "frame-layout-codes", + "members": [ + { + "columns": [ + "stat_var", + "financingProvider", + "financingRecipient", + "time_period", + "value", + "unit", + "measurementMethod", + "provenance", + "status", + "stat_var_crsCooperationModality", + "stat_var_crsPurpose", + "stat_var_crsTypeOfFinance", + "stat_var_crsTypeOfFlow" + ], + "rows": [ + [ + "dc/sv/oda_health_grant", + "country/DEU", + "country/NGA", + 2021, + 5.0, + "USDollar", + "Gross", + "provenance/CRS", + null, + "A02", + "CRSPurposeCode12220", + "CRSTypeOfFinance110", + "CRSTypeOfFlow10" + ], + [ + "dc/sv/oda_health_grant", + "country/DEU", + "country/NGA", + 2020, + 4.0, + "USDollar", + "Gross", + "provenance/CRS", + "provisional", + "A02", + "CRSPurposeCode12220", + "CRSTypeOfFinance110", + "CRSTypeOfFlow10" + ] + ] + } + ], + "columns": [ + "time_period", + "financingProvider", + "financingRecipient", + "stat_var_crsPurpose", + "value", + "unit", + "stat_var_crsTypeOfFlow", + "provenance", + "status", + "measurementMethod", + "stat_var_crsCooperationModality", + "stat_var_crsTypeOfFinance", + "stat_var" + ], + "rows": [ + [ + 2020, + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + 4.0, + "USDollar", + "CRSTypeOfFlow10", + "provenance/CRS", + "provisional", + "Gross", + "A02", + "CRSTypeOfFinance110", + "dc/sv/oda_health_grant" + ], + [ + 2021, + "country/DEU", + "country/NGA", + "CRSPurposeCode12220", + 5.0, + "USDollar", + "CRSTypeOfFlow10", + "provenance/CRS", + null, + "Gross", + "A02", + "CRSTypeOfFinance110", + "dc/sv/oda_health_grant" + ] + ] + }, + { + "name": "null-sorts-last", + "members": [ + { + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_label", + "dac2aAidType", + "dac2aAidType_label", + "time_period", + "value", + "unit", + "unit_label", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/oda_dac2a", + null, + null, + "unallocated", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2022, + 1.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ], + [ + "dc/sv/oda_dac2a", + "country/FRA", + "France", + "official", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2022, + 2.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ], + [ + "dc/sv/oda_dac2a", + "country/DEU", + "Germany", + "official", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2022, + 3.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ], + [ + "dc/sv/oda_dac2a", + "country/DEU", + "Germany", + "official", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2021, + 4.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ] + ] + } + ], + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "dac2aAidType", + "unit", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2022, + "France", + "Kenya", + 2.0, + "US dollars", + "ODA: Total Net", + "OECD DAC2a", + "country/FRA", + "official", + "country/KEN", + "DAC2aAidType206", + "USDollar", + "provenance/DAC2a", + "dc/sv/oda_dac2a" + ], + [ + 2021, + "Germany", + "Kenya", + 4.0, + "US dollars", + "ODA: Total Net", + "OECD DAC2a", + "country/DEU", + "official", + "country/KEN", + "DAC2aAidType206", + "USDollar", + "provenance/DAC2a", + "dc/sv/oda_dac2a" + ], + [ + 2022, + "Germany", + "Kenya", + 3.0, + "US dollars", + "ODA: Total Net", + "OECD DAC2a", + "country/DEU", + "official", + "country/KEN", + "DAC2aAidType206", + "USDollar", + "provenance/DAC2a", + "dc/sv/oda_dac2a" + ], + [ + 2022, + null, + "Kenya", + 1.0, + "US dollars", + "ODA: Total Net", + "OECD DAC2a", + null, + "unallocated", + "country/KEN", + "DAC2aAidType206", + "USDollar", + "provenance/DAC2a", + "dc/sv/oda_dac2a" + ] + ] + }, + { + "name": "stat-var-varying-keys-series", + "members": [ + { + "columns": [ + "stat_var", + "stat_var_label", + "observationAbout", + "observationAbout_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "base_year", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/gdp_ppp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 120.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2021, + 110.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp_ppp", + "GDP", + "country/KEN", + "Kenya", + 2021, + 118.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 113.4, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ] + ] + } + ], + "columns": [ + "time_period", + "observationAbout_label", + "value", + "unit_label", + "provenance_label", + "stat_var_label", + "observationAbout", + "unit", + "currency", + "base_year", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2021, + "Kenya", + 110.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + 2022, + "Kenya", + 113.4, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + 2021, + "Kenya", + 118.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp_ppp" + ], + [ + 2022, + "Kenya", + 120.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp_ppp" + ] + ] + }, + { + "name": "ties-keep-served-order", + "members": [ + { + "columns": [ + "stat_var", + "stat_var_label", + "observationAbout", + "observationAbout_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "base_year", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 2.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 1.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ], + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2021, + 3.0, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ] + ] + } + ], + "columns": [ + "time_period", + "observationAbout_label", + "value", + "unit_label", + "provenance_label", + "stat_var_label", + "observationAbout", + "unit", + "currency", + "base_year", + "provenance", + "stat_var" + ], + "rows": [ + [ + 2021, + "Kenya", + 3.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + 2022, + "Kenya", + 2.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + 2022, + "Kenya", + 1.0, + "US dollars", + "World Bank WDI", + "GDP", + "country/KEN", + "USDollar", + "USD", + null, + "provenance/WDI", + "dc/sv/gdp" + ] + ] + }, + { + "name": "stack-crs-and-dac2a", + "members": [ + { + "part": "health ODA", + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingRecipient", + "financingRecipient_label", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsPurpose_label", + "crsTypeOfFlow", + "crsTypeOfFlow_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/oda_health_grant", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2021, + 5.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ], + [ + "dc/sv/oda_health_grant", + "country/DEU", + "Germany", + "country/NGA", + "Nigeria", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "Basic health care", + "CRSTypeOfFlow10", + "ODA", + 2020, + 4.0, + "USDollar", + "US dollars", + "USD", + "provenance/CRS", + "OECD CRS" + ] + ] + }, + { + "part": "total ODA", + "columns": [ + "stat_var", + "financingProvider", + "financingProvider_label", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_label", + "dac2aAidType", + "dac2aAidType_label", + "time_period", + "value", + "unit", + "unit_label", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/oda_dac2a", + "country/DEU", + "Germany", + "official", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2021, + 40.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ], + [ + "dc/sv/oda_dac2a", + "country/DEU", + "Germany", + "official", + "country/KEN", + "Kenya", + "DAC2aAidType206", + "ODA: Total Net", + 2020, + 30.0, + "USDollar", + "US dollars", + "provenance/DAC2a", + "OECD DAC2a" + ] + ] + } + ], + "columns": [ + "part", + "time_period", + "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingRecipient", + "crsCooperationModality", + "crsCooperationModality_label", + "crsPurpose", + "crsTypeOfFlow", + "currency", + "provenance", + "financingProvider_kind", + "dac2aAidType", + "dac2aAidType_label", + "financingProvider", + "unit", + "stat_var" + ], + "rows": [ + [ + "health ODA", + 2020, + "Germany", + "Nigeria", + "Basic health care", + 4.0, + "US dollars", + "ODA", + "OECD CRS", + "country/NGA", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USD", + "provenance/CRS", + null, + null, + null, + "country/DEU", + "USDollar", + "dc/sv/oda_health_grant" + ], + [ + "health ODA", + 2021, + "Germany", + "Nigeria", + "Basic health care", + 5.0, + "US dollars", + "ODA", + "OECD CRS", + "country/NGA", + "A02", + "Sector budget support", + "CRSPurposeCode12220", + "CRSTypeOfFlow10", + "USD", + "provenance/CRS", + null, + null, + null, + "country/DEU", + "USDollar", + "dc/sv/oda_health_grant" + ], + [ + "total ODA", + 2020, + "Germany", + "Kenya", + null, + 30.0, + "US dollars", + null, + "OECD DAC2a", + "country/KEN", + null, + null, + null, + null, + null, + "provenance/DAC2a", + "official", + "DAC2aAidType206", + "ODA: Total Net", + "country/DEU", + "USDollar", + "dc/sv/oda_dac2a" + ], + [ + "total ODA", + 2021, + "Germany", + "Kenya", + null, + 40.0, + "US dollars", + null, + "OECD DAC2a", + "country/KEN", + null, + null, + null, + null, + null, + "provenance/DAC2a", + "official", + "DAC2aAidType206", + "ODA: Total Net", + "country/DEU", + "USDollar", + "dc/sv/oda_dac2a" + ] + ] + }, + { + "name": "stack-constant-per-member-varies-in-stack", + "members": [ + { + "part": "GDP in dollars", + "columns": [ + "stat_var", + "stat_var_label", + "observationAbout", + "observationAbout_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "base_year", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 113.4, + "USDollar", + "US dollars", + "USD", + null, + "provenance/WDI", + "World Bank WDI" + ] + ] + }, + { + "part": "GDP in shillings", + "columns": [ + "stat_var", + "stat_var_label", + "observationAbout", + "observationAbout_label", + "time_period", + "value", + "unit", + "unit_label", + "currency", + "base_year", + "provenance", + "provenance_label" + ], + "rows": [ + [ + "dc/sv/gdp", + "GDP", + "country/KEN", + "Kenya", + 2022, + 15.1, + "USDollar", + "US dollars", + "KES", + null, + "provenance/WDI", + "World Bank WDI" + ] + ] + } + ], + "columns": [ + "part", + "time_period", + "observationAbout_label", + "value", + "unit_label", + "provenance_label", + "currency", + "stat_var_label", + "observationAbout", + "unit", + "base_year", + "provenance", + "stat_var" + ], + "rows": [ + [ + "GDP in dollars", + 2022, + "Kenya", + 113.4, + "US dollars", + "World Bank WDI", + "USD", + "GDP", + "country/KEN", + "USDollar", + null, + "provenance/WDI", + "dc/sv/gdp" + ], + [ + "GDP in shillings", + 2022, + "Kenya", + 15.1, + "US dollars", + "World Bank WDI", + "KES", + "GDP", + "country/KEN", + "USDollar", + null, + "provenance/WDI", + "dc/sv/gdp" + ] + ] + }, + { + "name": "integer-and-float-are-one-value", + "members": [ + { + "columns": [ + "time_period", + "value", + "currency", + "base_year" + ], + "rows": [ + [ + 2021, + 5, + "USD", + 2020 + ], + [ + 2020, + 4, + "USD", + 2020.0 + ] + ] + } + ], + "columns": [ + "time_period", + "value", + "currency", + "base_year" + ], + "rows": [ + [ + 2020, + 4, + "USD", + 2020.0 + ], + [ + 2021, + 5, + "USD", + 2020 + ] + ] + }, + { + "name": "nested-cells-compare-by-value", + "members": [ + { + "columns": [ + "time_period", + "value", + "meta", + "tags", + "flags" + ], + "rows": [ + [ + 2021, + 5, + { + "a": 1, + "b": 2 + }, + [ + 1 + ], + [ + true + ] + ], + [ + 2020, + 4, + { + "b": 2, + "a": 1 + }, + [ + 1.0 + ], + [ + false + ] + ] + ] + } + ], + "columns": [ + "time_period", + "value", + "flags", + "meta", + "tags" + ], + "rows": [ + [ + 2020, + 4, + [ + false + ], + { + "b": 2, + "a": 1 + }, + [ + 1.0 + ] + ], + [ + 2021, + 5, + [ + true + ], + { + "a": 1, + "b": 2 + }, + [ + 1 + ] + ] + ] + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/role-keys.json b/python/client-contract/src/dfo_client_contract/rules/role-keys.json new file mode 100644 index 0000000..1ac9f25 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/role-keys.json @@ -0,0 +1,355 @@ +{ + "format_version": 1, + "rule": "role-keys", + "keys": { + "place": "observationAbout", + "origin": "origin", + "destination": "destination" + }, + "flow_keys": [ + "origin", + "destination" + ], + "vectors": [ + { + "name": "subject-place", + "identity": { + "entities": { + "observationAbout": [ + "country/KEN" + ] + } + }, + "roles": { + "place": [ + "country/KEN" + ] + }, + "entities": { + "place": [ + "country/NGA", + "country/GHA" + ] + }, + "posted": { + "observationAbout": [ + "country/NGA", + "country/GHA" + ] + } + }, + { + "name": "subject-wire-role-refused", + "identity": { + "entities": { + "observationAbout": [ + "country/KEN" + ] + } + }, + "roles": { + "place": [ + "country/KEN" + ] + }, + "entities": { + "observationAbout": [ + "country/NGA" + ] + }, + "error": { + "reason": "invalid_argument", + "names": [ + "observationAbout", + "place" + ] + } + }, + { + "name": "subject-flow-key-refused", + "identity": { + "entities": { + "observationAbout": [ + "country/KEN" + ] + } + }, + "roles": { + "place": [ + "country/KEN" + ] + }, + "entities": { + "origin": [ + "country/DEU" + ] + }, + "error": { + "reason": "invalid_argument", + "names": [ + "origin", + "place" + ] + } + }, + { + "name": "flow-origin-rebound", + "identity": { + "entities": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + } + }, + "roles": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + }, + "entities": { + "origin": [ + "country/FRA" + ] + }, + "posted": { + "origin": [ + "country/FRA" + ], + "destination": [ + "country/NGA" + ] + } + }, + { + "name": "flow-open-origin-bound", + "identity": { + "entities": { + "destination": [ + "country/KEN" + ] + }, + "open_roles": [ + "origin" + ] + }, + "roles": { + "destination": [ + "country/KEN" + ], + "origin": null + }, + "entities": { + "origin": [ + "country/GBR" + ], + "destination": [ + "country/ETH" + ] + }, + "posted": { + "destination": [ + "country/ETH" + ], + "origin": [ + "country/GBR" + ] + } + }, + { + "name": "flow-total-direction-bound", + "identity": { + "entities": { + "destination": [ + "country/ETH" + ] + } + }, + "roles": { + "destination": [ + "country/ETH" + ], + "origin": null + }, + "entities": { + "origin": [ + "country/USA" + ] + }, + "posted": { + "destination": [ + "country/ETH" + ], + "origin": [ + "country/USA" + ] + } + }, + { + "name": "flow-direction-dropped", + "identity": { + "entities": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + } + }, + "roles": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + }, + "entities": { + "destination": null + }, + "posted": { + "origin": [ + "country/DEU" + ] + } + }, + { + "name": "flow-column-name-refused", + "identity": { + "entities": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + } + }, + "roles": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + }, + "entities": { + "financingRecipient": [ + "country/GHA" + ] + }, + "error": { + "reason": "invalid_argument", + "names": [ + "financingRecipient", + "origin", + "destination" + ] + } + }, + { + "name": "flow-place-refused", + "identity": { + "entities": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + } + }, + "roles": { + "origin": [ + "country/DEU" + ], + "destination": [ + "country/NGA" + ] + }, + "entities": { + "place": [ + "country/GHA" + ] + }, + "error": { + "reason": "invalid_argument", + "names": [ + "place", + "origin", + "destination" + ] + } + }, + { + "name": "flow-open-destination-roles", + "identity": { + "entities": {}, + "open_roles": [ + "destination" + ] + }, + "roles": { + "destination": null, + "origin": null + } + }, + { + "name": "no-roles-place", + "identity": { + "entities": {} + }, + "roles": {}, + "entities": { + "place": [ + "country/KEN" + ] + }, + "posted": { + "observationAbout": [ + "country/KEN" + ] + } + }, + { + "name": "no-roles-other-key-posted", + "identity": { + "entities": {} + }, + "roles": {}, + "entities": { + "financingRecipient": [ + "country/KEN" + ] + }, + "posted": { + "financingRecipient": [ + "country/KEN" + ] + } + }, + { + "name": "no-roles-wire-role-refused", + "identity": {}, + "roles": {}, + "entities": { + "observationAbout": [ + "country/KEN" + ] + }, + "error": { + "reason": "invalid_argument", + "names": [ + "observationAbout", + "place" + ] + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/rolling-composition.json b/python/client-contract/src/dfo_client_contract/rules/rolling-composition.json new file mode 100644 index 0000000..b7f555a --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/rolling-composition.json @@ -0,0 +1,530 @@ +{ + "format_version": 1, + "rule": "rolling-composition", + "vectors": [ + { + "name": "one-rolling-step", + "description": "One rolling step takes the newest period at least half of every population's series hold: Kenya alone holds 2024, so 2023.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": {} + } + ] + } + ], + "period": "2023" + }, + { + "name": "two-steps-one-population", + "description": "Two rolling steps that state the same populations, however they list them, share one latest period.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": { + "destination": [ + "country/KEN", + "country/GHA" + ] + } + } + ] + }, + { + "step": "rank", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": { + "destination": [ + "country/GHA", + "country/KEN" + ] + } + } + ] + } + ], + "period": "2024" + }, + { + "name": "one-population-per-root", + "description": "A tree over two roots counts the latest period over both: each population must hold it.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": {} + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "bindings": {} + } + ] + }, + { + "step": "growth", + "population": [ + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "bindings": {} + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": {} + } + ] + } + ], + "period": "2023" + }, + { + "name": "same-year-other-series-refused", + "description": "Two rolling steps over different series are refused even when each would settle on the same year, 2024 here, so one latest period is never counted twice.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": { + "destination": [ + "country/KEN", + "country/GHA" + ] + } + } + ] + }, + { + "step": "rank", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": { + "destination": [ + "country/KEN", + "country/NGA" + ] + } + } + ] + } + ], + "error": "rolling_populations_differ" + }, + { + "name": "other-year-refused", + "description": "Two rolling steps over different populations that would settle on different years are refused.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": {} + } + ] + }, + { + "step": "growth", + "population": [ + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "bindings": {} + } + ] + } + ], + "error": "rolling_populations_differ" + }, + { + "name": "empty-population-unresolved", + "description": "A population no series falls in names no latest period.", + "series": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "coordinates": { + "destination": "country/NGA" + }, + "periods": [ + "2021", + "2022", + "2023" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/KEN" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + }, + { + "spec_id": "spec_1a3c5e7f9b2d4f6a8c0e2b4d6f8a0c3e", + "coordinates": { + "observationAbout": "country/GHA" + }, + "periods": [ + "2021", + "2022", + "2023", + "2024" + ] + } + ], + "steps": [ + { + "step": "growth", + "population": [ + { + "spec_id": "spec_6f0c2a9d4b7e1f3a5c8d0b2e4f6a8c1d", + "bindings": { + "destination": [ + "country/SOM" + ] + } + } + ] + } + ], + "error": "rolling_anchor_unresolved" + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/rules/rules.schema.json b/python/client-contract/src/dfo_client_contract/rules/rules.schema.json index 9a33072..e8dd62a 100644 --- a/python/client-contract/src/dfo_client_contract/rules/rules.schema.json +++ b/python/client-contract/src/dfo_client_contract/rules/rules.schema.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/DevelopmentFinanceObservatory/spec-kit/main/python/client-contract/src/dfo_client_contract/rules/rules.schema.json", "title": "Client rule vectors", - "description": "One file per rule, each holding the vectors a client's implementation of that rule must reproduce. rules/README.md states each rule.", + "description": "One file per rule, each holding the vectors an implementation of that rule must reproduce: a client's for the client rules, DFO's for the analysis and publication rules. rules/README.md states each rule.", "type": "object", "oneOf": [ { @@ -17,11 +17,44 @@ { "$ref": "#/$defs/key_lookup" }, + { + "$ref": "#/$defs/base_url" + }, { "$ref": "#/$defs/cache_dir" }, { "$ref": "#/$defs/cache_layout" + }, + { + "$ref": "#/$defs/reading_order" + }, + { + "$ref": "#/$defs/client_identity" + }, + { + "$ref": "#/$defs/role_keys" + }, + { + "$ref": "#/$defs/output_digest" + }, + { + "$ref": "#/$defs/output_comparison" + }, + { + "$ref": "#/$defs/operation_outputs" + }, + { + "$ref": "#/$defs/rolling_composition" + }, + { + "$ref": "#/$defs/publication" + }, + { + "$ref": "#/$defs/series_left_out" + }, + { + "$ref": "#/$defs/file_selector" } ], "$defs": { @@ -78,6 +111,10 @@ }, "layout": { "const": "frame" + }, + "without_party_totals": { + "description": "Present, and true, for a rendering with party totals left out.", + "const": true } } }, @@ -365,6 +402,74 @@ } } }, + "base_url": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "base-url" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "env" + ], + "oneOf": [ + { + "required": [ + "source", + "url" + ] + }, + { + "required": [ + "error" + ] + } + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "base_url": { + "description": "The base URL argument of the call or constructor, which may be the empty string a client refuses. Absent when the caller passes none.", + "type": "string" + }, + "env": { + "$ref": "#/$defs/env" + }, + "source": { + "enum": [ + "argument", + "environment", + "default" + ] + }, + "url": { + "type": "string", + "minLength": 1 + }, + "error": { + "type": "object", + "additionalProperties": false, + "required": [ + "reason" + ], + "properties": { + "reason": { + "const": "base_url_invalid" + } + } + } + } + } + } + } + }, "cache_dir": { "$ref": "#/$defs/suite", "additionalProperties": false, @@ -514,6 +619,1476 @@ } } } + }, + "client_identity": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "required": [ + "header", + "pattern" + ], + "properties": { + "format_version": true, + "rule": { + "const": "client-identity" + }, + "header": { + "const": "X-DFO-Client" + }, + "pattern": { + "description": "A regular expression every well-formed header value matches in full.", + "type": "string", + "minLength": 1 + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "oneOf": [ + { + "required": [ + "client", + "version", + "runtime", + "value" + ], + "not": { + "anyOf": [ + { + "required": [ + "via" + ] + }, + { + "required": [ + "headers" + ] + }, + { + "required": [ + "verdict" + ] + } + ] + } + }, + { + "required": [ + "via", + "headers", + "verdict" + ], + "not": { + "anyOf": [ + { + "required": [ + "client" + ] + }, + { + "required": [ + "version" + ] + }, + { + "required": [ + "runtime" + ] + }, + { + "required": [ + "value" + ] + } + ] + } + } + ], + "if": { + "properties": { + "verdict": { + "const": "refuse" + } + }, + "required": [ + "verdict" + ] + }, + "then": { + "required": [ + "reason" + ] + }, + "else": { + "not": { + "required": [ + "reason" + ] + } + }, + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "client": { + "enum": [ + "dfo-python", + "dfo-ts" + ] + }, + "version": { + "description": "The package version as the package states it.", + "type": "string", + "pattern": "^[0-9A-Za-z.+-]+$", + "$ref": "#/$defs/one_line" + }, + "runtime": { + "description": "The runtime's name and its full version, or `browser` or `other` with no version.", + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "properties": { + "name": { + "type": "string", + "pattern": "^[A-Za-z]+$", + "$ref": "#/$defs/one_line" + }, + "version": { + "type": "string", + "pattern": "^[0-9]+\\.[0-9]+" + } + }, + "if": { + "properties": { + "name": { + "enum": [ + "browser", + "other" + ] + } + } + }, + "then": { + "not": { + "required": [ + "version" + ] + } + }, + "else": { + "required": [ + "version" + ] + } + }, + "value": { + "description": "The X-DFO-Client value the client sends.", + "type": "string", + "minLength": 1 + }, + "via": { + "description": "Where the caller's headers come from: the client's headers argument, or the header provider it calls before each request.", + "enum": [ + "argument", + "provider" + ] + }, + "headers": { + "description": "The caller's headers, by name as the caller writes it.", + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "verdict": { + "enum": [ + "accept", + "refuse" + ] + }, + "reason": { + "const": "header_invalid" + } + } + } + } + } + }, + "reading_order": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "required": [ + "reading_columns" + ], + "properties": { + "format_version": true, + "rule": { + "const": "reading-order" + }, + "reading_columns": { + "description": "The reading slots in order. Each slot lists the column names that fill it, each long layout's label before its frame layout's code; the first name a dataset holds fills the slot.", + "type": "array", + "minItems": 1, + "items": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + }, + "uniqueItems": true + } + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "members", + "columns", + "rows" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "members": { + "description": "The members a dataset stacks, in member order, each its served table: its `part` value when the dataset has several members, its columns and its rows in the order the server wrote them.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "columns", + "rows" + ], + "properties": { + "part": { + "type": "string", + "minLength": 1 + }, + "columns": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + }, + "uniqueItems": true + }, + "rows": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": [ + "string", + "number", + "boolean", + "null", + "array", + "object" + ] + } + } + } + } + } + }, + "columns": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + }, + "uniqueItems": true, + "description": "The columns a client presents, in its order." + }, + "rows": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": [ + "string", + "number", + "boolean", + "null", + "array", + "object" + ] + } + }, + "description": "The rows a client presents, in its order, each in the presented column order." + } + } + } + } + } + }, + "role_keys": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "required": [ + "keys", + "flow_keys" + ], + "properties": { + "format_version": true, + "rule": { + "const": "role-keys" + }, + "keys": { + "description": "Each key a caller types in entities=, to the wire role it posts as.", + "type": "object", + "additionalProperties": false, + "required": [ + "place", + "origin", + "destination" + ], + "properties": { + "place": { + "const": "observationAbout" + }, + "origin": { + "const": "origin" + }, + "destination": { + "const": "destination" + } + } + }, + "flow_keys": { + "description": "The flow directions. A spec that names one takes both.", + "const": [ + "origin", + "destination" + ] + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "identity", + "roles" + ], + "oneOf": [ + { + "not": { + "anyOf": [ + { + "required": [ + "entities" + ] + }, + { + "required": [ + "posted" + ] + }, + { + "required": [ + "error" + ] + } + ] + } + }, + { + "required": [ + "entities", + "posted" + ], + "not": { + "required": [ + "error" + ] + } + }, + { + "required": [ + "entities", + "error" + ], + "not": { + "required": [ + "posted" + ] + } + } + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "identity": { + "description": "The spec's definition.identity members that name roles, by wire role.", + "type": "object", + "additionalProperties": false, + "properties": { + "entities": { + "type": "object", + "propertyNames": { + "enum": [ + "observationAbout", + "origin", + "destination" + ] + }, + "additionalProperties": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1 + } + } + }, + "open_roles": { + "type": "array", + "items": { + "enum": [ + "observationAbout", + "origin", + "destination" + ] + }, + "uniqueItems": true + } + } + }, + "roles": { + "description": "The caller keys the spec takes, in order, each to the dcids it binds or null: Spec.roles in Python, specRoles(spec) in TypeScript.", + "type": "object", + "propertyNames": { + "enum": [ + "place", + "origin", + "destination" + ] + }, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1 + } + }, + { + "type": "null" + } + ] + } + }, + "entities": { + "description": "The caller's entities=, each key to dcids or to null, which drops the role.", + "type": "object", + "minProperties": 1, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1 + } + }, + { + "type": "null" + } + ] + } + }, + "posted": { + "description": "The build's scope.entities, by wire role.", + "type": "object", + "additionalProperties": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1 + } + } + }, + "error": { + "type": "object", + "additionalProperties": false, + "required": [ + "reason", + "names" + ], + "properties": { + "reason": { + "const": "invalid_argument" + }, + "names": { + "description": "The keys the message names: the key typed, then the spec's caller keys, or place for observationAbout on a spec that names no role.", + "type": "array", + "minItems": 2, + "items": { + "type": "string", + "minLength": 1 + } + } + } + } + } + } + } + } + }, + "output_digest": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "output-digest" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "frame", + "document", + "digest" + ], + "dependentRequired": { + "variables": [ + "name_documents" + ] + }, + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "frame": { + "type": "object", + "additionalProperties": false, + "required": [ + "columns", + "rows" + ], + "properties": { + "columns": { + "description": "The frame's columns in the frame grammar's order.", + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + } + }, + "rows": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": [ + "string", + "number", + "null", + "object" + ] + } + } + } + }, + "description": "A frame as docs/frame-grammar-v1.md defines it, each row in its column order." + }, + "variables": { + "description": "The name of each variable an operation of the frame's recipe wrote, keyed by the operation's ops_ id, which the frame's stat_var cells spell after dcid:.", + "type": "object", + "propertyNames": { + "pattern": "^ops_[0-9a-f]{32}$" + }, + "additionalProperties": { + "type": "string", + "pattern": "^mv_[0-9a-f]{32}$" + } + }, + "name_documents": { + "description": "The name document of each name in variables and of each mv_ name those documents hold, as the UTF-8 text the name hashes, keyed by the name.", + "type": "object", + "minProperties": 1, + "propertyNames": { + "pattern": "^mv_[0-9a-f]{32}$" + }, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "document": { + "description": "The output digest document, as the UTF-8 text the digest hashes.", + "type": "string", + "minLength": 1 + }, + "digest": { + "type": "string", + "pattern": "^out_[0-9a-f]{32}$" + } + } + } + } + } + }, + "output_comparison": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "output-comparison" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "left", + "right", + "equal", + "same_data" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "left": { + "$ref": "../dataset-ref.schema.json" + }, + "right": { + "$ref": "../dataset-ref.schema.json" + }, + "equal": { + "description": "Whether the refs name the same versions, recipes and renderings: `==` in Python, `equals` in TypeScript.", + "type": "boolean" + }, + "same_data": { + "description": "Whether the refs hold the same output: `same_data` in Python, `sameData` in TypeScript.", + "type": "boolean" + } + } + } + } + } + }, + "operation_outputs": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "operation-outputs" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "description", + "verb", + "arguments", + "data" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "description": { + "type": "string", + "minLength": 1 + }, + "verb": { + "enum": [ + "share_of", + "join", + "compare", + "sum_of", + "growth" + ] + }, + "arguments": { + "type": "object", + "additionalProperties": false, + "required": [], + "properties": { + "on": { + "description": "Each role of data, to the role of other it pairs with.", + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "by": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1 + } + }, + "window": { + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": { + "type": "string", + "pattern": "^[0-9]{4}$" + } + }, + "change": { + "enum": [ + "relative", + "absolute" + ] + } + } + }, + "data": { + "type": "array", + "items": { + "type": "object", + "required": [ + "time_period", + "value", + "unit" + ], + "properties": { + "stat_var": { + "type": "string", + "pattern": "^dcid:." + }, + "time_period": { + "type": "string", + "minLength": 1 + }, + "value": { + "type": "number" + }, + "unit": { + "type": [ + "string", + "number", + "null", + "object" + ] + } + }, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "description": "The rows of the first reading, each its frame row by column name." + }, + "other": { + "type": "array", + "items": { + "type": "object", + "required": [ + "time_period", + "value", + "unit" + ], + "properties": { + "stat_var": { + "type": "string", + "pattern": "^dcid:." + }, + "time_period": { + "type": "string", + "minLength": 1 + }, + "value": { + "type": "number" + }, + "unit": { + "type": [ + "string", + "number", + "null", + "object" + ] + } + }, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "description": "The rows of the second reading, for an operation that takes one." + }, + "rows": { + "type": "array", + "items": { + "type": "object", + "required": [ + "time_period", + "value", + "unit" + ], + "properties": { + "stat_var": { + "type": "string", + "pattern": "^dcid:." + }, + "time_period": { + "type": "string", + "minLength": 1 + }, + "value": { + "type": "number" + }, + "unit": { + "type": [ + "string", + "number", + "null", + "object" + ] + } + }, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "description": "The output rows, in any order. A row leaves out stat_var when the operation writes a variable of its own." + }, + "error": { + "description": "The code the operation refuses with in place of rows.", + "enum": [ + "role_ambiguous" + ] + } + }, + "oneOf": [ + { + "required": [ + "rows" + ] + }, + { + "required": [ + "error" + ] + } + ] + } + } + } + }, + "rolling_composition": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "rolling-composition" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "description", + "series", + "steps" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "description": { + "type": "string", + "minLength": 1 + }, + "series": { + "description": "Every series of the roots, by its root's spec_id and coordinates, with the periods it holds a row at.", + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "spec_id", + "coordinates", + "periods" + ], + "properties": { + "spec_id": { + "type": "string", + "pattern": "^spec_[0-9a-f]{32}$" + }, + "coordinates": { + "type": "object", + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "periods": { + "type": "array", + "uniqueItems": true, + "items": { + "type": "string", + "pattern": "^[0-9]{4}$" + } + } + } + } + }, + "steps": { + "description": "The rolling steps of one recipe tree, each with the populations it counts the latest period over.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "step", + "population" + ], + "properties": { + "step": { + "type": "string", + "minLength": 1 + }, + "population": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "spec_id" + ], + "properties": { + "spec_id": { + "type": "string", + "pattern": "^spec_[0-9a-f]{32}$" + }, + "bindings": { + "type": "object", + "additionalProperties": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1 + } + } + } + } + } + } + } + } + }, + "period": { + "type": "string", + "pattern": "^[0-9]{4}$" + }, + "error": { + "enum": [ + "rolling_populations_differ", + "rolling_anchor_unresolved" + ] + } + }, + "oneOf": [ + { + "required": [ + "period" + ] + }, + { + "required": [ + "error" + ] + } + ] + } + } + } + }, + "publication": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "publication" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "description", + "steps", + "outcomes" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "description": { + "type": "string", + "minLength": 1 + }, + "steps": { + "type": "array", + "minItems": 1, + "items": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "op", + "share", + "latest", + "cards" + ], + "properties": { + "op": { + "const": "share" + }, + "share": { + "type": "string", + "minLength": 1 + }, + "latest": { + "type": "boolean" + }, + "cards": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^cmp_[0-9a-z]+$" + } + }, + "tokenless": { + "description": "The share holds no write token, as a guest's share migrates.", + "const": true + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "op", + "share", + "cards" + ], + "properties": { + "op": { + "const": "replace" + }, + "share": { + "type": "string", + "minLength": 1 + }, + "cards": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^cmp_[0-9a-z]+$" + } + }, + "write_token": { + "description": "`held` for the token the share's creation returned, `other` for another; absent sends none.", + "enum": [ + "held", + "other" + ] + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "op", + "share" + ], + "properties": { + "op": { + "const": "unshare" + }, + "share": { + "type": "string", + "minLength": 1 + }, + "write_token": { + "description": "`held` for the token the share's creation returned, `other` for another; absent sends none.", + "enum": [ + "held", + "other" + ] + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "op", + "share" + ], + "properties": { + "op": { + "const": "read" + }, + "share": { + "type": "string", + "minLength": 1 + } + } + } + ] + } + }, + "outcomes": { + "description": "One outcome per step.", + "type": "array", + "minItems": 1, + "items": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "share" + ], + "properties": { + "share": { + "type": "object", + "additionalProperties": false, + "required": [ + "latest", + "slots" + ], + "properties": { + "latest": { + "type": "boolean" + }, + "slots": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "slot", + "card" + ], + "properties": { + "slot": { + "type": "integer", + "minimum": 1 + }, + "card": { + "oneOf": [ + { + "type": "string", + "pattern": "^cmp_[0-9a-z]+$" + }, + { + "type": "null" + } + ] + } + } + } + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "removed" + ], + "properties": { + "removed": { + "const": true + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "error" + ], + "properties": { + "error": { + "enum": [ + "write_token_required", + "write_token_invalid", + "share_not_replaceable" + ] + } + } + } + ] + } + } + } + } + } + } + }, + "series_left_out": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "properties": { + "format_version": true, + "rule": { + "const": "series-left-out" + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "description", + "members", + "strict" + ], + "oneOf": [ + { + "required": [ + "warnings" + ] + }, + { + "required": [ + "raises" + ] + } + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "description": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + }, + "members": { + "description": "The left-out series of each derived member one fetch returns, in member order, as its derived manifest serves them.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "left_out", + "left_out_complete" + ], + "oneOf": [ + { + "properties": { + "left_out": { + "type": "null" + }, + "left_out_complete": { + "type": "null" + } + } + }, + { + "properties": { + "left_out": { + "type": "array" + }, + "left_out_complete": { + "type": "boolean" + } + } + } + ], + "properties": { + "left_out": { + "description": "Every series a step of the recipe dropped for a data gap; null for a recipe stored without them.", + "type": [ + "array", + "null" + ], + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "coordinates", + "step", + "side", + "reason", + "rows" + ], + "properties": { + "coordinates": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "string", + "minLength": 1 + } + }, + "step": { + "type": "object", + "additionalProperties": false, + "required": [ + "verb", + "node" + ], + "properties": { + "verb": { + "type": "string", + "minLength": 1 + }, + "node": { + "type": "integer", + "minimum": 0 + } + } + }, + "side": { + "enum": [ + "data", + "other" + ] + }, + "reason": { + "type": "string", + "minLength": 1 + }, + "rows": { + "type": "integer", + "minimum": 1 + } + } + } + }, + "left_out_complete": { + "description": "False when the steps dropped more series than `left_out` lists; null with a null `left_out`.", + "type": [ + "boolean", + "null" + ] + } + } + } + }, + "strict": { + "type": "boolean" + }, + "warnings": { + "description": "The warnings the fetch issues, each once.", + "type": "array", + "maxItems": 1, + "items": { + "enum": [ + "series_left_out" + ] + } + }, + "raises": { + "description": "The warning a strict fetch raises in place of its data.", + "enum": [ + "series_left_out" + ] + } + } + } + } + } + }, + "file_selector": { + "$ref": "#/$defs/suite", + "additionalProperties": false, + "required": [ + "query_keys" + ], + "properties": { + "format_version": true, + "rule": { + "const": "file-selector" + }, + "query_keys": { + "description": "The rendering keys the file selector reads, in the order a client writes them into its query.", + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "$ref": "#/$defs/one_line" + } + }, + "vectors": { + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "rendering", + "query" + ], + "properties": { + "name": { + "$ref": "#/$defs/name" + }, + "rendering": { + "$ref": "#/$defs/rendering" + }, + "query": { + "description": "The selector's query a client sends for the rendering, without the leading `?`.", + "type": "string", + "minLength": 1 + } + } + } + } + } } } } diff --git a/python/client-contract/src/dfo_client_contract/rules/series-left-out.json b/python/client-contract/src/dfo_client_contract/rules/series-left-out.json new file mode 100644 index 0000000..a0afe22 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/rules/series-left-out.json @@ -0,0 +1,199 @@ +{ + "format_version": 1, + "rule": "series-left-out", + "vectors": [ + { + "name": "growth-left-series-out", + "description": "A growth dropped Nigeria, which has no value at the window's end, so the fetch issues series_left_out and returns the frame.", + "members": [ + { + "left_out": [ + { + "coordinates": { + "financingRecipient": "country/NGA" + }, + "step": { + "verb": "growth", + "node": 0 + }, + "side": "data", + "reason": "endpoint_missing", + "rows": 1 + } + ], + "left_out_complete": true + } + ], + "strict": false, + "warnings": [ + "series_left_out" + ] + }, + { + "name": "nothing-left-out", + "description": "A manifest that lists no left-out series and says the list is complete issues nothing.", + "members": [ + { + "left_out": [], + "left_out_complete": true + } + ], + "strict": false, + "warnings": [] + }, + { + "name": "recipe-stored-without-left-out", + "description": "A recipe stored without its left-out series has a null left_out and left_out_complete and issues nothing.", + "members": [ + { + "left_out": null, + "left_out_complete": null + } + ], + "strict": false, + "warnings": [] + }, + { + "name": "list-incomplete", + "description": "A manifest whose list stops short of every left-out series, left_out_complete false, issues series_left_out.", + "members": [ + { + "left_out": [ + { + "coordinates": { + "financingRecipient": "country/GHA" + }, + "step": { + "verb": "quotient_of", + "node": 1 + }, + "side": "data", + "reason": "unmatched_key", + "rows": 2 + } + ], + "left_out_complete": false + } + ], + "strict": false, + "warnings": [ + "series_left_out" + ] + }, + { + "name": "incomplete-with-none-listed", + "description": "left_out_complete false issues series_left_out even when left_out lists no series.", + "members": [ + { + "left_out": [], + "left_out_complete": false + } + ], + "strict": false, + "warnings": [ + "series_left_out" + ] + }, + { + "name": "several-members-warn-once", + "description": "Two members whose manifests each list left-out series issue one series_left_out for the fetch.", + "members": [ + { + "left_out": [ + { + "coordinates": { + "financingRecipient": "country/NGA" + }, + "step": { + "verb": "growth", + "node": 0 + }, + "side": "data", + "reason": "endpoint_missing", + "rows": 1 + } + ], + "left_out_complete": true + }, + { + "left_out": null, + "left_out_complete": null + }, + { + "left_out": [ + { + "coordinates": { + "financingRecipient": "country/GHA" + }, + "step": { + "verb": "quotient_of", + "node": 1 + }, + "side": "data", + "reason": "unmatched_key", + "rows": 2 + }, + { + "coordinates": { + "financingRecipient": "country/NGA" + }, + "step": { + "verb": "quotient_of", + "node": 1 + }, + "side": "data", + "reason": "period_unmatched", + "rows": 1 + } + ], + "left_out_complete": true + } + ], + "strict": false, + "warnings": [ + "series_left_out" + ] + }, + { + "name": "strict-raises", + "description": "Under strict, a fetch whose member's manifest lists left-out series raises series_left_out in place of its data.", + "members": [ + { + "left_out": null, + "left_out_complete": null + }, + { + "left_out": [ + { + "coordinates": { + "financingRecipient": "country/GHA" + }, + "step": { + "verb": "growth", + "node": 0 + }, + "side": "data", + "reason": "zero_or_sign_crossing_base", + "rows": 2 + } + ], + "left_out_complete": true + } + ], + "strict": true, + "raises": "series_left_out" + }, + { + "name": "strict-without-left-out-series", + "description": "Under strict, a fetch whose manifests list no left-out series returns its data with no warning.", + "members": [ + { + "left_out": [], + "left_out_complete": true + } + ], + "strict": true, + "warnings": [] + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenario.schema.json b/python/client-contract/src/dfo_client_contract/scenario.schema.json index 5499a73..d730cd8 100644 --- a/python/client-contract/src/dfo_client_contract/scenario.schema.json +++ b/python/client-contract/src/dfo_client_contract/scenario.schema.json @@ -222,35 +222,44 @@ ] }, "step": { - "oneOf": [ - { - "$ref": "#/$defs/search_step" - }, - { - "$ref": "#/$defs/choose_step" - }, - { - "$ref": "#/$defs/fetch_answer_step" - }, - { - "$ref": "#/$defs/fetch_spec_step" - }, - { - "$ref": "#/$defs/fetch_ref_step" - }, - { - "$ref": "#/$defs/newest_version_step" - }, - { - "$ref": "#/$defs/mint_step" - }, - { - "$ref": "#/$defs/operation_step" - }, - { - "$ref": "#/$defs/result_step" - } - ] + "description": "A step with a `ref` is a fetch of that ref, so a broken ref is reported at the member that breaks it; any other step is one of the other kinds.", + "if": { + "type": "object", + "required": [ + "ref" + ] + }, + "then": { + "$ref": "#/$defs/fetch_ref_step" + }, + "else": { + "oneOf": [ + { + "$ref": "#/$defs/search_step" + }, + { + "$ref": "#/$defs/choose_step" + }, + { + "$ref": "#/$defs/fetch_answer_step" + }, + { + "$ref": "#/$defs/fetch_spec_step" + }, + { + "$ref": "#/$defs/newest_version_step" + }, + { + "$ref": "#/$defs/mint_step" + }, + { + "$ref": "#/$defs/operation_step" + }, + { + "$ref": "#/$defs/result_step" + } + ] + } }, "search_step": { "type": "object", @@ -326,6 +335,10 @@ }, "strict": { "type": "boolean" + }, + "party_totals": { + "description": "True keeps the group totals of the identity's open party roles, so the build posts no `without_party_totals`. Absent or false leaves them out.", + "type": "boolean" } } }, @@ -350,6 +363,10 @@ "strict": { "type": "boolean" }, + "party_totals": { + "description": "True keeps the group totals of the identity's open party roles, so the build posts no `without_party_totals`. Absent or false leaves them out.", + "type": "boolean" + }, "window": true, "entities": true, "facets": true @@ -368,7 +385,19 @@ "const": "fetch" }, "ref": { - "$ref": "dataset-ref.schema.json" + "allOf": [ + { + "$ref": "dataset-ref.schema.json" + }, + { + "properties": { + "format_version": { + "description": "Format 1, a ref of dataset members. No scenario serves a derived member yet, so a format 2 ref fails here.", + "const": 1 + } + } + } + ] } } }, @@ -390,7 +419,7 @@ } }, "mint_step": { - "description": "A mint of a spec, which starts its build and answers the operation; it takes a layout and no narrowing.", + "description": "A mint of a spec, which starts its build and answers the operation; it takes a layout and party totals as a fetch of a spec does, and no narrowing.", "type": "object", "additionalProperties": false, "required": [ @@ -406,6 +435,10 @@ }, "layout": { "$ref": "#/$defs/layout" + }, + "party_totals": { + "description": "True keeps the group totals of the identity's open party roles, so the build posts no `without_party_totals`. Absent or false leaves them out.", + "type": "boolean" } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-candidate-specs-not-a-list.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-candidate-specs-not-a-list.json index 20a8675..00abc2b 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-candidate-specs-not-a-list.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-candidate-specs-not-a-list.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -41,22 +41,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -67,28 +67,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -810,7 +810,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1593,6 +1593,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1809,6 +1818,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2344,7 +2362,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2579,7 +2597,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0" + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-candidates-choose-fetch.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-candidates-choose-fetch.json index 524967e..6da0d9d 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-candidates-choose-fetch.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-candidates-choose-fetch.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:07Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:01Z" }, "call": [ { @@ -38,7 +38,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -49,22 +49,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -75,28 +75,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -818,7 +818,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1601,6 +1601,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1817,6 +1826,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2352,7 +2370,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2633,7 +2651,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bb50ec0b45244567bf259cee9e6de857" + "x-dfo-trace-id": "97f100748b194ca09afc06c9dc1ac81d" }, "json": { "schema_version": 1, @@ -2642,11 +2660,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bb50ec0b45244567bf259cee9e6de857", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "97f100748b194ca09afc06c9dc1ac81d", "warnings": [], "timing_ms": { - "total": 486, + "total": 345, "by_step": { "coverage": 0 } @@ -2659,19 +2677,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 436, + "total": 284, "by_step": { - "validate": 1, - "walk": 182, + "validate": 0, + "walk": 0, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 158, - "bind": 1, + "shape": 174, + "bind": 0, "where_places": 0, - "materialise": 93, - "answer": 1, - "options": 0 + "materialise": 107, + "answer": 2, + "options": 1 } }, "entity_refs": [], @@ -3190,7 +3208,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -3403,7 +3421,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3411,7 +3430,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "cbcf0bde995644728537c2ed3abc531e" + "x-dfo-trace-id": "1607014b90c1472db06c480ad25623d1" }, "json": { "schema_version": 1, @@ -3619,7 +3638,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:17.280841+00:00" + "confirmed_at": "2026-10-26T09:17:50.952592+00:00" }, "warnings": [], "diagnostics": [ @@ -3635,7 +3654,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 5000, "media_type": "application/vnd.apache.parquet", @@ -3657,7 +3677,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -3796,7 +3816,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-22T17:34:00.253769Z" @@ -3814,7 +3834,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A38f1b5db24567c68f9791ac00bce803082c607f23c78688edf2d95e0778a070a?X-Goog-Signature=redacted", - "x-dfo-trace-id": "86f0988c26f742f59efa2fba3ebcf738" + "x-dfo-trace-id": "08870d34b30642f8b5dc651859af6425" } } }, @@ -3865,21 +3885,21 @@ "dataset": { "rows": 10, "columns": [ - "stat_var", - "stat_var_label", - "observationAbout", - "observationAbout_label", "time_period", + "observationAbout_label", "value", - "unit", "unit_label", + "provenance_label", + "stat_var_label", + "observationAbout", + "unit", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-catalogue.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-catalogue.json index a15c230..d55b3e7 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-catalogue.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-catalogue.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:26Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:10Z" }, "call": [ { @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "a1f54c790c9c4278af93e96086733481" + "x-dfo-trace-id": "f384c8a7105445ed8da7d4f21c168a64" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "a1f54c790c9c4278af93e96086733481", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "f384c8a7105445ed8da7d4f21c168a64", "warnings": [], "timing_ms": { - "total": 1079, + "total": 969, "by_step": { "coverage": 0, - "extract": 1079, + "extract": 967, "entity_check": 0, "cache_lookup": 0 } @@ -84,7 +84,7 @@ "error": { "class": "DeclinedError", "code": "answer_declined", - "trace_id": "a1f54c790c9c4278af93e96086733481" + "trace_id": "f384c8a7105445ed8da7d4f21c168a64" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-declined.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-declined.json index dfd7f8d..1e7c21c 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-declined.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-declined.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:24Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:09Z" }, "call": [ { @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "5b2bc45dc7f7419cb3dfb98ed9bf68a3" + "x-dfo-trace-id": "518bf1ffcf3d4739b1a9d866a0cab91a" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "5b2bc45dc7f7419cb3dfb98ed9bf68a3", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "518bf1ffcf3d4739b1a9d866a0cab91a", "warnings": [], "timing_ms": { - "total": 1050, + "total": 902, "by_step": { - "coverage": 1, - "extract": 1049, + "coverage": 0, + "extract": 901, "entity_check": 0, "cache_lookup": 0 } @@ -90,7 +90,7 @@ "error": { "class": "DeclinedError", "code": "answer_declined", - "trace_id": "5b2bc45dc7f7419cb3dfb98ed9bf68a3" + "trace_id": "518bf1ffcf3d4739b1a9d866a0cab91a" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-definite-fetch.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-definite-fetch.json index 4275254..7d66613 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-definite-fetch.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-definite-fetch.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:01Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:21:59Z" }, "call": [ { @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -42,26 +42,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -71,30 +68,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -541,7 +538,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1194,7 +1191,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3784,7 +3781,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3792,7 +3790,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4193,7 +4191,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4215,7 +4213,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4238,7 +4237,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4521,7 +4520,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4538,7 +4537,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -4573,32 +4572,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-definition-spec-id-mismatch.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-definition-spec-id-mismatch.json index 9c3c84d..5b84639 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-definition-spec-id-mismatch.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-definition-spec-id-mismatch.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -38,26 +38,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -67,30 +64,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -537,7 +534,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1190,7 +1187,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3707,7 +3704,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9" + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-empty-parts.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-empty-parts.json index 35da4e4..7b3098b 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-empty-parts.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-empty-parts.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -41,14 +41,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -77,6 +77,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -100,7 +101,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "304fe92b998b45849ff11a547aadc3e8" + "trace_id": "01cf4a9623c449759f24d249f8783de1" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-folded-parts-unnamed.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-folded-parts-unnamed.json index 40f0122..9a7302c 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-folded-parts-unnamed.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-folded-parts-unnamed.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "74e9acc99aff40f7a653093976c55e3b" + "x-dfo-trace-id": "5e0f0d6a5a6442a38932b02d7f72ed6c" }, "json": { "schema_version": 1, @@ -41,26 +41,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", "normalized_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "74e9acc99aff40f7a653093976c55e3b", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5e0f0d6a5a6442a38932b02d7f72ed6c", "warnings": [], "timing_ms": { - "total": 4632, + "total": 8, "by_step": { "coverage": 0, - "extract": 1209, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 215, - "cached_tokens": 842, - "jev_input_tokens": 25612, - "jev_output_tokens": 5740 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -69,30 +66,30 @@ { "warnings": [], "timing_ms": { - "total": 3118, + "total": 4401, "by_step": { - "recall": 753, + "recall": 1271, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 75, - "shape_probe_coverage": 3, - "shape_probe": 164, - "shape_groups": 44, - "shape_labels": 9, + "shape_arcs": 108, + "shape_probe_coverage": 4, + "shape_probe": 113, + "shape_groups": 72, + "shape_labels": 12, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 695, + "shape": 524, "phase_prune_and_rank": 0, - "bind": 275, - "phase_bind": 190, - "phase_settle": 91, - "phase_place_bindings": 19, + "bind": 482, + "phase_bind": 205, + "phase_settle": 57, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 616, - "phase_materialise": 0, - "answer": 20, - "options": 307, - "phase_outcome": 152 + "materialise": 686, + "phase_materialise": 225, + "answer": 28, + "options": 685, + "phase_outcome": 204 } }, "restatement": { @@ -1436,7 +1433,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3764,30 +3761,30 @@ } ], "timing_ms": { - "total": 3017, + "total": 4561, "by_step": { - "recall": 704, - "phase_resolve_places": 1, + "recall": 1310, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 82, - "shape_probe_coverage": 2, - "shape_probe": 99, - "shape_groups": 47, - "shape_labels": 10, + "shape_arcs": 117, + "shape_probe_coverage": 28, + "shape_probe": 126, + "shape_groups": 83, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 365, + "shape": 753, "phase_prune_and_rank": 0, - "bind": 402, - "phase_bind": 196, - "phase_settle": 88, - "phase_place_bindings": 44, + "bind": 309, + "phase_bind": 209, + "phase_settle": 82, + "phase_place_bindings": 25, "phase_facet_requests": 0, - "materialise": 512, - "phase_materialise": 0, - "answer": 58, - "options": 511, - "phase_outcome": 136 + "materialise": 1061, + "phase_materialise": 233, + "answer": 29, + "options": 385, + "phase_outcome": 165 } }, "restatement": { @@ -5131,7 +5128,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -7464,30 +7461,30 @@ } ], "timing_ms": { - "total": 1849, + "total": 2851, "by_step": { - "recall": 391, + "recall": 360, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, "shape_probe_coverage": 2, - "shape_probe": 126, - "shape_groups": 55, - "shape_labels": 10, - "shape_admission": 1, + "shape_probe": 121, + "shape_groups": 841, + "shape_labels": 17, + "shape_admission": 4, "phase_derive_shapes": 0, - "shape": 311, + "shape": 1100, "phase_prune_and_rank": 0, - "bind": 139, - "phase_bind": 15, - "phase_settle": 27, - "phase_place_bindings": 426, + "bind": 261, + "phase_bind": 16, + "phase_settle": 7, + "phase_place_bindings": 554, "phase_facet_requests": 0, - "materialise": 93, - "phase_materialise": 0, - "answer": 244, - "options": 193, - "phase_outcome": 10 + "materialise": 96, + "phase_materialise": 17, + "answer": 239, + "options": 188, + "phase_outcome": 13 } }, "restatement": { @@ -8101,7 +8098,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -8924,7 +8921,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -9457,7 +9454,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -10030,7 +10027,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -10038,7 +10036,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "fc2f3ffbd0f448d2a6dcade27ac985bb" + "x-dfo-trace-id": "9b2918c16ee54353a38b2401d62c118a" }, "json": { "schema_version": 1, @@ -14638,7 +14636,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:28.193872+00:00" + "confirmed_at": "2026-10-26T09:18:03.336477+00:00" }, "warnings": [], "diagnostics": [ @@ -14654,7 +14652,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 16208, "media_type": "application/vnd.apache.parquet", @@ -14677,7 +14676,7 @@ ], "series_count": 108, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -14960,7 +14959,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T15:55:47.734809Z" @@ -14978,7 +14977,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa189b171faab41729ddc1317add46add9b5eefb5fcac62a98cb735cb5f20f0f8?X-Goog-Signature=redacted", - "x-dfo-trace-id": "c0e0b1a4539f4af3b08693b51621e3cb" + "x-dfo-trace-id": "b06362d84bb845e786735d92808b1b48" } } }, @@ -15063,7 +15062,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -15071,7 +15071,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "8afa9ef2f2b54f89b6c8c836c84efd26" + "x-dfo-trace-id": "4e3b97d6e0814dc7ae89509dcb63b46b" }, "json": { "schema_version": 1, @@ -15248,7 +15248,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:29.978390+00:00" + "confirmed_at": "2026-10-26T09:18:05.070107+00:00" }, "warnings": [], "diagnostics": [ @@ -15264,7 +15264,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 6545, "media_type": "application/vnd.apache.parquet", @@ -15286,7 +15287,7 @@ ], "series_count": 2, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -15440,7 +15441,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T15:55:49.978519Z" @@ -15458,7 +15459,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9ff8d7d0df398e2b298b22f54828fb187c705888db1816d223137c950d037839?X-Goog-Signature=redacted", - "x-dfo-trace-id": "af76c3e1b057409e8bbc7a9647e9aacc" + "x-dfo-trace-id": "dd6c261d3d2e4566bdc4071ef3834710" } } }, @@ -15495,35 +15496,35 @@ "rows": 644, "columns": [ "part", - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "observationAbout_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", + "provenance", + "stat_var_label", + "observationAbout", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", - "provenance", - "provenance_label", - "stat_var_label", - "observationAbout", - "observationAbout_label" + "stat_var" ], "part_rows": [ { diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-partial.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-partial.json index 70fb08f..e20fbd8 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-partial.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-partial.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:20Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:08Z" }, "call": [ { @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "5974af081db54d0babd26f8b1de79beb" + "x-dfo-trace-id": "bffd660858754e239446c0e2c32a7593" }, "json": { "schema_version": 1, @@ -46,15 +46,15 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "5974af081db54d0babd26f8b1de79beb", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "bffd660858754e239446c0e2c32a7593", "warnings": [], "timing_ms": { - "total": 328, + "total": 389, "by_step": { "coverage": 0, "extract": 0, - "cache_lookup": 1 + "cache_lookup": 0 } }, "llm_usage": { @@ -72,30 +72,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 323, + "total": 385, "by_step": { "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 2, - "shape_probe_coverage": 15, - "shape_probe": 16, - "shape_groups": 5, - "shape_labels": 10, - "shape_admission": 0, + "shape_arcs": 3, + "shape_probe_coverage": 20, + "shape_probe": 22, + "shape_groups": 6, + "shape_labels": 11, + "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 104, + "shape": 127, "phase_prune_and_rank": 0, - "bind": 101, - "phase_bind": 11, - "phase_settle": 12, - "phase_place_bindings": 11, + "bind": 102, + "phase_bind": 15, + "phase_settle": 6, + "phase_place_bindings": 13, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 2, - "options": 7, - "phase_outcome": 65 + "materialise": 11, + "phase_materialise": 37, + "answer": 3, + "options": 9, + "phase_outcome": 61 } }, "restatement": { @@ -570,7 +570,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1288,7 +1288,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3273,18 +3273,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 15, + "total": 18, "by_step": { - "recall": 1, + "recall": 0, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 4, - "shape_groups": 2, + "shape_probe": 2, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 14, + "shape": 18, "phase_prune_and_rank": 0 } }, @@ -3715,7 +3715,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3723,7 +3724,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "b8b4bbcd49e14cd49ecea658aea24ec5" + "x-dfo-trace-id": "a6a95e93a4134cbca3ef315e6bb5c6fb" }, "json": { "schema_version": 1, @@ -3963,7 +3964,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261012T000033Z-84777283", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:21.793459+00:00" + "confirmed_at": "2026-10-26T09:17:55.468442+00:00" }, "warnings": [ { @@ -3985,7 +3986,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 6095, "media_type": "application/vnd.apache.parquet", @@ -4008,7 +4010,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4184,7 +4186,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4201,7 +4203,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aab0bfec276a23aaefea6dbc3f1b57854ea520b3a7ae5feeade202f292446ed2c?X-Goog-Signature=redacted", - "x-dfo-trace-id": "1ac27caa67d74511b31d6ca9016617df" + "x-dfo-trace-id": "a835d2fa775c43b181cc16faf6a63b45" } } }, @@ -4242,22 +4244,22 @@ "dataset": { "rows": 55, "columns": [ - "stat_var", + "time_period", + "financingRecipient_label", + "value", + "unit_label", + "provenance_label", "stat_var_label", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": false, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-raises.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-raises.json index 254d0f6..b5cb905 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-raises.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-mixed-fetch-raises.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:18Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:08Z" }, "call": [ { @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -45,21 +45,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -71,30 +71,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -569,7 +569,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1287,7 +1287,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3272,18 +3272,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, @@ -3653,7 +3653,7 @@ "error": { "class": "IncompleteAnswerError", "code": "answer_incomplete", - "trace_id": "1e95940d311043d0b63bd27a87996439" + "trace_id": "86b361f85db44a29b3d711350962ca73" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-nested-specs-unchecked.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-nested-specs-unchecked.json index 7cd013e..29aeeb4 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-nested-specs-unchecked.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-nested-specs-unchecked.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6c90259bc5dc43649740c2e47cb555bc" + "x-dfo-trace-id": "989481cf76744b4cbddf1aff890cda9c" }, "json": { "schema_version": 1, @@ -45,8 +45,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "989481cf76744b4cbddf1aff890cda9c", "warnings": [], "timing_ms": { "total": 1, @@ -68,9 +68,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -90,6 +90,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -186,7 +187,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -215,7 +217,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ], "flagged_match": { @@ -251,6 +254,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -303,7 +307,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "67793b9246794fc18d48484eb317d58f" + "x-dfo-trace-id": "7cbd24b2c1a0457da3df6702b8caf3aa" }, "json": { "schema_version": 1, @@ -312,11 +316,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "67793b9246794fc18d48484eb317d58f", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "7cbd24b2c1a0457da3df6702b8caf3aa", "warnings": [], "timing_ms": { - "total": 932, + "total": 1050, "by_step": { "coverage": 0 } @@ -342,30 +346,30 @@ } ], "timing_ms": { - "total": 907, + "total": 1045, "by_step": { "recall": 0, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe_coverage": 6, - "shape_probe": 109, - "shape_groups": 41, + "shape_probe_coverage": 10, + "shape_probe": 104, + "shape_groups": 56, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 387, + "shape": 473, "phase_prune_and_rank": 0, - "bind": 111, - "phase_bind": 175, - "phase_settle": 9, - "phase_place_bindings": 67, + "bind": 105, + "phase_bind": 185, + "phase_settle": 8, + "phase_place_bindings": 77, "phase_facet_requests": 0, - "materialise": 15, - "phase_materialise": 0, - "answer": 10, - "options": 13, - "phase_outcome": 120 + "materialise": 17, + "phase_materialise": 5, + "answer": 9, + "options": 26, + "phase_outcome": 140 } }, "restatement": { @@ -769,7 +773,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1223,7 +1227,7 @@ "start_year": 2018, "end_year": 2018 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-operations-not-a-list.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-operations-not-a-list.json index a9a4400..8ca97a3 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-operations-not-a-list.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-operations-not-a-list.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -41,26 +41,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -70,30 +67,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -544,7 +541,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1197,7 +1194,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3787,7 +3784,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3795,7 +3793,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4196,7 +4194,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4218,7 +4216,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4241,7 +4240,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4524,7 +4523,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4541,7 +4540,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -4576,32 +4575,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-part-status-not-a-string.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-part-status-not-a-string.json index 08f9ae8..7f3c983 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-part-status-not-a-string.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-part-status-not-a-string.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -41,21 +41,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -67,30 +67,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -565,7 +565,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1283,7 +1283,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3268,18 +3268,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, @@ -3631,7 +3631,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "1e95940d311043d0b63bd27a87996439" + "trace_id": "86b361f85db44a29b3d711350962ca73" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-several-readings-folded.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-several-readings-folded.json index 93272d8..66e1e7f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-several-readings-folded.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-several-readings-folded.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:22Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:10Z" }, "call": [ { @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "74e9acc99aff40f7a653093976c55e3b" + "x-dfo-trace-id": "5e0f0d6a5a6442a38932b02d7f72ed6c" }, "json": { "schema_version": 1, @@ -42,26 +42,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", "normalized_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "74e9acc99aff40f7a653093976c55e3b", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5e0f0d6a5a6442a38932b02d7f72ed6c", "warnings": [], "timing_ms": { - "total": 4632, + "total": 8, "by_step": { "coverage": 0, - "extract": 1209, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 215, - "cached_tokens": 842, - "jev_input_tokens": 25612, - "jev_output_tokens": 5740 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -71,30 +68,30 @@ "variable_text": "health ODA disbursements", "warnings": [], "timing_ms": { - "total": 3118, + "total": 4401, "by_step": { - "recall": 753, + "recall": 1271, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 75, - "shape_probe_coverage": 3, - "shape_probe": 164, - "shape_groups": 44, - "shape_labels": 9, + "shape_arcs": 108, + "shape_probe_coverage": 4, + "shape_probe": 113, + "shape_groups": 72, + "shape_labels": 12, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 695, + "shape": 524, "phase_prune_and_rank": 0, - "bind": 275, - "phase_bind": 190, - "phase_settle": 91, - "phase_place_bindings": 19, + "bind": 482, + "phase_bind": 205, + "phase_settle": 57, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 616, - "phase_materialise": 0, - "answer": 20, - "options": 307, - "phase_outcome": 152 + "materialise": 686, + "phase_materialise": 225, + "answer": 28, + "options": 685, + "phase_outcome": 204 } }, "restatement": { @@ -1438,7 +1435,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3766,30 +3763,30 @@ } ], "timing_ms": { - "total": 3017, + "total": 4561, "by_step": { - "recall": 704, - "phase_resolve_places": 1, + "recall": 1310, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 82, - "shape_probe_coverage": 2, - "shape_probe": 99, - "shape_groups": 47, - "shape_labels": 10, + "shape_arcs": 117, + "shape_probe_coverage": 28, + "shape_probe": 126, + "shape_groups": 83, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 365, + "shape": 753, "phase_prune_and_rank": 0, - "bind": 402, - "phase_bind": 196, - "phase_settle": 88, - "phase_place_bindings": 44, + "bind": 309, + "phase_bind": 209, + "phase_settle": 82, + "phase_place_bindings": 25, "phase_facet_requests": 0, - "materialise": 512, - "phase_materialise": 0, - "answer": 58, - "options": 511, - "phase_outcome": 136 + "materialise": 1061, + "phase_materialise": 233, + "answer": 29, + "options": 385, + "phase_outcome": 165 } }, "restatement": { @@ -5133,7 +5130,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -7466,30 +7463,30 @@ } ], "timing_ms": { - "total": 1849, + "total": 2851, "by_step": { - "recall": 391, + "recall": 360, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, "shape_probe_coverage": 2, - "shape_probe": 126, - "shape_groups": 55, - "shape_labels": 10, - "shape_admission": 1, + "shape_probe": 121, + "shape_groups": 841, + "shape_labels": 17, + "shape_admission": 4, "phase_derive_shapes": 0, - "shape": 311, + "shape": 1100, "phase_prune_and_rank": 0, - "bind": 139, - "phase_bind": 15, - "phase_settle": 27, - "phase_place_bindings": 426, + "bind": 261, + "phase_bind": 16, + "phase_settle": 7, + "phase_place_bindings": 554, "phase_facet_requests": 0, - "materialise": 93, - "phase_materialise": 0, - "answer": 244, - "options": 193, - "phase_outcome": 10 + "materialise": 96, + "phase_materialise": 17, + "answer": 239, + "options": 188, + "phase_outcome": 13 } }, "restatement": { @@ -8103,7 +8100,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -8926,7 +8923,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -9459,7 +9456,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -10032,7 +10029,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -10040,7 +10038,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "fc2f3ffbd0f448d2a6dcade27ac985bb" + "x-dfo-trace-id": "9b2918c16ee54353a38b2401d62c118a" }, "json": { "schema_version": 1, @@ -14640,7 +14638,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:28.193872+00:00" + "confirmed_at": "2026-10-26T09:18:03.336477+00:00" }, "warnings": [], "diagnostics": [ @@ -14656,7 +14654,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 16208, "media_type": "application/vnd.apache.parquet", @@ -14679,7 +14678,7 @@ ], "series_count": 108, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -14962,7 +14961,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T15:55:47.734809Z" @@ -14980,7 +14979,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa189b171faab41729ddc1317add46add9b5eefb5fcac62a98cb735cb5f20f0f8?X-Goog-Signature=redacted", - "x-dfo-trace-id": "c0e0b1a4539f4af3b08693b51621e3cb" + "x-dfo-trace-id": "b06362d84bb845e786735d92808b1b48" } } }, @@ -15065,7 +15064,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -15073,7 +15073,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "8afa9ef2f2b54f89b6c8c836c84efd26" + "x-dfo-trace-id": "4e3b97d6e0814dc7ae89509dcb63b46b" }, "json": { "schema_version": 1, @@ -15250,7 +15250,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:29.978390+00:00" + "confirmed_at": "2026-10-26T09:18:05.070107+00:00" }, "warnings": [], "diagnostics": [ @@ -15266,7 +15266,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 6545, "media_type": "application/vnd.apache.parquet", @@ -15288,7 +15289,7 @@ ], "series_count": 2, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -15442,7 +15443,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T15:55:49.978519Z" @@ -15460,7 +15461,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9ff8d7d0df398e2b298b22f54828fb187c705888db1816d223137c950d037839?X-Goog-Signature=redacted", - "x-dfo-trace-id": "af76c3e1b057409e8bbc7a9647e9aacc" + "x-dfo-trace-id": "dd6c261d3d2e4566bdc4071ef3834710" } } }, @@ -15497,35 +15498,35 @@ "rows": 644, "columns": [ "part", - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "observationAbout_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", + "provenance", + "stat_var_label", + "observationAbout", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", - "provenance", - "provenance_label", - "stat_var_label", - "observationAbout", - "observationAbout_label" + "stat_var" ], "part_rows": [ { diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-identity.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-identity.json index ddf62a5..14974bf 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-identity.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-identity.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -38,26 +38,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -67,30 +64,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -474,7 +471,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1127,7 +1124,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3644,7 +3641,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9" + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-resolved.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-resolved.json index 67702d7..6056087 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-resolved.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-spec-without-resolved.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -38,26 +38,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -67,30 +64,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -424,7 +421,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9" + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out-keeps-reasons.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out-keeps-reasons.json new file mode 100644 index 0000000..6a320eb --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out-keeps-reasons.json @@ -0,0 +1,3659 @@ +{ + "format_version": 1, + "name": "answer-timed-out-keeps-reasons", + "description": "A no_data part's gap kind comes from its own reason, whatever diagnostics.timed_out says. A part Search resolved before its time limit keeps its reason's gap kind, and fetching the answer raises IncompleteAnswerError.", + "origin": { + "kind": "synthetic", + "derived_from": "answer-mixed-fetch-raises", + "edit": "The answer's diagnostics.timed_out becomes true, part 1 keeps its reason variable_not_resolved, and its response carries x-dfo-trace-id 55555555555555555555555555555555." + }, + "call": [ + { + "op": "search", + "question": "debt service of Kenya and the price of tea in Kenya" + }, + { + "op": "fetch" + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "raw_text", + "query": "debt service of Kenya and the price of tea in Kenya" + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "55555555555555555555555555555555" + }, + "json": { + "schema_version": 1, + "query_echo": { + "entry_path": "raw_text", + "raw_query": "debt service of Kenya and the price of tea in Kenya", + "normalized_query": "debt service of Kenya and the price of tea in Kenya", + "extract_skipped": false + }, + "diagnostics": { + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", + "warnings": [], + "timing_ms": { + "total": 456, + "by_step": { + "coverage": 0, + "extract": 0, + "cache_lookup": 0 + } + }, + "llm_usage": { + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 5250, + "jev_output_tokens": 1104 + }, + "timed_out": true + }, + "kind": "resolution", + "parts": [ + { + "variable_text": "debt service", + "warnings": [], + "timing_ms": { + "total": 452, + "by_step": { + "recall": 1, + "phase_resolve_places": 0, + "phase_check_entities": 0, + "shape_arcs": 2, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, + "shape_admission": 0, + "phase_derive_shapes": 0, + "shape": 225, + "phase_prune_and_rank": 0, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, + "phase_facet_requests": 0, + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 + } + }, + "restatement": { + "variable_text": "debt service", + "entities": [ + { + "text": "Kenya" + } + ], + "sources": [], + "operations": [] + }, + "entity_refs": [ + { + "text": "Kenya", + "refs": [ + "country/KEN" + ] + } + ], + "operations": [], + "departures": [], + "assumptions": [ + { + "kind": "open_role", + "binding_ref": "origin:", + "text": "every creditor", + "refs": [] + } + ], + "status": "definite", + "spec": { + "spec_id": "spec_78d04fefc3b3f1ef71acd622957e4995", + "definition": { + "schema_version": 1, + "spec_id": "spec_78d04fefc3b3f1ef71acd622957e4995", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "ServicingPayment" + } + }, + { + "property": "financingResource", + "binding": { + "state": "unbound" + } + }, + { + "property": "guaranteeStatus", + "binding": { + "state": "unbound" + } + }, + { + "property": "idsConcessionality", + "binding": { + "state": "unbound" + } + }, + { + "property": "idsCreditorType", + "binding": { + "state": "unbound" + } + }, + { + "property": "idsDebtorType", + "binding": { + "state": "unbound" + } + }, + { + "property": "includesIMFCredit", + "binding": { + "state": "unbound" + } + }, + { + "property": "maturityBand", + "binding": { + "state": "unbound" + } + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "requested_window": { + "kind": "all" + } + } + }, + "resolved": { + "shape": { + "shape_id": "financialflow_amount_measuredvalue_nominal_by_role", + "label": "Debt service on external debt", + "population_type": { + "ref": "FinancialFlow", + "label": "financial flow" + }, + "measured_property": { + "ref": "amount", + "label": "amount" + }, + "stat_type": { + "ref": "measuredValue", + "label": "measuredValue" + }, + "measurement_qualifier": { + "ref": "Nominal", + "label": "Nominal" + }, + "measurement_denominator_unresolved": false, + "slot_keys": [ + { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + { + "axis": "how", + "property": { + "ref": "financingResource", + "label": "financing resource" + }, + "label": "financing resource" + }, + { + "axis": "how", + "property": { + "ref": "guaranteeStatus", + "label": "guarantee status" + }, + "label": "guarantee status" + }, + { + "axis": "how", + "property": { + "ref": "idsDebtorType", + "label": "IDS debtor type" + }, + "label": "IDS debtor type" + }, + { + "axis": "how", + "property": { + "ref": "idsCreditorType", + "label": "IDS creditor type" + }, + "label": "IDS creditor type" + }, + { + "axis": "how", + "property": { + "ref": "idsConcessionality", + "label": "IDS concessionality" + }, + "label": "IDS concessionality" + }, + { + "axis": "how", + "property": { + "ref": "includesIMFCredit", + "label": "includes IMF credit" + }, + "label": "includes IMF credit" + }, + { + "axis": "how", + "property": { + "ref": "maturityBand", + "label": "maturity band" + }, + "label": "maturity band" + }, + { + "axis": "where", + "label": "place" + }, + { + "axis": "when", + "label": "time window" + }, + { + "axis": "source", + "label": "data source" + } + ], + "member_count": 65, + "member_digest": "b66541c53125ee32113e51b4f52a3a428043b5506e15f96277f37b89547fc1c4", + "refine_supported": true, + "data_commons": [ + { + "id": "dfo", + "label": "Development Finance Observatory" + } + ] + }, + "slots": [ + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "ServicingPayment", + "label": "servicing payment" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financingResource", + "label": "financing resource" + }, + "label": "financing resource" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "guaranteeStatus", + "label": "guarantee status" + }, + "label": "guarantee status" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsDebtorType", + "label": "IDS debtor type" + }, + "label": "IDS debtor type" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsCreditorType", + "label": "IDS creditor type" + }, + "label": "IDS creditor type" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsConcessionality", + "label": "IDS concessionality" + }, + "label": "IDS concessionality" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "includesIMFCredit", + "label": "includes IMF credit" + }, + "label": "includes IMF credit" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "maturityBand", + "label": "maturity band" + }, + "label": "maturity band" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "country/KEN", + "label": "Kenya" + }, + "value_kind": "entity" + } + }, + "summable": false + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + } + ], + "inapplicable_slots": [ + "financingResource", + "guaranteeStatus", + "idsDebtorType", + "idsCreditorType", + "idsConcessionality", + "includesIMFCredit", + "maturityBand" + ], + "unresolved_slots": [], + "stat_vars": { + "member_count": 1, + "member_count_exact": false, + "served_count": 1, + "data_date_range": { + "start": "1970", + "end": "2024" + }, + "source_codes": [ + { + "provenance": "provenance/WorldBankInternationalDebtStatistics", + "code": "DT.TDS.DECT.CD", + "exact": true + } + ] + }, + "entities": [ + { + "ref": "country/KEN", + "label": "Kenya", + "entity_type": "Country", + "role": "destination" + } + ], + "coverage": { + "kind": "exact", + "has_data": true, + "observation_count": 55, + "evidence": "facet_metadata", + "dimensions": [ + { + "label": "sources", + "count": 1 + }, + { + "label": "observations", + "count": 55 + }, + { + "label": "variables", + "count": 1 + } + ] + }, + "headline": { + "year": 2024, + "year_named": false, + "observations": 1, + "series": 1, + "role_counts": { + "financingRecipient": 1 + }, + "recent_years": [ + { + "year": 2024, + "observations": 1, + "series": 1, + "role_counts": { + "financingRecipient": 1 + } + }, + { + "year": 2023, + "observations": 1, + "series": 1, + "role_counts": { + "financingRecipient": 1 + } + }, + { + "year": 2022, + "observations": 1, + "series": 1, + "role_counts": { + "financingRecipient": 1 + } + } + ] + }, + "download": { + "rows": 55, + "rows_exact": true, + "series": 1, + "window": { + "start_year": 1970, + "end_year": 2024 + }, + "licence": { + "label": "Creative Commons Attribution License" + } + }, + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", + "facets": { + "provenance": { + "basis": "fallback", + "ref": "provenance/WorldBankInternationalDebtStatistics", + "label": "World Bank International Debt Statistics (IDS)", + "source": { + "ref": "dc/s/WorldBank", + "label": "World Bank" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/WorldBankInternationalDebtStatistics" + }, + "unit": { + "basis": "fallback", + "ref": "CurrencyUSD", + "label": "US Dollar", + "currency": "USD" + } + }, + "slot_options": [ + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "options": [ + { + "value": { + "ref": { + "ref": "ServicingPayment", + "label": "servicing payment" + }, + "value_kind": "enum_value" + } + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "guaranteeStatus", + "label": "guarantee status" + }, + "label": "guarantee status" + }, + "options": [ + { + "value": { + "ref": { + "ref": "PublicOrPubliclyGuaranteed", + "label": "public or publicly guaranteed" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "NotPubliclyGuaranteed", + "label": "not publicly guaranteed" + }, + "value_kind": "enum_value" + }, + "served_rows": 56, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsDebtorType", + "label": "IDS debtor type" + }, + "label": "IDS debtor type" + }, + "options": [ + { + "value": { + "ref": { + "ref": "IDSDebtorTypePS", + "label": "public sector (IDS debtor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "IDSDebtorTypeGG", + "label": "general government (IDS debtor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "IDSDebtorTypeOPS", + "label": "other public sector (IDS debtor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "IDSDebtorTypePrivateSector", + "label": "private sector (IDS debtor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 56, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsCreditorType", + "label": "IDS creditor type" + }, + "label": "IDS creditor type" + }, + "options": [ + { + "value": { + "ref": { + "ref": "IDSCreditorTypePROP", + "label": "other private creditors (IDS creditor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 46, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "IDSCreditorTypePBND", + "label": "bondholders (IDS creditor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 31, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "IDSCreditorTypePRVT", + "label": "private creditors (IDS creditor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + }, + { + "value": { + "ref": { + "ref": "IDSCreditorTypeOFFT", + "label": "official creditors (IDS creditor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 55, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "IDSCreditorTypeMLAT", + "label": "multilateral (IDS creditor type)" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsConcessionality", + "label": "IDS concessionality" + }, + "label": "IDS concessionality" + }, + "options": [ + { + "value": { + "ref": { + "ref": "IDSConcessional", + "label": "concessional (IDS concessionality)" + }, + "value_kind": "enum_value" + }, + "served_rows": 118, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "includesIMFCredit", + "label": "includes IMF credit" + }, + "label": "includes IMF credit" + }, + "options": [ + { + "value": { + "ref": { + "ref": "WithoutIMFCredit", + "label": "without IMF credit" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "maturityBand", + "label": "maturity band" + }, + "label": "maturity band" + }, + "options": [ + { + "value": { + "ref": { + "ref": "LongTermMaturity", + "label": "long-term maturity" + }, + "value_kind": "enum_value" + }, + "served_rows": 63, + "served_rows_exact": false + } + ], + "truncated": true, + "source": "observed" + } + ], + "entity_options": [], + "facet_options": [ + { + "axis": "provenance", + "options": [ + { + "value": { + "ref": { + "ref": "provenance/WorldBankInternationalDebtStatistics", + "label": "World Bank International Debt Statistics (IDS)" + }, + "value_kind": "source" + } + } + ], + "truncated": false, + "source": "observed" + }, + { + "axis": "measurementMethod", + "options": [ + { + "value": { + "value_kind": "enum_value" + } + }, + { + "value": { + "ref": { + "ref": "Estimated", + "label": "estimated" + }, + "value_kind": "enum_value" + } + } + ], + "truncated": false, + "source": "observed" + }, + { + "axis": "unit", + "options": [ + { + "value": { + "ref": { + "ref": "CurrencyUSD", + "label": "US Dollar" + }, + "value_kind": "enum_value" + } + } + ], + "truncated": false, + "source": "observed" + } + ], + "rendered_sentence": "Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS).", + "sentence_plan": [ + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ] + }, + { + "kind": "text", + "text": "'s", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "external debt service", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "axis:populationType", + "axis:measuredProperty", + "axis:statType" + ] + }, + { + "kind": "text", + "text": " owed to ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "all creditors", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financingResource", + "binding_ref": "how:financingResource", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "guaranteeStatus", + "binding_ref": "how:guaranteeStatus", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsConcessionality", + "binding_ref": "how:idsConcessionality", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsCreditorType", + "binding_ref": "how:idsCreditorType", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsDebtorType", + "binding_ref": "how:idsDebtorType", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "includesIMFCredit", + "binding_ref": "how:includesIMFCredit", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "maturityBand", + "binding_ref": "how:maturityBand", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": ", ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "1970 to 2024", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ] + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "World Bank International Debt Statistics (IDS)", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [] + } + ], + "trace": { + "resolved_entities": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "resolved_sources": [ + { + "ref": "provenance/WorldBankInternationalDebtStatistics", + "label": "World Bank International Debt Statistics (IDS)", + "source": { + "ref": "dc/s/WorldBank", + "label": "World Bank" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/WorldBankInternationalDebtStatistics" + } + ], + "slot_filters": [ + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "ServicingPayment", + "label": "servicing payment" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financingResource", + "label": "financing resource" + }, + "label": "financing resource" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "guaranteeStatus", + "label": "guarantee status" + }, + "label": "guarantee status" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsDebtorType", + "label": "IDS debtor type" + }, + "label": "IDS debtor type" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsCreditorType", + "label": "IDS creditor type" + }, + "label": "IDS creditor type" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "idsConcessionality", + "label": "IDS concessionality" + }, + "label": "IDS concessionality" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "includesIMFCredit", + "label": "includes IMF credit" + }, + "label": "includes IMF credit" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "maturityBand", + "label": "maturity band" + }, + "label": "maturity band" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + } + ], + "query_qualifiers": [], + "capability_version": "cap_765c9d2a0c7529b9", + "default_reading": { + "applied": true, + "removed_stat_vars": [] + } + }, + "confidence": "high", + "available_at_place": true, + "measure_label_axes": [], + "measure_axis_values": [], + "unit_conversion": { + "currencies": [ + { + "code": "AED", + "noun": "UAE dirhams", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AFN", + "noun": "Afghan afghanis", + "current_years": { + "start_year": 2002, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2002, + "base_year_max": 2025 + }, + { + "code": "ALL", + "noun": "Albanian leks", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AMD", + "noun": "Armenian drams", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "AOA", + "noun": "Angolan kwanzas", + "current_years": { + "start_year": 1991, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "ARS", + "noun": "Argentine pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AUD", + "noun": "Australian dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AWG", + "noun": "Aruban florins", + "current_years": { + "start_year": 1995, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1995, + "base_year_max": 2025 + }, + { + "code": "AZN", + "noun": "Azerbaijani manats", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "BAM", + "noun": "Bosnia-Herzegovina convertible marks", + "current_years": { + "start_year": 1996, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1996, + "base_year_max": 2025 + }, + { + "code": "BBD", + "noun": "Barbadian dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BDT", + "noun": "Bangladeshi taka", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BHD", + "noun": "Bahraini dinars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BIF", + "noun": "Burundian francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BND", + "noun": "Brunei dollars", + "current_years": { + "start_year": 1985, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1985, + "base_year_max": 2025 + }, + { + "code": "BOB", + "noun": "Bolivian bolivianos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BRL", + "noun": "Brazilian reais", + "current_years": { + "start_year": 1986, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "BSD", + "noun": "Bahamian dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BTN", + "noun": "Bhutanese ngultrums", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BWP", + "noun": "Botswana pula", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BYN", + "noun": "Belarusian rubles", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "BZD", + "noun": "Belize dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CAD", + "noun": "Canadian dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CDF", + "noun": "Congolese francs", + "current_years": { + "start_year": 1991, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "CHF", + "noun": "Swiss francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CLP", + "noun": "Chilean pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CNY", + "noun": "Chinese yuan", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "COP", + "noun": "Colombian pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CRC", + "noun": "Costa Rican colones", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CVE", + "noun": "Cape Verdean escudos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CZK", + "noun": "Czech korunas", + "current_years": { + "start_year": 1995, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1995, + "base_year_max": 2025 + }, + { + "code": "DJF", + "noun": "Djiboutian francs", + "current_years": { + "start_year": 1990, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "DKK", + "noun": "Danish kroner", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "DOP", + "noun": "Dominican pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "DZD", + "noun": "Algerian dinars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EGP", + "noun": "Egyptian pounds", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ERN", + "noun": "Eritrean nakfas", + "current_years": { + "start_year": 1992, + "end_year": 2019 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2019 + }, + { + "code": "ETB", + "noun": "Ethiopian birr", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EUR", + "noun": "euros", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "FJD", + "noun": "Fijian dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GBP", + "noun": "British pounds", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GEL", + "noun": "Georgian lari", + "current_years": { + "start_year": 1994, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1994, + "base_year_max": 2025 + }, + { + "code": "GHS", + "noun": "Ghanaian cedis", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GMD", + "noun": "Gambian dalasis", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GNF", + "noun": "Guinean francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GTQ", + "noun": "Guatemalan quetzales", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GYD", + "noun": "Guyanese dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HKD", + "noun": "Hong Kong dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HNL", + "noun": "Honduran lempiras", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HTG", + "noun": "Haitian gourdes", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HUF", + "noun": "Hungarian forints", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "IDR", + "noun": "Indonesian rupiahs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ILS", + "noun": "Israeli new shekels", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "INR", + "noun": "Indian rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "IQD", + "noun": "Iraqi dinars", + "current_years": { + "start_year": 2000, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2000, + "base_year_max": 2025 + }, + { + "code": "IRR", + "noun": "Iranian rials", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ISK", + "noun": "Icelandic krónur", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JMD", + "noun": "Jamaican dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JOD", + "noun": "Jordanian dinars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JPY", + "noun": "Japanese yen", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KES", + "noun": "Kenyan shillings", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KGS", + "noun": "Kyrgyz soms", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "KHR", + "noun": "Cambodian riels", + "current_years": { + "start_year": 1986, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "KMF", + "noun": "Comorian francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KRW", + "noun": "South Korean won", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KWD", + "noun": "Kuwaiti dinars", + "current_years": { + "start_year": 1981, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1981, + "base_year_max": 2025 + }, + { + "code": "KZT", + "noun": "Kazakh tenge", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "LAK", + "noun": "Lao kip", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LBP", + "noun": "Lebanese pounds", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LKR", + "noun": "Sri Lankan rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LSL", + "noun": "Lesotho maloti", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LYD", + "noun": "Libyan dinars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MAD", + "noun": "Moroccan dirhams", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MDL", + "noun": "Moldovan lei", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MGA", + "noun": "Malagasy ariary", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MKD", + "noun": "Macedonian denars", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MMK", + "noun": "Myanmar kyats", + "current_years": { + "start_year": 1998, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "MNT", + "noun": "Mongolian tugriks", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MOP", + "noun": "Macanese patacas", + "current_years": { + "start_year": 2001, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2001, + "base_year_max": 2025 + }, + { + "code": "MRU", + "noun": "Mauritanian ouguiyas", + "current_years": { + "start_year": 1990, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "MUR", + "noun": "Mauritian rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MVR", + "noun": "Maldivian rufiyaa", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MWK", + "noun": "Malawian kwachas", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MXN", + "noun": "Mexican pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MYR", + "noun": "Malaysian ringgits", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MZN", + "noun": "Mozambican meticais", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NAD", + "noun": "Namibian dollars", + "current_years": { + "start_year": 1989, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1989, + "base_year_max": 2025 + }, + { + "code": "NGN", + "noun": "Nigerian naira", + "current_years": { + "start_year": 1990, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "NIO", + "noun": "Nicaraguan córdobas", + "current_years": { + "start_year": 1987, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1987, + "base_year_max": 2025 + }, + { + "code": "NOK", + "noun": "Norwegian kroner", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NPR", + "noun": "Nepalese rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NZD", + "noun": "New Zealand dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "OMR", + "noun": "Omani rials", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PEN", + "noun": "Peruvian soles", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PGK", + "noun": "Papua New Guinean kina", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PHP", + "noun": "Philippine pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PKR", + "noun": "Pakistani rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PLN", + "noun": "Polish zlotys", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PYG", + "noun": "Paraguayan guaranís", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "QAR", + "noun": "Qatari riyals", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "RON", + "noun": "Romanian lei", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "RSD", + "noun": "Serbian dinars", + "current_years": { + "start_year": 1997, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "RUB", + "noun": "Russian rubles", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "RWF", + "noun": "Rwandan francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SAR", + "noun": "Saudi riyals", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SBD", + "noun": "Solomon Islands dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SCR", + "noun": "Seychellois rupees", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SDG", + "noun": "Sudanese pounds", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SEK", + "noun": "Swedish kronor", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SGD", + "noun": "Singapore dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SLE", + "noun": "Sierra Leonean leones", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SRD", + "noun": "Surinamese dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SSP", + "noun": "South Sudanese pounds", + "current_years": { + "start_year": 2011, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2011, + "base_year_max": 2025 + }, + { + "code": "STN", + "noun": "São Tomé and Príncipe dobras", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SYP", + "noun": "Syrian pounds", + "current_years": { + "start_year": 1980, + "end_year": 2010 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2010 + }, + { + "code": "SZL", + "noun": "Swazi emalangeni", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "THB", + "noun": "Thai baht", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TJS", + "noun": "Tajikistani somonis", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TMT", + "noun": "Turkmenistani manat", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TND", + "noun": "Tunisian dinars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TOP", + "noun": "Tongan paʻanga", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TRY", + "noun": "Turkish lira", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TTD", + "noun": "Trinidad and Tobago dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TWD", + "noun": "New Taiwan dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TZS", + "noun": "Tanzanian shillings", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UAH", + "noun": "Ukrainian hryvnias", + "current_years": { + "start_year": 1991, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "UGX", + "noun": "Ugandan shillings", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "USD", + "noun": "US dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UYU", + "noun": "Uruguayan pesos", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UZS", + "noun": "Uzbekistani soms", + "current_years": { + "start_year": 1992, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "VES", + "noun": "Venezuelan bolívares", + "current_years": { + "start_year": 2017, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2017, + "base_year_max": 2025 + }, + { + "code": "VND", + "noun": "Vietnamese dong", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "VUV", + "noun": "Vanuatu vatu", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "WST", + "noun": "Samoan tala", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XAF", + "noun": "Central African CFA francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XCD", + "noun": "East Caribbean dollars", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XOF", + "noun": "West African CFA francs", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "YER", + "noun": "Yemeni rials", + "current_years": { + "start_year": 2014, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 2014, + "base_year_max": 2025 + }, + { + "code": "ZAR", + "noun": "South African rand", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ZMW", + "noun": "Zambian kwachas", + "current_years": { + "start_year": 1980, + "end_year": 2024 + }, + "constant_years": { + "start_year": 1980, + "end_year": 2024 + }, + "base_year_min": 1980, + "base_year_max": 2025 + } + ], + "default_base_year": 2024, + "role_currencies": { + "destination": "KES" + } + } + } + } + }, + { + "variable_text": "price of tea", + "warnings": [], + "timing_ms": { + "total": 121, + "by_step": { + "recall": 0, + "phase_resolve_places": 0, + "phase_check_entities": 0, + "shape_arcs": 1, + "shape_probe": 96, + "shape_groups": 1, + "shape_labels": 0, + "shape_admission": 0, + "phase_derive_shapes": 0, + "shape": 121, + "phase_prune_and_rank": 0 + } + }, + "rendered_sentence": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "sentence_plan": [ + { + "kind": "text", + "text": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "segment": "heading", + "entity_refs": [], + "states": [] + } + ], + "entity_refs": [], + "operations": [], + "departures": [], + "assumptions": [], + "status": "no_data", + "no_data": { + "reason": "variable_not_resolved", + "category": "unresolved", + "message": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "entity_terms": [], + "ambiguous_entities": [], + "source_terms": [], + "ambiguous_sources": [], + "unbound_qualifiers": [], + "options": [] + } + } + ], + "rendered_sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. No match for price of tea. Try a different wording or a related indicator.", + "sentence_plan": [ + { + "kind": "text", + "text": "Answered 1 of 2 parts.", + "segment": "coverage", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "coverage", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": "'s", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "external debt service", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "axis:populationType", + "axis:measuredProperty", + "axis:statType" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " owed to ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "all creditors", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financingResource", + "binding_ref": "how:financingResource", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "guaranteeStatus", + "binding_ref": "how:guaranteeStatus", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsConcessionality", + "binding_ref": "how:idsConcessionality", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsCreditorType", + "binding_ref": "how:idsCreditorType", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "idsDebtorType", + "binding_ref": "how:idsDebtorType", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "includesIMFCredit", + "binding_ref": "how:includesIMFCredit", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "maturityBand", + "binding_ref": "how:maturityBand", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": ", ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "1970 to 2024", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "World Bank International Debt Statistics (IDS)", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "Assumed: ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "every creditor", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": ".", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": "No match for ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 1 + }, + { + "kind": "slot", + "text": "price of tea", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 1 + }, + { + "kind": "text", + "text": ". Try a different wording or a related indicator.", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 1 + } + ], + "status": "mixed" + } + } + } + ], + "expect": [ + { + "answer": { + "kind": "resolution", + "status": "mixed", + "sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. No match for price of tea. Try a different wording or a related indicator.", + "complete": false, + "specs": [ + "spec_78d04fefc3b3f1ef71acd622957e4995" + ], + "gaps": [ + { + "part": 1, + "kind": "not_understood", + "options": 0 + } + ] + } + }, + { + "error": { + "class": "IncompleteAnswerError", + "code": "answer_incomplete", + "trace_id": "55555555555555555555555555555555" + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out.json index 54b6e48..f1593e9 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-timed-out.json @@ -1,11 +1,11 @@ { "format_version": 1, "name": "answer-timed-out", - "description": "A no_data part of an answer whose diagnostics say Search timed out is a timed_out gap, whatever its reason, and fetching the answer raises IncompleteAnswerError, which describes the answer and carries its response's trace id.", + "description": "A no_data part with reason timed_out, the reason Search gives a part it left unresolved at its time limit, is a timed_out gap, and fetching the answer raises IncompleteAnswerError, which describes the answer and carries its response's trace id.", "origin": { "kind": "synthetic", "derived_from": "answer-mixed-fetch-raises", - "edit": "The answer's diagnostics.timed_out becomes true, and its response carries x-dfo-trace-id 55555555555555555555555555555555." + "edit": "The answer's diagnostics.timed_out becomes true, part 1's no_data reason becomes timed_out with the message that reason renders, and its response carries x-dfo-trace-id 55555555555555555555555555555555." }, "call": [ { @@ -44,21 +44,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -70,30 +70,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -568,7 +568,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1286,7 +1286,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3271,26 +3271,26 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, - "rendered_sentence": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "rendered_sentence": "Search reached its time limit before answering this part. Try again, or ask a narrower question.", "sentence_plan": [ { "kind": "text", - "text": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "text": "Search reached its time limit before answering this part. Try again, or ask a narrower question.", "segment": "heading", "entity_refs": [], "states": [] @@ -3302,9 +3302,9 @@ "assumptions": [], "status": "no_data", "no_data": { - "reason": "variable_not_resolved", + "reason": "timed_out", "category": "unresolved", - "message": "Could not match \"price of tea\" to a supported indicator. Try a different wording or a related indicator.", + "message": "Search reached its time limit before answering this part. Try again, or ask a narrower question.", "entity_terms": [], "ambiguous_entities": [], "source_terms": [], @@ -3314,7 +3314,7 @@ } } ], - "rendered_sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. No match for price of tea. Try a different wording or a related indicator.", + "rendered_sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. Search reached its time limit before answering this part. Try again, or ask a narrower question.", "sentence_plan": [ { "kind": "text", @@ -3600,24 +3600,7 @@ }, { "kind": "text", - "text": "No match for ", - "segment": "heading", - "entity_refs": [], - "states": [], - "part_index": 1 - }, - { - "kind": "slot", - "text": "price of tea", - "axis": "what", - "segment": "heading", - "entity_refs": [], - "states": [], - "part_index": 1 - }, - { - "kind": "text", - "text": ". Try a different wording or a related indicator.", + "text": "Search reached its time limit before answering this part. Try again, or ask a narrower question.", "segment": "heading", "entity_refs": [], "states": [], @@ -3634,7 +3617,7 @@ "answer": { "kind": "resolution", "status": "mixed", - "sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. No match for price of tea. Try a different wording or a related indicator.", + "sentence": "Answered 1 of 2 parts. Kenya's external debt service owed to all creditors, 1970 to 2024. Current US dollars · World Bank International Debt Statistics (IDS). Assumed: every creditor. Search reached its time limit before answering this part. Try again, or ask a narrower question.", "complete": false, "specs": [ "spec_78d04fefc3b3f1ef71acd622957e4995" diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-enum-value.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-enum-value.json index d55e0ec..ba5c570 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-enum-value.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-enum-value.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -41,14 +41,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -65,9 +65,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -87,6 +87,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -183,7 +184,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -212,7 +214,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -234,6 +237,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-kind.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-kind.json index bf2b909..3f2cee4 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-kind.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-kind.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -41,14 +41,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -65,9 +65,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -87,6 +87,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -183,7 +184,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -212,7 +214,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -234,6 +237,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -257,7 +261,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "304fe92b998b45849ff11a547aadc3e8" + "trace_id": "01cf4a9623c449759f24d249f8783de1" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-part-status.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-part-status.json index 1ca39fd..ff7c4f0 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-part-status.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-unknown-part-status.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -44,21 +44,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -70,30 +70,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -568,7 +568,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1286,7 +1286,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3271,18 +3271,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, @@ -3652,7 +3652,7 @@ "error": { "class": "IncompleteAnswerError", "code": "answer_incomplete", - "trace_id": "1e95940d311043d0b63bd27a87996439" + "trace_id": "86b361f85db44a29b3d711350962ca73" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/answer-unsupported-part.json b/python/client-contract/src/dfo_client_contract/scenarios/answer-unsupported-part.json index 3471fd6..26eaf48 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/answer-unsupported-part.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/answer-unsupported-part.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -44,21 +44,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -70,30 +70,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -568,7 +568,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1286,7 +1286,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3271,18 +3271,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, @@ -3646,7 +3646,7 @@ "error": { "class": "IncompleteAnswerError", "code": "answer_incomplete", - "trace_id": "1e95940d311043d0b63bd27a87996439" + "trace_id": "86b361f85db44a29b3d711350962ca73" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-assumed-reading.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-assumed-reading.json index d4a4d89..0888ccb 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-assumed-reading.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-assumed-reading.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -45,22 +45,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -71,28 +71,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -814,7 +814,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1597,6 +1597,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1813,6 +1822,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2348,7 +2366,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2647,7 +2665,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bb50ec0b45244567bf259cee9e6de857" + "x-dfo-trace-id": "97f100748b194ca09afc06c9dc1ac81d" }, "json": { "schema_version": 1, @@ -2656,11 +2674,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bb50ec0b45244567bf259cee9e6de857", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "97f100748b194ca09afc06c9dc1ac81d", "warnings": [], "timing_ms": { - "total": 486, + "total": 345, "by_step": { "coverage": 0 } @@ -2673,19 +2691,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 436, + "total": 284, "by_step": { - "validate": 1, - "walk": 182, + "validate": 0, + "walk": 0, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 158, - "bind": 1, + "shape": 174, + "bind": 0, "where_places": 0, - "materialise": 93, - "answer": 1, - "options": 0 + "materialise": 107, + "answer": 2, + "options": 1 } }, "entity_refs": [], @@ -3213,7 +3231,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-by-label.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-by-label.json index 6219a29..39aad61 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-by-label.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-by-label.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -45,22 +45,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -71,28 +71,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -814,7 +814,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1597,6 +1597,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1813,6 +1822,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2348,7 +2366,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2629,7 +2647,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bb50ec0b45244567bf259cee9e6de857" + "x-dfo-trace-id": "97f100748b194ca09afc06c9dc1ac81d" }, "json": { "schema_version": 1, @@ -2638,11 +2656,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bb50ec0b45244567bf259cee9e6de857", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "97f100748b194ca09afc06c9dc1ac81d", "warnings": [], "timing_ms": { - "total": 486, + "total": 345, "by_step": { "coverage": 0 } @@ -2655,19 +2673,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 436, + "total": 284, "by_step": { - "validate": 1, - "walk": 182, + "validate": 0, + "walk": 0, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 158, - "bind": 1, + "shape": 174, + "bind": 0, "where_places": 0, - "materialise": 93, - "answer": 1, - "options": 0 + "materialise": 107, + "answer": 2, + "options": 1 } }, "entity_refs": [], @@ -3186,7 +3204,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-echoes-assumptions.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-echoes-assumptions.json index c0a28da..9c99077 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-echoes-assumptions.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-echoes-assumptions.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:32:01Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:52Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "fef53a976b3e4e1cb64f3bc36f96f076" + "x-dfo-trace-id": "77333f3756d7434fa7476addd497eff0" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "fef53a976b3e4e1cb64f3bc36f96f076", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "77333f3756d7434fa7476addd497eff0", "warnings": [], "timing_ms": { - "total": 3026, + "total": 4076, "by_step": { "coverage": 0, - "extract": 1213, + "extract": 1095, "cache_lookup": 0 } }, @@ -62,7 +62,7 @@ "input_tokens": 2242, "output_tokens": 197, "cached_tokens": 843, - "jev_input_tokens": 21367, + "jev_input_tokens": 21368, "jev_output_tokens": 4607 }, "timed_out": false @@ -79,28 +79,28 @@ } ], "timing_ms": { - "total": 1634, + "total": 2673, "by_step": { "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 3, - "shape_probe": 138, - "shape_groups": 5, + "shape_arcs": 5, + "shape_probe": 91, + "shape_groups": 6, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 249, + "shape": 214, "phase_prune_and_rank": 0, - "bind": 135, - "phase_bind": 11, - "phase_settle": 7, - "phase_place_bindings": 28, + "bind": 134, + "phase_bind": 13, + "phase_settle": 1, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 116, - "phase_materialise": 0, - "answer": 57, - "phase_outcome": 1030 + "materialise": 106, + "phase_materialise": 29, + "answer": 54, + "phase_outcome": 2087 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022. You named Kenya as a recipient. This measure has no recipients, so this shows the place itself.", @@ -844,7 +844,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1627,6 +1627,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1843,6 +1852,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2378,7 +2396,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2414,30 +2432,30 @@ } ], "timing_ms": { - "total": 1809, + "total": 2976, "by_step": { - "recall": 452, - "phase_resolve_places": 0, + "recall": 771, + "phase_resolve_places": 1, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 10, - "shape_probe": 142, - "shape_groups": 40, + "shape_arcs": 2, + "shape_probe_coverage": 21, + "shape_probe": 111, + "shape_groups": 66, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 666, + "shape": 621, "phase_prune_and_rank": 0, - "bind": 129, - "phase_bind": 297, - "phase_settle": 13, - "phase_place_bindings": 76, + "bind": 681, + "phase_bind": 341, + "phase_settle": 11, + "phase_place_bindings": 111, "phase_facet_requests": 0, - "materialise": 16, - "phase_materialise": 0, - "answer": 8, - "options": 33, - "phase_outcome": 119 + "materialise": 20, + "phase_materialise": 4, + "answer": 257, + "options": 11, + "phase_outcome": 147 } }, "restatement": { @@ -2864,7 +2882,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -3241,7 +3259,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -3312,7 +3330,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -6073,7 +6091,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6255e04219bd4208bca0a6f26fbe89b2" + "x-dfo-trace-id": "73fbf797c20a40a4851f5d88882cf25a" }, "json": { "schema_version": 1, @@ -6082,11 +6100,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6255e04219bd4208bca0a6f26fbe89b2", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "73fbf797c20a40a4851f5d88882cf25a", "warnings": [], "timing_ms": { - "total": 1746, + "total": 1963, "by_step": { "coverage": 0 } @@ -6099,19 +6117,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 262, + "total": 277, "by_step": { "validate": 0, "walk": 0, - "reread_arcs": 1, + "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 169, - "bind": 0, + "shape": 178, + "bind": 1, "where_places": 0, - "materialise": 91, + "materialise": 96, "answer": 1, - "options": 0 + "options": 1 } }, "entity_refs": [], @@ -6630,7 +6648,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -6679,19 +6697,19 @@ "variable_text": "aid flows", "warnings": [], "timing_ms": { - "total": 1695, + "total": 1896, "by_step": { "validate": 0, - "walk": 10, + "walk": 13, "reread_arcs": 1, "detect_edits": 0, "resolve_survivors": 0, - "shape": 1660, - "bind": 0, + "shape": 1838, + "bind": 1, "where_places": 0, - "materialise": 15, - "answer": 7, - "options": 2 + "materialise": 18, + "answer": 8, + "options": 17 } }, "entity_refs": [], @@ -7092,7 +7110,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -7540,7 +7558,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts-2.parquet b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts-2.parquet index 81ccb11..957b873 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts-2.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts-2.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.json index cbcd88d..2b3c0e5 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:37Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:24Z" }, "call": [ { @@ -39,7 +39,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "542dac0d7c43477cb66c1841c20e169d" + "x-dfo-trace-id": "94058d9236554b299cad827b56912349" }, "json": { "schema_version": 1, @@ -50,14 +50,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "542dac0d7c43477cb66c1841c20e169d", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "94058d9236554b299cad827b56912349", "warnings": [], "timing_ms": { - "total": 8962, + "total": 10762, "by_step": { "coverage": 0, - "extract": 1389, + "extract": 1197, "cache_lookup": 0 } }, @@ -65,7 +65,7 @@ "input_tokens": 2245, "output_tokens": 202, "cached_tokens": 844, - "jev_input_tokens": 26840, + "jev_input_tokens": 26837, "jev_output_tokens": 5603 }, "timed_out": false @@ -76,28 +76,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 7499, + "total": 9466, "by_step": { "recall": 1, - "phase_resolve_places": 83, + "phase_resolve_places": 72, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe": 98, - "shape_groups": 5, + "shape_probe": 94, + "shape_groups": 6, "shape_labels": 1, - "shape_admission": 75, + "shape_admission": 91, "phase_derive_shapes": 0, - "shape": 417, + "shape": 456, "phase_prune_and_rank": 0, - "bind": 125, - "phase_bind": 17, - "phase_settle": 7, - "phase_place_bindings": 23, + "bind": 131, + "phase_bind": 19, + "phase_settle": 1, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 285, - "phase_materialise": 0, - "answer": 46, - "phase_outcome": 6495 + "materialise": 434, + "phase_materialise": 50, + "answer": 78, + "phase_outcome": 8190 } }, "rendered_sentence": "3 readings. Deaths rate per capita in the 48 countries in Sub-Saharan Africa. World Development Indicators · 2013 to 2022. Number of deaths in the 48 countries in Sub-Saharan Africa. Kenya Census · 2018 to 2022. Count of mortality event (as fraction of count birth event live birth) in the 48 countries in Sub-Saharan Africa. Nigeria Statistics · 2014.", @@ -1977,7 +1977,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3427,6 +3427,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -3643,6 +3652,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -4547,7 +4565,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -5489,6 +5507,15 @@ "label": "age" }, "options": [ + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -6136,7 +6163,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -6162,30 +6189,30 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 1683, + "total": 1533, "by_step": { - "recall": 0, - "phase_resolve_places": 103, + "recall": 1, + "phase_resolve_places": 92, "phase_check_entities": 0, - "shape_arcs": 3, - "shape_probe_coverage": 3, - "shape_probe": 5, - "shape_groups": 48, - "shape_labels": 10, - "shape_admission": 112, + "shape_arcs": 4, + "shape_probe_coverage": 1, + "shape_probe": 107, + "shape_groups": 78, + "shape_labels": 14, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 447, + "shape": 503, "phase_prune_and_rank": 0, - "bind": 143, - "phase_bind": 12, - "phase_settle": 36, - "phase_place_bindings": 7, + "bind": 145, + "phase_bind": 14, + "phase_settle": 8, + "phase_place_bindings": 20, "phase_facet_requests": 0, - "materialise": 337, - "phase_materialise": 0, - "answer": 33, - "options": 426, - "phase_outcome": 139 + "materialise": 199, + "phase_materialise": 19, + "answer": 61, + "options": 144, + "phase_outcome": 327 } }, "restatement": { @@ -7510,7 +7537,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -8687,7 +8714,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -10256,7 +10283,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -11011,7 +11038,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "fed0402857db4d198957a2377d91a266" + "x-dfo-trace-id": "600d857b0d8d4a1396104f5a8b22da07" }, "json": { "schema_version": 1, @@ -11020,11 +11047,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "fed0402857db4d198957a2377d91a266", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "600d857b0d8d4a1396104f5a8b22da07", "warnings": [], "timing_ms": { - "total": 1060, + "total": 1385, "by_step": { "coverage": 0 } @@ -11037,19 +11064,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 548, + "total": 565, "by_step": { - "validate": 1, - "walk": 0, + "validate": 0, + "walk": 1, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 157, + "shape": 187, "bind": 0, "where_places": 0, - "materialise": 256, - "answer": 8, - "options": 126 + "materialise": 240, + "answer": 9, + "options": 128 } }, "entity_refs": [], @@ -12619,7 +12646,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -12633,19 +12660,19 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 937, + "total": 1244, "by_step": { "validate": 0, "walk": 0, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 657, + "shape": 915, "bind": 0, - "where_places": 1, - "materialise": 181, - "answer": 7, - "options": 91 + "where_places": 0, + "materialise": 204, + "answer": 8, + "options": 117 } }, "entity_refs": [], @@ -13812,7 +13839,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -14777,7 +14804,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -15209,7 +15236,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -15217,7 +15245,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "4afea2b786664a9fa9659b020e0016d6" + "x-dfo-trace-id": "eda4b58fea364fa8b3fbbdd89e20714d" }, "json": { "schema_version": 1, @@ -15478,7 +15506,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", "served_from": "head", - "confirmed_at": "2026-10-24T16:31:48.627461+00:00" + "confirmed_at": "2026-10-26T09:23:37.491397+00:00" }, "warnings": [], "diagnostics": [ @@ -15490,17 +15518,18 @@ ], "artifacts": [ { - "artifact_digest": "sha256:0652c59a04403a740ed0a4a75360cc04900bbcce57dfc4d7f7b7d2b21a9404dd", + "artifact_digest": "sha256:b8b53c89363b53919fb9d1501ad052892123e12fb4e788f67af0513b6e938937", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 9259, + "bytes": 9281, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T17:40:52.293546+00:00", + "created_at": "2026-10-25T14:21:12.672660+00:00", "tier": "working", - "url": "/api/datasets/spec_176fd5e63bb2a122c4ad00831497b863/cv_6c43e754af661c87eb22e7a21595a732/files/sha256:0652c59a04403a740ed0a4a75360cc04900bbcce57dfc4d7f7b7d2b21a9404dd" + "url": "/api/datasets/spec_176fd5e63bb2a122c4ad00831497b863/cv_6c43e754af661c87eb22e7a21595a732/files/sha256:b8b53c89363b53919fb9d1501ad052892123e12fb4e788f67af0513b6e938937" }, { "artifact_digest": "sha256:bd259d52a10e62d1a97e04ca5ad722c043dd38997750e8dc7a785dcd5b697056", @@ -15516,7 +15545,7 @@ ], "series_count": 48, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -15676,7 +15705,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-22T17:40:52.293692Z" @@ -15687,21 +15716,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_176fd5e63bb2a122c4ad00831497b863/cv_6c43e754af661c87eb22e7a21595a732/files/sha256:0652c59a04403a740ed0a4a75360cc04900bbcce57dfc4d7f7b7d2b21a9404dd" + "origin_path": "/api/datasets/spec_176fd5e63bb2a122c4ad00831497b863/cv_6c43e754af661c87eb22e7a21595a732/files/sha256:b8b53c89363b53919fb9d1501ad052892123e12fb4e788f67af0513b6e938937" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A0652c59a04403a740ed0a4a75360cc04900bbcce57dfc4d7f7b7d2b21a9404dd?X-Goog-Signature=redacted", - "x-dfo-trace-id": "4035ed942dbd417b883890979c32af7b" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ab8b53c89363b53919fb9d1501ad052892123e12fb4e788f67af0513b6e938937?X-Goog-Signature=redacted", + "x-dfo-trace-id": "26ade94d58da490982f1d8f48d45f655" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A0652c59a04403a740ed0a4a75360cc04900bbcce57dfc4d7f7b7d2b21a9404dd?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ab8b53c89363b53919fb9d1501ad052892123e12fb4e788f67af0513b6e938937?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -15827,7 +15856,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -15835,7 +15865,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "0dd46708d45b47a39b91408cc8700637" + "x-dfo-trace-id": "bccfe916299a4c878ffd63c2ab263027" }, "json": { "schema_version": 1, @@ -16060,7 +16090,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", "served_from": "head", - "confirmed_at": "2026-10-24T16:31:49.948810+00:00" + "confirmed_at": "2026-10-26T09:23:38.856824+00:00" }, "warnings": [ { @@ -16078,17 +16108,18 @@ ], "artifacts": [ { - "artifact_digest": "sha256:18a2e84a4829808841c3967941a92c555ac6c8c30856fe2113ba38d89e121cf7", + "artifact_digest": "sha256:af952b05a7ac047731b039b51f8876806d9c89beba61b9e94c5f552a75f175ef", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 10233, + "bytes": 10255, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T17:40:54.208385+00:00", + "created_at": "2026-10-25T14:21:14.682748+00:00", "tier": "working", - "url": "/api/datasets/spec_310c03c5f99c73c41d12226098714f81/cv_7a402f88c10de233e81d664ab22037a5/files/sha256:18a2e84a4829808841c3967941a92c555ac6c8c30856fe2113ba38d89e121cf7" + "url": "/api/datasets/spec_310c03c5f99c73c41d12226098714f81/cv_7a402f88c10de233e81d664ab22037a5/files/sha256:af952b05a7ac047731b039b51f8876806d9c89beba61b9e94c5f552a75f175ef" }, { "artifact_digest": "sha256:45c12e425c4cda8e0f4d577367ccb74e0613cb3b78fdc06b17a263fc4f803ae8", @@ -16104,7 +16135,7 @@ ], "series_count": 47, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -16264,7 +16295,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-22T17:40:54.208730Z" @@ -16275,21 +16306,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_310c03c5f99c73c41d12226098714f81/cv_7a402f88c10de233e81d664ab22037a5/files/sha256:18a2e84a4829808841c3967941a92c555ac6c8c30856fe2113ba38d89e121cf7" + "origin_path": "/api/datasets/spec_310c03c5f99c73c41d12226098714f81/cv_7a402f88c10de233e81d664ab22037a5/files/sha256:af952b05a7ac047731b039b51f8876806d9c89beba61b9e94c5f552a75f175ef" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A18a2e84a4829808841c3967941a92c555ac6c8c30856fe2113ba38d89e121cf7?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7646f48ce8414ae8aefdf014523b025c" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aaf952b05a7ac047731b039b51f8876806d9c89beba61b9e94c5f552a75f175ef?X-Goog-Signature=redacted", + "x-dfo-trace-id": "c85bcbaef6224ecb95aedae047d77704" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A18a2e84a4829808841c3967941a92c555ac6c8c30856fe2113ba38d89e121cf7?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aaf952b05a7ac047731b039b51f8876806d9c89beba61b9e94c5f552a75f175ef?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -16337,21 +16368,21 @@ "rows": 943, "columns": [ "part", - "stat_var", - "stat_var_label", - "observationAbout", - "observationAbout_label", "time_period", + "observationAbout_label", "value", - "unit", "unit_label", + "provenance_label", + "stat_var_label", + "observationAbout", + "unit", "currency", "price_basis", - "base_year", "measurementMethod", "measurementMethod_label", + "base_year", "provenance", - "provenance_label" + "stat_var" ], "part_rows": [ { diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.parquet b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.parquet index 088e609..bcfd067 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/choose-candidate-two-parts.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-echo-drops-malformed-items.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-echo-drops-malformed-items.json index 8e13000..52fe91d 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-echo-drops-malformed-items.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-echo-drops-malformed-items.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "fef53a976b3e4e1cb64f3bc36f96f076" + "x-dfo-trace-id": "77333f3756d7434fa7476addd497eff0" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "fef53a976b3e4e1cb64f3bc36f96f076", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "77333f3756d7434fa7476addd497eff0", "warnings": [], "timing_ms": { - "total": 3026, + "total": 4076, "by_step": { "coverage": 0, - "extract": 1213, + "extract": 1095, "cache_lookup": 0 } }, @@ -61,7 +61,7 @@ "input_tokens": 2242, "output_tokens": 197, "cached_tokens": 843, - "jev_input_tokens": 21367, + "jev_input_tokens": 21368, "jev_output_tokens": 4607 }, "timed_out": false @@ -78,28 +78,28 @@ } ], "timing_ms": { - "total": 1634, + "total": 2673, "by_step": { "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 3, - "shape_probe": 138, - "shape_groups": 5, + "shape_arcs": 5, + "shape_probe": 91, + "shape_groups": 6, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 249, + "shape": 214, "phase_prune_and_rank": 0, - "bind": 135, - "phase_bind": 11, - "phase_settle": 7, - "phase_place_bindings": 28, + "bind": 134, + "phase_bind": 13, + "phase_settle": 1, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 116, - "phase_materialise": 0, - "answer": 57, - "phase_outcome": 1030 + "materialise": 106, + "phase_materialise": 29, + "answer": 54, + "phase_outcome": 2087 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022. You named Kenya as a recipient. This measure has no recipients, so this shows the place itself.", @@ -843,7 +843,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1626,6 +1626,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1842,6 +1851,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2377,7 +2395,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2420,30 +2438,30 @@ } ], "timing_ms": { - "total": 1809, + "total": 2976, "by_step": { - "recall": 452, - "phase_resolve_places": 0, + "recall": 771, + "phase_resolve_places": 1, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 10, - "shape_probe": 142, - "shape_groups": 40, + "shape_arcs": 2, + "shape_probe_coverage": 21, + "shape_probe": 111, + "shape_groups": 66, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 666, + "shape": 621, "phase_prune_and_rank": 0, - "bind": 129, - "phase_bind": 297, - "phase_settle": 13, - "phase_place_bindings": 76, + "bind": 681, + "phase_bind": 341, + "phase_settle": 11, + "phase_place_bindings": 111, "phase_facet_requests": 0, - "materialise": 16, - "phase_materialise": 0, - "answer": 8, - "options": 33, - "phase_outcome": 119 + "materialise": 20, + "phase_materialise": 4, + "answer": 257, + "options": 11, + "phase_outcome": 147 } }, "restatement": { @@ -2903,7 +2921,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -3280,7 +3298,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -3351,7 +3369,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -6131,7 +6149,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6255e04219bd4208bca0a6f26fbe89b2" + "x-dfo-trace-id": "73fbf797c20a40a4851f5d88882cf25a" }, "json": { "schema_version": 1, @@ -6140,11 +6158,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6255e04219bd4208bca0a6f26fbe89b2", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "73fbf797c20a40a4851f5d88882cf25a", "warnings": [], "timing_ms": { - "total": 1746, + "total": 1963, "by_step": { "coverage": 0 } @@ -6157,19 +6175,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 262, + "total": 277, "by_step": { "validate": 0, "walk": 0, - "reread_arcs": 1, + "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 169, - "bind": 0, + "shape": 178, + "bind": 1, "where_places": 0, - "materialise": 91, + "materialise": 96, "answer": 1, - "options": 0 + "options": 1 } }, "entity_refs": [], @@ -6688,7 +6706,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -6737,19 +6755,19 @@ "variable_text": "aid flows", "warnings": [], "timing_ms": { - "total": 1695, + "total": 1896, "by_step": { "validate": 0, - "walk": 10, + "walk": 13, "reread_arcs": 1, "detect_edits": 0, "resolve_survivors": 0, - "shape": 1660, - "bind": 0, + "shape": 1838, + "bind": 1, "where_places": 0, - "materialise": 15, - "answer": 7, - "options": 2 + "materialise": 18, + "answer": 8, + "options": 17 } }, "entity_refs": [], @@ -7150,7 +7168,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -7598,7 +7616,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-echoes-separate-and-operations.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-echoes-separate-and-operations.json index 0799c24..355b698 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-echoes-separate-and-operations.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-echoes-separate-and-operations.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "009afd1dc09e44299c2004e111d17991" + "x-dfo-trace-id": "44089cedfffd4c0ca025f3d8faba0ca7" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "009afd1dc09e44299c2004e111d17991", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "44089cedfffd4c0ca025f3d8faba0ca7", "warnings": [], "timing_ms": { - "total": 2627, + "total": 2666, "by_step": { "coverage": 0, - "extract": 1263, + "extract": 1154, "cache_lookup": 0 } }, @@ -61,7 +61,7 @@ "input_tokens": 2239, "output_tokens": 164, "cached_tokens": 842, - "jev_input_tokens": 17402, + "jev_input_tokens": 17398, "jev_output_tokens": 3500 }, "timed_out": false @@ -72,30 +72,30 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 1129, + "total": 1283, "by_step": { "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe_coverage": 4, - "shape_probe": 151, - "shape_groups": 45, - "shape_labels": 10, + "shape_probe_coverage": 5, + "shape_probe": 116, + "shape_groups": 64, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 407, + "shape": 463, "phase_prune_and_rank": 0, - "bind": 127, - "phase_bind": 10, - "phase_settle": 24, - "phase_place_bindings": 8, + "bind": 125, + "phase_bind": 12, + "phase_settle": 6, + "phase_place_bindings": 10, "phase_facet_requests": 0, - "materialise": 144, - "phase_materialise": 0, - "answer": 185, - "options": 98, - "phase_outcome": 124 + "materialise": 173, + "phase_materialise": 18, + "answer": 207, + "options": 140, + "phase_outcome": 127 } }, "restatement": { @@ -993,7 +993,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -1965,7 +1965,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2884,7 +2884,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2988,24 +2988,24 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 619, + "total": 598, "by_step": { - "recall": 315, + "recall": 301, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 5, - "shape_probe": 107, + "shape_arcs": 2, + "shape_probe_coverage": 6, + "shape_probe": 99, "shape_groups": 3, "shape_labels": 0, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 196, + "shape": 189, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 2, - "phase_settle": 9, - "phase_place_bindings": 0 + "bind": 91, + "phase_bind": 3, + "phase_settle": 7, + "phase_place_bindings": 7 } }, "rendered_sentence": "Could not tell who is giving and who is receiving among \"G20\", \"Germany\", and \"India\". Put \"from\" or \"to\" before a name.", @@ -3575,7 +3575,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a837108e9cb4b74b197a33c82a6e92f" + "x-dfo-trace-id": "4f781d285d73459fa3f2f981f5828776" }, "json": { "schema_version": 1, @@ -3584,11 +3584,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6a837108e9cb4b74b197a33c82a6e92f", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "4f781d285d73459fa3f2f981f5828776", "warnings": [], "timing_ms": { - "total": 953, + "total": 1276, "by_step": { "coverage": 0 } @@ -3608,19 +3608,19 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 847, + "total": 1161, "by_step": { "validate": 0, "walk": 1, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 638, - "bind": 0, + "shape": 921, + "bind": 1, "where_places": 0, - "materialise": 110, + "materialise": 120, "answer": 4, - "options": 94 + "options": 114 } }, "entity_refs": [], @@ -4375,7 +4375,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -5135,7 +5135,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -5210,30 +5210,30 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 339, + "total": 399, "by_step": { - "recall": 1, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, + "shape_arcs": 2, "shape_probe_coverage": 6, - "shape_probe": 95, + "shape_probe": 97, "shape_groups": 3, "shape_labels": 0, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 169, + "shape": 181, "phase_prune_and_rank": 0, - "bind": 84, - "phase_bind": 9, + "bind": 100, + "phase_bind": 12, "phase_settle": 8, "phase_place_bindings": 1, "phase_facet_requests": 0, - "materialise": 17, - "phase_materialise": 0, - "answer": 9, - "options": 6, - "phase_outcome": 34 + "materialise": 20, + "phase_materialise": 5, + "answer": 15, + "options": 7, + "phase_outcome": 50 } }, "restatement": { @@ -5809,7 +5809,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -6276,7 +6276,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -6347,7 +6347,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-beside-definite.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-beside-definite.json index 45ca40c..6e9c25a 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-beside-definite.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-beside-definite.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:56Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:47Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "009afd1dc09e44299c2004e111d17991" + "x-dfo-trace-id": "44089cedfffd4c0ca025f3d8faba0ca7" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "009afd1dc09e44299c2004e111d17991", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "44089cedfffd4c0ca025f3d8faba0ca7", "warnings": [], "timing_ms": { - "total": 2627, + "total": 2666, "by_step": { "coverage": 0, - "extract": 1263, + "extract": 1154, "cache_lookup": 0 } }, @@ -62,7 +62,7 @@ "input_tokens": 2239, "output_tokens": 164, "cached_tokens": 842, - "jev_input_tokens": 17402, + "jev_input_tokens": 17398, "jev_output_tokens": 3500 }, "timed_out": false @@ -73,30 +73,30 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 1129, + "total": 1283, "by_step": { "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe_coverage": 4, - "shape_probe": 151, - "shape_groups": 45, - "shape_labels": 10, + "shape_probe_coverage": 5, + "shape_probe": 116, + "shape_groups": 64, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 407, + "shape": 463, "phase_prune_and_rank": 0, - "bind": 127, - "phase_bind": 10, - "phase_settle": 24, - "phase_place_bindings": 8, + "bind": 125, + "phase_bind": 12, + "phase_settle": 6, + "phase_place_bindings": 10, "phase_facet_requests": 0, - "materialise": 144, - "phase_materialise": 0, - "answer": 185, - "options": 98, - "phase_outcome": 124 + "materialise": 173, + "phase_materialise": 18, + "answer": 207, + "options": 140, + "phase_outcome": 127 } }, "restatement": { @@ -988,7 +988,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -1960,7 +1960,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2879,7 +2879,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2982,24 +2982,24 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 619, + "total": 598, "by_step": { - "recall": 315, + "recall": 301, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 5, - "shape_probe": 107, + "shape_arcs": 2, + "shape_probe_coverage": 6, + "shape_probe": 99, "shape_groups": 3, "shape_labels": 0, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 196, + "shape": 189, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 2, - "phase_settle": 9, - "phase_place_bindings": 0 + "bind": 91, + "phase_bind": 3, + "phase_settle": 7, + "phase_place_bindings": 7 } }, "rendered_sentence": "Could not tell who is giving and who is receiving among \"G20\", \"Germany\", and \"India\". Put \"from\" or \"to\" before a name.", @@ -3561,7 +3561,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a837108e9cb4b74b197a33c82a6e92f" + "x-dfo-trace-id": "4f781d285d73459fa3f2f981f5828776" }, "json": { "schema_version": 1, @@ -3570,11 +3570,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6a837108e9cb4b74b197a33c82a6e92f", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "4f781d285d73459fa3f2f981f5828776", "warnings": [], "timing_ms": { - "total": 953, + "total": 1276, "by_step": { "coverage": 0 } @@ -3594,19 +3594,19 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 847, + "total": 1161, "by_step": { "validate": 0, "walk": 1, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 638, - "bind": 0, + "shape": 921, + "bind": 1, "where_places": 0, - "materialise": 110, + "materialise": 120, "answer": 4, - "options": 94 + "options": 114 } }, "entity_refs": [], @@ -4361,7 +4361,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -5121,7 +5121,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -5196,30 +5196,30 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 339, + "total": 399, "by_step": { - "recall": 1, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, + "shape_arcs": 2, "shape_probe_coverage": 6, - "shape_probe": 95, + "shape_probe": 97, "shape_groups": 3, "shape_labels": 0, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 169, + "shape": 181, "phase_prune_and_rank": 0, - "bind": 84, - "phase_bind": 9, + "bind": 100, + "phase_bind": 12, "phase_settle": 8, "phase_place_bindings": 1, "phase_facet_requests": 0, - "materialise": 17, - "phase_materialise": 0, - "answer": 9, - "options": 6, - "phase_outcome": 34 + "materialise": 20, + "phase_materialise": 5, + "answer": 15, + "options": 7, + "phase_outcome": 50 } }, "restatement": { @@ -5795,7 +5795,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -6262,7 +6262,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -6333,7 +6333,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-two-part-query.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-two-part-query.json index 61d4757..73b992c 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-two-part-query.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-two-part-query.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6c90259bc5dc43649740c2e47cb555bc" + "x-dfo-trace-id": "989481cf76744b4cbddf1aff890cda9c" }, "json": { "schema_version": 1, @@ -45,8 +45,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "989481cf76744b4cbddf1aff890cda9c", "warnings": [], "timing_ms": { "total": 1, @@ -68,9 +68,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -90,6 +90,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -204,7 +205,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -233,7 +235,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -255,6 +258,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -294,7 +298,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc" + "trace_id": "989481cf76744b4cbddf1aff890cda9c" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-without-data-query.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-without-data-query.json index f2c90c7..85140b0 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-option-without-data-query.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-option-without-data-query.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6c90259bc5dc43649740c2e47cb555bc" + "x-dfo-trace-id": "989481cf76744b4cbddf1aff890cda9c" }, "json": { "schema_version": 1, @@ -45,8 +45,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "989481cf76744b4cbddf1aff890cda9c", "warnings": [], "timing_ms": { "total": 1, @@ -68,9 +68,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -90,6 +90,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -163,7 +164,8 @@ "mentions": [ "Congo" ], - "label": "Congo [DRC]" + "label": "Congo [DRC]", + "entity_choice": 0 }, { "mentions": [ @@ -192,7 +194,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -214,6 +217,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -253,7 +257,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc" + "trace_id": "989481cf76744b4cbddf1aff890cda9c" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-part-by-mention.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-part-by-mention.json index 926c0bb..2224c1f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-part-by-mention.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-part-by-mention.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:31Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:17Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "79f55e76540244d18a4543d497614a89" + "x-dfo-trace-id": "b303ed69d4a54228aadf721bfa9f73c1" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "79f55e76540244d18a4543d497614a89", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "b303ed69d4a54228aadf721bfa9f73c1", "warnings": [], "timing_ms": { "total": 1, "by_step": { "coverage": 0, - "response_cache_layer1": 1 + "response_cache_layer1": 0 } }, "llm_usage": { @@ -70,9 +70,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -92,6 +92,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -188,7 +189,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -217,7 +219,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -239,6 +242,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -291,7 +295,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "0a622ff00fe449c19cedbae4fd61f52b" + "x-dfo-trace-id": "a94d701882c9454da1459e8f957b8309" }, "json": { "schema_version": 1, @@ -300,8 +304,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "0a622ff00fe449c19cedbae4fd61f52b", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "a94d701882c9454da1459e8f957b8309", "warnings": [], "timing_ms": { "total": 3, @@ -329,30 +333,30 @@ } ], "timing_ms": { - "total": 907, + "total": 1045, "by_step": { "recall": 0, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe_coverage": 6, - "shape_probe": 109, - "shape_groups": 41, + "shape_probe_coverage": 10, + "shape_probe": 104, + "shape_groups": 56, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 387, + "shape": 473, "phase_prune_and_rank": 0, - "bind": 111, - "phase_bind": 175, - "phase_settle": 9, - "phase_place_bindings": 67, + "bind": 105, + "phase_bind": 185, + "phase_settle": 8, + "phase_place_bindings": 77, "phase_facet_requests": 0, - "materialise": 15, - "phase_materialise": 0, - "answer": 10, - "options": 13, - "phase_outcome": 120 + "materialise": 17, + "phase_materialise": 5, + "answer": 9, + "options": 26, + "phase_outcome": 140 } }, "restatement": { @@ -756,7 +760,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1210,7 +1214,7 @@ "start_year": 2018, "end_year": 2018 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-place-one-part.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-place-one-part.json index 81ed866..64b25c4 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-place-one-part.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-place-one-part.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:30Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:16Z" }, "call": [ { @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6c90259bc5dc43649740c2e47cb555bc" + "x-dfo-trace-id": "989481cf76744b4cbddf1aff890cda9c" }, "json": { "schema_version": 1, @@ -46,8 +46,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "989481cf76744b4cbddf1aff890cda9c", "warnings": [], "timing_ms": { "total": 1, @@ -69,9 +69,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -91,6 +91,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -187,7 +188,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -216,7 +218,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -238,6 +241,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -290,7 +294,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "67793b9246794fc18d48484eb317d58f" + "x-dfo-trace-id": "7cbd24b2c1a0457da3df6702b8caf3aa" }, "json": { "schema_version": 1, @@ -299,11 +303,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "67793b9246794fc18d48484eb317d58f", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "7cbd24b2c1a0457da3df6702b8caf3aa", "warnings": [], "timing_ms": { - "total": 932, + "total": 1050, "by_step": { "coverage": 0 } @@ -329,30 +333,30 @@ } ], "timing_ms": { - "total": 907, + "total": 1045, "by_step": { "recall": 0, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe_coverage": 6, - "shape_probe": 109, - "shape_groups": 41, + "shape_probe_coverage": 10, + "shape_probe": 104, + "shape_groups": 56, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 387, + "shape": 473, "phase_prune_and_rank": 0, - "bind": 111, - "phase_bind": 175, - "phase_settle": 9, - "phase_place_bindings": 67, + "bind": 105, + "phase_bind": 185, + "phase_settle": 8, + "phase_place_bindings": 77, "phase_facet_requests": 0, - "materialise": 15, - "phase_materialise": 0, - "answer": 10, - "options": 13, - "phase_outcome": 120 + "materialise": 17, + "phase_materialise": 5, + "answer": 9, + "options": 26, + "phase_outcome": 140 } }, "restatement": { @@ -756,7 +760,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1210,7 +1214,7 @@ "start_year": 2018, "end_year": 2018 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-place-two-parts.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-place-two-parts.json index 4ba7c6a..3b9e216 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-place-two-parts.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-place-two-parts.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:32Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:18Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "a49188b7f477461e9942a92b389f8671" + "x-dfo-trace-id": "2e930444ceab49e29354b073af99327e" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "a49188b7f477461e9942a92b389f8671", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2e930444ceab49e29354b073af99327e", "warnings": [], "timing_ms": { "total": 1, "by_step": { "coverage": 0, - "response_cache_layer1": 1 + "response_cache_layer1": 0 } }, "llm_usage": { @@ -92,6 +92,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -199,7 +200,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -232,7 +234,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -241,9 +244,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 1, + "total": 2, "by_step": { - "recall": 1, + "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -263,6 +266,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -370,7 +374,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -403,7 +408,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -424,6 +430,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -503,7 +510,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "e4482a207d27410a9a03cb6f915ae3df" + "x-dfo-trace-id": "729685f37628473f8bb70255010dc59a" }, "json": { "schema_version": 1, @@ -512,11 +519,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "e4482a207d27410a9a03cb6f915ae3df", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "729685f37628473f8bb70255010dc59a", "warnings": [], "timing_ms": { - "total": 984, + "total": 1092, "by_step": { "coverage": 0 } @@ -538,14 +545,14 @@ { "code": "time_window_defaulted", "severity": "warn", - "message": "no time_text was given; the answer was resolved across every period with data." + "message": "No years were given, so the answer covers every year with data." } ], "timing_ms": { "total": 1, "by_step": { - "recall": 1, - "phase_resolve_places": 0, + "recall": 0, + "phase_resolve_places": 1, "phase_check_entities": 0 } }, @@ -564,6 +571,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -671,7 +679,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -704,7 +713,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -720,34 +730,34 @@ { "code": "time_window_defaulted", "severity": "warn", - "message": "no time_text was given; the answer was resolved across every period with data." + "message": "No years were given, so the answer covers every year with data." } ], "timing_ms": { - "total": 980, + "total": 1087, "by_step": { "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe_coverage": 6, - "shape_probe": 84, - "shape_groups": 40, + "shape_probe_coverage": 7, + "shape_probe": 104, + "shape_groups": 59, "shape_labels": 1, - "shape_admission": 0, + "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 402, + "shape": 451, "phase_prune_and_rank": 0, - "bind": 106, - "phase_bind": 230, - "phase_settle": 9, - "phase_place_bindings": 83, + "bind": 105, + "phase_bind": 247, + "phase_settle": 8, + "phase_place_bindings": 77, "phase_facet_requests": 0, - "materialise": 13, - "phase_materialise": 0, - "answer": 7, + "materialise": 16, + "phase_materialise": 5, + "answer": 9, "options": 6, - "phase_outcome": 123 + "phase_outcome": 162 } }, "restatement": { @@ -1160,7 +1170,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1633,7 +1643,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3788,6 +3798,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-posts-unsupported-placeholder.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-posts-unsupported-placeholder.json index 0c8ad89..22fb908 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-posts-unsupported-placeholder.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-posts-unsupported-placeholder.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "009afd1dc09e44299c2004e111d17991" + "x-dfo-trace-id": "44089cedfffd4c0ca025f3d8faba0ca7" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "009afd1dc09e44299c2004e111d17991", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "44089cedfffd4c0ca025f3d8faba0ca7", "warnings": [], "timing_ms": { - "total": 2627, + "total": 2666, "by_step": { "coverage": 0, - "extract": 1263, + "extract": 1154, "cache_lookup": 0 } }, @@ -61,7 +61,7 @@ "input_tokens": 2239, "output_tokens": 164, "cached_tokens": 842, - "jev_input_tokens": 17402, + "jev_input_tokens": 17398, "jev_output_tokens": 3500 }, "timed_out": false @@ -72,30 +72,30 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 1129, + "total": 1283, "by_step": { "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe_coverage": 4, - "shape_probe": 151, - "shape_groups": 45, - "shape_labels": 10, + "shape_probe_coverage": 5, + "shape_probe": 116, + "shape_groups": 64, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 407, + "shape": 463, "phase_prune_and_rank": 0, - "bind": 127, - "phase_bind": 10, - "phase_settle": 24, - "phase_place_bindings": 8, + "bind": 125, + "phase_bind": 12, + "phase_settle": 6, + "phase_place_bindings": 10, "phase_facet_requests": 0, - "materialise": 144, - "phase_materialise": 0, - "answer": 185, - "options": 98, - "phase_outcome": 124 + "materialise": 173, + "phase_materialise": 18, + "answer": 207, + "options": 140, + "phase_outcome": 127 } }, "entity_refs": [ @@ -157,24 +157,24 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 619, + "total": 598, "by_step": { - "recall": 315, + "recall": 301, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 5, - "shape_probe": 107, + "shape_arcs": 2, + "shape_probe_coverage": 6, + "shape_probe": 99, "shape_groups": 3, "shape_labels": 0, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 196, + "shape": 189, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 2, - "phase_settle": 9, - "phase_place_bindings": 0 + "bind": 91, + "phase_bind": 3, + "phase_settle": 7, + "phase_place_bindings": 7 } }, "rendered_sentence": "Could not tell who is giving and who is receiving among \"G20\", \"Germany\", and \"India\". Put \"from\" or \"to\" before a name.", @@ -660,7 +660,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a837108e9cb4b74b197a33c82a6e92f" + "x-dfo-trace-id": "4f781d285d73459fa3f2f981f5828776" }, "json": { "schema_version": 1, @@ -669,11 +669,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6a837108e9cb4b74b197a33c82a6e92f", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "4f781d285d73459fa3f2f981f5828776", "warnings": [], "timing_ms": { - "total": 953, + "total": 1276, "by_step": { "coverage": 0 } @@ -693,19 +693,19 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 847, + "total": 1161, "by_step": { "validate": 0, "walk": 1, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 638, - "bind": 0, + "shape": 921, + "bind": 1, "where_places": 0, - "materialise": 110, + "materialise": 120, "answer": 4, - "options": 94 + "options": 114 } }, "entity_refs": [], @@ -1460,7 +1460,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -2220,7 +2220,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -2295,30 +2295,30 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 339, + "total": 399, "by_step": { - "recall": 1, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, + "shape_arcs": 2, "shape_probe_coverage": 6, - "shape_probe": 95, + "shape_probe": 97, "shape_groups": 3, "shape_labels": 0, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 169, + "shape": 181, "phase_prune_and_rank": 0, - "bind": 84, - "phase_bind": 9, + "bind": 100, + "phase_bind": 12, "phase_settle": 8, "phase_place_bindings": 1, "phase_facet_requests": 0, - "materialise": 17, - "phase_materialise": 0, - "answer": 9, - "options": 6, - "phase_outcome": 34 + "materialise": 20, + "phase_materialise": 5, + "answer": 15, + "options": 7, + "phase_outcome": 50 } }, "restatement": { @@ -2894,7 +2894,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -3361,7 +3361,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -3432,7 +3432,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-answered-part.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-answered-part.json index 17760cf..e5437b3 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-answered-part.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-answered-part.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -46,21 +46,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -72,30 +72,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -570,7 +570,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1288,7 +1288,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3273,18 +3273,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-label-in-two-groups.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-label-in-two-groups.json new file mode 100644 index 0000000..fb234df --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-label-in-two-groups.json @@ -0,0 +1,395 @@ +{ + "format_version": 1, + "name": "choose-refused-label-in-two-groups", + "description": "Both entity_choice groups offer an option labelled Congo [DRC], so the label matches two options and the choice raises InvalidRequestError with no request. A choice in such a part names its option by index.", + "origin": { + "kind": "synthetic", + "derived_from": "search-place-ambiguous-twice", + "edit": "A choice of \"Congo [DRC]\" follows the search." + }, + "call": [ + { + "op": "search", + "question": "aid from Congo to Congo" + }, + { + "op": "choose", + "option": "Congo [DRC]" + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "raw_text", + "query": "aid from Congo to Congo" + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "14c23c7a7a01430eb3f485e658ae7e51" + }, + "json": { + "schema_version": 1, + "query_echo": { + "entry_path": "raw_text", + "raw_query": "aid from Congo to Congo", + "normalized_query": "aid from Congo to Congo", + "extract_skipped": false + }, + "diagnostics": { + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "14c23c7a7a01430eb3f485e658ae7e51", + "warnings": [], + "timing_ms": { + "total": 1535, + "by_step": { + "coverage": 0, + "extract": 1020, + "cache_lookup": 0 + } + }, + "llm_usage": { + "input_tokens": 2226, + "output_tokens": 106, + "cached_tokens": 840 + }, + "timed_out": false + }, + "kind": "resolution", + "parts": [ + { + "variable_text": "aid", + "warnings": [], + "timing_ms": { + "total": 377, + "by_step": { + "recall": 377, + "phase_resolve_places": 0, + "phase_check_entities": 0 + } + }, + "rendered_sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "sentence_plan": [ + { + "kind": "text", + "text": "\"", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Congo", + "segment": "heading", + "role": "origin", + "entity_choice": 0, + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": "\" (from) and \"", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Congo", + "axis": "where", + "segment": "heading", + "role": "destination", + "entity_choice": 1, + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": "\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "segment": "heading", + "entity_refs": [], + "states": [] + } + ], + "restatement": { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + }, + "entity_refs": [ + { + "text": "Congo", + "refs": [] + }, + { + "text": "Congo", + "refs": [] + } + ], + "operations": [], + "departures": [], + "assumptions": [], + "status": "no_data", + "no_data": { + "reason": "entity_ambiguous", + "category": "unresolved", + "message": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "entity_terms": [ + "Congo", + "Congo" + ], + "ambiguous_entities": [ + { + "text": "Congo", + "candidates": [ + { + "ref": "country/COD", + "label": "Congo [DRC]", + "kind": "country" + }, + { + "ref": "country/COG", + "label": "Congo [Republic]", + "kind": "country" + } + ] + }, + { + "text": "Congo", + "candidates": [ + { + "ref": "country/COD", + "label": "Congo [DRC]", + "kind": "country" + }, + { + "ref": "country/COG", + "label": "Congo [Republic]", + "kind": "country" + } + ] + } + ], + "source_terms": [], + "ambiguous_sources": [], + "unbound_qualifiers": [], + "options": [ + { + "mentions": [ + "Congo" + ], + "label": "Congo [DRC]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "role_text": "from", + "dcid": "country/COD" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 0 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [Republic]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "role_text": "from", + "dcid": "country/COG" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 0 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [DRC]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "role_text": "to", + "dcid": "country/COD" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 1 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [Republic]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "role_text": "to", + "dcid": "country/COG" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 1 + } + ] + } + } + ], + "rendered_sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "sentence_plan": [ + { + "kind": "text", + "text": "\"", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Congo", + "segment": "heading", + "role": "origin", + "entity_choice": 0, + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "\" (from) and \"", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Congo", + "axis": "where", + "segment": "heading", + "role": "destination", + "entity_choice": 1, + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + } + ], + "status": "no_data" + } + } + } + ], + "expect": [ + { + "answer": { + "kind": "resolution", + "status": "no_data", + "sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "complete": false, + "specs": [], + "gaps": [ + { + "part": 0, + "kind": "choice", + "options": 4 + } + ] + } + }, + { + "error": { + "class": "InvalidRequestError", + "code": "choice_invalid", + "trace_id": null + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-mention-names-two-parts.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-mention-names-two-parts.json index e8343b6..a6964dc 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-mention-names-two-parts.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-mention-names-two-parts.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "4b576ea3cd364ddeb664b3b219dffb19" + "x-dfo-trace-id": "5df79c87dd3c43589df5c2921f3c93e3" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "4b576ea3cd364ddeb664b3b219dffb19", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5df79c87dd3c43589df5c2921f3c93e3", "warnings": [], "timing_ms": { - "total": 1260, + "total": 1209, "by_step": { "coverage": 0, - "extract": 1255, + "extract": 1202, "cache_lookup": 0 } }, @@ -92,6 +92,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -199,7 +200,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -232,7 +234,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -241,9 +244,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 1, + "total": 2, "by_step": { - "recall": 1, + "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -263,6 +266,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -370,7 +374,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -403,7 +408,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -424,6 +430,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-choice.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-choice.json index f77a9d9..6025dd5 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-choice.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-choice.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -42,26 +42,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -71,30 +68,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -541,7 +538,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1194,7 +1191,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-options.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-options.json index 1ec2f9f..0ea3c80 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-options.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-no-options.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -69,9 +69,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -91,6 +91,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -178,6 +179,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-option-out-of-range.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-option-out-of-range.json index ac800b2..5900087 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-option-out-of-range.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-option-out-of-range.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -45,22 +45,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -71,28 +71,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -814,7 +814,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1597,6 +1597,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1813,6 +1822,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2348,7 +2366,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-part-without-variable-text.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-part-without-variable-text.json index 07027a2..31f75cf 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-part-without-variable-text.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-part-without-variable-text.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "009afd1dc09e44299c2004e111d17991" + "x-dfo-trace-id": "44089cedfffd4c0ca025f3d8faba0ca7" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "009afd1dc09e44299c2004e111d17991", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "44089cedfffd4c0ca025f3d8faba0ca7", "warnings": [], "timing_ms": { - "total": 2627, + "total": 2666, "by_step": { "coverage": 0, - "extract": 1263, + "extract": 1154, "cache_lookup": 0 } }, @@ -61,7 +61,7 @@ "input_tokens": 2239, "output_tokens": 164, "cached_tokens": 842, - "jev_input_tokens": 17402, + "jev_input_tokens": 17398, "jev_output_tokens": 3500 }, "timed_out": false @@ -71,30 +71,30 @@ { "warnings": [], "timing_ms": { - "total": 1129, + "total": 1283, "by_step": { "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe_coverage": 4, - "shape_probe": 151, - "shape_groups": 45, - "shape_labels": 10, + "shape_probe_coverage": 5, + "shape_probe": 116, + "shape_groups": 64, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 407, + "shape": 463, "phase_prune_and_rank": 0, - "bind": 127, - "phase_bind": 10, - "phase_settle": 24, - "phase_place_bindings": 8, + "bind": 125, + "phase_bind": 12, + "phase_settle": 6, + "phase_place_bindings": 10, "phase_facet_requests": 0, - "materialise": 144, - "phase_materialise": 0, - "answer": 185, - "options": 98, - "phase_outcome": 124 + "materialise": 173, + "phase_materialise": 18, + "answer": 207, + "options": 140, + "phase_outcome": 127 } }, "restatement": { @@ -986,7 +986,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -1958,7 +1958,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2877,7 +2877,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2980,24 +2980,24 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 619, + "total": 598, "by_step": { - "recall": 315, + "recall": 301, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 5, - "shape_probe": 107, + "shape_arcs": 2, + "shape_probe_coverage": 6, + "shape_probe": 99, "shape_groups": 3, "shape_labels": 0, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 196, + "shape": 189, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 2, - "phase_settle": 9, - "phase_place_bindings": 0 + "bind": 91, + "phase_bind": 3, + "phase_settle": 7, + "phase_place_bindings": 7 } }, "rendered_sentence": "Could not tell who is giving and who is receiving among \"G20\", \"Germany\", and \"India\". Put \"from\" or \"to\" before a name.", diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-candidate-label.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-candidate-label.json index 91e3f40..85414ff 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-candidate-label.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-candidate-label.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -45,22 +45,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -71,28 +71,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -814,7 +814,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1597,6 +1597,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1813,6 +1822,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2293,7 +2311,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-label.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-label.json index fad67ba..6886548 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-label.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-shared-label.json @@ -45,8 +45,8 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "6c90259bc5dc43649740c2e47cb555bc", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "989481cf76744b4cbddf1aff890cda9c", "warnings": [], "timing_ms": { "total": 1, @@ -68,9 +68,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -90,6 +90,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -186,7 +187,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -215,7 +217,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -237,6 +240,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-two-choices.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-two-choices.json index ae28288..b078e79 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-two-choices.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-two-choices.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "4b576ea3cd364ddeb664b3b219dffb19" + "x-dfo-trace-id": "5df79c87dd3c43589df5c2921f3c93e3" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "4b576ea3cd364ddeb664b3b219dffb19", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5df79c87dd3c43589df5c2921f3c93e3", "warnings": [], "timing_ms": { - "total": 1260, + "total": 1209, "by_step": { "coverage": 0, - "extract": 1255, + "extract": 1202, "cache_lookup": 0 } }, @@ -91,6 +91,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -198,7 +199,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -231,7 +233,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -240,9 +243,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 1, + "total": 2, "by_step": { - "recall": 1, + "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -262,6 +265,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -369,7 +373,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -402,7 +407,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -423,6 +429,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-candidate-label.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-candidate-label.json index 8a9eadc..893fd5f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-candidate-label.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-candidate-label.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "21bb19c92c824a70b2eb378e85a9cfd0" + "x-dfo-trace-id": "2de415a9f27d4ce69e4f1eb97b5951af" }, "json": { "schema_version": 1, @@ -45,22 +45,22 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "21bb19c92c824a70b2eb378e85a9cfd0", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2de415a9f27d4ce69e4f1eb97b5951af", "warnings": [], "timing_ms": { - "total": 8287, + "total": 4676, "by_step": { - "coverage": 0, - "extract": 1330, - "cache_lookup": 2 + "coverage": 1, + "extract": 1, + "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 143, - "cached_tokens": 842, - "jev_input_tokens": 13358, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, + "jev_input_tokens": 13359, "jev_output_tokens": 2855 }, "timed_out": false @@ -71,28 +71,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 6951, + "total": 4672, "by_step": { - "recall": 261, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 7, - "shape_probe": 122, + "shape_arcs": 6, + "shape_probe": 120, "shape_groups": 6, - "shape_labels": 1, + "shape_labels": 2, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 241, + "shape": 258, "phase_prune_and_rank": 0, - "bind": 142, - "phase_bind": 17, - "phase_settle": 6, - "phase_place_bindings": 490, + "bind": 154, + "phase_bind": 18, + "phase_settle": 1, + "phase_place_bindings": 39, "phase_facet_requests": 0, - "materialise": 115, - "phase_materialise": 1, - "answer": 243, - "phase_outcome": 5435 + "materialise": 202, + "phase_materialise": 33, + "answer": 30, + "phase_outcome": 3936 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -814,7 +814,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1597,6 +1597,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1813,6 +1822,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2348,7 +2366,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-label.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-label.json index e99807c..49de559 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-label.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-label.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -69,9 +69,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -91,6 +91,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -187,7 +188,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -216,7 +218,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -238,6 +241,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-part.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-part.json index 8442bcc..57beb8e 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-part.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-part.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1e95940d311043d0b63bd27a87996439" + "x-dfo-trace-id": "86b361f85db44a29b3d711350962ca73" }, "json": { "schema_version": 1, @@ -46,21 +46,21 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "1e95940d311043d0b63bd27a87996439", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "86b361f85db44a29b3d711350962ca73", "warnings": [], "timing_ms": { - "total": 2200, + "total": 456, "by_step": { "coverage": 0, - "extract": 1432, + "extract": 0, "cache_lookup": 0 } }, "llm_usage": { - "input_tokens": 2232, - "output_tokens": 103, - "cached_tokens": 841, + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0, "jev_input_tokens": 5250, "jev_output_tokens": 1104 }, @@ -72,30 +72,30 @@ "variable_text": "debt service", "warnings": [], "timing_ms": { - "total": 757, + "total": 452, "by_step": { - "recall": 316, + "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 8, - "shape_probe": 72, - "shape_groups": 5, - "shape_labels": 10, + "shape_probe_coverage": 17, + "shape_probe": 101, + "shape_groups": 8, + "shape_labels": 14, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 179, + "shape": 225, "phase_prune_and_rank": 0, - "bind": 89, - "phase_bind": 10, - "phase_settle": 13, - "phase_place_bindings": 12, + "bind": 77, + "phase_bind": 17, + "phase_settle": 5, + "phase_place_bindings": 14, "phase_facet_requests": 0, - "materialise": 9, - "phase_materialise": 0, - "answer": 1, - "options": 79, - "phase_outcome": 49 + "materialise": 11, + "phase_materialise": 23, + "answer": 2, + "options": 9, + "phase_outcome": 68 } }, "restatement": { @@ -570,7 +570,7 @@ "label": "Creative Commons Attribution License" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1288,7 +1288,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3273,18 +3273,18 @@ "variable_text": "price of tea", "warnings": [], "timing_ms": { - "total": 330, + "total": 121, "by_step": { - "recall": 216, - "phase_resolve_places": 1, + "recall": 0, + "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, - "shape_probe": 108, - "shape_groups": 2, + "shape_probe": 96, + "shape_groups": 1, "shape_labels": 0, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 113, + "shape": 121, "phase_prune_and_rank": 0 } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-status.json b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-status.json index f6404bf..151f531 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-status.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/choose-refused-unknown-status.json @@ -35,7 +35,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "009afd1dc09e44299c2004e111d17991" + "x-dfo-trace-id": "44089cedfffd4c0ca025f3d8faba0ca7" }, "json": { "schema_version": 1, @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "009afd1dc09e44299c2004e111d17991", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "44089cedfffd4c0ca025f3d8faba0ca7", "warnings": [], "timing_ms": { - "total": 2627, + "total": 2666, "by_step": { "coverage": 0, - "extract": 1263, + "extract": 1154, "cache_lookup": 0 } }, @@ -61,7 +61,7 @@ "input_tokens": 2239, "output_tokens": 164, "cached_tokens": 842, - "jev_input_tokens": 17402, + "jev_input_tokens": 17398, "jev_output_tokens": 3500 }, "timed_out": false @@ -72,30 +72,30 @@ "variable_text": "GDP", "warnings": [], "timing_ms": { - "total": 1129, + "total": 1283, "by_step": { "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe_coverage": 4, - "shape_probe": 151, - "shape_groups": 45, - "shape_labels": 10, + "shape_probe_coverage": 5, + "shape_probe": 116, + "shape_groups": 64, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 407, + "shape": 463, "phase_prune_and_rank": 0, - "bind": 127, - "phase_bind": 10, - "phase_settle": 24, - "phase_place_bindings": 8, + "bind": 125, + "phase_bind": 12, + "phase_settle": 6, + "phase_place_bindings": 10, "phase_facet_requests": 0, - "materialise": 144, - "phase_materialise": 0, - "answer": 185, - "options": 98, - "phase_outcome": 124 + "materialise": 173, + "phase_materialise": 18, + "answer": 207, + "options": 140, + "phase_outcome": 127 } }, "entity_refs": [ @@ -970,7 +970,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -1942,7 +1942,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2861,7 +2861,7 @@ "start_year": 2015, "end_year": 2015 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -2964,24 +2964,24 @@ "variable_text": "ODA", "warnings": [], "timing_ms": { - "total": 619, + "total": 598, "by_step": { - "recall": 315, + "recall": 301, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 1, - "shape_probe_coverage": 5, - "shape_probe": 107, + "shape_arcs": 2, + "shape_probe_coverage": 6, + "shape_probe": 99, "shape_groups": 3, "shape_labels": 0, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 196, + "shape": 189, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 2, - "phase_settle": 9, - "phase_place_bindings": 0 + "bind": 91, + "phase_bind": 3, + "phase_settle": 7, + "phase_place_bindings": 7 } }, "rendered_sentence": "Could not tell who is giving and who is receiving among \"G20\", \"Germany\", and \"India\". Put \"from\" or \"to\" before a name.", diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-frame-several-readings.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-frame-several-readings.json index a526083..775f678 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-frame-several-readings.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-frame-several-readings.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "74e9acc99aff40f7a653093976c55e3b" + "x-dfo-trace-id": "5e0f0d6a5a6442a38932b02d7f72ed6c" }, "json": { "schema_version": 1, @@ -42,26 +42,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", "normalized_query": "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "74e9acc99aff40f7a653093976c55e3b", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5e0f0d6a5a6442a38932b02d7f72ed6c", "warnings": [], "timing_ms": { - "total": 4632, + "total": 8, "by_step": { "coverage": 0, - "extract": 1209, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2237, - "output_tokens": 215, - "cached_tokens": 842, - "jev_input_tokens": 25612, - "jev_output_tokens": 5740 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -71,30 +68,30 @@ "variable_text": "health ODA disbursements", "warnings": [], "timing_ms": { - "total": 3118, + "total": 4401, "by_step": { - "recall": 753, + "recall": 1271, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 75, - "shape_probe_coverage": 3, - "shape_probe": 164, - "shape_groups": 44, - "shape_labels": 9, + "shape_arcs": 108, + "shape_probe_coverage": 4, + "shape_probe": 113, + "shape_groups": 72, + "shape_labels": 12, "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 695, + "shape": 524, "phase_prune_and_rank": 0, - "bind": 275, - "phase_bind": 190, - "phase_settle": 91, - "phase_place_bindings": 19, + "bind": 482, + "phase_bind": 205, + "phase_settle": 57, + "phase_place_bindings": 34, "phase_facet_requests": 0, - "materialise": 616, - "phase_materialise": 0, - "answer": 20, - "options": 307, - "phase_outcome": 152 + "materialise": 686, + "phase_materialise": 225, + "answer": 28, + "options": 685, + "phase_outcome": 204 } }, "restatement": { @@ -1438,7 +1435,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3766,30 +3763,30 @@ } ], "timing_ms": { - "total": 3017, + "total": 4561, "by_step": { - "recall": 704, - "phase_resolve_places": 1, + "recall": 1310, + "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 82, - "shape_probe_coverage": 2, - "shape_probe": 99, - "shape_groups": 47, - "shape_labels": 10, + "shape_arcs": 117, + "shape_probe_coverage": 28, + "shape_probe": 126, + "shape_groups": 83, + "shape_labels": 12, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 365, + "shape": 753, "phase_prune_and_rank": 0, - "bind": 402, - "phase_bind": 196, - "phase_settle": 88, - "phase_place_bindings": 44, + "bind": 309, + "phase_bind": 209, + "phase_settle": 82, + "phase_place_bindings": 25, "phase_facet_requests": 0, - "materialise": 512, - "phase_materialise": 0, - "answer": 58, - "options": 511, - "phase_outcome": 136 + "materialise": 1061, + "phase_materialise": 233, + "answer": 29, + "options": 385, + "phase_outcome": 165 } }, "restatement": { @@ -5133,7 +5130,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -7466,30 +7463,30 @@ } ], "timing_ms": { - "total": 1849, + "total": 2851, "by_step": { - "recall": 391, + "recall": 360, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, "shape_probe_coverage": 2, - "shape_probe": 126, - "shape_groups": 55, - "shape_labels": 10, - "shape_admission": 1, + "shape_probe": 121, + "shape_groups": 841, + "shape_labels": 17, + "shape_admission": 4, "phase_derive_shapes": 0, - "shape": 311, + "shape": 1100, "phase_prune_and_rank": 0, - "bind": 139, - "phase_bind": 15, - "phase_settle": 27, - "phase_place_bindings": 426, + "bind": 261, + "phase_bind": 16, + "phase_settle": 7, + "phase_place_bindings": 554, "phase_facet_requests": 0, - "materialise": 93, - "phase_materialise": 0, - "answer": 244, - "options": 193, - "phase_outcome": 10 + "materialise": 96, + "phase_materialise": 17, + "answer": 239, + "options": 188, + "phase_outcome": 13 } }, "restatement": { @@ -8103,7 +8100,7 @@ "value": { "ref": { "ref": "FinancialActivities", - "label": "FinancialActivities" + "label": "Financial activities" }, "value_kind": "enum_value" } @@ -8926,7 +8923,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ @@ -9459,7 +9456,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-null-member.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-null-member.json index e1916ab..d8795d5 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-null-member.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-null-member.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -41,26 +41,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -70,30 +67,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -540,7 +537,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1193,7 +1190,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3783,7 +3780,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3791,7 +3789,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4192,7 +4190,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4214,7 +4212,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4237,7 +4236,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4521,7 +4520,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4538,7 +4537,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -4573,32 +4572,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-partial-nothing-answered.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-partial-nothing-answered.json index 86968a3..2adeebd 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-partial-nothing-answered.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-partial-nothing-answered.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -45,14 +45,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -69,9 +69,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -91,6 +91,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -187,7 +188,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -216,7 +218,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -238,6 +241,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -277,7 +281,7 @@ "error": { "class": "IncompleteAnswerError", "code": "answer_incomplete", - "trace_id": "304fe92b998b45849ff11a547aadc3e8" + "trace_id": "01cf4a9623c449759f24d249f8783de1" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-artifact-unlisted.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-artifact-unlisted.json index 44e3548..f77ae51 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-artifact-unlisted.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-artifact-unlisted.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -111,7 +112,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a9a38fa39014e7282da94b82d6ba3ee" + "x-dfo-trace-id": "6bc9cad45e3c48769587db03ca0826a9" }, "json": { "schema_version": 1, @@ -527,6 +528,20 @@ } ], "artifacts": [ + { + "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": "frame", + "without_party_totals": true + }, + "bytes": 5627, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-20T20:44:10.228016+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104" + }, { "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", "rendering": { @@ -643,7 +658,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -943,7 +958,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -966,32 +981,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-file-selector.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-file-selector.json index dff0cc7..d5debee 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-file-selector.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-file-selector.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -111,7 +112,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a9a38fa39014e7282da94b82d6ba3ee" + "x-dfo-trace-id": "6bc9cad45e3c48769587db03ca0826a9" }, "json": { "schema_version": 1, @@ -527,6 +528,21 @@ } ], "artifacts": [ + { + "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4, + "without_party_totals": true + }, + "bytes": 9842, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-23T16:56:32.058938+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, { "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { @@ -541,6 +557,20 @@ "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, + { + "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": "frame", + "without_party_totals": true + }, + "bytes": 5627, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-20T20:44:10.228016+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104" + }, { "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", "rendering": { @@ -657,7 +687,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -979,7 +1009,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -1002,32 +1032,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-newer-version.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-newer-version.json index 35c65cb..beb1daf 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-newer-version.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-newer-version.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -203,7 +204,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -211,7 +213,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -612,7 +614,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -634,7 +636,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -657,7 +660,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -940,7 +943,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-not-reproducible.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-not-reproducible.json index 517e8c4..84a42e1 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-not-reproducible.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-not-reproducible.json @@ -91,7 +91,8 @@ "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -201,7 +202,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -231,7 +233,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6f93d15300c8413191296f90fbc4b8b0" + "x-dfo-trace-id": "7675f31f4bd548c782dad4c6001a7666" }, "json": { "schema_version": 1, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-other-spec.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-other-spec.json index d8e8c73..aac6fda 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-other-spec.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-other-spec.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -203,7 +204,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -211,7 +213,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -612,7 +614,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -634,7 +636,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -657,7 +660,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -940,7 +943,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -953,7 +956,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "42460c90e7274dd19ecba24a6a0377e9" + "trace_id": "4c3f35c8c0d44b99941a73a9519b41d2" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-same-version.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-same-version.json index 87efd88..680ad6d 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-same-version.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-expired-rebuild-same-version.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -203,7 +204,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -211,7 +213,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -612,7 +614,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -634,7 +636,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -657,7 +660,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -940,7 +943,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -957,7 +960,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -980,32 +983,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-newest-version.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-newest-version.json index 8916674..609e0ee 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-newest-version.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-newest-version.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:32:01Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:52Z" }, "call": [ { @@ -94,7 +94,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6f93d15300c8413191296f90fbc4b8b0" + "x-dfo-trace-id": "7675f31f4bd548c782dad4c6001a7666" }, "json": { "schema_version": 1, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused-twice.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused-twice.json index 85a980e..5d12fa5 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused-twice.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused-twice.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -111,7 +112,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a9a38fa39014e7282da94b82d6ba3ee" + "x-dfo-trace-id": "6bc9cad45e3c48769587db03ca0826a9" }, "json": { "schema_version": 1, @@ -527,6 +528,21 @@ } ], "artifacts": [ + { + "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4, + "without_party_totals": true + }, + "bytes": 9842, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-23T16:56:32.058938+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, { "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { @@ -541,6 +557,20 @@ "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, + { + "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": "frame", + "without_party_totals": true + }, + "bytes": 5627, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-20T20:44:10.228016+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104" + }, { "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", "rendering": { @@ -657,7 +687,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -957,7 +987,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -983,7 +1013,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused.json index 23646db..a379515 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-signed-url-refused.json @@ -93,7 +93,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -111,7 +112,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a9a38fa39014e7282da94b82d6ba3ee" + "x-dfo-trace-id": "6bc9cad45e3c48769587db03ca0826a9" }, "json": { "schema_version": 1, @@ -527,6 +528,21 @@ } ], "artifacts": [ + { + "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4, + "without_party_totals": true + }, + "bytes": 9842, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-23T16:56:32.058938+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, { "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { @@ -541,6 +557,20 @@ "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, + { + "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": "frame", + "without_party_totals": true + }, + "bytes": 5627, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-20T20:44:10.228016+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104" + }, { "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", "rendering": { @@ -657,7 +687,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -957,7 +987,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -983,7 +1013,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -1006,32 +1036,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-unknown-version.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-unknown-version.json index 50d58fe..e594531 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-unknown-version.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref-unknown-version.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:53Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:34Z" }, "call": [ { @@ -94,7 +94,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -112,7 +113,7 @@ "status": 404, "headers": { "content-type": "application/problem+json", - "x-dfo-trace-id": "5228b20a07a147e9861e20170e7363f1" + "x-dfo-trace-id": "72953e5cc1f1470dac2c80cf5889052d" }, "json": { "type": "/codes/snapshot_not_found", @@ -120,8 +121,8 @@ "status": 404, "detail": "'spec_774d6c001a6250aa67ca676ab067895e' has no version at content_version 'cv_00000000000000000000000000000000'", "code": "snapshot_not_found", - "trace_id": "5228b20a07a147e9861e20170e7363f1", - "capability_version": "dset_cc753cefa9199881", + "trace_id": "72953e5cc1f1470dac2c80cf5889052d", + "capability_version": "dset_76f5fa7244511c58", "retryable": false } } @@ -132,7 +133,7 @@ "error": { "class": "NotFoundError", "code": "snapshot_not_found", - "trace_id": "5228b20a07a147e9861e20170e7363f1" + "trace_id": "72953e5cc1f1470dac2c80cf5889052d" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref.json index 856d5c5..5c43080 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-ref.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:51Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:32Z" }, "call": [ { @@ -94,7 +94,8 @@ "digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } } @@ -112,7 +113,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6a9a38fa39014e7282da94b82d6ba3ee" + "x-dfo-trace-id": "6bc9cad45e3c48769587db03ca0826a9" }, "json": { "schema_version": 1, @@ -528,6 +529,21 @@ } ], "artifacts": [ + { + "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4, + "without_party_totals": true + }, + "bytes": 9842, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-23T16:56:32.058938+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, { "artifact_digest": "sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "rendering": { @@ -542,6 +558,20 @@ "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, + { + "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": "frame", + "without_party_totals": true + }, + "bytes": 5627, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-20T20:44:10.228016+00:00", + "tier": "working", + "url": "/api/datasets/spec_774d6c001a6250aa67ca676ab067895e/cv_b92d94774702c7f05f2075d0b03dd756/files/sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104" + }, { "artifact_digest": "sha256:f4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104", "rendering": { @@ -658,7 +688,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -958,7 +988,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "7e3ff4f01985498db43aebbf83d13839" + "x-dfo-trace-id": "9db04233957a46068e3999162a93c70c" } } }, @@ -981,32 +1011,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-renamed-strict-field.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-renamed-strict-field.json index 5e6acb1..cdf37fa 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-renamed-strict-field.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-renamed-strict-field.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -41,26 +41,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -70,30 +67,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -540,7 +537,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1193,7 +1190,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3783,7 +3780,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3791,7 +3789,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4191,7 +4189,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4213,7 +4211,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4236,7 +4235,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4519,7 +4518,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true }, @@ -4545,7 +4544,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "42460c90e7274dd19ecba24a6a0377e9" + "trace_id": "4c3f35c8c0d44b99941a73a9519b41d2" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-retry-after-504.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-retry-after-504.json index 6e10641..cd3d71e 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-retry-after-504.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-retry-after-504.json @@ -33,7 +33,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -41,26 +41,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -70,30 +67,30 @@ "variable_text": "ODA disbursements for basic health care", "warnings": [], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -540,7 +537,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1193,7 +1190,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3783,7 +3780,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3883,7 +3881,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3891,7 +3890,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4292,7 +4291,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4314,7 +4313,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4337,7 +4337,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4620,7 +4620,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4637,7 +4637,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -4672,32 +4672,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-503-then-504.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-503-then-504.json index b3db0e5..701f36e 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-503-then-504.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-503-then-504.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -269,7 +270,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -371,7 +373,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -379,7 +382,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -786,7 +789,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -808,7 +811,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -831,7 +835,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1099,7 +1103,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -1117,7 +1121,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -1140,32 +1144,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-retry-after.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-retry-after.json index 05c7be4..11761b6 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-retry-after.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-retry-after.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -270,7 +271,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -278,7 +280,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -685,7 +687,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -707,7 +709,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -730,7 +733,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -998,7 +1001,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -1016,7 +1019,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -1039,32 +1042,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-then-503.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-then-503.json index 7f82896..dbd3d1f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-then-503.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-then-503.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -269,7 +270,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-twice.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-twice.json index 22f9d24..9326930 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-twice.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-504-twice.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -269,7 +270,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-accepted-running.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-accepted-running.json index 8f0d10e..ab81f2c 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-accepted-running.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-accepted-running.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,24 +503,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -528,24 +529,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -554,24 +555,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -580,24 +633,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -810,14 +889,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -839,14 +918,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -965,13 +1044,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -980,8 +1059,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -989,15 +1068,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1009,34 +1088,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1060,7 +1139,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1300,7 +1379,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1324,7 +1403,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1383,10 +1462,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1395,21 +1474,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1423,47 +1502,50 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-async-build.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-async-build.json index ecdf113..bcc6706 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-async-build.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-async-build.json @@ -5,16 +5,16 @@ "origin": { "kind": "synthetic", "derived_from": "mint-async-build", - "edit": "The mint and result steps become one fetch of the spec, which makes the same exchanges." + "edit": "The mint and result steps become one fetch of the spec with party_totals, which makes the same exchanges." }, "call": [ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,24 +503,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -528,24 +529,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -554,24 +555,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -580,24 +633,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -810,14 +889,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -839,14 +918,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -965,13 +1044,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -980,8 +1059,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -989,15 +1068,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1009,34 +1088,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1060,7 +1139,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1300,7 +1379,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1324,7 +1403,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1383,10 +1462,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1395,21 +1474,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1423,47 +1502,50 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-capability-unavailable.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-capability-unavailable.json index f5495bf..694c332 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-capability-unavailable.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-capability-unavailable.json @@ -165,7 +165,8 @@ "options": { "format": "parquet", "bundle": false, - "layout": "frame" + "layout": "frame", + "without_party_totals": true } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-digest-mismatch.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-digest-mismatch.json index ab4e913..2a98b3b 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-digest-mismatch.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-digest-mismatch.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -913,7 +915,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -940,7 +942,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-empty-narrowing.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-empty-narrowing.json index 38256c0..7095cf3 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-empty-narrowing.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-empty-narrowing.json @@ -169,7 +169,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -177,7 +178,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -584,7 +585,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -606,7 +607,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -629,7 +631,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -897,7 +899,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -915,7 +917,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -938,32 +940,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.json index 6d1c279..166b104 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:35Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:17Z" }, "call": [ { @@ -174,7 +174,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -182,7 +183,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "1855a065f7494cd6ba7eecd9fd790858" + "x-dfo-trace-id": "dfa8f009e43a48bd9113306c1946b4b7" }, "json": { "schema_version": 1, @@ -584,7 +585,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261020T170016Z-691bf0da", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:35.784659+00:00" + "confirmed_at": "2026-10-26T09:18:11.197140+00:00" }, "warnings": [], "diagnostics": [ @@ -596,17 +597,18 @@ ], "artifacts": [ { - "artifact_digest": "sha256:582f586bcb05d6807c91e6b5d88779c13c5c0962cd42e793184e8c6c8c90d2d9", + "artifact_digest": "sha256:e8092d8c75768ba31abe07fa60d85c1d13f7ed8b2159acb7785e0041f44e771a", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 13398, + "bytes": 13418, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T16:58:55.692876+00:00", + "created_at": "2026-10-25T14:19:44.744913+00:00", "tier": "working", - "url": "/api/datasets/spec_4c228c73be6c18efd0939eabb3b3c456/cv_1868d697e8e7d56da6a83740a940c2e5/files/sha256:582f586bcb05d6807c91e6b5d88779c13c5c0962cd42e793184e8c6c8c90d2d9", + "url": "/api/datasets/spec_4c228c73be6c18efd0939eabb3b3c456/cv_1868d697e8e7d56da6a83740a940c2e5/files/sha256:e8092d8c75768ba31abe07fa60d85c1d13f7ed8b2159acb7785e0041f44e771a", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { @@ -623,7 +625,7 @@ ], "series_count": 58, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1024,7 +1026,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-20T06:50:59.874372Z" @@ -1113,7 +1115,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "37247e5873b241a7aee7ac321de35814" + "x-dfo-trace-id": "de250b63d7d54077a2e7ced00745249b" }, "json": { "schema_version": 1, @@ -1122,11 +1124,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "37247e5873b241a7aee7ac321de35814", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "de250b63d7d54077a2e7ced00745249b", "warnings": [], "timing_ms": { - "total": 442, + "total": 2, "by_step": { "coverage": 0 } @@ -1141,23 +1143,13 @@ { "code": "role_scope_defaulted", "severity": "warn", - "message": "no provider was named; the answer was resolved across every provider with data." + "message": "No provider was named, so the answer covers every provider with data." } ], "timing_ms": { - "total": 321, + "total": 0, "by_step": { - "validate": 3, - "walk": 227, - "reread_arcs": 1, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 4, - "bind": 0, - "where_places": 0, - "materialise": 33, - "answer": 42, - "options": 11 + "response_cache_part_memo": 0 } }, "entity_refs": [], @@ -1572,7 +1564,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -2301,7 +2293,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -2554,43 +2546,44 @@ ], "open_axis_facts": [ { - "property": "crsCooperationModality", + "property": "crsTypeOfFinance", "varies": true, - "distinct_count": 14, - "undeclared_count": 2, + "distinct_count": 2, + "undeclared_count": 0, "values": [ { "value": { - "ref": "CRSCooperationModalityC01", - "label": "project-type intervention" - }, - "count": 5 - }, - { - "value": { - "ref": "CRSCooperationModalityA02", - "label": "sector budget support" + "ref": "CRSTypeOfFinance110", + "label": "standard grant" }, - "count": 2 + "count": 9 }, { "value": { - "ref": "CRSCooperationModalityB04", - "label": "basket fund and pooled funding" + "ref": "CRSTypeOfFinance421", + "label": "standard loan" }, "count": 1 - }, + } + ] + }, + { + "property": "crsCooperationModality", + "varies": true, + "distinct_count": 8, + "undeclared_count": 1, + "values": [ { "value": { - "ref": "CRSCooperationModalityB031", - "label": "contribution to multi-donor and multi-entity funding mechanism" + "ref": "CRSCooperationModalityC01", + "label": "project-type intervention" }, - "count": 1 + "count": 2 }, { "value": { - "ref": "CRSCooperationModalityB032", - "label": "contribution to multi-donor and single-entity funding mechanism" + "ref": "CRSCooperationModalityB04", + "label": "basket fund and pooled funding" }, "count": 1 }, @@ -2622,27 +2615,6 @@ }, "count": 1 }, - { - "value": { - "ref": "CRSCooperationModalityA01", - "label": "general budget support" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE02", - "label": "imputed student cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityG01", - "label": "other administrative cost" - }, - "count": 1 - }, { "value": { "ref": "CRSCooperationModalityD02", @@ -2658,49 +2630,6 @@ "count": 1 } ] - }, - { - "property": "crsTypeOfFinance", - "varies": true, - "distinct_count": 5, - "undeclared_count": 0, - "values": [ - { - "value": { - "ref": "CRSTypeOfFinance110", - "label": "standard grant" - }, - "count": 14 - }, - { - "value": { - "ref": "CRSTypeOfFinance421", - "label": "standard loan" - }, - "count": 4 - }, - { - "value": { - "ref": "CRSTypeOfFinance510", - "label": "common equity" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance210", - "label": "interest subsidy" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance520", - "label": "share in collective investment vehicle" - }, - "count": 1 - } - ] } ], "role_axis_facts": [ @@ -4926,21 +4855,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_4c228c73be6c18efd0939eabb3b3c456/cv_1868d697e8e7d56da6a83740a940c2e5/files/sha256:582f586bcb05d6807c91e6b5d88779c13c5c0962cd42e793184e8c6c8c90d2d9" + "origin_path": "/api/datasets/spec_4c228c73be6c18efd0939eabb3b3c456/cv_1868d697e8e7d56da6a83740a940c2e5/files/sha256:e8092d8c75768ba31abe07fa60d85c1d13f7ed8b2159acb7785e0041f44e771a" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A582f586bcb05d6807c91e6b5d88779c13c5c0962cd42e793184e8c6c8c90d2d9?X-Goog-Signature=redacted", - "x-dfo-trace-id": "c76ebbf49c774bb7acee93d8877f450d" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ae8092d8c75768ba31abe07fa60d85c1d13f7ed8b2159acb7785e0041f44e771a?X-Goog-Signature=redacted", + "x-dfo-trace-id": "3971fe2094964eb69642432194306145" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A582f586bcb05d6807c91e6b5d88779c13c5c0962cd42e793184e8c6c8c90d2d9?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ae8092d8c75768ba31abe07fa60d85c1d13f7ed8b2159acb7785e0041f44e771a?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -4956,32 +4885,32 @@ "dataset": { "rows": 309, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "crsCooperationModality", "crsCooperationModality_label", - "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", + "crsPurpose", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.parquet b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.parquet index 878d002..ecb1ada 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-dropped.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-rebound.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-rebound.json index 9cf73b6..d246838 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-rebound.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-entity-rebound.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:33Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:15Z" }, "call": [ { @@ -179,7 +179,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -187,7 +188,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "102dc8c602004615becaa7905509e259" + "x-dfo-trace-id": "042e6c2ebb034091b3fbbfcef0565c48" }, "json": { "schema_version": 1, @@ -592,7 +593,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261020T170016Z-691bf0da", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:33.522693+00:00" + "confirmed_at": "2026-10-26T09:18:08.892616+00:00" }, "warnings": [], "diagnostics": [ @@ -608,7 +609,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9116, "media_type": "application/vnd.apache.parquet", @@ -631,7 +633,7 @@ ], "series_count": 2, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -914,7 +916,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-20T06:50:59.083841Z" @@ -1006,7 +1008,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "c379d5c730dc4f85b1a5630c2a8b85f8" + "x-dfo-trace-id": "155a086b6b2b46fe899457496517c746" }, "json": { "schema_version": 1, @@ -1015,11 +1017,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "c379d5c730dc4f85b1a5630c2a8b85f8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "155a086b6b2b46fe899457496517c746", "warnings": [], "timing_ms": { - "total": 461, + "total": 3, "by_step": { "coverage": 0 } @@ -1032,19 +1034,9 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 405, + "total": 1, "by_step": { - "validate": 4, - "walk": 190, - "reread_arcs": 2, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 4, - "bind": 0, - "where_places": 93, - "materialise": 14, - "answer": 95, - "options": 3 + "response_cache_part_memo": 1 } }, "entity_refs": [], @@ -1124,7 +1116,7 @@ "resolved": { "shape": { "shape_id": "financialflow_amount_measuredvalue_nominal", - "label": "gross disbursements (basic health care, official development assistance)", + "label": "gross disbursements (basic health care, official development assistance, standard grant)", "population_type": { "ref": "FinancialFlow", "label": "financial flow" @@ -1458,7 +1450,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1732,6 +1724,7 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -2057,7 +2050,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -2318,65 +2311,9 @@ { "property": "crsCooperationModality", "varies": true, - "distinct_count": 14, - "undeclared_count": 2, + "distinct_count": 2, + "undeclared_count": 0, "values": [ - { - "value": { - "ref": "CRSCooperationModalityC01", - "label": "project-type intervention" - }, - "count": 5 - }, - { - "value": { - "ref": "CRSCooperationModalityA02", - "label": "sector budget support" - }, - "count": 2 - }, - { - "value": { - "ref": "CRSCooperationModalityB04", - "label": "basket fund and pooled funding" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB031", - "label": "contribution to multi-donor and multi-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB032", - "label": "contribution to multi-donor and single-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB033", - "label": "contribution to single-donor funding mechanism or earmarked contribution to a specific funding window or geographical location" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB03", - "label": "contribution to specific-purpose programme and fund managed by an international organisation" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB01", - "label": "core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes" - }, - "count": 1 - }, { "value": { "ref": "CRSCooperationModalityD01", @@ -2386,36 +2323,8 @@ }, { "value": { - "ref": "CRSCooperationModalityA01", - "label": "general budget support" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE02", - "label": "imputed student cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityG01", - "label": "other administrative cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityD02", - "label": "other technical assistance" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE01", - "label": "scholarship and training in donor country" + "ref": "CRSCooperationModalityC01", + "label": "project-type intervention" }, "count": 1 } @@ -2423,8 +2332,12 @@ }, { "property": "crsTypeOfFinance", - "varies": true, - "distinct_count": 5, + "varies": false, + "value": { + "ref": "CRSTypeOfFinance110", + "label": "standard grant" + }, + "distinct_count": 1, "undeclared_count": 0, "values": [ { @@ -2432,35 +2345,7 @@ "ref": "CRSTypeOfFinance110", "label": "standard grant" }, - "count": 14 - }, - { - "value": { - "ref": "CRSTypeOfFinance421", - "label": "standard loan" - }, - "count": 4 - }, - { - "value": { - "ref": "CRSTypeOfFinance510", - "label": "common equity" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance210", - "label": "interest subsidy" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance520", - "label": "share in collective investment vehicle" - }, - "count": 1 + "count": 2 } ] } @@ -4474,6 +4359,7 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -4664,7 +4550,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A13a775f3b4811a104717ac226f173d614d685e533def09482b83c55e06505c4c?X-Goog-Signature=redacted", - "x-dfo-trace-id": "9e24d0570f8f4988a32209d5670538c9" + "x-dfo-trace-id": "1300eead85a84452b36be91c751c6206" } } }, @@ -4687,32 +4573,32 @@ "dataset": { "rows": 2, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-no-content.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-no-content.json index 4130d61..e364bed 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-no-content.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-no-content.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -911,7 +913,7 @@ "response": { "status": 204, "headers": { - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } } @@ -921,7 +923,7 @@ "error": { "class": "ContractError", "code": "unexpected_response", - "trace_id": "8261c9be5a7c406a80692fac6b28ee05" + "trace_id": "cf3f86713afb44c98d21fbf9731d66e3" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-serves-file.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-serves-file.json index 818ed05..dc2c862 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-serves-file.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-serves-file.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -912,7 +914,7 @@ "status": 200, "headers": { "content-type": "application/vnd.apache.parquet", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" }, "body_file": "fetch-spec-known-sentence.parquet" } @@ -923,32 +925,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-temporary-redirect.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-temporary-redirect.json index c24d0d2..32dc6ef 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-temporary-redirect.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-files-route-temporary-redirect.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -912,7 +914,7 @@ "status": 307, "headers": { "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } } @@ -922,7 +924,7 @@ "error": { "class": "ContractError", "code": "unexpected_response", - "trace_id": "8261c9be5a7c406a80692fac6b28ee05" + "trace_id": "cf3f86713afb44c98d21fbf9731d66e3" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-foreign-artifact-url.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-foreign-artifact-url.json index 6e83998..1424844 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-foreign-artifact-url.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-foreign-artifact-url.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -909,7 +911,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "trace_id": "b89cda3bb794448abf06b56c955d754e" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-frame-layout.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-frame-layout.json index 728a173..bd68531 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-frame-layout.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-frame-layout.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:23Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:08Z" }, "call": [ { @@ -166,7 +166,8 @@ "options": { "format": "parquet", "bundle": false, - "layout": "frame" + "layout": "frame", + "without_party_totals": true } } }, @@ -174,7 +175,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bb332867f3894a6291e202e0b4666901" + "x-dfo-trace-id": "b236e3739a984c4da5c2a1ba3fe66cc9" }, "json": { "schema_version": 1, @@ -575,7 +576,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -597,7 +598,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": "frame" + "layout": "frame", + "without_party_totals": true }, "bytes": 5627, "media_type": "application/vnd.apache.parquet", @@ -619,7 +621,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -902,7 +904,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -919,7 +921,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Af4026e80af4b8683d3bb86645ab3aca83dff75cd03fdd4c73ec54ba638922104?X-Goog-Signature=redacted", - "x-dfo-trace-id": "1621b753678947a1ab313a52be193785" + "x-dfo-trace-id": "5abca8a9651b4bca8e2d422e6b8c02a1" } } }, @@ -942,19 +944,19 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", + "time_period", "financingProvider", "financingRecipient", - "time_period", + "stat_var_crsPurpose", "value", "unit", - "measurementMethod", + "stat_var_crsTypeOfFlow", "provenance", - "status", "stat_var_crsCooperationModality", - "stat_var_crsPurpose", + "measurementMethod", + "status", "stat_var_crsTypeOfFinance", - "stat_var_crsTypeOfFlow" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.json index 9a5b59c..136dde5 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:37Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:26Z" }, "call": [ { @@ -241,7 +241,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -249,7 +250,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "031e93c3d83b4ef9bbead5c617599dee" + "x-dfo-trace-id": "c658ba6641be4c4183f82bbab7cabc5a" }, "json": { "schema_version": 1, @@ -535,7 +536,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261021T070020Z-7262111e", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:41.604527+00:00" + "confirmed_at": "2026-10-26T09:18:33.682273+00:00" }, "warnings": [ { @@ -553,17 +554,18 @@ ], "artifacts": [ { - "artifact_digest": "sha256:a5198249a7b16b895b1b7e8127605c644edaf2cda240201e391fc377aaf4dafd", + "artifact_digest": "sha256:87ee2f60de69561349a468cc45a902dc585b7726affc5fa2d080750df7ce9a72", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 8777, + "bytes": 8810, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T16:59:04.339940+00:00", + "created_at": "2026-10-25T14:20:02.459303+00:00", "tier": "working", - "url": "/api/datasets/spec_f04936b8378cde70bc6289ac1e45ee9b/cv_daec84e443d4e1b37291242d90fc8fa8/files/sha256:a5198249a7b16b895b1b7e8127605c644edaf2cda240201e391fc377aaf4dafd", + "url": "/api/datasets/spec_f04936b8378cde70bc6289ac1e45ee9b/cv_daec84e443d4e1b37291242d90fc8fa8/files/sha256:87ee2f60de69561349a468cc45a902dc585b7726affc5fa2d080750df7ce9a72", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { @@ -580,7 +582,7 @@ ], "series_count": 43, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -869,7 +871,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-20T07:08:48.808348Z" @@ -999,7 +1001,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "11177bfa56c249feaec154d376c0e318" + "x-dfo-trace-id": "b0a96c8e862d4a3ea8b0dea282c6ada4" }, "json": { "schema_version": 1, @@ -1008,13 +1010,13 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "11177bfa56c249feaec154d376c0e318", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "b0a96c8e862d4a3ea8b0dea282c6ada4", "warnings": [], "timing_ms": { - "total": 1198, + "total": 2, "by_step": { - "coverage": 0 + "coverage": 1 } }, "timed_out": false @@ -1025,19 +1027,9 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 1118, + "total": 0, "by_step": { - "validate": 0, - "walk": 11, - "reread_arcs": 0, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 1069, - "bind": 0, - "where_places": 1, - "materialise": 24, - "answer": 11, - "options": 2 + "response_cache_part_memo": 0 } }, "entity_refs": [], @@ -2030,7 +2022,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -2872,7 +2864,7 @@ "start_year": 2020, "end_year": 2023 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -5305,21 +5297,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_f04936b8378cde70bc6289ac1e45ee9b/cv_daec84e443d4e1b37291242d90fc8fa8/files/sha256:a5198249a7b16b895b1b7e8127605c644edaf2cda240201e391fc377aaf4dafd" + "origin_path": "/api/datasets/spec_f04936b8378cde70bc6289ac1e45ee9b/cv_daec84e443d4e1b37291242d90fc8fa8/files/sha256:87ee2f60de69561349a468cc45a902dc585b7726affc5fa2d080750df7ce9a72" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa5198249a7b16b895b1b7e8127605c644edaf2cda240201e391fc377aaf4dafd?X-Goog-Signature=redacted", - "x-dfo-trace-id": "f34686e7f0454a87b31b8f3a23ee43a4" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A87ee2f60de69561349a468cc45a902dc585b7726affc5fa2d080750df7ce9a72?X-Goog-Signature=redacted", + "x-dfo-trace-id": "83e8a270332a4fdd9ae6127e98ee0d0d" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa5198249a7b16b895b1b7e8127605c644edaf2cda240201e391fc377aaf4dafd?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A87ee2f60de69561349a468cc45a902dc585b7726affc5fa2d080750df7ce9a72?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -5335,26 +5327,26 @@ "dataset": { "rows": 170, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingRecipient", + "financingProvider", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.parquet b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.parquet index b7ceb83..001b8f4 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-narrowed.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.json index e961827..f5318d4 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:44Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:27Z" }, "call": [ { @@ -288,7 +288,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -296,7 +297,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "be9bb0e6abf54384a9226f44f6f09902" + "x-dfo-trace-id": "e32f376d58684753b9bc195b1429569b" }, "json": { "schema_version": 1, @@ -580,7 +581,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261021T070020Z-7262111e", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:44.568363+00:00" + "confirmed_at": "2026-10-26T09:18:36.478834+00:00" }, "warnings": [], "diagnostics": [ @@ -592,17 +593,18 @@ ], "artifacts": [ { - "artifact_digest": "sha256:a382b8f9e3ba634c150c6588e21f79413722d53f7bcdcb9dfd4e929aa4f80993", + "artifact_digest": "sha256:e9513ce3953896d5b833a71546a5987a82e3f18219cd43eb987fc69dfab80321", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 21088, + "bytes": 21121, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T16:59:11.517434+00:00", + "created_at": "2026-10-25T14:20:06.098634+00:00", "tier": "working", - "url": "/api/datasets/spec_c2f21170dfa9f31705e47ed60c8fa305/cv_66a0a4a0331d59b519aa1629dade3d99/files/sha256:a382b8f9e3ba634c150c6588e21f79413722d53f7bcdcb9dfd4e929aa4f80993", + "url": "/api/datasets/spec_c2f21170dfa9f31705e47ed60c8fa305/cv_66a0a4a0331d59b519aa1629dade3d99/files/sha256:e9513ce3953896d5b833a71546a5987a82e3f18219cd43eb987fc69dfab80321", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { @@ -619,7 +621,7 @@ ], "series_count": 44, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -908,7 +910,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-20T07:08:52.858156Z" @@ -1036,7 +1038,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "de81189cf3d441c69f9bc70f4786a3d4" + "x-dfo-trace-id": "76155faedf434e46b697c5bd58c318f8" }, "json": { "schema_version": 1, @@ -1045,11 +1047,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "de81189cf3d441c69f9bc70f4786a3d4", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "76155faedf434e46b697c5bd58c318f8", "warnings": [], "timing_ms": { - "total": 2566, + "total": 2, "by_step": { "coverage": 0 } @@ -1062,19 +1064,9 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 2493, + "total": 0, "by_step": { - "validate": 1, - "walk": 10, - "reread_arcs": 1, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 1756, - "bind": 1, - "where_places": 0, - "materialise": 20, - "answer": 702, - "options": 2 + "response_cache_part_memo": 0 } }, "entity_refs": [], @@ -2045,7 +2037,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -2883,7 +2875,7 @@ } ], "query_qualifiers": [], - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -5304,21 +5296,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_c2f21170dfa9f31705e47ed60c8fa305/cv_66a0a4a0331d59b519aa1629dade3d99/files/sha256:a382b8f9e3ba634c150c6588e21f79413722d53f7bcdcb9dfd4e929aa4f80993" + "origin_path": "/api/datasets/spec_c2f21170dfa9f31705e47ed60c8fa305/cv_66a0a4a0331d59b519aa1629dade3d99/files/sha256:e9513ce3953896d5b833a71546a5987a82e3f18219cd43eb987fc69dfab80321" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa382b8f9e3ba634c150c6588e21f79413722d53f7bcdcb9dfd4e929aa4f80993?X-Goog-Signature=redacted", - "x-dfo-trace-id": "d62572e0146a43f89d89c138493238e1" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ae9513ce3953896d5b833a71546a5987a82e3f18219cd43eb987fc69dfab80321?X-Goog-Signature=redacted", + "x-dfo-trace-id": "75d565fbb9c948c8b131be201b1cf923" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa382b8f9e3ba634c150c6588e21f79413722d53f7bcdcb9dfd4e929aa4f80993?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ae9513ce3953896d5b833a71546a5987a82e3f18219cd43eb987fc69dfab80321?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -5334,26 +5326,26 @@ "dataset": { "rows": 2114, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingRecipient", + "financingProvider", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.parquet b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.parquet index accd075..6c76c4a 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-group-origin-rebound.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-known-sentence.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-known-sentence.json index f58acdc..ecc048b 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-known-sentence.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-known-sentence.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:48Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:29Z" }, "call": [ { @@ -168,7 +168,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -176,7 +177,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -583,7 +584,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -605,7 +606,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -628,7 +630,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -896,7 +898,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -914,7 +916,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -937,32 +939,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-identity.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-identity.json index 5f174b5..2e809f4 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-identity.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-identity.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -517,7 +518,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -539,7 +540,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -562,7 +564,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -830,7 +832,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -844,7 +846,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "trace_id": "b89cda3bb794448abf06b56c955d754e" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-schema-version.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-schema-version.json index 40803a3..ef7f6ec 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-schema-version.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-manifest-without-schema-version.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -581,7 +582,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -603,7 +604,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -626,7 +628,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -894,7 +896,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -908,7 +910,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "trace_id": "b89cda3bb794448abf06b56c955d754e" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-member-not-resolvable.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-member-not-resolvable.json index ff7635a..73dcf74 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-member-not-resolvable.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-member-not-resolvable.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-narrowed.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-narrowed.json index 3e6c833..69f12eb 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-narrowed.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-narrowed.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:30Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:13Z" }, "call": [ { @@ -175,7 +175,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -183,7 +184,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "a1fe53d87b784d23bcc55926b9530dec" + "x-dfo-trace-id": "250fbb95951e4a8c8d847cbcde0ae4f9" }, "json": { "schema_version": 1, @@ -590,7 +591,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -612,7 +613,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -635,7 +637,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -903,7 +905,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -997,7 +999,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "c4313642ac68484fb1d13d6f7b206d66" + "x-dfo-trace-id": "ff646392059e46e68f83a2753e921d6a" }, "json": { "schema_version": 1, @@ -1006,11 +1008,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "c4313642ac68484fb1d13d6f7b206d66", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "ff646392059e46e68f83a2753e921d6a", "warnings": [], "timing_ms": { - "total": 340, + "total": 1, "by_step": { "coverage": 0 } @@ -1023,19 +1025,9 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 256, + "total": 0, "by_step": { - "validate": 4, - "walk": 219, - "reread_arcs": 1, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 3, - "bind": 1, - "where_places": 0, - "materialise": 12, - "answer": 14, - "options": 2 + "response_cache_part_memo": 0 } }, "entity_refs": [], @@ -1117,7 +1109,7 @@ "resolved": { "shape": { "shape_id": "financialflow_amount_measuredvalue_nominal", - "label": "gross disbursements (basic health care, official development assistance)", + "label": "gross disbursements (basic health care, official development assistance, core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes, standard grant)", "population_type": { "ref": "FinancialFlow", "label": "financial flow" @@ -1469,7 +1461,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1703,6 +1695,8 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsCooperationModality", + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -2032,7 +2026,7 @@ "start_year": 2020, "end_year": 2023 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -2296,114 +2290,31 @@ "open_axis_facts": [ { "property": "crsCooperationModality", - "varies": true, - "distinct_count": 14, - "undeclared_count": 2, + "varies": false, + "value": { + "ref": "CRSCooperationModalityB01", + "label": "core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes" + }, + "distinct_count": 1, + "undeclared_count": 0, "values": [ - { - "value": { - "ref": "CRSCooperationModalityC01", - "label": "project-type intervention" - }, - "count": 5 - }, - { - "value": { - "ref": "CRSCooperationModalityA02", - "label": "sector budget support" - }, - "count": 2 - }, - { - "value": { - "ref": "CRSCooperationModalityB04", - "label": "basket fund and pooled funding" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB031", - "label": "contribution to multi-donor and multi-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB032", - "label": "contribution to multi-donor and single-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB033", - "label": "contribution to single-donor funding mechanism or earmarked contribution to a specific funding window or geographical location" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB03", - "label": "contribution to specific-purpose programme and fund managed by an international organisation" - }, - "count": 1 - }, { "value": { "ref": "CRSCooperationModalityB01", "label": "core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes" }, "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityD01", - "label": "donor country personnel" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityA01", - "label": "general budget support" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE02", - "label": "imputed student cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityG01", - "label": "other administrative cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityD02", - "label": "other technical assistance" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE01", - "label": "scholarship and training in donor country" - }, - "count": 1 } ] }, { "property": "crsTypeOfFinance", - "varies": true, - "distinct_count": 5, + "varies": false, + "value": { + "ref": "CRSTypeOfFinance110", + "label": "standard grant" + }, + "distinct_count": 1, "undeclared_count": 0, "values": [ { @@ -2411,34 +2322,6 @@ "ref": "CRSTypeOfFinance110", "label": "standard grant" }, - "count": 14 - }, - { - "value": { - "ref": "CRSTypeOfFinance421", - "label": "standard loan" - }, - "count": 4 - }, - { - "value": { - "ref": "CRSTypeOfFinance510", - "label": "common equity" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance210", - "label": "interest subsidy" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance520", - "label": "share in collective investment vehicle" - }, "count": 1 } ] @@ -4450,6 +4333,8 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsCooperationModality", + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -4640,7 +4525,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "2f0c6beb58394dd390dc7e490b5e6c00" + "x-dfo-trace-id": "6a74097c16e04e21857031860d5c9de1" } } }, @@ -4663,32 +4548,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-operation-failed-resubmitted.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-operation-failed-resubmitted.json index 1fec5d0..919f817 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-operation-failed-resubmitted.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-operation-failed-resubmitted.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,20 +503,20 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "failed", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 }, "error": { @@ -539,10 +540,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -754,7 +755,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -770,15 +771,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -787,24 +788,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -813,24 +814,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -839,24 +840,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -865,24 +918,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -1095,14 +1174,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -1124,14 +1203,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -1250,13 +1329,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -1265,8 +1344,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -1274,15 +1353,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1294,34 +1373,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1345,7 +1424,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1585,7 +1664,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1609,7 +1688,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1668,10 +1747,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1680,21 +1759,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1708,48 +1787,51 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 1.5, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 1 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.json new file mode 100644 index 0000000..90b582d --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.json @@ -0,0 +1,4460 @@ +{ + "format_version": 1, + "name": "fetch-spec-party-totals-kept", + "description": "Fetching the spec of fetch-spec-party-totals-left-out with party_totals=True posts no without_party_totals, so the build keeps the donor group totals and answers the plain version, with more rows.", + "origin": { + "kind": "recorded", + "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:22Z" + }, + "call": [ + { + "op": "fetch", + "spec": { + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "definition": { + "schema_version": 1, + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/AFG", + "country/AGO", + "country/ARE", + "country/ARG", + "country/ATG", + "country/AZE", + "country/BDI", + "country/BEN", + "country/BFA", + "country/BGD", + "country/BHR", + "country/BHS", + "country/BLZ", + "country/BOL", + "country/BRA", + "country/BRB", + "country/BRN", + "country/BTN", + "country/BWA", + "country/CAF", + "country/CHL", + "country/CHN", + "country/CIV", + "country/CMR", + "country/COD", + "country/COG", + "country/COL", + "country/COM", + "country/CPV", + "country/CRI", + "country/CUB", + "country/DJI", + "country/DMA", + "country/DOM", + "country/DZA", + "country/ECU", + "country/EGY", + "country/ERI", + "country/ETH", + "country/FJI", + "country/FSM", + "country/GAB", + "country/GHA", + "country/GIN", + "country/GMB", + "country/GNB", + "country/GNQ", + "country/GRD", + "country/GTM", + "country/GUY", + "country/HND", + "country/HTI", + "country/IDN", + "country/IND", + "country/IRN", + "country/IRQ", + "country/JAM", + "country/JOR", + "country/KEN", + "country/KHM", + "country/KIR", + "country/KNA", + "country/KWT", + "country/LAO", + "country/LBN", + "country/LBR", + "country/LBY", + "country/LCA", + "country/LKA", + "country/LSO", + "country/MAR", + "country/MDG", + "country/MDV", + "country/MHL", + "country/MLI", + "country/MMR", + "country/MNG", + "country/MOZ", + "country/MRT", + "country/MUS", + "country/MWI", + "country/MYS", + "country/NAM", + "country/NER", + "country/NGA", + "country/NIC", + "country/NPL", + "country/NRU", + "country/OMN", + "country/PAK", + "country/PAN", + "country/PER", + "country/PHL", + "country/PNG", + "country/PRK", + "country/PRY", + "country/QAT", + "country/RWA", + "country/SAU", + "country/SDN", + "country/SEN", + "country/SGP", + "country/SLB", + "country/SLE", + "country/SLV", + "country/SOM", + "country/SSD", + "country/STP", + "country/SUR", + "country/SWZ", + "country/SYC", + "country/SYR", + "country/TCD", + "country/TGO", + "country/THA", + "country/TJK", + "country/TKM", + "country/TLS", + "country/TON", + "country/TTO", + "country/TUN", + "country/TZA", + "country/UGA", + "country/URY", + "country/VCT", + "country/VEN", + "country/VNM", + "country/VUT", + "country/WSM", + "country/YEM", + "country/ZAF", + "country/ZMB", + "country/ZWE" + ], + "origin": [ + "country/AUS", + "country/AUT", + "country/BEL", + "country/CAN", + "country/CHE", + "country/CZE", + "country/DEU", + "country/DNK", + "country/ESP", + "country/EST", + "country/FIN", + "country/FRA", + "country/GBR", + "country/GRC", + "country/HUN", + "country/IRL", + "country/ISL", + "country/ITA", + "country/JPN", + "country/KOR", + "country/LTU", + "country/LUX", + "country/LVA", + "country/NLD", + "country/NOR", + "country/NZL", + "country/POL", + "country/PRT", + "country/SVK", + "country/SVN", + "country/SWE", + "country/USA" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 1967, + "end_year": 2024 + } + } + }, + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", + "entity_groups": { + "destination": { + "label": "G77 countries", + "group_id": "groups/G77" + }, + "origin": { + "label": "DAC countries", + "group_id": "groups/OECD.DAC" + } + } + }, + "window": [ + 2022, + 2022 + ], + "entities": { + "origin": null, + "destination": [ + "country/KEN" + ] + }, + "party_totals": true + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/datasets", + "json": { + "schema_version": 1, + "input": { + "kind": "spec_id", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "definition": { + "schema_version": 1, + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/AFG", + "country/AGO", + "country/ARE", + "country/ARG", + "country/ATG", + "country/AZE", + "country/BDI", + "country/BEN", + "country/BFA", + "country/BGD", + "country/BHR", + "country/BHS", + "country/BLZ", + "country/BOL", + "country/BRA", + "country/BRB", + "country/BRN", + "country/BTN", + "country/BWA", + "country/CAF", + "country/CHL", + "country/CHN", + "country/CIV", + "country/CMR", + "country/COD", + "country/COG", + "country/COL", + "country/COM", + "country/CPV", + "country/CRI", + "country/CUB", + "country/DJI", + "country/DMA", + "country/DOM", + "country/DZA", + "country/ECU", + "country/EGY", + "country/ERI", + "country/ETH", + "country/FJI", + "country/FSM", + "country/GAB", + "country/GHA", + "country/GIN", + "country/GMB", + "country/GNB", + "country/GNQ", + "country/GRD", + "country/GTM", + "country/GUY", + "country/HND", + "country/HTI", + "country/IDN", + "country/IND", + "country/IRN", + "country/IRQ", + "country/JAM", + "country/JOR", + "country/KEN", + "country/KHM", + "country/KIR", + "country/KNA", + "country/KWT", + "country/LAO", + "country/LBN", + "country/LBR", + "country/LBY", + "country/LCA", + "country/LKA", + "country/LSO", + "country/MAR", + "country/MDG", + "country/MDV", + "country/MHL", + "country/MLI", + "country/MMR", + "country/MNG", + "country/MOZ", + "country/MRT", + "country/MUS", + "country/MWI", + "country/MYS", + "country/NAM", + "country/NER", + "country/NGA", + "country/NIC", + "country/NPL", + "country/NRU", + "country/OMN", + "country/PAK", + "country/PAN", + "country/PER", + "country/PHL", + "country/PNG", + "country/PRK", + "country/PRY", + "country/QAT", + "country/RWA", + "country/SAU", + "country/SDN", + "country/SEN", + "country/SGP", + "country/SLB", + "country/SLE", + "country/SLV", + "country/SOM", + "country/SSD", + "country/STP", + "country/SUR", + "country/SWZ", + "country/SYC", + "country/SYR", + "country/TCD", + "country/TGO", + "country/THA", + "country/TJK", + "country/TKM", + "country/TLS", + "country/TON", + "country/TTO", + "country/TUN", + "country/TZA", + "country/UGA", + "country/URY", + "country/VCT", + "country/VEN", + "country/VNM", + "country/VUT", + "country/WSM", + "country/YEM", + "country/ZAF", + "country/ZMB", + "country/ZWE" + ], + "origin": [ + "country/AUS", + "country/AUT", + "country/BEL", + "country/CAN", + "country/CHE", + "country/CZE", + "country/DEU", + "country/DNK", + "country/ESP", + "country/EST", + "country/FIN", + "country/FRA", + "country/GBR", + "country/GRC", + "country/HUN", + "country/IRL", + "country/ISL", + "country/ITA", + "country/JPN", + "country/KOR", + "country/LTU", + "country/LUX", + "country/LVA", + "country/NLD", + "country/NOR", + "country/NZL", + "country/POL", + "country/PRT", + "country/SVK", + "country/SVN", + "country/SWE", + "country/USA" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 1967, + "end_year": 2024 + } + } + }, + "scope": { + "window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + }, + "entities": { + "destination": [ + "country/KEN" + ] + } + } + }, + "options": { + "format": "parquet", + "bundle": false + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "7459cea38c154e609d1f2142b94f4cbe" + }, + "json": { + "schema_version": 1, + "definition": { + "schema_version": 1, + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "identity": { + "shape": { + "kind": "statvar_schema", + "stat_type": "measuredValue", + "population_type": "FinancialFlow", + "measured_property": "amount", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "binding": { + "ref": "DAC2aAidType206", + "state": "value" + }, + "property": "dac2aAidType" + }, + { + "binding": { + "state": "unbound" + }, + "property": "dacTypeOfFinance" + }, + { + "binding": { + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" + }, + { + "binding": { + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "identity_version": "spec1", + "requested_window": { + "kind": "range", + "end_year": 2022, + "start_year": 2022 + } + } + }, + "content_version": "cv_2aa4cc2fd852cd9b84347c61d07d609a", + "prepared_at": "2026-10-12T18:33:25Z", + "coverage": { + "rows": 111, + "dimensions": [ + { + "label": "financingProvider", + "count": 111, + "axis": "origin", + "noun": { + "singular": "provider", + "plural": "providers" + } + }, + { + "label": "financingRecipient", + "count": 1, + "axis": "destination", + "noun": { + "singular": "recipient", + "plural": "recipients" + } + }, + { + "label": "years", + "count": 1 + } + ], + "window": { + "kind": "range", + "origin": "query", + "end_year": 2022, + "start_year": 2022 + } + }, + "citation": { + "text": "OECD DAC2A via ONE DFO, retrieved 2026-10-12", + "sources": [ + { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A", + "license": "OECD Terms and Conditions", + "license_type": "OECDTermsAndConditions", + "license_url": "https://www.oecd.org/en/about/terms-conditions.html", + "url": "https://data-explorer.oecd.org/vis?tm=dac2a&pg=0&snb=1&df[ds]=dsDisseminateFinalDMZ&df[id]=DSD_DAC2%40DF_DAC2A&df[ag]=OECD.DCD.FSD&df[vs]=1.6&dq=.DPGC.206.USD.Q&lom=LASTNPERIODS&lo=5&to[TIME_PERIOD]=false&isAvailabilityDisabled=false&page=0", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": { + "publisher": "Organisation for Economic Co-operation and Development (OECD)", + "description": "DAC2A is the OECD Development Assistance Committee's table of ODA disbursements by donor and recipient country or region. It covers bilateral donors and multilateral agencies reporting their own outflows, and includes the OECD's estimate of each donor's core multilateral contributions attributed to recipients.", + "source_data_url": "https://webfs-dcd.oecd.org/files/dotStat/DSD_DAC2/Table2a_Data.zip", + "attribution": "OECD (2026), OECD DAC2A, OECD Data Explorer, https://webfs-dcd.oecd.org/files/dotStat/DSD_DAC2/Table2a_Data.zip (accessed on 2026-10-12)", + "update_cadence": "Quarterly", + "last_update": "2026-10-12", + "years_covered": "1960-2025", + "granularity": "Annual, by provider and recipient", + "currencies": [ + "USD" + ], + "transformation_log": [ + { + "transformation": "Download the source data", + "description": "The OECD publishes DAC2A in April, June, September and December. The complete table is downloaded from the OECD each time it is republished, and replaces the previous copy, so the data reflects the OECD's latest release." + }, + { + "transformation": "Standardise providers and recipients", + "description": "Provider and recipient codes are standardised to Data Commons IDs used across the platform, so data can be compared across datasets. The totals DAC2A reports for groups of providers, such as all DAC countries, and for groups of recipients, such as regions and income groups, are included as providers and recipients of their own. Where two provider codes name the same organisation and report the same figures, the figures are counted once. Where two recipient codes name the same organisation, such as IDA and IDA's Multilateral Debt Relief Initiative, their flows are added." + }, + { + "transformation": "Select figures", + "description": "Only figures in current US dollars are included, not constant US dollars. Only flows to recipients on Part I of the DAC List are included: the developing countries, and the multilateral organisations DAC2A reports as recipients, such as development banks and UN agencies. Flows to Part II recipients, the countries in transition, are excluded." + }, + { + "transformation": "Convert to whole units", + "description": "DAC2A reports amounts in millions, which are converted to whole units. ODA per person and ODA as a share of GNI are kept as reported." + }, + { + "transformation": "Select dimensions", + "description": "Each figure is reported by year, provider, recipient and type of aid, such as grants, repayments, interest received, or debt forgiven or rescheduled. DAC2A is a table of disbursements, so every figure, repayments and interest included, is recorded as a disbursement. Figures are marked as gross or net, except where DAC2A does not say which, and as grants or non-grants where the type of aid is wholly one or the other. Imputed multilateral ODA, the OECD's estimate of each provider's share of multilateral organisations' ODA, is marked as modelled. ODA as a share of GNI records the recipient's GNI as its denominator, and ODA per person records the recipient's population." + } + ], + "version_history": [ + { + "date": "2026-10-14", + "title": "First release", + "changes": [ + { + "kind": "added", + "note": "ODA disbursements by donor and recipient country or region, including the OECD's imputed multilateral contributions, and ODA per person and as a share of recipient GNI." + } + ] + } + ] + } + } + ] + }, + "trace": { + "facets": [ + { + "ref": "provenance/OECDDAC2A", + "axis": "provenance", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "sole_candidate", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + }, + { + "ref": null, + "axis": "measurementMethod", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "absent_on_selected_facet", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + }, + { + "ref": "CurrencyUSD", + "axis": "unit", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "sole_candidate", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + } + ], + "applied_window": { + "kind": "range", + "origin": "query", + "end_year": 2022, + "start_year": 2022 + }, + "graph_revision": "dfo-dev-dc-datacommons-service-00242-zfr", + "normalisations": [], + "code_index_stamp": { + "exported_at": "2026-10-11T15:49:13Z", + "content_hash": "952a221e6d43254927fcb75abbf12ce7ba03285fbfeef7d9913e2ddcf49f904c", + "graph_endpoint": "https://dfo-dev-dc-datacommons-service-zms5lj4zhq-nn.a.run.app" + }, + "source_revisions": [ + { + "source": "provenance/OECDDAC2A", + "revision": "2026-10-12" + } + ], + "capability_version": "dset_5c3f5b0bf4cd9569", + "materialiser_build": "cd25922ac45834a7c7d0193d3b3ee3d7e9cc7b55", + "resolved_stat_vars": [ + { + "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", + "roles": [ + "financingProvider", + "financingRecipient" + ], + "observation_count": 111 + } + ], + "family_member_count": 1, + "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", + "frame_grammar_version": "1.1.0", + "origin_catalogue_build_id": "20261012T163104Z-67f98b87", + "served_from": "head", + "confirmed_at": "2026-10-26T09:18:21.103020+00:00" + }, + "warnings": [], + "diagnostics": [ + { + "code": "processing_level_undeclared", + "severity": "info", + "message": "A source StatVar lacks a processing-level declaration; the citation credits the source and omits the processing qualifier." + } + ], + "artifacts": [ + { + "artifact_digest": "sha256:05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4 + }, + "bytes": 10932, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-25T13:51:32.998697+00:00", + "tier": "working", + "url": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_2aa4cc2fd852cd9b84347c61d07d609a/files/sha256:05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, + { + "artifact_digest": "sha256:8682dea270c469411916b6758279fdda0f4c989dac40b35509802141c8f7de22", + "rendering": { + "statistics": "series" + }, + "bytes": 10113, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-12T18:21:54.977252+00:00", + "tier": "working", + "url": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_2aa4cc2fd852cd9b84347c61d07d609a/files/sha256:8682dea270c469411916b6758279fdda0f4c989dac40b35509802141c8f7de22" + } + ], + "series_count": 111, + "output_options": { + "capability_version": "dset_76f5fa7244511c58", + "columns": [ + { + "name": "stat_var", + "kind": "grain", + "droppable": false, + "distinct": 1 + }, + { + "name": "financingProvider", + "kind": "role", + "droppable": false, + "distinct": 111 + }, + { + "name": "financingRecipient", + "kind": "role", + "droppable": false, + "distinct": 1 + }, + { + "name": "time_period", + "kind": "grain", + "droppable": false, + "distinct": 1 + }, + { + "name": "value", + "kind": "value", + "droppable": true + }, + { + "name": "unit", + "kind": "unit", + "droppable": false, + "distinct": 1 + }, + { + "name": "measurementMethod", + "kind": "facet", + "droppable": true, + "distinct": 1 + }, + { + "name": "provenance", + "kind": "facet", + "droppable": false, + "distinct": 1 + }, + { + "name": "status", + "kind": "status", + "droppable": true + } + ], + "formats": [ + { + "name": "csv", + "state": "available" + }, + { + "name": "xlsx", + "state": "available" + }, + { + "name": "parquet", + "state": "available" + } + ], + "bundle": { + "state": "available" + }, + "enrichments": [ + { + "column": "financingProvider", + "field": "iso3", + "token": "financingProvider_iso3", + "state": "partial", + "reason": "enrichment_unavailable", + "coverage": { + "entities": 111, + "covered": 39 + }, + "missing": [ + { + "ref": "AdaptationFund", + "label": "Adaptation Fund" + }, + { + "ref": "AfricanDevelopmentBank", + "label": "African Development Bank" + }, + { + "ref": "AfricanDevelopmentBankGroup", + "label": "African Development Bank Group" + }, + { + "ref": "AfricanDevelopmentFund", + "label": "African Development Fund" + }, + { + "ref": "BezosEarthFund", + "label": "Bezos Earth Fund" + }, + { + "ref": "BloombergPhilanthropies", + "label": "Bloomberg Philanthropies" + }, + { + "ref": "CentralEmergencyResponseFund", + "label": "Central Emergency Response Fund" + }, + { + "ref": "ChildrensInvestmentFundFoundation", + "label": "Children's Investment Fund Foundation" + }, + { + "ref": "ClimateInvestmentFunds", + "label": "Climate Investment Funds" + }, + { + "ref": "ConradNHiltonFoundation", + "label": "Conrad N. Hilton Foundation" + } + ], + "truncated": true + }, + { + "column": "financingProvider", + "field": "iso2", + "token": "financingProvider_iso2", + "state": "partial", + "reason": "enrichment_unavailable", + "coverage": { + "entities": 111, + "covered": 39 + }, + "missing": [ + { + "ref": "AdaptationFund", + "label": "Adaptation Fund" + }, + { + "ref": "AfricanDevelopmentBank", + "label": "African Development Bank" + }, + { + "ref": "AfricanDevelopmentBankGroup", + "label": "African Development Bank Group" + }, + { + "ref": "AfricanDevelopmentFund", + "label": "African Development Fund" + }, + { + "ref": "BezosEarthFund", + "label": "Bezos Earth Fund" + }, + { + "ref": "BloombergPhilanthropies", + "label": "Bloomberg Philanthropies" + }, + { + "ref": "CentralEmergencyResponseFund", + "label": "Central Emergency Response Fund" + }, + { + "ref": "ChildrensInvestmentFundFoundation", + "label": "Children's Investment Fund Foundation" + }, + { + "ref": "ClimateInvestmentFunds", + "label": "Climate Investment Funds" + }, + { + "ref": "ConradNHiltonFoundation", + "label": "Conrad N. Hilton Foundation" + } + ], + "truncated": true + }, + { + "column": "financingProvider", + "field": "label", + "token": "financingProvider_label", + "state": "available", + "coverage": { + "entities": 111, + "covered": 111 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "type", + "token": "financingProvider_type", + "state": "available", + "coverage": { + "entities": 111, + "covered": 111 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_kind", + "token": "financingProvider_party_kind", + "state": "available", + "coverage": { + "entities": 111, + "covered": 111 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_class", + "token": "financingProvider_party_class", + "state": "available", + "coverage": { + "entities": 111, + "covered": 111 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_whole", + "token": "financingProvider_party_whole", + "state": "available", + "coverage": { + "entities": 111, + "covered": 111 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "iso3", + "token": "financingRecipient_iso3", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "iso2", + "token": "financingRecipient_iso2", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "label", + "token": "financingRecipient_label", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "type", + "token": "financingRecipient_type", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_kind", + "token": "financingRecipient_party_kind", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_class", + "token": "financingRecipient_party_class", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_whole", + "token": "financingRecipient_party_whole", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + } + ], + "long_layout_defaults": [ + "financingProvider_label", + "financingProvider_kind", + "financingRecipient_label", + "financingRecipient_kind" + ], + "long_layout_totals": [], + "pivots": [], + "groupby": { + "state": "available", + "axes": [ + { + "name": "crsCooperationModality" + }, + { + "name": "crsPurpose" + }, + { + "name": "crsTypeOfFinance" + }, + { + "name": "crsTypeOfFlow" + } + ] + } + }, + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", + "retention": { + "pinned": true + } + } + } + }, + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "dataset_resubmit", + "entries": [ + { + "entry_kind": "identity", + "variable_text": "dataset", + "identity": { + "shape": { + "kind": "statvar_schema", + "stat_type": "measuredValue", + "population_type": "FinancialFlow", + "measured_property": "amount", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "binding": { + "ref": "DAC2aAidType206", + "state": "value" + }, + "property": "dac2aAidType" + }, + { + "binding": { + "state": "unbound" + }, + "property": "dacTypeOfFinance" + }, + { + "binding": { + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" + }, + { + "binding": { + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "identity_version": "spec1", + "requested_window": { + "kind": "range", + "end_year": 2022, + "start_year": 2022 + } + } + } + ] + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "ccdbf2ec39f8429ab9c3b8b0f3bb1a92" + }, + "json": { + "schema_version": 1, + "query_echo": { + "entry_path": "dataset_resubmit", + "extract_skipped": true + }, + "diagnostics": { + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "ccdbf2ec39f8429ab9c3b8b0f3bb1a92", + "warnings": [], + "timing_ms": { + "total": 2, + "by_step": { + "coverage": 0 + } + }, + "timed_out": false + }, + "kind": "resolution", + "parts": [ + { + "variable_text": "dataset", + "warnings": [ + { + "code": "role_scope_defaulted", + "severity": "warn", + "message": "No provider was named, so the answer covers every provider with data." + }, + { + "code": "whole_total_available", + "severity": "info", + "message": "The source publishes the total of every official donor as official donors (DAC); a sum of the listed donors includes private donors and differs from it.", + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "params": { + "total": "DACOfficialDonors", + "total_label": "official donors (DAC)", + "whole_of": "official" + } + } + ], + "timing_ms": { + "total": 0, + "by_step": { + "response_cache_part_memo": 0 + } + }, + "entity_refs": [], + "operations": [], + "departures": [], + "assumptions": [ + { + "kind": "open_role", + "binding_ref": "origin:", + "text": "every provider", + "refs": [] + } + ], + "status": "definite", + "spec": { + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "definition": { + "schema_version": 1, + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + } + } + }, + "resolved": { + "shape": { + "shape_id": "financialflow_amount_measuredvalue_nominal_family_f096b1ba", + "label": "gross disbursements (official development assistance)", + "population_type": { + "ref": "FinancialFlow", + "label": "financial flow" + }, + "measured_property": { + "ref": "amount", + "label": "amount" + }, + "stat_type": { + "ref": "measuredValue", + "label": "measuredValue" + }, + "measurement_qualifier": { + "ref": "Nominal", + "label": "Nominal" + }, + "measurement_denominator_unresolved": false, + "slot_keys": [ + { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + { + "axis": "where", + "label": "place" + }, + { + "axis": "when", + "label": "time window" + }, + { + "axis": "source", + "label": "data source" + } + ], + "member_count": 1, + "member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", + "refine_supported": true, + "data_commons": [ + { + "id": "dfo", + "label": "Development Finance Observatory" + } + ] + }, + "slots": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "DAC2aAidType206", + "label": "official development assistance" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "Disbursement", + "label": "disbursement" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "Gross", + "label": "gross" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "country/KEN", + "label": "Kenya" + }, + "value_kind": "entity" + } + }, + "summable": false + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding": { + "kind": "value", + "value": { + "value_kind": "time_window", + "time_window": { + "start_year": 2022, + "end_year": 2022 + } + } + }, + "summable": false + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + } + ], + "inapplicable_slots": [ + "dacTypeOfFinance" + ], + "unresolved_slots": [], + "stat_vars": { + "member_count": 1, + "member_count_exact": true, + "served_count": 1, + "data_date_range": { + "start": "2022", + "end": "2022" + } + }, + "entities": [ + { + "ref": "country/KEN", + "label": "Kenya", + "entity_type": "Country", + "role": "destination" + } + ], + "coverage": { + "kind": "exact", + "has_data": true, + "observation_count": 111, + "evidence": "facet_metadata", + "dimensions": [ + { + "label": "sources", + "count": 1 + }, + { + "label": "observations", + "count": 1 + }, + { + "label": "variables", + "count": 1 + } + ], + "window": { + "start_year": 2022, + "end_year": 2022 + } + }, + "headline": { + "year": 2022, + "year_named": true, + "named_window": { + "start_year": 2022, + "end_year": 2022 + }, + "observations": 111, + "series": 111, + "role_counts": { + "financingProvider": 111, + "financingRecipient": 1 + }, + "recent_years": [ + { + "year": 2022, + "observations": 111, + "series": 111, + "role_counts": { + "financingProvider": 111, + "financingRecipient": 1 + } + }, + { + "year": 2021, + "observations": 103, + "series": 103, + "role_counts": { + "financingProvider": 103, + "financingRecipient": 1 + } + }, + { + "year": 2020, + "observations": 97, + "series": 97, + "role_counts": { + "financingProvider": 97, + "financingRecipient": 1 + } + } + ] + }, + "download": { + "rows": 111, + "rows_exact": true, + "series": 111, + "window": { + "start_year": 2022, + "end_year": 2022 + }, + "licence": { + "label": "OECD Terms and Conditions" + } + }, + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", + "facets": { + "provenance": { + "basis": "fallback", + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/OECDDAC2A", + "account": "provider" + }, + "unit": { + "basis": "fallback", + "ref": "CurrencyUSD", + "label": "US Dollar", + "currency": "USD" + } + }, + "slot_options": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "options": [ + { + "value": { + "ref": { + "ref": "DAC2aAidType218", + "label": "ODA loans and other long-term capital" + }, + "value_kind": "enum_value" + }, + "relation": "child", + "parent": "DAC2aAidType206", + "member_count": 2, + "served_rows": 27, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType201", + "label": "ODA grant" + }, + "value_kind": "enum_value" + }, + "relation": "child", + "parent": "DAC2aAidType206", + "member_count": 1, + "served_rows": 109, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType207", + "label": "free-standing technical co-operation" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 45, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType213", + "label": "developmental food aid" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 15, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType216", + "label": "humanitarian aid" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 42, + "served_rows_exact": true + } + ], + "truncated": false, + "source": "observed" + } + ], + "entity_options": [ + { + "role": "origin", + "options": [], + "truncated": true, + "source": "observed" + } + ], + "facet_options": [ + { + "axis": "provenance", + "options": [ + { + "value": { + "ref": { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A" + }, + "value_kind": "source" + } + } + ], + "truncated": false, + "source": "observed" + }, + { + "axis": "unit", + "options": [ + { + "value": { + "ref": { + "ref": "CurrencyUSD", + "label": "US Dollar" + }, + "value_kind": "enum_value" + } + } + ], + "truncated": false, + "source": "observed" + } + ], + "rendered_sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a.", + "sentence_plan": [ + { + "kind": "slot", + "text": "Gross disbursements", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "how:nettingBasis", + "axis:populationType", + "axis:measuredProperty" + ] + }, + { + "kind": "text", + "text": " to ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ] + }, + { + "kind": "text", + "text": " from ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "96 providers", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ] + }, + { + "kind": "text", + "text": " for ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "official development assistance", + "axis": "how", + "segment": "heading", + "property_dcid": "dac2aAidType", + "binding_ref": "how:dac2aAidType", + "entity_refs": [], + "states": [ + "how:dac2aAidType" + ] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "dacTypeOfFinance", + "binding_ref": "how:dacTypeOfFinance", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "nettingBasis", + "binding_ref": "how:nettingBasis", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "in 2022", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ] + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "OECD DAC2a", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [] + } + ], + "trace": { + "resolved_entities": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "resolved_sources": [ + { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2a", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/OECDDAC2A" + } + ], + "slot_filters": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "DAC2aAidType206", + "label": "official development assistance" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "Disbursement", + "label": "disbursement" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "Gross", + "label": "gross" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding_kind": "value", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + } + ], + "query_qualifiers": [], + "requested_window": { + "start_year": 2022, + "end_year": 2022 + }, + "capability_version": "cap_765c9d2a0c7529b9" + }, + "confidence": "high", + "available_at_place": true, + "alternative_measures": [ + { + "definition": { + "schema_version": 1, + "spec_id": "spec_e14699f9dcd167f0211081a726ae1031", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Net" + } + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + } + } + }, + "label": "net disbursements", + "axis": "what", + "measure_axis_values": [ + { + "axis": "populationType", + "value": { + "ref": "FinancialFlow", + "label": "financial flow" + } + }, + { + "axis": "measuredProperty", + "value": { + "ref": "amount", + "label": "amount" + } + }, + { + "axis": "measurementQualifier", + "value": { + "ref": "Nominal", + "label": "nominal" + } + }, + { + "axis": "financialFlowType", + "value": { + "ref": "Disbursement", + "label": "disbursement" + } + }, + { + "axis": "nettingBasis", + "value": { + "ref": "Net", + "label": "net" + } + } + ], + "served_rows": 111, + "served_rows_exact": true + } + ], + "measure_label": "gross disbursements", + "measure_label_axes": [ + "populationType", + "measuredProperty", + "measurementQualifier", + "financialFlowType", + "nettingBasis" + ], + "measure_axis_values": [ + { + "axis": "populationType", + "value": { + "ref": "FinancialFlow", + "label": "financial flow" + } + }, + { + "axis": "measuredProperty", + "value": { + "ref": "amount", + "label": "amount" + } + }, + { + "axis": "measurementQualifier", + "value": { + "ref": "Nominal", + "label": "nominal" + } + }, + { + "axis": "financialFlowType", + "value": { + "ref": "Disbursement", + "label": "disbursement" + } + }, + { + "axis": "nettingBasis", + "value": { + "ref": "Gross", + "label": "gross" + } + } + ], + "role_axis_facts": [ + { + "role": "origin", + "member_count": 96, + "members_available": true + } + ], + "unit_conversion": { + "currencies": [ + { + "code": "AED", + "noun": "UAE dirhams", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "AFN", + "noun": "Afghan afghanis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2002, + "base_year_max": 2025 + }, + { + "code": "ALL", + "noun": "Albanian leks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AMD", + "noun": "Armenian drams", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "AOA", + "noun": "Angolan kwanzas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "ARS", + "noun": "Argentine pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AUD", + "noun": "Australian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "AWG", + "noun": "Aruban florins", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1995, + "base_year_max": 2025 + }, + { + "code": "AZN", + "noun": "Azerbaijani manats", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "BAM", + "noun": "Bosnia-Herzegovina convertible marks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1996, + "base_year_max": 2025 + }, + { + "code": "BBD", + "noun": "Barbadian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BDT", + "noun": "Bangladeshi taka", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BHD", + "noun": "Bahraini dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BIF", + "noun": "Burundian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BND", + "noun": "Brunei dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1985, + "base_year_max": 2025 + }, + { + "code": "BOB", + "noun": "Bolivian bolivianos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BRL", + "noun": "Brazilian reais", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "BSD", + "noun": "Bahamian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BTN", + "noun": "Bhutanese ngultrums", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BWP", + "noun": "Botswana pula", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BYN", + "noun": "Belarusian rubles", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "BZD", + "noun": "Belize dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CAD", + "noun": "Canadian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "CDF", + "noun": "Congolese francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "CHF", + "noun": "Swiss francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "CLP", + "noun": "Chilean pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CNY", + "noun": "Chinese yuan", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "COP", + "noun": "Colombian pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CRC", + "noun": "Costa Rican colones", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CVE", + "noun": "Cape Verdean escudos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CZK", + "noun": "Czech korunas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "DJF", + "noun": "Djiboutian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "DKK", + "noun": "Danish kroner", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "DOP", + "noun": "Dominican pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "DZD", + "noun": "Algerian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EGP", + "noun": "Egyptian pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ERN", + "noun": "Eritrean nakfas", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2019, + "current_years": null + }, + { + "code": "ETB", + "noun": "Ethiopian birr", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EUR", + "noun": "euros", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1973, + "base_year_max": 2025 + }, + { + "code": "FJD", + "noun": "Fijian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GBP", + "noun": "British pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "GEL", + "noun": "Georgian lari", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1994, + "base_year_max": 2025 + }, + { + "code": "GHS", + "noun": "Ghanaian cedis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GMD", + "noun": "Gambian dalasis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GNF", + "noun": "Guinean francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GTQ", + "noun": "Guatemalan quetzales", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GYD", + "noun": "Guyanese dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HKD", + "noun": "Hong Kong dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HNL", + "noun": "Honduran lempiras", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HTG", + "noun": "Haitian gourdes", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HUF", + "noun": "Hungarian forints", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2003, + "base_year_max": 2025 + }, + { + "code": "IDR", + "noun": "Indonesian rupiahs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ILS", + "noun": "Israeli new shekels", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "INR", + "noun": "Indian rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "IQD", + "noun": "Iraqi dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2000, + "base_year_max": 2025 + }, + { + "code": "IRR", + "noun": "Iranian rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ISK", + "noun": "Icelandic krónur", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "JMD", + "noun": "Jamaican dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JOD", + "noun": "Jordanian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JPY", + "noun": "Japanese yen", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "KES", + "noun": "Kenyan shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KGS", + "noun": "Kyrgyz soms", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "KHR", + "noun": "Cambodian riels", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "KMF", + "noun": "Comorian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KRW", + "noun": "South Korean won", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1987, + "base_year_max": 2025 + }, + { + "code": "KWD", + "noun": "Kuwaiti dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1995, + "base_year_max": 2024 + }, + { + "code": "KZT", + "noun": "Kazakh tenge", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2013, + "base_year_max": 2024 + }, + { + "code": "LAK", + "noun": "Lao kip", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LBP", + "noun": "Lebanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LKR", + "noun": "Sri Lankan rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2024 + }, + { + "code": "LSL", + "noun": "Lesotho maloti", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LYD", + "noun": "Libyan dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MAD", + "noun": "Moroccan dirhams", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MDL", + "noun": "Moldovan lei", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MGA", + "noun": "Malagasy ariary", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MKD", + "noun": "Macedonian denars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MMK", + "noun": "Myanmar kyats", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "MNT", + "noun": "Mongolian tugriks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MOP", + "noun": "Macanese patacas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2001, + "base_year_max": 2025 + }, + { + "code": "MRU", + "noun": "Mauritanian ouguiyas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "MUR", + "noun": "Mauritian rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MVR", + "noun": "Maldivian rufiyaa", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MWK", + "noun": "Malawian kwachas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MXN", + "noun": "Mexican pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MYR", + "noun": "Malaysian ringgits", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MZN", + "noun": "Mozambican meticais", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NAD", + "noun": "Namibian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1989, + "base_year_max": 2025 + }, + { + "code": "NGN", + "noun": "Nigerian naira", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "NIO", + "noun": "Nicaraguan córdobas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1987, + "base_year_max": 2025 + }, + { + "code": "NOK", + "noun": "Norwegian kroner", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "NPR", + "noun": "Nepalese rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NZD", + "noun": "New Zealand dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1961, + "base_year_max": 2025 + }, + { + "code": "OMR", + "noun": "Omani rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PEN", + "noun": "Peruvian soles", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PGK", + "noun": "Papua New Guinean kina", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PHP", + "noun": "Philippine pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PKR", + "noun": "Pakistani rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PLN", + "noun": "Polish zlotys", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "PYG", + "noun": "Paraguayan guaranís", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "QAR", + "noun": "Qatari riyals", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "RON", + "noun": "Romanian lei", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2013, + "base_year_max": 2025 + }, + { + "code": "RSD", + "noun": "Serbian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "RUB", + "noun": "Russian rubles", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2010, + "base_year_max": 2020, + "current_years": null + }, + { + "code": "RWF", + "noun": "Rwandan francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SAR", + "noun": "Saudi riyals", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2000, + "base_year_max": 2023 + }, + { + "code": "SBD", + "noun": "Solomon Islands dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SCR", + "noun": "Seychellois rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SDG", + "noun": "Sudanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SEK", + "noun": "Swedish kronor", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "SGD", + "noun": "Singapore dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SLE", + "noun": "Sierra Leonean leones", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SRD", + "noun": "Surinamese dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SSP", + "noun": "South Sudanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2011, + "base_year_max": 2025 + }, + { + "code": "STN", + "noun": "São Tomé and Príncipe dobras", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SYP", + "noun": "Syrian pounds", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2010, + "current_years": null + }, + { + "code": "SZL", + "noun": "Swazi emalangeni", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "THB", + "noun": "Thai baht", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2018, + "base_year_max": 2024 + }, + { + "code": "TJS", + "noun": "Tajikistani somonis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TMT", + "noun": "Turkmenistani manat", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TND", + "noun": "Tunisian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TOP", + "noun": "Tongan paʻanga", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TRY", + "noun": "Turkish lira", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "TTD", + "noun": "Trinidad and Tobago dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TWD", + "noun": "New Taiwan dollars", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2025, + "current_years": null + }, + { + "code": "TZS", + "noun": "Tanzanian shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UAH", + "noun": "Ukrainian hryvnias", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "UGX", + "noun": "Ugandan shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "USD", + "noun": "US dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "UYU", + "noun": "Uruguayan pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UZS", + "noun": "Uzbekistani soms", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "VES", + "noun": "Venezuelan bolívares", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2017, + "base_year_max": 2025 + }, + { + "code": "VND", + "noun": "Vietnamese dong", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "VUV", + "noun": "Vanuatu vatu", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "WST", + "noun": "Samoan tala", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XAF", + "noun": "Central African CFA francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XCD", + "noun": "East Caribbean dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XOF", + "noun": "West African CFA francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "YER", + "noun": "Yemeni rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2014, + "base_year_max": 2025 + }, + { + "code": "ZAR", + "noun": "South African rand", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ZMW", + "noun": "Zambian kwachas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + } + ], + "default_base_year": 2024, + "role_currencies": { + "destination": "KES" + } + } + } + } + } + ], + "rendered_sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. Assumed: every provider.", + "sentence_plan": [ + { + "kind": "slot", + "text": "Gross disbursements", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "how:nettingBasis", + "axis:populationType", + "axis:measuredProperty" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " to ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " from ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "96 providers", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " for ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "official development assistance", + "axis": "how", + "segment": "heading", + "property_dcid": "dac2aAidType", + "binding_ref": "how:dac2aAidType", + "entity_refs": [], + "states": [ + "how:dac2aAidType" + ], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "dacTypeOfFinance", + "binding_ref": "how:dacTypeOfFinance", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "nettingBasis", + "binding_ref": "how:nettingBasis", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "in 2022", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ] + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "OECD DAC2a", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "Assumed: ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "every provider", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": ".", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + } + ], + "status": "definite" + } + } + }, + { + "request": { + "method": "GET", + "origin_path": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_2aa4cc2fd852cd9b84347c61d07d609a/files/sha256:05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426" + }, + "response": { + "status": 302, + "headers": { + "content-type": "text/html", + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426?X-Goog-Signature=redacted", + "x-dfo-trace-id": "3c0742cfbe724089ab0e8e2a15bfa9ce" + } + } + }, + { + "request": { + "method": "GET", + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A05d0cbf59e8dddd7162104bf8e2edc406fc402a0b8d0d971452d01b07b0ee426?X-Goog-Signature=redacted" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/vnd.apache.parquet" + }, + "body_file": "fetch-spec-party-totals-kept.parquet" + } + } + ], + "expect": [ + { + "dataset": { + "rows": 111, + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", + "dac2aAidType", + "unit", + "currency", + "price_basis", + "base_year", + "measurementMethod", + "measurementMethod_label", + "provenance", + "stat_var" + ], + "complete": true, + "members": [ + { + "parts": [], + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "content_version": "cv_2aa4cc2fd852cd9b84347c61d07d609a", + "sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a." + } + ], + "missing": [] + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.parquet b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.parquet new file mode 100644 index 0000000..e39f858 Binary files /dev/null and b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-kept.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.json new file mode 100644 index 0000000..957c35b --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.json @@ -0,0 +1,4484 @@ +{ + "format_version": 1, + "name": "fetch-spec-party-totals-left-out", + "description": "Fetching a spec of DAC2a ODA to Kenya in 2022 with its origin open posts without_party_totals: true in the build's options, as every fetch does by default, so the build leaves out the donor group totals beside their members and answers a version of its own, with fewer rows than fetch-spec-party-totals-kept.", + "origin": { + "kind": "recorded", + "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:19Z" + }, + "call": [ + { + "op": "fetch", + "spec": { + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "definition": { + "schema_version": 1, + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/AFG", + "country/AGO", + "country/ARE", + "country/ARG", + "country/ATG", + "country/AZE", + "country/BDI", + "country/BEN", + "country/BFA", + "country/BGD", + "country/BHR", + "country/BHS", + "country/BLZ", + "country/BOL", + "country/BRA", + "country/BRB", + "country/BRN", + "country/BTN", + "country/BWA", + "country/CAF", + "country/CHL", + "country/CHN", + "country/CIV", + "country/CMR", + "country/COD", + "country/COG", + "country/COL", + "country/COM", + "country/CPV", + "country/CRI", + "country/CUB", + "country/DJI", + "country/DMA", + "country/DOM", + "country/DZA", + "country/ECU", + "country/EGY", + "country/ERI", + "country/ETH", + "country/FJI", + "country/FSM", + "country/GAB", + "country/GHA", + "country/GIN", + "country/GMB", + "country/GNB", + "country/GNQ", + "country/GRD", + "country/GTM", + "country/GUY", + "country/HND", + "country/HTI", + "country/IDN", + "country/IND", + "country/IRN", + "country/IRQ", + "country/JAM", + "country/JOR", + "country/KEN", + "country/KHM", + "country/KIR", + "country/KNA", + "country/KWT", + "country/LAO", + "country/LBN", + "country/LBR", + "country/LBY", + "country/LCA", + "country/LKA", + "country/LSO", + "country/MAR", + "country/MDG", + "country/MDV", + "country/MHL", + "country/MLI", + "country/MMR", + "country/MNG", + "country/MOZ", + "country/MRT", + "country/MUS", + "country/MWI", + "country/MYS", + "country/NAM", + "country/NER", + "country/NGA", + "country/NIC", + "country/NPL", + "country/NRU", + "country/OMN", + "country/PAK", + "country/PAN", + "country/PER", + "country/PHL", + "country/PNG", + "country/PRK", + "country/PRY", + "country/QAT", + "country/RWA", + "country/SAU", + "country/SDN", + "country/SEN", + "country/SGP", + "country/SLB", + "country/SLE", + "country/SLV", + "country/SOM", + "country/SSD", + "country/STP", + "country/SUR", + "country/SWZ", + "country/SYC", + "country/SYR", + "country/TCD", + "country/TGO", + "country/THA", + "country/TJK", + "country/TKM", + "country/TLS", + "country/TON", + "country/TTO", + "country/TUN", + "country/TZA", + "country/UGA", + "country/URY", + "country/VCT", + "country/VEN", + "country/VNM", + "country/VUT", + "country/WSM", + "country/YEM", + "country/ZAF", + "country/ZMB", + "country/ZWE" + ], + "origin": [ + "country/AUS", + "country/AUT", + "country/BEL", + "country/CAN", + "country/CHE", + "country/CZE", + "country/DEU", + "country/DNK", + "country/ESP", + "country/EST", + "country/FIN", + "country/FRA", + "country/GBR", + "country/GRC", + "country/HUN", + "country/IRL", + "country/ISL", + "country/ITA", + "country/JPN", + "country/KOR", + "country/LTU", + "country/LUX", + "country/LVA", + "country/NLD", + "country/NOR", + "country/NZL", + "country/POL", + "country/PRT", + "country/SVK", + "country/SVN", + "country/SWE", + "country/USA" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 1967, + "end_year": 2024 + } + } + }, + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", + "entity_groups": { + "destination": { + "label": "G77 countries", + "group_id": "groups/G77" + }, + "origin": { + "label": "DAC countries", + "group_id": "groups/OECD.DAC" + } + } + }, + "window": [ + 2022, + 2022 + ], + "entities": { + "origin": null, + "destination": [ + "country/KEN" + ] + } + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/datasets", + "json": { + "schema_version": 1, + "input": { + "kind": "spec_id", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "definition": { + "schema_version": 1, + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/AFG", + "country/AGO", + "country/ARE", + "country/ARG", + "country/ATG", + "country/AZE", + "country/BDI", + "country/BEN", + "country/BFA", + "country/BGD", + "country/BHR", + "country/BHS", + "country/BLZ", + "country/BOL", + "country/BRA", + "country/BRB", + "country/BRN", + "country/BTN", + "country/BWA", + "country/CAF", + "country/CHL", + "country/CHN", + "country/CIV", + "country/CMR", + "country/COD", + "country/COG", + "country/COL", + "country/COM", + "country/CPV", + "country/CRI", + "country/CUB", + "country/DJI", + "country/DMA", + "country/DOM", + "country/DZA", + "country/ECU", + "country/EGY", + "country/ERI", + "country/ETH", + "country/FJI", + "country/FSM", + "country/GAB", + "country/GHA", + "country/GIN", + "country/GMB", + "country/GNB", + "country/GNQ", + "country/GRD", + "country/GTM", + "country/GUY", + "country/HND", + "country/HTI", + "country/IDN", + "country/IND", + "country/IRN", + "country/IRQ", + "country/JAM", + "country/JOR", + "country/KEN", + "country/KHM", + "country/KIR", + "country/KNA", + "country/KWT", + "country/LAO", + "country/LBN", + "country/LBR", + "country/LBY", + "country/LCA", + "country/LKA", + "country/LSO", + "country/MAR", + "country/MDG", + "country/MDV", + "country/MHL", + "country/MLI", + "country/MMR", + "country/MNG", + "country/MOZ", + "country/MRT", + "country/MUS", + "country/MWI", + "country/MYS", + "country/NAM", + "country/NER", + "country/NGA", + "country/NIC", + "country/NPL", + "country/NRU", + "country/OMN", + "country/PAK", + "country/PAN", + "country/PER", + "country/PHL", + "country/PNG", + "country/PRK", + "country/PRY", + "country/QAT", + "country/RWA", + "country/SAU", + "country/SDN", + "country/SEN", + "country/SGP", + "country/SLB", + "country/SLE", + "country/SLV", + "country/SOM", + "country/SSD", + "country/STP", + "country/SUR", + "country/SWZ", + "country/SYC", + "country/SYR", + "country/TCD", + "country/TGO", + "country/THA", + "country/TJK", + "country/TKM", + "country/TLS", + "country/TON", + "country/TTO", + "country/TUN", + "country/TZA", + "country/UGA", + "country/URY", + "country/VCT", + "country/VEN", + "country/VNM", + "country/VUT", + "country/WSM", + "country/YEM", + "country/ZAF", + "country/ZMB", + "country/ZWE" + ], + "origin": [ + "country/AUS", + "country/AUT", + "country/BEL", + "country/CAN", + "country/CHE", + "country/CZE", + "country/DEU", + "country/DNK", + "country/ESP", + "country/EST", + "country/FIN", + "country/FRA", + "country/GBR", + "country/GRC", + "country/HUN", + "country/IRL", + "country/ISL", + "country/ITA", + "country/JPN", + "country/KOR", + "country/LTU", + "country/LUX", + "country/LVA", + "country/NLD", + "country/NOR", + "country/NZL", + "country/POL", + "country/PRT", + "country/SVK", + "country/SVN", + "country/SWE", + "country/USA" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 1967, + "end_year": 2024 + } + } + }, + "scope": { + "window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + }, + "entities": { + "destination": [ + "country/KEN" + ] + } + } + }, + "options": { + "format": "parquet", + "bundle": false, + "without_party_totals": true + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "32e9805e9f0641c38c9b29ba614120b5" + }, + "json": { + "schema_version": 1, + "definition": { + "schema_version": 1, + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "identity": { + "shape": { + "kind": "statvar_schema", + "stat_type": "measuredValue", + "population_type": "FinancialFlow", + "measured_property": "amount", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "binding": { + "ref": "DAC2aAidType206", + "state": "value" + }, + "property": "dac2aAidType" + }, + { + "binding": { + "state": "unbound" + }, + "property": "dacTypeOfFinance" + }, + { + "binding": { + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" + }, + { + "binding": { + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "identity_version": "spec1", + "requested_window": { + "kind": "range", + "end_year": 2022, + "start_year": 2022 + } + } + }, + "content_version": "cv_fd93b9bd8095076557d0e500bb15514f", + "prepared_at": "2026-10-12T18:33:25Z", + "coverage": { + "rows": 96, + "dimensions": [ + { + "label": "financingProvider", + "count": 96, + "axis": "origin", + "noun": { + "singular": "provider", + "plural": "providers" + } + }, + { + "label": "financingRecipient", + "count": 1, + "axis": "destination", + "noun": { + "singular": "recipient", + "plural": "recipients" + } + }, + { + "label": "years", + "count": 1 + } + ], + "window": { + "kind": "range", + "origin": "query", + "end_year": 2022, + "start_year": 2022 + } + }, + "citation": { + "text": "OECD DAC2A via ONE DFO, retrieved 2026-10-12", + "sources": [ + { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A", + "license": "OECD Terms and Conditions", + "license_type": "OECDTermsAndConditions", + "license_url": "https://www.oecd.org/en/about/terms-conditions.html", + "url": "https://data-explorer.oecd.org/vis?tm=dac2a&pg=0&snb=1&df[ds]=dsDisseminateFinalDMZ&df[id]=DSD_DAC2%40DF_DAC2A&df[ag]=OECD.DCD.FSD&df[vs]=1.6&dq=.DPGC.206.USD.Q&lom=LASTNPERIODS&lo=5&to[TIME_PERIOD]=false&isAvailabilityDisabled=false&page=0", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": { + "publisher": "Organisation for Economic Co-operation and Development (OECD)", + "description": "DAC2A is the OECD Development Assistance Committee's table of ODA disbursements by donor and recipient country or region. It covers bilateral donors and multilateral agencies reporting their own outflows, and includes the OECD's estimate of each donor's core multilateral contributions attributed to recipients.", + "source_data_url": "https://webfs-dcd.oecd.org/files/dotStat/DSD_DAC2/Table2a_Data.zip", + "attribution": "OECD (2026), OECD DAC2A, OECD Data Explorer, https://webfs-dcd.oecd.org/files/dotStat/DSD_DAC2/Table2a_Data.zip (accessed on 2026-10-12)", + "update_cadence": "Quarterly", + "last_update": "2026-10-12", + "years_covered": "1960-2025", + "granularity": "Annual, by provider and recipient", + "currencies": [ + "USD" + ], + "transformation_log": [ + { + "transformation": "Download the source data", + "description": "The OECD publishes DAC2A in April, June, September and December. The complete table is downloaded from the OECD each time it is republished, and replaces the previous copy, so the data reflects the OECD's latest release." + }, + { + "transformation": "Standardise providers and recipients", + "description": "Provider and recipient codes are standardised to Data Commons IDs used across the platform, so data can be compared across datasets. The totals DAC2A reports for groups of providers, such as all DAC countries, and for groups of recipients, such as regions and income groups, are included as providers and recipients of their own. Where two provider codes name the same organisation and report the same figures, the figures are counted once. Where two recipient codes name the same organisation, such as IDA and IDA's Multilateral Debt Relief Initiative, their flows are added." + }, + { + "transformation": "Select figures", + "description": "Only figures in current US dollars are included, not constant US dollars. Only flows to recipients on Part I of the DAC List are included: the developing countries, and the multilateral organisations DAC2A reports as recipients, such as development banks and UN agencies. Flows to Part II recipients, the countries in transition, are excluded." + }, + { + "transformation": "Convert to whole units", + "description": "DAC2A reports amounts in millions, which are converted to whole units. ODA per person and ODA as a share of GNI are kept as reported." + }, + { + "transformation": "Select dimensions", + "description": "Each figure is reported by year, provider, recipient and type of aid, such as grants, repayments, interest received, or debt forgiven or rescheduled. DAC2A is a table of disbursements, so every figure, repayments and interest included, is recorded as a disbursement. Figures are marked as gross or net, except where DAC2A does not say which, and as grants or non-grants where the type of aid is wholly one or the other. Imputed multilateral ODA, the OECD's estimate of each provider's share of multilateral organisations' ODA, is marked as modelled. ODA as a share of GNI records the recipient's GNI as its denominator, and ODA per person records the recipient's population." + } + ], + "version_history": [ + { + "date": "2026-10-14", + "title": "First release", + "changes": [ + { + "kind": "added", + "note": "ODA disbursements by donor and recipient country or region, including the OECD's imputed multilateral contributions, and ODA per person and as a share of recipient GNI." + } + ] + } + ] + } + } + ] + }, + "trace": { + "facets": [ + { + "ref": "provenance/OECDDAC2A", + "axis": "provenance", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "sole_candidate", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + }, + { + "ref": null, + "axis": "measurementMethod", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "absent_on_selected_facet", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + }, + { + "ref": "CurrencyUSD", + "axis": "unit", + "basis": "deployment_fallback", + "default_policy": "dfo/facet-policy/most-observations", + "selection_reason": "sole_candidate", + "dispatch_facet_id": "1962969911839422253", + "default_policy_version": "1" + } + ], + "applied_window": { + "kind": "range", + "origin": "query", + "end_year": 2022, + "start_year": 2022 + }, + "graph_revision": "dfo-dev-dc-datacommons-service-00242-zfr", + "normalisations": [], + "code_index_stamp": { + "exported_at": "2026-10-11T15:49:13Z", + "content_hash": "952a221e6d43254927fcb75abbf12ce7ba03285fbfeef7d9913e2ddcf49f904c", + "graph_endpoint": "https://dfo-dev-dc-datacommons-service-zms5lj4zhq-nn.a.run.app" + }, + "source_revisions": [ + { + "source": "provenance/OECDDAC2A", + "revision": "2026-10-12" + } + ], + "capability_version": "dset_5c3f5b0bf4cd9569", + "materialiser_build": "cd25922ac45834a7c7d0193d3b3ee3d7e9cc7b55", + "resolved_stat_vars": [ + { + "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", + "roles": [ + "financingProvider", + "financingRecipient" + ], + "observation_count": 111 + } + ], + "family_member_count": 1, + "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", + "frame_grammar_version": "1.1.0", + "party_totals_left_out": [ + { + "role": "financingProvider", + "entities": [ + "AfricanDevelopmentBankGroup", + "DACCountriesTotal", + "DACEUMembers", + "DACEUMembersAndEUInstitutions", + "DACMultilaterals", + "DACNonDACCountries", + "DACOfficialDonors", + "DACOtherMultilateralOrganisations", + "DACPrivateDonors", + "DACRegionalDevelopmentBanks", + "DACUnitedNations", + "DACWorldBank", + "GroupOfSeven", + "PostcodeLotteryGroup", + "WorldBankGroup" + ], + "provenance": "provenance/OECDDAC2A" + } + ], + "origin_catalogue_build_id": "20261012T163104Z-67f98b87", + "served_from": "head", + "confirmed_at": "2026-10-26T09:18:21.103020+00:00" + }, + "warnings": [], + "diagnostics": [ + { + "code": "processing_level_undeclared", + "severity": "info", + "message": "A source StatVar lacks a processing-level declaration; the citation credits the source and omits the processing qualifier." + } + ], + "artifacts": [ + { + "artifact_digest": "sha256:586f5aaffd96c46feb4d7bf09625d4c1a259faf51daf2d88967566fd4de07b4e", + "rendering": { + "bundle": false, + "format": "parquet", + "layout": 4, + "without_party_totals": true + }, + "bytes": 10346, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-25T13:52:32.232191+00:00", + "tier": "working", + "url": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_fd93b9bd8095076557d0e500bb15514f/files/sha256:586f5aaffd96c46feb4d7bf09625d4c1a259faf51daf2d88967566fd4de07b4e", + "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" + }, + { + "artifact_digest": "sha256:f59e46a097f5837a3cc787e5dfe8b8bb562ad4dbc106f811575e322a15cc2db4", + "rendering": { + "statistics": "series" + }, + "bytes": 9488, + "media_type": "application/vnd.apache.parquet", + "created_at": "2026-10-15T09:58:17.967502+00:00", + "tier": "working", + "url": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_fd93b9bd8095076557d0e500bb15514f/files/sha256:f59e46a097f5837a3cc787e5dfe8b8bb562ad4dbc106f811575e322a15cc2db4" + } + ], + "series_count": 96, + "output_options": { + "capability_version": "dset_76f5fa7244511c58", + "columns": [ + { + "name": "stat_var", + "kind": "grain", + "droppable": false, + "distinct": 1 + }, + { + "name": "financingProvider", + "kind": "role", + "droppable": false, + "distinct": 96 + }, + { + "name": "financingRecipient", + "kind": "role", + "droppable": false, + "distinct": 1 + }, + { + "name": "time_period", + "kind": "grain", + "droppable": false, + "distinct": 1 + }, + { + "name": "value", + "kind": "value", + "droppable": true + }, + { + "name": "unit", + "kind": "unit", + "droppable": false, + "distinct": 1 + }, + { + "name": "measurementMethod", + "kind": "facet", + "droppable": true, + "distinct": 1 + }, + { + "name": "provenance", + "kind": "facet", + "droppable": false, + "distinct": 1 + }, + { + "name": "status", + "kind": "status", + "droppable": true + } + ], + "formats": [ + { + "name": "csv", + "state": "available" + }, + { + "name": "xlsx", + "state": "available" + }, + { + "name": "parquet", + "state": "available" + } + ], + "bundle": { + "state": "available" + }, + "enrichments": [ + { + "column": "financingProvider", + "field": "iso3", + "token": "financingProvider_iso3", + "state": "partial", + "reason": "enrichment_unavailable", + "coverage": { + "entities": 96, + "covered": 39 + }, + "missing": [ + { + "ref": "AdaptationFund", + "label": "Adaptation Fund" + }, + { + "ref": "AfricanDevelopmentBank", + "label": "African Development Bank" + }, + { + "ref": "AfricanDevelopmentFund", + "label": "African Development Fund" + }, + { + "ref": "BezosEarthFund", + "label": "Bezos Earth Fund" + }, + { + "ref": "BloombergPhilanthropies", + "label": "Bloomberg Philanthropies" + }, + { + "ref": "CentralEmergencyResponseFund", + "label": "Central Emergency Response Fund" + }, + { + "ref": "ChildrensInvestmentFundFoundation", + "label": "Children's Investment Fund Foundation" + }, + { + "ref": "ClimateInvestmentFunds", + "label": "Climate Investment Funds" + }, + { + "ref": "ConradNHiltonFoundation", + "label": "Conrad N. Hilton Foundation" + }, + { + "ref": "DavidAndLucilePackardFoundation", + "label": "David and Lucile Packard Foundation" + } + ], + "truncated": true + }, + { + "column": "financingProvider", + "field": "iso2", + "token": "financingProvider_iso2", + "state": "partial", + "reason": "enrichment_unavailable", + "coverage": { + "entities": 96, + "covered": 39 + }, + "missing": [ + { + "ref": "AdaptationFund", + "label": "Adaptation Fund" + }, + { + "ref": "AfricanDevelopmentBank", + "label": "African Development Bank" + }, + { + "ref": "AfricanDevelopmentFund", + "label": "African Development Fund" + }, + { + "ref": "BezosEarthFund", + "label": "Bezos Earth Fund" + }, + { + "ref": "BloombergPhilanthropies", + "label": "Bloomberg Philanthropies" + }, + { + "ref": "CentralEmergencyResponseFund", + "label": "Central Emergency Response Fund" + }, + { + "ref": "ChildrensInvestmentFundFoundation", + "label": "Children's Investment Fund Foundation" + }, + { + "ref": "ClimateInvestmentFunds", + "label": "Climate Investment Funds" + }, + { + "ref": "ConradNHiltonFoundation", + "label": "Conrad N. Hilton Foundation" + }, + { + "ref": "DavidAndLucilePackardFoundation", + "label": "David and Lucile Packard Foundation" + } + ], + "truncated": true + }, + { + "column": "financingProvider", + "field": "label", + "token": "financingProvider_label", + "state": "available", + "coverage": { + "entities": 96, + "covered": 96 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "type", + "token": "financingProvider_type", + "state": "available", + "coverage": { + "entities": 96, + "covered": 96 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_kind", + "token": "financingProvider_party_kind", + "state": "available", + "coverage": { + "entities": 96, + "covered": 96 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_class", + "token": "financingProvider_party_class", + "state": "available", + "coverage": { + "entities": 96, + "covered": 96 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingProvider", + "field": "party_whole", + "token": "financingProvider_party_whole", + "state": "available", + "coverage": { + "entities": 96, + "covered": 96 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "iso3", + "token": "financingRecipient_iso3", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "iso2", + "token": "financingRecipient_iso2", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "label", + "token": "financingRecipient_label", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "type", + "token": "financingRecipient_type", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_kind", + "token": "financingRecipient_party_kind", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_class", + "token": "financingRecipient_party_class", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + }, + { + "column": "financingRecipient", + "field": "party_whole", + "token": "financingRecipient_party_whole", + "state": "available", + "coverage": { + "entities": 1, + "covered": 1 + }, + "missing": [], + "truncated": false + } + ], + "long_layout_defaults": [ + "financingProvider_label", + "financingProvider_kind", + "financingRecipient_label", + "financingRecipient_kind" + ], + "long_layout_totals": [], + "pivots": [], + "groupby": { + "state": "available", + "axes": [ + { + "name": "crsCooperationModality" + }, + { + "name": "crsPurpose" + }, + { + "name": "crsTypeOfFinance" + }, + { + "name": "crsTypeOfFlow" + } + ] + } + }, + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", + "retention": { + "pinned": true + } + } + } + }, + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "dataset_resubmit", + "entries": [ + { + "entry_kind": "identity", + "variable_text": "dataset", + "identity": { + "shape": { + "kind": "statvar_schema", + "stat_type": "measuredValue", + "population_type": "FinancialFlow", + "measured_property": "amount", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "binding": { + "ref": "DAC2aAidType206", + "state": "value" + }, + "property": "dac2aAidType" + }, + { + "binding": { + "state": "unbound" + }, + "property": "dacTypeOfFinance" + }, + { + "binding": { + "ref": "Disbursement", + "state": "value" + }, + "property": "financialFlowType" + }, + { + "binding": { + "ref": "Gross", + "state": "value" + }, + "property": "nettingBasis" + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "identity_version": "spec1", + "requested_window": { + "kind": "range", + "end_year": 2022, + "start_year": 2022 + } + } + } + ] + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "66a2d2508ed04bddb8e7bb1aabaa88db" + }, + "json": { + "schema_version": 1, + "query_echo": { + "entry_path": "dataset_resubmit", + "extract_skipped": true + }, + "diagnostics": { + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "66a2d2508ed04bddb8e7bb1aabaa88db", + "warnings": [], + "timing_ms": { + "total": 2, + "by_step": { + "coverage": 0 + } + }, + "timed_out": false + }, + "kind": "resolution", + "parts": [ + { + "variable_text": "dataset", + "warnings": [ + { + "code": "role_scope_defaulted", + "severity": "warn", + "message": "No provider was named, so the answer covers every provider with data." + }, + { + "code": "whole_total_available", + "severity": "info", + "message": "The source publishes the total of every official donor as official donors (DAC); a sum of the listed donors includes private donors and differs from it.", + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "params": { + "total": "DACOfficialDonors", + "total_label": "official donors (DAC)", + "whole_of": "official" + } + } + ], + "timing_ms": { + "total": 0, + "by_step": { + "response_cache_part_memo": 0 + } + }, + "entity_refs": [], + "operations": [], + "departures": [], + "assumptions": [ + { + "kind": "open_role", + "binding_ref": "origin:", + "text": "every provider", + "refs": [] + } + ], + "status": "definite", + "spec": { + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "definition": { + "schema_version": 1, + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Gross" + } + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + } + } + }, + "resolved": { + "shape": { + "shape_id": "financialflow_amount_measuredvalue_nominal_family_f096b1ba", + "label": "gross disbursements (official development assistance)", + "population_type": { + "ref": "FinancialFlow", + "label": "financial flow" + }, + "measured_property": { + "ref": "amount", + "label": "amount" + }, + "stat_type": { + "ref": "measuredValue", + "label": "measuredValue" + }, + "measurement_qualifier": { + "ref": "Nominal", + "label": "Nominal" + }, + "measurement_denominator_unresolved": false, + "slot_keys": [ + { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + { + "axis": "where", + "label": "place" + }, + { + "axis": "when", + "label": "time window" + }, + { + "axis": "source", + "label": "data source" + } + ], + "member_count": 1, + "member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", + "refine_supported": true, + "data_commons": [ + { + "id": "dfo", + "label": "Development Finance Observatory" + } + ] + }, + "slots": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "DAC2aAidType206", + "label": "official development assistance" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "Disbursement", + "label": "disbursement" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "Gross", + "label": "gross" + }, + "value_kind": "enum_value" + } + }, + "summable": false + }, + { + "key": { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding": { + "kind": "value", + "value": { + "ref": { + "ref": "country/KEN", + "label": "Kenya" + }, + "value_kind": "entity" + } + }, + "summable": false + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding": { + "kind": "value", + "value": { + "value_kind": "time_window", + "time_window": { + "start_year": 2022, + "end_year": 2022 + } + } + }, + "summable": false + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding": { + "kind": "unbound" + }, + "summable": false + } + ], + "inapplicable_slots": [ + "dacTypeOfFinance" + ], + "unresolved_slots": [], + "stat_vars": { + "member_count": 1, + "member_count_exact": true, + "served_count": 1, + "data_date_range": { + "start": "2022", + "end": "2022" + } + }, + "entities": [ + { + "ref": "country/KEN", + "label": "Kenya", + "entity_type": "Country", + "role": "destination" + } + ], + "coverage": { + "kind": "exact", + "has_data": true, + "observation_count": 111, + "evidence": "facet_metadata", + "dimensions": [ + { + "label": "sources", + "count": 1 + }, + { + "label": "observations", + "count": 1 + }, + { + "label": "variables", + "count": 1 + } + ], + "window": { + "start_year": 2022, + "end_year": 2022 + } + }, + "headline": { + "year": 2022, + "year_named": true, + "named_window": { + "start_year": 2022, + "end_year": 2022 + }, + "observations": 111, + "series": 111, + "role_counts": { + "financingProvider": 111, + "financingRecipient": 1 + }, + "recent_years": [ + { + "year": 2022, + "observations": 111, + "series": 111, + "role_counts": { + "financingProvider": 111, + "financingRecipient": 1 + } + }, + { + "year": 2021, + "observations": 103, + "series": 103, + "role_counts": { + "financingProvider": 103, + "financingRecipient": 1 + } + }, + { + "year": 2020, + "observations": 97, + "series": 97, + "role_counts": { + "financingProvider": 97, + "financingRecipient": 1 + } + } + ] + }, + "download": { + "rows": 111, + "rows_exact": true, + "series": 111, + "window": { + "start_year": 2022, + "end_year": 2022 + }, + "licence": { + "label": "OECD Terms and Conditions" + } + }, + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", + "facets": { + "provenance": { + "basis": "fallback", + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/OECDDAC2A", + "account": "provider" + }, + "unit": { + "basis": "fallback", + "ref": "CurrencyUSD", + "label": "US Dollar", + "currency": "USD" + } + }, + "slot_options": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "options": [ + { + "value": { + "ref": { + "ref": "DAC2aAidType218", + "label": "ODA loans and other long-term capital" + }, + "value_kind": "enum_value" + }, + "relation": "child", + "parent": "DAC2aAidType206", + "member_count": 2, + "served_rows": 27, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType201", + "label": "ODA grant" + }, + "value_kind": "enum_value" + }, + "relation": "child", + "parent": "DAC2aAidType206", + "member_count": 1, + "served_rows": 109, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType207", + "label": "free-standing technical co-operation" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 45, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType213", + "label": "developmental food aid" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 15, + "served_rows_exact": true + }, + { + "value": { + "ref": { + "ref": "DAC2aAidType216", + "label": "humanitarian aid" + }, + "value_kind": "enum_value" + }, + "relation": "root", + "member_count": 1, + "served_rows": 42, + "served_rows_exact": true + } + ], + "truncated": false, + "source": "observed" + } + ], + "entity_options": [ + { + "role": "origin", + "options": [], + "truncated": true, + "source": "observed" + } + ], + "facet_options": [ + { + "axis": "provenance", + "options": [ + { + "value": { + "ref": { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2A" + }, + "value_kind": "source" + } + } + ], + "truncated": false, + "source": "observed" + }, + { + "axis": "unit", + "options": [ + { + "value": { + "ref": { + "ref": "CurrencyUSD", + "label": "US Dollar" + }, + "value_kind": "enum_value" + } + } + ], + "truncated": false, + "source": "observed" + } + ], + "rendered_sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a.", + "sentence_plan": [ + { + "kind": "slot", + "text": "Gross disbursements", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "how:nettingBasis", + "axis:populationType", + "axis:measuredProperty" + ] + }, + { + "kind": "text", + "text": " to ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ] + }, + { + "kind": "text", + "text": " from ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "96 providers", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ] + }, + { + "kind": "text", + "text": " for ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "official development assistance", + "axis": "how", + "segment": "heading", + "property_dcid": "dac2aAidType", + "binding_ref": "how:dac2aAidType", + "entity_refs": [], + "states": [ + "how:dac2aAidType" + ] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "dacTypeOfFinance", + "binding_ref": "how:dacTypeOfFinance", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "nettingBasis", + "binding_ref": "how:nettingBasis", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "in 2022", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ] + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "OECD DAC2a", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [] + } + ], + "trace": { + "resolved_entities": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "resolved_sources": [ + { + "ref": "provenance/OECDDAC2A", + "label": "OECD DAC2a", + "source": { + "ref": "dc/s/OrganisationForEconomicCo-operationAndDevelopmentOecd", + "label": "Organisation for Economic Co-operation and Development (OECD)" + }, + "data_commons": { + "id": "dfo", + "label": "Development Finance Observatory" + }, + "documentation": "/api/graph/documentation/provenance/OECDDAC2A" + } + ], + "slot_filters": [ + { + "key": { + "axis": "how", + "property": { + "ref": "dac2aAidType", + "label": "DAC 2a aid type" + }, + "label": "aid type" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "DAC2aAidType206", + "label": "official development assistance" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "financialFlowType", + "label": "financial flow type" + }, + "label": "financial flow type" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "Disbursement", + "label": "disbursement" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "nettingBasis", + "label": "netting basis" + }, + "label": "netting basis" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "Gross", + "label": "gross" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "how", + "property": { + "ref": "dacTypeOfFinance", + "label": "DAC type of finance" + }, + "label": "DAC type of finance" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "where", + "label": "place" + }, + "binding_kind": "value", + "refs": [ + { + "ref": "country/KEN", + "label": "Kenya" + } + ], + "provenance": "binder" + }, + { + "key": { + "axis": "when", + "label": "time window" + }, + "binding_kind": "value", + "refs": [], + "provenance": "binder" + }, + { + "key": { + "axis": "source", + "label": "data source" + }, + "binding_kind": "unbound", + "refs": [], + "provenance": "binder" + } + ], + "query_qualifiers": [], + "requested_window": { + "start_year": 2022, + "end_year": 2022 + }, + "capability_version": "cap_765c9d2a0c7529b9" + }, + "confidence": "high", + "available_at_place": true, + "alternative_measures": [ + { + "definition": { + "schema_version": 1, + "spec_id": "spec_e14699f9dcd167f0211081a726ae1031", + "identity": { + "identity_version": "spec1", + "shape": { + "kind": "statvar_schema", + "population_type": "FinancialFlow", + "measured_property": "amount", + "stat_type": "measuredValue", + "measurement_qualifier": "Nominal" + }, + "slots": [ + { + "property": "dac2aAidType", + "binding": { + "state": "value", + "ref": "DAC2aAidType206" + } + }, + { + "property": "dacTypeOfFinance", + "binding": { + "state": "unbound" + } + }, + { + "property": "financialFlowType", + "binding": { + "state": "value", + "ref": "Disbursement" + } + }, + { + "property": "nettingBasis", + "binding": { + "state": "value", + "ref": "Net" + } + } + ], + "entities": { + "destination": [ + "country/KEN" + ] + }, + "requested_window": { + "kind": "range", + "start_year": 2022, + "end_year": 2022 + } + } + }, + "label": "net disbursements", + "axis": "what", + "measure_axis_values": [ + { + "axis": "populationType", + "value": { + "ref": "FinancialFlow", + "label": "financial flow" + } + }, + { + "axis": "measuredProperty", + "value": { + "ref": "amount", + "label": "amount" + } + }, + { + "axis": "measurementQualifier", + "value": { + "ref": "Nominal", + "label": "nominal" + } + }, + { + "axis": "financialFlowType", + "value": { + "ref": "Disbursement", + "label": "disbursement" + } + }, + { + "axis": "nettingBasis", + "value": { + "ref": "Net", + "label": "net" + } + } + ], + "served_rows": 111, + "served_rows_exact": true + } + ], + "measure_label": "gross disbursements", + "measure_label_axes": [ + "populationType", + "measuredProperty", + "measurementQualifier", + "financialFlowType", + "nettingBasis" + ], + "measure_axis_values": [ + { + "axis": "populationType", + "value": { + "ref": "FinancialFlow", + "label": "financial flow" + } + }, + { + "axis": "measuredProperty", + "value": { + "ref": "amount", + "label": "amount" + } + }, + { + "axis": "measurementQualifier", + "value": { + "ref": "Nominal", + "label": "nominal" + } + }, + { + "axis": "financialFlowType", + "value": { + "ref": "Disbursement", + "label": "disbursement" + } + }, + { + "axis": "nettingBasis", + "value": { + "ref": "Gross", + "label": "gross" + } + } + ], + "role_axis_facts": [ + { + "role": "origin", + "member_count": 96, + "members_available": true + } + ], + "unit_conversion": { + "currencies": [ + { + "code": "AED", + "noun": "UAE dirhams", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "AFN", + "noun": "Afghan afghanis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2002, + "base_year_max": 2025 + }, + { + "code": "ALL", + "noun": "Albanian leks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AMD", + "noun": "Armenian drams", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "AOA", + "noun": "Angolan kwanzas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "ARS", + "noun": "Argentine pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "AUD", + "noun": "Australian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "AWG", + "noun": "Aruban florins", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1995, + "base_year_max": 2025 + }, + { + "code": "AZN", + "noun": "Azerbaijani manats", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "BAM", + "noun": "Bosnia-Herzegovina convertible marks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1996, + "base_year_max": 2025 + }, + { + "code": "BBD", + "noun": "Barbadian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BDT", + "noun": "Bangladeshi taka", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BHD", + "noun": "Bahraini dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BIF", + "noun": "Burundian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BND", + "noun": "Brunei dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1985, + "base_year_max": 2025 + }, + { + "code": "BOB", + "noun": "Bolivian bolivianos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BRL", + "noun": "Brazilian reais", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "BSD", + "noun": "Bahamian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BTN", + "noun": "Bhutanese ngultrums", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BWP", + "noun": "Botswana pula", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "BYN", + "noun": "Belarusian rubles", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "BZD", + "noun": "Belize dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CAD", + "noun": "Canadian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "CDF", + "noun": "Congolese francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "CHF", + "noun": "Swiss francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "CLP", + "noun": "Chilean pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CNY", + "noun": "Chinese yuan", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "COP", + "noun": "Colombian pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CRC", + "noun": "Costa Rican colones", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CVE", + "noun": "Cape Verdean escudos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "CZK", + "noun": "Czech korunas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "DJF", + "noun": "Djiboutian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "DKK", + "noun": "Danish kroner", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "DOP", + "noun": "Dominican pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "DZD", + "noun": "Algerian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EGP", + "noun": "Egyptian pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ERN", + "noun": "Eritrean nakfas", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2019, + "current_years": null + }, + { + "code": "ETB", + "noun": "Ethiopian birr", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "EUR", + "noun": "euros", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1973, + "base_year_max": 2025 + }, + { + "code": "FJD", + "noun": "Fijian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GBP", + "noun": "British pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "GEL", + "noun": "Georgian lari", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1994, + "base_year_max": 2025 + }, + { + "code": "GHS", + "noun": "Ghanaian cedis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GMD", + "noun": "Gambian dalasis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GNF", + "noun": "Guinean francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GTQ", + "noun": "Guatemalan quetzales", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "GYD", + "noun": "Guyanese dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HKD", + "noun": "Hong Kong dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HNL", + "noun": "Honduran lempiras", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HTG", + "noun": "Haitian gourdes", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "HUF", + "noun": "Hungarian forints", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2003, + "base_year_max": 2025 + }, + { + "code": "IDR", + "noun": "Indonesian rupiahs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ILS", + "noun": "Israeli new shekels", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "INR", + "noun": "Indian rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "IQD", + "noun": "Iraqi dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2000, + "base_year_max": 2025 + }, + { + "code": "IRR", + "noun": "Iranian rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ISK", + "noun": "Icelandic krónur", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "JMD", + "noun": "Jamaican dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JOD", + "noun": "Jordanian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "JPY", + "noun": "Japanese yen", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "KES", + "noun": "Kenyan shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KGS", + "noun": "Kyrgyz soms", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "KHR", + "noun": "Cambodian riels", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1986, + "base_year_max": 2025 + }, + { + "code": "KMF", + "noun": "Comorian francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "KRW", + "noun": "South Korean won", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1987, + "base_year_max": 2025 + }, + { + "code": "KWD", + "noun": "Kuwaiti dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1995, + "base_year_max": 2024 + }, + { + "code": "KZT", + "noun": "Kazakh tenge", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2013, + "base_year_max": 2024 + }, + { + "code": "LAK", + "noun": "Lao kip", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LBP", + "noun": "Lebanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LKR", + "noun": "Sri Lankan rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2024 + }, + { + "code": "LSL", + "noun": "Lesotho maloti", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "LYD", + "noun": "Libyan dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MAD", + "noun": "Moroccan dirhams", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MDL", + "noun": "Moldovan lei", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MGA", + "noun": "Malagasy ariary", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MKD", + "noun": "Macedonian denars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "MMK", + "noun": "Myanmar kyats", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "MNT", + "noun": "Mongolian tugriks", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MOP", + "noun": "Macanese patacas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2001, + "base_year_max": 2025 + }, + { + "code": "MRU", + "noun": "Mauritanian ouguiyas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "MUR", + "noun": "Mauritian rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MVR", + "noun": "Maldivian rufiyaa", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MWK", + "noun": "Malawian kwachas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MXN", + "noun": "Mexican pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MYR", + "noun": "Malaysian ringgits", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "MZN", + "noun": "Mozambican meticais", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NAD", + "noun": "Namibian dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1989, + "base_year_max": 2025 + }, + { + "code": "NGN", + "noun": "Nigerian naira", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "NIO", + "noun": "Nicaraguan córdobas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1987, + "base_year_max": 2025 + }, + { + "code": "NOK", + "noun": "Norwegian kroner", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "NPR", + "noun": "Nepalese rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "NZD", + "noun": "New Zealand dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1961, + "base_year_max": 2025 + }, + { + "code": "OMR", + "noun": "Omani rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PEN", + "noun": "Peruvian soles", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PGK", + "noun": "Papua New Guinean kina", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PHP", + "noun": "Philippine pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PKR", + "noun": "Pakistani rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "PLN", + "noun": "Polish zlotys", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1998, + "base_year_max": 2025 + }, + { + "code": "PYG", + "noun": "Paraguayan guaranís", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "QAR", + "noun": "Qatari riyals", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2023, + "current_years": null + }, + { + "code": "RON", + "noun": "Romanian lei", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2013, + "base_year_max": 2025 + }, + { + "code": "RSD", + "noun": "Serbian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1997, + "base_year_max": 2025 + }, + { + "code": "RUB", + "noun": "Russian rubles", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2010, + "base_year_max": 2020, + "current_years": null + }, + { + "code": "RWF", + "noun": "Rwandan francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SAR", + "noun": "Saudi riyals", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2000, + "base_year_max": 2023 + }, + { + "code": "SBD", + "noun": "Solomon Islands dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SCR", + "noun": "Seychellois rupees", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SDG", + "noun": "Sudanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SEK", + "noun": "Swedish kronor", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "SGD", + "noun": "Singapore dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SLE", + "noun": "Sierra Leonean leones", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SRD", + "noun": "Surinamese dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SSP", + "noun": "South Sudanese pounds", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2011, + "base_year_max": 2025 + }, + { + "code": "STN", + "noun": "São Tomé and Príncipe dobras", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "SYP", + "noun": "Syrian pounds", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2010, + "current_years": null + }, + { + "code": "SZL", + "noun": "Swazi emalangeni", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "THB", + "noun": "Thai baht", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2018, + "base_year_max": 2024 + }, + { + "code": "TJS", + "noun": "Tajikistani somonis", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TMT", + "noun": "Turkmenistani manat", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "TND", + "noun": "Tunisian dinars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TOP", + "noun": "Tongan paʻanga", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TRY", + "noun": "Turkish lira", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1990, + "base_year_max": 2025 + }, + { + "code": "TTD", + "noun": "Trinidad and Tobago dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "TWD", + "noun": "New Taiwan dollars", + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2023, + "base_year_max": 2025, + "current_years": null + }, + { + "code": "TZS", + "noun": "Tanzanian shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UAH", + "noun": "Ukrainian hryvnias", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1991, + "base_year_max": 2025 + }, + { + "code": "UGX", + "noun": "Ugandan shillings", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "USD", + "noun": "US dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1960, + "base_year_max": 2025 + }, + { + "code": "UYU", + "noun": "Uruguayan pesos", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "UZS", + "noun": "Uzbekistani soms", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1992, + "base_year_max": 2025 + }, + { + "code": "VES", + "noun": "Venezuelan bolívares", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2017, + "base_year_max": 2025 + }, + { + "code": "VND", + "noun": "Vietnamese dong", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "VUV", + "noun": "Vanuatu vatu", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "WST", + "noun": "Samoan tala", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XAF", + "noun": "Central African CFA francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XCD", + "noun": "East Caribbean dollars", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "XOF", + "noun": "West African CFA francs", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "YER", + "noun": "Yemeni rials", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 2014, + "base_year_max": 2025 + }, + { + "code": "ZAR", + "noun": "South African rand", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + }, + { + "code": "ZMW", + "noun": "Zambian kwachas", + "current_years": { + "start_year": 2022, + "end_year": 2022 + }, + "constant_years": { + "start_year": 2022, + "end_year": 2022 + }, + "base_year_min": 1980, + "base_year_max": 2025 + } + ], + "default_base_year": 2024, + "role_currencies": { + "destination": "KES" + } + } + } + } + } + ], + "rendered_sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. Assumed: every provider.", + "sentence_plan": [ + { + "kind": "slot", + "text": "Gross disbursements", + "axis": "what", + "segment": "heading", + "entity_refs": [], + "states": [ + "how:financialFlowType", + "how:nettingBasis", + "axis:populationType", + "axis:measuredProperty" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " to ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Kenya", + "axis": "where", + "segment": "heading", + "role": "destination", + "binding_ref": "where:", + "entity_refs": [ + "country/KEN" + ], + "states": [ + "where:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " from ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "96 providers", + "segment": "heading", + "role": "origin", + "binding_ref": "origin:", + "entity_refs": [], + "states": [ + "origin:" + ], + "part_index": 0 + }, + { + "kind": "text", + "text": " for ", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "official development assistance", + "axis": "how", + "segment": "heading", + "property_dcid": "dac2aAidType", + "binding_ref": "how:dac2aAidType", + "entity_refs": [], + "states": [ + "how:dac2aAidType" + ], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "dacTypeOfFinance", + "binding_ref": "how:dacTypeOfFinance", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "financialFlowType", + "binding_ref": "how:financialFlowType", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "", + "axis": "how", + "segment": "heading", + "property_dcid": "nettingBasis", + "binding_ref": "how:nettingBasis", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": " ", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "in 2022", + "axis": "when", + "segment": "heading", + "binding_ref": "when:", + "entity_refs": [], + "states": [ + "when:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Current US dollars", + "axis": "unit", + "segment": "byline", + "binding_ref": "unit:", + "entity_refs": [], + "states": [ + "unit:", + "axis:measurementQualifier" + ] + }, + { + "kind": "text", + "text": " · ", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "OECD DAC2a", + "axis": "source", + "segment": "byline", + "entity_refs": [], + "states": [ + "source:" + ] + }, + { + "kind": "text", + "text": ".", + "segment": "byline", + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": " ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "Assumed: ", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "every provider", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": ".", + "segment": "assumption", + "entity_refs": [], + "states": [], + "part_index": 0 + } + ], + "status": "definite" + } + } + }, + { + "request": { + "method": "GET", + "origin_path": "/api/datasets/spec_73897fdfb0206fcf27c5898b353348fb/cv_fd93b9bd8095076557d0e500bb15514f/files/sha256:586f5aaffd96c46feb4d7bf09625d4c1a259faf51daf2d88967566fd4de07b4e" + }, + "response": { + "status": 302, + "headers": { + "content-type": "text/html", + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A586f5aaffd96c46feb4d7bf09625d4c1a259faf51daf2d88967566fd4de07b4e?X-Goog-Signature=redacted", + "x-dfo-trace-id": "cd279a74b4ef4b9689cf4a809b87c9c9" + } + } + }, + { + "request": { + "method": "GET", + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A586f5aaffd96c46feb4d7bf09625d4c1a259faf51daf2d88967566fd4de07b4e?X-Goog-Signature=redacted" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/vnd.apache.parquet" + }, + "body_file": "fetch-spec-party-totals-left-out.parquet" + } + } + ], + "expect": [ + { + "dataset": { + "rows": 96, + "columns": [ + "time_period", + "financingProvider_label", + "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", + "dac2aAidType", + "unit", + "currency", + "price_basis", + "base_year", + "measurementMethod", + "measurementMethod_label", + "provenance", + "stat_var" + ], + "complete": true, + "members": [ + { + "parts": [], + "spec_id": "spec_73897fdfb0206fcf27c5898b353348fb", + "content_version": "cv_fd93b9bd8095076557d0e500bb15514f", + "sentence": "Gross disbursements to Kenya from 96 providers for official development assistance in 2022. Current US dollars · OECD DAC2a." + } + ], + "missing": [] + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.parquet b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.parquet new file mode 100644 index 0000000..6898a12 Binary files /dev/null and b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-party-totals-left-out.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-not-found.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-not-found.json index 96921e6..4fb6e44 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-not-found.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-not-found.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,7 +503,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 404, @@ -530,7 +531,7 @@ "trace_id": "00000000000000000000000000000000", "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5 ], diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-pacing-changes.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-pacing-changes.json index 099416b..636e329 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-pacing-changes.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-pacing-changes.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,24 +503,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 2500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -528,24 +529,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 4000, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -554,24 +555,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1000, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -580,24 +581,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -810,14 +811,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -839,14 +840,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -965,13 +966,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -980,8 +981,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -989,15 +990,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1009,34 +1010,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1060,7 +1061,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1300,7 +1301,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1324,7 +1325,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1383,10 +1384,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1395,21 +1396,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1423,42 +1424,42 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 2.5, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited-not-retryable.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited-not-retryable.json index eb3fdc5..0c01f71 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited-not-retryable.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited-not-retryable.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,7 +503,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 429, @@ -527,7 +528,7 @@ "trace_id": "00000000000000000000000000000000", "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5 ], diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited.json index 5dc6d95..ea02336 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-rate-limited.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,7 +503,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 429, @@ -514,24 +515,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -540,24 +541,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -566,24 +567,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -592,24 +645,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -822,14 +901,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -851,14 +930,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -977,13 +1056,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -992,8 +1071,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -1001,15 +1080,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1021,34 +1100,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1072,7 +1151,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1312,7 +1391,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1336,7 +1415,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1395,10 +1474,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1407,21 +1486,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1435,48 +1514,51 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 3.0, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-backoff.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-backoff.json index ed5514f..a6ede46 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-backoff.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-backoff.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,14 +486,14 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" } } } @@ -500,22 +501,22 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -524,7 +525,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 503, @@ -546,22 +547,22 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -570,22 +571,46 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -594,24 +619,72 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -824,14 +897,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -853,14 +926,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -979,13 +1052,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -994,8 +1067,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -1003,15 +1076,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1023,34 +1096,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1074,7 +1147,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1314,7 +1387,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1338,7 +1411,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1397,10 +1470,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1409,21 +1482,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1437,47 +1510,50 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.0, 2.0, 4.0, 8.0, + 8.0, + 8.0, + 8.0, 8.0 ], "resubmissions": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-retry-after.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-retry-after.json index 02becd9..322e6f8 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-retry-after.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused-retry-after.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,7 +503,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 503, @@ -525,24 +526,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -551,24 +552,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -577,24 +578,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -603,24 +656,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -833,14 +912,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -862,14 +941,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -988,13 +1067,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -1003,8 +1082,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -1012,15 +1091,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1032,34 +1111,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1083,7 +1162,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1323,7 +1402,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1347,7 +1426,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1406,10 +1485,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1418,21 +1497,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1446,48 +1525,51 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 5.0, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused.json index b1f3dcf..a629db0 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-poll-refused.json @@ -11,10 +11,10 @@ { "op": "fetch", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -226,12 +226,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -242,7 +242,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true } ], "exchanges": [ @@ -254,10 +255,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -469,7 +470,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -485,15 +486,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -502,7 +503,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 503, @@ -524,24 +525,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -550,24 +551,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -576,7 +577,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 503, @@ -598,7 +599,7 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 503, @@ -620,24 +621,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -646,24 +699,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -876,14 +955,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -905,14 +984,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -1031,13 +1110,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -1046,8 +1125,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -1055,15 +1134,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1075,34 +1154,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1126,7 +1205,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1366,7 +1445,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1390,7 +1469,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1449,10 +1528,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1461,21 +1540,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1489,42 +1568,42 @@ "expect": [ { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 1.5, @@ -1532,7 +1611,10 @@ 1.5, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-relative-location.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-relative-location.json index 1b2da24..9a95b17 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-relative-location.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-relative-location.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -913,7 +915,7 @@ "headers": { "content-type": "text/html", "location": "bytes", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -936,32 +938,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-request-invalid.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-request-invalid.json index 29a3f26..c5dc5c9 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-request-invalid.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-request-invalid.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:51Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:31Z" }, "call": [ { @@ -174,7 +174,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -182,17 +183,38 @@ "status": 400, "headers": { "content-type": "application/problem+json", - "x-dfo-trace-id": "e229106e38d04e07af32b2895e017913" + "x-dfo-trace-id": "827e62e72957420a91edcc1564671a13" }, "json": { "type": "/codes/request_invalid", "title": "Bad Request", "status": 400, - "detail": "Value error, unknown facet axis/axes: colour", + "detail": "unknown facet axis 'colour'. The facet axes are provenance, measurementMethod, unit.", "code": "request_invalid", - "pointer": "/input/scope", - "trace_id": "e229106e38d04e07af32b2895e017913", - "capability_version": "dset_cc753cefa9199881", + "pointer": "/input/scope/facets/colour", + "issues": [ + { + "pointer": "/input/scope/facets/colour", + "axis": "colour", + "alternatives": [ + { + "ref": "provenance", + "label": "provenance" + }, + { + "ref": "measurementMethod", + "label": "measurementMethod" + }, + { + "ref": "unit", + "label": "unit" + } + ], + "truncated": false + } + ], + "trace_id": "827e62e72957420a91edcc1564671a13", + "capability_version": "dset_76f5fa7244511c58", "retryable": false } } @@ -203,7 +225,7 @@ "error": { "class": "InvalidRequestError", "code": "request_invalid", - "trace_id": "e229106e38d04e07af32b2895e017913" + "trace_id": "827e62e72957420a91edcc1564671a13" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-resubmit-other-spec.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-resubmit-other-spec.json index e3b2dfb..8393370 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-resubmit-other-spec.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-resubmit-other-spec.json @@ -174,7 +174,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -182,7 +183,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "a1fe53d87b784d23bcc55926b9530dec" + "x-dfo-trace-id": "250fbb95951e4a8c8d847cbcde0ae4f9" }, "json": { "schema_version": 1, @@ -589,7 +590,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -611,7 +612,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -634,7 +636,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -902,7 +904,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -996,7 +998,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "c4313642ac68484fb1d13d6f7b206d66" + "x-dfo-trace-id": "ff646392059e46e68f83a2753e921d6a" }, "json": { "schema_version": 1, @@ -1005,11 +1007,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "c4313642ac68484fb1d13d6f7b206d66", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "ff646392059e46e68f83a2753e921d6a", "warnings": [], "timing_ms": { - "total": 340, + "total": 1, "by_step": { "coverage": 0 } @@ -1022,19 +1024,9 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 256, + "total": 0, "by_step": { - "validate": 4, - "walk": 219, - "reread_arcs": 1, - "detect_edits": 0, - "resolve_survivors": 0, - "shape": 3, - "bind": 1, - "where_places": 0, - "materialise": 12, - "answer": 14, - "options": 2 + "response_cache_part_memo": 0 } }, "entity_refs": [], @@ -1116,7 +1108,7 @@ "resolved": { "shape": { "shape_id": "financialflow_amount_measuredvalue_nominal", - "label": "gross disbursements (basic health care, official development assistance)", + "label": "gross disbursements (basic health care, official development assistance, core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes, standard grant)", "population_type": { "ref": "FinancialFlow", "label": "financial flow" @@ -1468,7 +1460,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1702,6 +1694,8 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsCooperationModality", + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -2031,7 +2025,7 @@ "start_year": 2020, "end_year": 2023 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -2295,114 +2289,31 @@ "open_axis_facts": [ { "property": "crsCooperationModality", - "varies": true, - "distinct_count": 14, - "undeclared_count": 2, + "varies": false, + "value": { + "ref": "CRSCooperationModalityB01", + "label": "core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes" + }, + "distinct_count": 1, + "undeclared_count": 0, "values": [ - { - "value": { - "ref": "CRSCooperationModalityC01", - "label": "project-type intervention" - }, - "count": 5 - }, - { - "value": { - "ref": "CRSCooperationModalityA02", - "label": "sector budget support" - }, - "count": 2 - }, - { - "value": { - "ref": "CRSCooperationModalityB04", - "label": "basket fund and pooled funding" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB031", - "label": "contribution to multi-donor and multi-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB032", - "label": "contribution to multi-donor and single-entity funding mechanism" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB033", - "label": "contribution to single-donor funding mechanism or earmarked contribution to a specific funding window or geographical location" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityB03", - "label": "contribution to specific-purpose programme and fund managed by an international organisation" - }, - "count": 1 - }, { "value": { "ref": "CRSCooperationModalityB01", "label": "core support to non-governmental organisations, other private bodies, public-private partnerships, and research institutes" }, "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityD01", - "label": "donor country personnel" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityA01", - "label": "general budget support" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE02", - "label": "imputed student cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityG01", - "label": "other administrative cost" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityD02", - "label": "other technical assistance" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSCooperationModalityE01", - "label": "scholarship and training in donor country" - }, - "count": 1 } ] }, { "property": "crsTypeOfFinance", - "varies": true, - "distinct_count": 5, + "varies": false, + "value": { + "ref": "CRSTypeOfFinance110", + "label": "standard grant" + }, + "distinct_count": 1, "undeclared_count": 0, "values": [ { @@ -2410,34 +2321,6 @@ "ref": "CRSTypeOfFinance110", "label": "standard grant" }, - "count": 14 - }, - { - "value": { - "ref": "CRSTypeOfFinance421", - "label": "standard loan" - }, - "count": 4 - }, - { - "value": { - "ref": "CRSTypeOfFinance510", - "label": "common equity" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance210", - "label": "interest subsidy" - }, - "count": 1 - }, - { - "value": { - "ref": "CRSTypeOfFinance520", - "label": "share in collective investment vehicle" - }, "count": 1 } ] @@ -4449,6 +4332,8 @@ "segment": "heading", "entity_refs": [], "states": [ + "how:crsCooperationModality", + "how:crsTypeOfFinance", "how:crsTypeOfFlow", "how:financialFlowType", "how:nettingBasis", @@ -4635,7 +4520,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "c4313642ac68484fb1d13d6f7b206d66" + "trace_id": "ff646392059e46e68f83a2753e921d6a" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-same-origin-artifact-url.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-same-origin-artifact-url.json index 2878965..a796ffd 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-same-origin-artifact-url.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-same-origin-artifact-url.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -913,7 +915,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -936,32 +938,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-signed-fetch-partial.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-signed-fetch-partial.json index dc97e7e..dcf6a62 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-signed-fetch-partial.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-signed-fetch-partial.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -913,7 +915,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -940,7 +942,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ac44200969f19d73104d1bc24ebca905e59f749c2939234cdde10ba6f5a131170?X-Goog-Signature=redacted", - "x-dfo-trace-id": "8261c9be5a7c406a80692fac6b28ee05" + "x-dfo-trace-id": "cf3f86713afb44c98d21fbf9731d66e3" } } }, @@ -963,32 +965,32 @@ "dataset": { "rows": 4, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "crsCooperationModality", "crsCooperationModality_label", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-storage-artifact-url.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-storage-artifact-url.json index 29d6a08..575a504 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-storage-artifact-url.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-storage-artifact-url.json @@ -167,7 +167,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -175,7 +176,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "x-dfo-trace-id": "b89cda3bb794448abf06b56c955d754e" }, "json": { "schema_version": 1, @@ -582,7 +583,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -604,7 +605,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -627,7 +629,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -895,7 +897,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" @@ -909,7 +911,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "bfc5cc090c4347f489fffcc62fc3d2bf" + "trace_id": "b89cda3bb794448abf06b56c955d754e" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-strict-warning.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-strict-warning.json index f5755d2..03d5a61 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-strict-warning.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-spec-strict-warning.json @@ -168,7 +168,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -583,7 +584,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:31.570035+00:00" + "confirmed_at": "2026-10-26T09:18:06.976633+00:00" }, "warnings": [ { @@ -605,7 +606,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9599, "media_type": "application/vnd.apache.parquet", @@ -628,7 +630,7 @@ ], "series_count": 1, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -896,7 +898,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, "held_until": "2026-11-19T13:59:18.051742Z" diff --git a/python/client-contract/src/dfo_client_contract/scenarios/fetch-strict-info-notice.json b/python/client-contract/src/dfo_client_contract/scenarios/fetch-strict-info-notice.json index 1463a62..5652a17 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/fetch-strict-info-notice.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/fetch-strict-info-notice.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "bf6613bf82be4bddb4d60879aaf907d9" + "x-dfo-trace-id": "c7cb40f3790f4a15a0ddcad0cc93c9bb" }, "json": { "schema_version": 1, @@ -42,26 +42,23 @@ "entry_path": "raw_text", "raw_query": "ODA disbursements from Germany to Nigeria for basic health care", "normalized_query": "ODA disbursements from Germany to Nigeria for basic health care", - "extract_skipped": false + "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "bf6613bf82be4bddb4d60879aaf907d9", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "c7cb40f3790f4a15a0ddcad0cc93c9bb", "warnings": [], "timing_ms": { - "total": 2620, + "total": 3, "by_step": { "coverage": 0, - "extract": 1068, - "cache_lookup": 0 + "response_cache_layer1": 1 } }, "llm_usage": { - "input_tokens": 2231, - "output_tokens": 102, - "cached_tokens": 841, - "jev_input_tokens": 6586, - "jev_output_tokens": 2060 + "input_tokens": 0, + "output_tokens": 0, + "cached_tokens": 0 }, "timed_out": false }, @@ -77,30 +74,30 @@ } ], "timing_ms": { - "total": 1548, + "total": 3011, "by_step": { - "recall": 437, + "recall": 457, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 2, - "shape_probe_coverage": 4, - "shape_probe": 161, - "shape_groups": 45, + "shape_probe_coverage": 8, + "shape_probe": 114, + "shape_groups": 579, "shape_labels": 1, - "shape_admission": 3, + "shape_admission": 2, "phase_derive_shapes": 0, - "shape": 326, + "shape": 1247, "phase_prune_and_rank": 0, - "bind": 225, - "phase_bind": 158, - "phase_settle": 56, - "phase_place_bindings": 18, + "bind": 266, + "phase_bind": 327, + "phase_settle": 46, + "phase_place_bindings": 57, "phase_facet_requests": 0, - "materialise": 41, - "phase_materialise": 0, - "answer": 186, - "options": 40, - "phase_outcome": 61 + "materialise": 44, + "phase_materialise": 200, + "answer": 220, + "options": 55, + "phase_outcome": 92 } }, "restatement": { @@ -547,7 +544,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -1200,7 +1197,7 @@ "selected": [] } ], - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3790,7 +3787,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3798,7 +3796,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "42460c90e7274dd19ecba24a6a0377e9" + "x-dfo-trace-id": "4c3f35c8c0d44b99941a73a9519b41d2" }, "json": { "schema_version": 1, @@ -4199,7 +4197,7 @@ "frame_grammar_version": "1.1.0", "origin_catalogue_build_id": "20261011T194155Z-80081f08", "served_from": "head", - "confirmed_at": "2026-10-24T16:30:05.017544+00:00" + "confirmed_at": "2026-10-26T09:17:22.037029+00:00" }, "warnings": [ { @@ -4221,7 +4219,8 @@ "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, "bytes": 9842, "media_type": "application/vnd.apache.parquet", @@ -4244,7 +4243,7 @@ ], "series_count": 5, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -4527,7 +4526,7 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": true } @@ -4544,7 +4543,7 @@ "headers": { "content-type": "text/html", "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A421efacc2dc010af38112a087b0b49ff5539f61d2afd12a88272b27e99b5f5db?X-Goog-Signature=redacted", - "x-dfo-trace-id": "b9b4b3a90d6546e09cc738a29bbbb1d6" + "x-dfo-trace-id": "333ab72de55449ed8288a5f458406a4c" } } }, @@ -4579,32 +4578,32 @@ "dataset": { "rows": 30, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", - "financingRecipient_kind", + "crsPurpose_label", + "value", + "unit_label", + "crsTypeOfFlow_label", + "provenance_label", "crsCooperationModality", "crsCooperationModality_label", + "financingProvider", + "financingProvider_kind", + "financingRecipient", + "financingRecipient_kind", "crsPurpose", - "crsPurpose_label", "crsTypeOfFinance", "crsTypeOfFinance_label", "crsTypeOfFlow", - "crsTypeOfFlow_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.json b/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.json index 8fc7eea..4a1c656 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.json @@ -5,17 +5,17 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:30:53Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:34Z" }, "call": [ { "op": "mint", "spec": { - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -227,12 +227,12 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } }, - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a.", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a.", "entity_groups": { "destination": { "label": "G77 countries", @@ -243,7 +243,8 @@ "group_id": "groups/OECD.DAC" } } - } + }, + "party_totals": true }, { "op": "result" @@ -258,10 +259,10 @@ "schema_version": 1, "input": { "kind": "spec_id", - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "identity_version": "spec1", "shape": { @@ -473,7 +474,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1963, + "start_year": 1967, "end_year": 2024 } } @@ -489,15 +490,15 @@ "status": 202, "headers": { "content-type": "application/json", - "location": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "location": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry-after": "2", - "x-dfo-trace-id": "a6d5bb2eff5a48578dce7fbf91715755" + "x-dfo-trace-id": "9e96509866ff4da09a698df1f19a5cbd" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500 } } @@ -506,24 +507,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "71ecc596c6034e27b8ba5dd0a5d2c744" + "x-dfo-trace-id": "6185b86217ef499bb5b6932299df1207" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 109318 + "estimate_rows": 107252 } } } @@ -532,24 +533,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "760a3bb2de5e430fb316b3f6a4059c67" + "x-dfo-trace-id": "1ebea3ca2d15409c953cd716bdc1fccf" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 61927, + "estimate_rows": 107252 } } } @@ -558,24 +559,76 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "15f9386c84484af983d33c6d183e5349" + "x-dfo-trace-id": "94bb5a8a7c2041969d9a9118b75dc4aa" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "running", - "poll": "/api/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 109318 + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "c78757a54b9f4b6786a784f8a47ad5b4" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 1784, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "914bfd57942740b893ba9ca1b4f4591b" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2289, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 } } } @@ -584,24 +637,50 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e" + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "2cc73cd4fbaa4a28a350025111f70a79" + "retry-after": "3", + "x-dfo-trace-id": "217766e0425f4916a393331295c4a7e7" }, "json": { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "status": "running", + "poll": "/api/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", + "retry_after_ms": 2934, + "progress": { + "rows_written": 107252, + "estimate_rows": 107252 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "c501ba8d087b4561b2ff0e423c719ab0" + }, + "json": { + "operation": { + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", "identity": { "shape": { "kind": "statvar_schema", @@ -814,14 +893,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } } }, - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "prepared_at": "2026-10-23T18:38:28Z", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "prepared_at": "2026-10-26T09:22:44Z", "coverage": { - "rows": 109318, + "rows": 107252, "dimensions": [ { "label": "financingProvider", @@ -843,14 +922,14 @@ }, { "label": "years", - "count": 62 + "count": 58 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 } }, "citation": { @@ -969,13 +1048,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1963 + "start_year": 1967 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": null, "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -984,8 +1063,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -993,15 +1072,15 @@ "financingProvider", "financingRecipient" ], - "observation_count": 109318 + "observation_count": 107252 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:30:59.952146+00:00" + "confirmed_at": "2026-10-26T09:22:46.854499+00:00" }, "warnings": [], "diagnostics": [ @@ -1013,34 +1092,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "artifact_digest": "sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 473679, + "bytes": 466531, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65", + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39", + "artifact_digest": "sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231", "rendering": { "statistics": "series" }, - "bytes": 144274, + "bytes": 144985, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:38:31.580044+00:00", + "created_at": "2026-10-26T09:22:46.786678+00:00", "tier": "working", - "url": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:099d1ce36b440868b72174fb9cdff503cc4c7a0f0ef902e66484c5bfbd270f39" + "url": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:5247bafa55f84fe2f8bc348601acb955162c39a1a082e939945336470e5c5231" } ], "series_count": 3656, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -1064,7 +1143,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 62 + "distinct": 58 }, { "name": "value", @@ -1304,7 +1383,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1328,7 +1407,7 @@ "varying": [ { "column": "time_period", - "distinct": 62 + "distinct": 58 } ] }, @@ -1387,10 +1466,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:38:31.579178Z" + "held_until": "2026-11-25T09:22:46.785219Z" } } } @@ -1399,21 +1478,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_fbca0f29755230d84729c1243b1d9b45/cv_3d58d26a874dc30b1bba7d5ad6dd90ee/files/sha256:9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65" + "origin_path": "/api/datasets/spec_70ddf0ce6a9b3f147063f2adf37fab9e/cv_fafc60a507e03b0d42689fb1936127b9/files/sha256:a8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted", - "x-dfo-trace-id": "fb09161c837d4874970eaf5a188845da" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted", + "x-dfo-trace-id": "22f0d88923c14d74825b5f14d41c05b8" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A9242d69bedb460996f21f782b1a85ee5cd7eb967174e3b64a4ec5ade5bebab65?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Aa8c742f8f416b2f3414c7a569f7c18b337c5ee9ae3012483583bad0b238f5c61?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -1427,53 +1506,56 @@ "expect": [ { "operation": { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "status": "pending" } }, { "dataset": { - "rows": 109318, + "rows": 107252, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_fbca0f29755230d84729c1243b1d9b45", - "content_version": "cv_3d58d26a874dc30b1bba7d5ad6dd90ee", - "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1963 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_70ddf0ce6a9b3f147063f2adf37fab9e", + "content_version": "cv_fafc60a507e03b0d42689fb1936127b9", + "sentence": "Gross disbursements to the 133 G77 countries from the 32 DAC countries for official development assistance, 1967 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_cc325e96d3ac6fdb6721c2468fd261f221bc87ee5eb9d8e82cf4d1f3720bdf6e", + "operation_id": "op_510787ac0c09a1323e90ae20eafc3eb0246bcd04333aa2ca10792d48dae2cc6f", "poll_waits": [ 1.5, 1.5, 1.5, - 1.5 + 1.5, + 1.784, + 2.289, + 2.934 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.parquet b/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.parquet index 212b05d..a432036 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/mint-async-build.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/newest-version-unreadable-page.json b/python/client-contract/src/dfo_client_contract/scenarios/newest-version-unreadable-page.json index 7226186..402884f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/newest-version-unreadable-page.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/newest-version-unreadable-page.json @@ -93,7 +93,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "6f93d15300c8413191296f90fbc4b8b0" + "x-dfo-trace-id": "7675f31f4bd548c782dad4c6001a7666" }, "json": { "schema_version": 1, @@ -145,7 +145,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "6f93d15300c8413191296f90fbc4b8b0" + "trace_id": "7675f31f4bd548c782dad4c6001a7666" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/operation-failed-without-error.json b/python/client-contract/src/dfo_client_contract/scenarios/operation-failed-without-error.json index b716c5c..f6ba391 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/operation-failed-without-error.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/operation-failed-without-error.json @@ -10,7 +10,7 @@ "call": [ { "op": "operation", - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, { "op": "result" @@ -20,24 +20,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "7a8fcbad75fa4232a0fd7b55729da907" + "x-dfo-trace-id": "3503ecbc88d84ff3b8fd26ec06762f34" }, "json": { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "running", - "poll": "/api/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 112203 + "estimate_rows": 110887 } } } @@ -46,20 +46,20 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "09873539d7e1412baa820fb5f7421322" + "x-dfo-trace-id": "b53e0a1669e045e18d440f6189c13db3" }, "json": { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "failed", - "poll": "/api/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "retry_after_ms": 1500 } } @@ -69,7 +69,7 @@ "expect": [ { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "running" } }, @@ -77,10 +77,10 @@ "error": { "class": "ServerError", "code": "operation_failed", - "trace_id": "09873539d7e1412baa820fb5f7421322", + "trace_id": "b53e0a1669e045e18d440f6189c13db3", "long_builds": [ { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "poll_waits": [ 1.5 ], diff --git a/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.json b/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.json index b34c17c..3c31a70 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.json @@ -5,13 +5,13 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:08Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:22:51Z" }, "call": [ { "op": "operation", - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, { "op": "result" @@ -21,24 +21,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "7a8fcbad75fa4232a0fd7b55729da907" + "x-dfo-trace-id": "3503ecbc88d84ff3b8fd26ec06762f34" }, "json": { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "running", - "poll": "/api/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "retry_after_ms": 1500, "progress": { "rows_written": 0, - "estimate_rows": 112203 + "estimate_rows": 110887 } } } @@ -47,24 +47,24 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, "response": { "status": 200, "headers": { "content-type": "application/json", "retry-after": "2", - "x-dfo-trace-id": "09873539d7e1412baa820fb5f7421322" + "x-dfo-trace-id": "b53e0a1669e045e18d440f6189c13db3" }, "json": { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "running", - "poll": "/api/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "retry_after_ms": 1500, "progress": { - "rows_written": 0, - "estimate_rows": 112203 + "rows_written": 33440, + "estimate_rows": 110887 } } } @@ -73,24 +73,128 @@ { "request": { "method": "GET", - "path": "/datasets/operations/op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5" + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" }, "response": { "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "c5978ef07c5340e0ae21c8881deb8130" + "retry-after": "2", + "x-dfo-trace-id": "549a9e0bbec342a99c0176fd523c76e1" }, "json": { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "status": "running", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "retry_after_ms": 1500, + "progress": { + "rows_written": 106679, + "estimate_rows": 110887 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "2bb0086a22c149bdb666fe3834159c36" + }, + "json": { + "operation": { + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "status": "running", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "retry_after_ms": 1500, + "progress": { + "rows_written": 110887, + "estimate_rows": 110887 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "2", + "x-dfo-trace-id": "d0b6893e4ca84abcaa449b70756d3a76" + }, + "json": { + "operation": { + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "status": "running", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "retry_after_ms": 1832, + "progress": { + "rows_written": 110887, + "estimate_rows": 110887 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "retry-after": "3", + "x-dfo-trace-id": "09d934a8ec1e4169971b1e0c61e5af38" + }, + "json": { + "operation": { + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "status": "running", + "poll": "/api/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", + "retry_after_ms": 2348, + "progress": { + "rows_written": 110887, + "estimate_rows": 110887 + } + } + } + } + }, + { + "request": { + "method": "GET", + "path": "/datasets/operations/op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "ddc5828be1d84a159d4579ad5ca8833d" + }, + "json": { + "operation": { + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "done" }, "manifest": { "schema_version": 1, "definition": { "schema_version": 1, - "spec_id": "spec_8150d26a25222b2043e8c12b250b764d", + "spec_id": "spec_dee41c61dd0335d145f4c65e451817cb", "identity": { "shape": { "kind": "statvar_schema", @@ -309,14 +413,14 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1966 + "start_year": 1968 } } }, - "content_version": "cv_fe2e676a79899730eac6110f58213472", - "prepared_at": "2026-10-23T18:38:59Z", + "content_version": "cv_c29903dc477674b87b3e5e0097f7a828", + "prepared_at": "2026-10-26T09:23:00Z", "coverage": { - "rows": 112203, + "rows": 110887, "dimensions": [ { "label": "financingProvider", @@ -338,14 +442,14 @@ }, { "label": "years", - "count": 59 + "count": 57 } ], "window": { "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1966 + "start_year": 1968 } }, "citation": { @@ -464,13 +568,13 @@ "kind": "range", "origin": "query", "end_year": 2024, - "start_year": 1966 + "start_year": 1968 }, - "graph_revision": "dfo-dev-dc-datacommons-service-00264-2s9", + "graph_revision": "dfo-dev-dc-datacommons-service-00273-mmh", "normalisations": [], "code_index_stamp": { - "exported_at": "2026-10-23T18:14:12Z", - "content_hash": "29eb81d7b6c88641b1d4e8eb5e850ca36de66b99d6e9e09680aac020507a6392", + "exported_at": "2026-10-24T19:20:27Z", + "content_hash": "3d946d8f4986dbbe0fa3e5c4c917200381f529d89ce417c11daeb22df56eed99", "graph_endpoint": "graph_release" }, "source_revisions": [ @@ -479,8 +583,8 @@ "revision": "2026-10-15" } ], - "capability_version": "dset_cc753cefa9199881", - "materialiser_build": "035d29cd05fb31d7cb47cfb33197d4002f773df6", + "capability_version": "dset_76f5fa7244511c58", + "materialiser_build": "d80df9f5fd598b7345255470932d202ee5d49942", "resolved_stat_vars": [ { "ref": "dcid:Amount_FinancialFlow_Gross_Disbursement_DAC2aAidType206_Nominal", @@ -488,21 +592,30 @@ "financingProvider", "financingRecipient" ], - "observation_count": 112203 + "observation_count": 110887 } ], "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", - "origin_catalogue_build_id": "20261023T061534Z-06fa7b89", + "origin_catalogue_build_id": "20261026T000018Z-0af2cac5", "served_from": "materialise", - "confirmed_at": "2026-10-24T16:31:14.752129+00:00" + "confirmed_at": "2026-10-26T09:23:03.079042+00:00" }, "warnings": [ { "code": "coverage_below_scope", "severity": "warn", - "message": "The served coverage falls short of the requested scope. No observations were served for country/CHL, country/COL, country/CRI, country/MEX." + "message": "The served coverage falls short of the requested scope. No observations were served for 4 requested entities.", + "unserved_entities": { + "count": 4, + "refs": [ + "dcid:country/CHL", + "dcid:country/COL", + "dcid:country/CRI", + "dcid:country/MEX" + ] + } } ], "diagnostics": [ @@ -514,34 +627,34 @@ ], "artifacts": [ { - "artifact_digest": "sha256:55f95f11ffe010e99407d279d159e956d4dfa6a6ba89f6d73aa32fb952fccd1d", + "artifact_digest": "sha256:3aae1d10e41e6cd40e33a2a65757641a4c74244ffcdd47974df72ab6dfa60779", "rendering": { "bundle": false, "format": "parquet", "layout": 4 }, - "bytes": 489647, + "bytes": 486086, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:39:02.714887+00:00", + "created_at": "2026-10-26T09:23:03.005638+00:00", "tier": "working", - "url": "/api/datasets/spec_8150d26a25222b2043e8c12b250b764d/cv_fe2e676a79899730eac6110f58213472/files/sha256:55f95f11ffe010e99407d279d159e956d4dfa6a6ba89f6d73aa32fb952fccd1d", + "url": "/api/datasets/spec_dee41c61dd0335d145f4c65e451817cb/cv_c29903dc477674b87b3e5e0097f7a828/files/sha256:3aae1d10e41e6cd40e33a2a65757641a4c74244ffcdd47974df72ab6dfa60779", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:39345f0fc420a15e6262bf58781d46ebda6e7e9a0f11fc0c7540826b151a1d0e", + "artifact_digest": "sha256:871390d2ad08d6c429938b930ac69f71d7b5d88cda0170a0433158f75dbdc167", "rendering": { "statistics": "series" }, - "bytes": 153622, + "bytes": 153793, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T18:39:02.714887+00:00", + "created_at": "2026-10-26T09:23:03.005638+00:00", "tier": "working", - "url": "/api/datasets/spec_8150d26a25222b2043e8c12b250b764d/cv_fe2e676a79899730eac6110f58213472/files/sha256:39345f0fc420a15e6262bf58781d46ebda6e7e9a0f11fc0c7540826b151a1d0e" + "url": "/api/datasets/spec_dee41c61dd0335d145f4c65e451817cb/cv_c29903dc477674b87b3e5e0097f7a828/files/sha256:871390d2ad08d6c429938b930ac69f71d7b5d88cda0170a0433158f75dbdc167" } ], "series_count": 3907, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -565,7 +678,7 @@ "name": "time_period", "kind": "grain", "droppable": false, - "distinct": 59 + "distinct": 57 }, { "name": "value", @@ -805,7 +918,7 @@ "varying": [ { "column": "time_period", - "distinct": 59 + "distinct": 57 } ] }, @@ -829,7 +942,7 @@ "varying": [ { "column": "time_period", - "distinct": 59 + "distinct": 57 } ] }, @@ -888,10 +1001,10 @@ ] } }, - "series_coverage_stamp": "0b8c9a18fcc5af19013b21e865c64481503b6a6542f7d2e6628999e6559a47de", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-22T18:39:02.713360Z" + "held_until": "2026-11-25T09:23:03.004652Z" } } } @@ -1127,7 +1240,7 @@ "requested_window": { "kind": "range", "end_year": 2024, - "start_year": 1966 + "start_year": 1968 } } } @@ -1139,7 +1252,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "0a02312a9fba4338a5c67e61e4f0c8fd" + "x-dfo-trace-id": "2a8974d28a894ee0aebb24df0ca7dc67" }, "json": { "schema_version": 1, @@ -1148,11 +1261,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "0a02312a9fba4338a5c67e61e4f0c8fd", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "2a8974d28a894ee0aebb24df0ca7dc67", "warnings": [], "timing_ms": { - "total": 2806, + "total": 2071, "by_step": { "coverage": 0 } @@ -1165,19 +1278,19 @@ "variable_text": "dataset", "warnings": [], "timing_ms": { - "total": 2075, + "total": 1091, "by_step": { "validate": 1, - "walk": 10, + "walk": 19, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 1995, + "shape": 1000, "bind": 0, - "where_places": 1, - "materialise": 24, - "answer": 11, - "options": 33 + "where_places": 2, + "materialise": 28, + "answer": 13, + "options": 28 } }, "entity_refs": [], @@ -1192,10 +1305,10 @@ "assumptions": [], "status": "definite", "spec": { - "spec_id": "spec_8150d26a25222b2043e8c12b250b764d", + "spec_id": "spec_dee41c61dd0335d145f4c65e451817cb", "definition": { "schema_version": 1, - "spec_id": "spec_8150d26a25222b2043e8c12b250b764d", + "spec_id": "spec_dee41c61dd0335d145f4c65e451817cb", "identity": { "identity_version": "spec1", "shape": { @@ -1413,7 +1526,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1966, + "start_year": 1968, "end_year": 2024 } } @@ -2526,7 +2639,7 @@ "value": { "value_kind": "time_window", "time_window": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 } } @@ -2553,7 +2666,7 @@ "member_count_exact": true, "served_count": 1, "data_date_range": { - "start": "1966", + "start": "1968", "end": "2024" } }, @@ -3588,7 +3701,7 @@ "coverage": { "kind": "exact", "has_data": true, - "observation_count": 112203, + "observation_count": 110887, "evidence": "facet_metadata", "dimensions": [ { @@ -3597,7 +3710,7 @@ }, { "label": "observations", - "count": 59 + "count": 57 }, { "label": "variables", @@ -3605,7 +3718,7 @@ } ], "window": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 } }, @@ -3613,7 +3726,7 @@ "year": 2024, "year_named": true, "named_window": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "observations": 2629, @@ -3653,18 +3766,18 @@ ] }, "download": { - "rows": 112203, + "rows": 110887, "rows_exact": true, "series": 3907, "window": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "licence": { "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -3710,7 +3823,7 @@ "relation": "child", "parent": "DAC2aAidType206", "member_count": 2, - "served_rows": 18000, + "served_rows": 17503, "served_rows_exact": true }, { @@ -3724,7 +3837,7 @@ "relation": "child", "parent": "DAC2aAidType206", "member_count": 1, - "served_rows": 111362, + "served_rows": 110146, "served_rows_exact": true }, { @@ -3737,7 +3850,7 @@ }, "relation": "root", "member_count": 1, - "served_rows": 88283, + "served_rows": 87187, "served_rows_exact": true }, { @@ -3806,7 +3919,7 @@ "source": "observed" } ], - "rendered_sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1966 to 2024. Current US dollars · OECD DAC2a.", + "rendered_sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1968 to 2024. Current US dollars · OECD DAC2a.", "sentence_plan": [ { "kind": "slot", @@ -4089,7 +4202,7 @@ }, { "kind": "slot", - "text": "1966 to 2024", + "text": "1968 to 2024", "axis": "when", "segment": "heading", "binding_ref": "when:", @@ -5482,10 +5595,10 @@ ], "query_qualifiers": [], "requested_window": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -5493,7 +5606,7 @@ { "definition": { "schema_version": 1, - "spec_id": "spec_52f77b1b2faab0669b4e550e53379c22", + "spec_id": "spec_6a878829c5a95abb6c08dc9bb0668022", "identity": { "identity_version": "spec1", "shape": { @@ -5711,7 +5824,7 @@ }, "requested_window": { "kind": "range", - "start_year": 1966, + "start_year": 1968, "end_year": 2024 } } @@ -5755,7 +5868,7 @@ } } ], - "served_rows": 112830, + "served_rows": 111506, "served_rows_exact": true } ], @@ -5814,7 +5927,7 @@ "end_year": 2021 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2023, @@ -5828,7 +5941,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2002, @@ -5842,7 +5955,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -5856,7 +5969,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -5870,7 +5983,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1991, @@ -5884,7 +5997,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -5894,11 +6007,11 @@ "code": "AUD", "noun": "Australian dollars", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -5912,7 +6025,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1995, @@ -5926,7 +6039,7 @@ "end_year": 2019 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2023, @@ -5940,7 +6053,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1996, @@ -5954,7 +6067,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -5968,7 +6081,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -5982,7 +6095,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -5996,7 +6109,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6010,7 +6123,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1985, @@ -6024,7 +6137,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6038,7 +6151,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1986, @@ -6052,7 +6165,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6066,7 +6179,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6080,7 +6193,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6094,7 +6207,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -6108,7 +6221,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6118,11 +6231,11 @@ "code": "CAD", "noun": "Canadian dollars", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -6136,7 +6249,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1991, @@ -6146,11 +6259,11 @@ "code": "CHF", "noun": "Swiss francs", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -6164,7 +6277,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6178,7 +6291,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6192,7 +6305,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6206,7 +6319,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6220,7 +6333,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6234,7 +6347,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1998, @@ -6248,7 +6361,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1990, @@ -6258,11 +6371,11 @@ "code": "DKK", "noun": "Danish kroner", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -6276,7 +6389,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6290,7 +6403,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6304,7 +6417,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6318,7 +6431,7 @@ "end_year": 2019 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -6332,7 +6445,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6346,7 +6459,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1973, @@ -6360,7 +6473,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6370,11 +6483,11 @@ "code": "GBP", "noun": "British pounds", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -6388,7 +6501,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1994, @@ -6402,7 +6515,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6416,7 +6529,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6430,7 +6543,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6444,7 +6557,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6458,7 +6571,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6472,7 +6585,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6486,7 +6599,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6500,7 +6613,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6514,7 +6627,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2003, @@ -6528,7 +6641,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6542,7 +6655,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1997, @@ -6556,7 +6669,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6570,7 +6683,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2000, @@ -6584,7 +6697,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6598,7 +6711,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1997, @@ -6612,7 +6725,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6626,7 +6739,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6636,11 +6749,11 @@ "code": "JPY", "noun": "Japanese yen", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -6654,7 +6767,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6668,7 +6781,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -6682,7 +6795,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1986, @@ -6696,7 +6809,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6710,7 +6823,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1987, @@ -6724,7 +6837,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1995, @@ -6738,7 +6851,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2013, @@ -6752,7 +6865,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6766,7 +6879,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6780,7 +6893,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6794,7 +6907,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6808,7 +6921,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6822,7 +6935,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6836,7 +6949,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -6850,7 +6963,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6864,7 +6977,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -6878,7 +6991,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1998, @@ -6892,7 +7005,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6906,7 +7019,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2001, @@ -6920,7 +7033,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1990, @@ -6934,7 +7047,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6948,7 +7061,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6962,7 +7075,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6976,7 +7089,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -6990,7 +7103,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7004,7 +7117,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7018,7 +7131,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1989, @@ -7032,7 +7145,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1990, @@ -7046,7 +7159,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1987, @@ -7056,11 +7169,11 @@ "code": "NOK", "noun": "Norwegian kroner", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -7074,7 +7187,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7084,11 +7197,11 @@ "code": "NZD", "noun": "New Zealand dollars", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1961, @@ -7102,7 +7215,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7116,7 +7229,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7130,7 +7243,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7144,7 +7257,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7158,7 +7271,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7172,7 +7285,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1998, @@ -7186,7 +7299,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7200,7 +7313,7 @@ "end_year": 1990 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2023, @@ -7214,7 +7327,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2013, @@ -7228,7 +7341,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1997, @@ -7242,7 +7355,7 @@ "end_year": 2020 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2010, @@ -7256,7 +7369,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7270,7 +7383,7 @@ "end_year": 2023 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2000, @@ -7284,7 +7397,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7298,7 +7411,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7312,7 +7425,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7322,11 +7435,11 @@ "code": "SEK", "noun": "Swedish kronor", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -7340,7 +7453,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7354,7 +7467,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7368,7 +7481,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7382,7 +7495,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2011, @@ -7396,7 +7509,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7410,7 +7523,7 @@ "end_year": 2010 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7424,7 +7537,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7438,7 +7551,7 @@ "end_year": 2016 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2018, @@ -7452,7 +7565,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -7466,7 +7579,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -7480,7 +7593,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7494,7 +7607,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7508,7 +7621,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1990, @@ -7522,7 +7635,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7536,7 +7649,7 @@ "end_year": 2016 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2023, @@ -7550,7 +7663,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7564,7 +7677,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1991, @@ -7578,7 +7691,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7588,11 +7701,11 @@ "code": "USD", "noun": "US dollars", "current_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1960, @@ -7606,7 +7719,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7620,7 +7733,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1992, @@ -7634,7 +7747,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2017, @@ -7648,7 +7761,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7662,7 +7775,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7676,7 +7789,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7690,7 +7803,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7704,7 +7817,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7718,7 +7831,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7732,7 +7845,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 2014, @@ -7746,7 +7859,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7760,7 +7873,7 @@ "end_year": 2024 }, "constant_years": { - "start_year": 1966, + "start_year": 1968, "end_year": 2024 }, "base_year_min": 1980, @@ -7774,7 +7887,7 @@ } } ], - "rendered_sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1966 to 2024. Current US dollars · OECD DAC2a. You asked for Chile, Colombia, Costa Rica, and Mexico. The data holds no values for them.", + "rendered_sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1968 to 2024. Current US dollars · OECD DAC2a. You asked for Chile, Colombia, Costa Rica, and Mexico. The data holds no values for them.", "sentence_plan": [ { "kind": "slot", @@ -8067,7 +8180,7 @@ }, { "kind": "slot", - "text": "1966 to 2024", + "text": "1968 to 2024", "axis": "when", "segment": "heading", "binding_ref": "when:", @@ -8150,21 +8263,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_8150d26a25222b2043e8c12b250b764d/cv_fe2e676a79899730eac6110f58213472/files/sha256:55f95f11ffe010e99407d279d159e956d4dfa6a6ba89f6d73aa32fb952fccd1d" + "origin_path": "/api/datasets/spec_dee41c61dd0335d145f4c65e451817cb/cv_c29903dc477674b87b3e5e0097f7a828/files/sha256:3aae1d10e41e6cd40e33a2a65757641a4c74244ffcdd47974df72ab6dfa60779" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A55f95f11ffe010e99407d279d159e956d4dfa6a6ba89f6d73aa32fb952fccd1d?X-Goog-Signature=redacted", - "x-dfo-trace-id": "51314836477547c6aff574e07f6b19ab" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A3aae1d10e41e6cd40e33a2a65757641a4c74244ffcdd47974df72ab6dfa60779?X-Goog-Signature=redacted", + "x-dfo-trace-id": "e8ac5b39c0c94474b741fefd66f9ef86" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A55f95f11ffe010e99407d279d159e956d4dfa6a6ba89f6d73aa32fb952fccd1d?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3A3aae1d10e41e6cd40e33a2a65757641a4c74244ffcdd47974df72ab6dfa60779?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -8178,51 +8291,55 @@ "expect": [ { "operation": { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "status": "running" } }, { "dataset": { - "rows": 112203, + "rows": 110887, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", - "financingProvider_kind", - "financingRecipient", "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", + "financingRecipient", + "financingProvider_kind", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ { "parts": [], - "spec_id": "spec_8150d26a25222b2043e8c12b250b764d", - "content_version": "cv_fe2e676a79899730eac6110f58213472", - "sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1966 to 2024. Current US dollars · OECD DAC2a." + "spec_id": "spec_dee41c61dd0335d145f4c65e451817cb", + "content_version": "cv_c29903dc477674b87b3e5e0097f7a828", + "sentence": "Gross disbursements to 133 recipients from 38 providers for official development assistance, 1968 to 2024. Current US dollars · OECD DAC2a." } ], "missing": [], "long_builds": [ { - "operation_id": "op_d3596ce124b0257af0dfded6707dc8179b5e046e67080dd6995f7e0de23f7fe5", + "operation_id": "op_a50d083649ee12fbede446887afa771e76446419365e74ade9d5239d128e6846", "poll_waits": [ 1.5, - 1.5 + 1.5, + 1.5, + 1.5, + 1.832, + 2.348 ], "resubmissions": 0 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.parquet b/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.parquet index b39d4a8..c952971 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/operation-resume.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/scenarios/resubmit-candidates-sibling.json b/python/client-contract/src/dfo_client_contract/scenarios/resubmit-candidates-sibling.json index 7854333..80e7e42 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/resubmit-candidates-sibling.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/resubmit-candidates-sibling.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:50Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:39Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "a42078d26ea943a488b4cb642719e808" + "x-dfo-trace-id": "3d449a99fb6941a8a036d4fd4ccd2555" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "a42078d26ea943a488b4cb642719e808", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "3d449a99fb6941a8a036d4fd4ccd2555", "warnings": [], "timing_ms": { - "total": 4608, + "total": 6074, "by_step": { "coverage": 0, - "extract": 1312, + "extract": 1151, "cache_lookup": 0 } }, @@ -62,7 +62,7 @@ "input_tokens": 2241, "output_tokens": 197, "cached_tokens": 843, - "jev_input_tokens": 26474, + "jev_input_tokens": 26478, "jev_output_tokens": 5727 }, "timed_out": false @@ -73,28 +73,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 2048, + "total": 3027, "by_step": { "recall": 1, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 4, - "shape_probe": 96, - "shape_groups": 5, + "shape_probe": 99, + "shape_groups": 6, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 246, + "shape": 206, "phase_prune_and_rank": 0, - "bind": 131, - "phase_bind": 11, - "phase_settle": 7, - "phase_place_bindings": 29, + "bind": 135, + "phase_bind": 13, + "phase_settle": 1, + "phase_place_bindings": 32, "phase_facet_requests": 0, - "materialise": 113, - "phase_materialise": 0, - "answer": 106, - "phase_outcome": 1404 + "materialise": 108, + "phase_materialise": 26, + "answer": 749, + "phase_outcome": 1756 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -816,7 +816,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1599,6 +1599,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1815,6 +1824,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2350,7 +2368,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2388,29 +2406,29 @@ } ], "timing_ms": { - "total": 3238, + "total": 4844, "by_step": { - "recall": 321, + "recall": 440, "phase_resolve_places": 0, "phase_check_entities": 0, - "shape_arcs": 86, - "shape_probe_coverage": 32, - "shape_probe": 102, - "shape_groups": 22, + "shape_arcs": 114, + "shape_probe_coverage": 1, + "shape_probe": 110, + "shape_groups": 48, "shape_labels": 1, - "shape_admission": 1, + "shape_admission": 0, "phase_derive_shapes": 0, - "shape": 316, + "shape": 430, "phase_prune_and_rank": 0, - "bind": 125, - "phase_bind": 26, - "phase_settle": 56, - "phase_place_bindings": 1829, + "bind": 142, + "phase_bind": 32, + "phase_settle": 18, + "phase_place_bindings": 3159, "phase_facet_requests": 0, - "materialise": 162, - "phase_materialise": 0, - "answer": 104, - "phase_outcome": 299 + "materialise": 156, + "phase_materialise": 145, + "answer": 88, + "phase_outcome": 234 } }, "rendered_sentence": "2 readings. Kenya's annual amount of emissions: carbon dioxide. Air and GHG emissions · 2013 to 2020. Kenya's annual amount of emissions: carbon dioxide. Air and GHG emissions · 2013 to 2020.", @@ -3048,7 +3066,7 @@ "value": { "ref": { "ref": "CarbonDioxide", - "label": "CarbonDioxide" + "label": "Carbon Dioxide (CO2)" }, "value_kind": "enum_value" }, @@ -3289,7 +3307,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3305,7 +3323,7 @@ "varies": false, "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" + "label": "Carbon Dioxide (CO2)" }, "distinct_count": 1, "undeclared_count": 0, @@ -3313,7 +3331,7 @@ { "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" + "label": "Carbon Dioxide (CO2)" }, "count": 1 } @@ -3570,7 +3588,7 @@ "value": { "ref": { "ref": "CarbonDioxide", - "label": "CarbonDioxide" + "label": "Carbon Dioxide (CO2)" }, "value_kind": "enum_value" }, @@ -3788,7 +3806,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -3805,7 +3823,7 @@ "varies": false, "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" + "label": "Carbon Dioxide (CO2)" }, "distinct_count": 1, "undeclared_count": 0, @@ -3813,7 +3831,7 @@ { "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" + "label": "Carbon Dioxide (CO2)" }, "count": 1 } @@ -4375,7 +4393,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "775b92aeffce4d859d3170a45daccd25" + "x-dfo-trace-id": "3b1149548ca74cb588cb0a6041e4fb22" }, "json": { "schema_version": 1, @@ -4384,11 +4402,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "775b92aeffce4d859d3170a45daccd25", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "3b1149548ca74cb588cb0a6041e4fb22", "warnings": [], "timing_ms": { - "total": 811, + "total": 698, "by_step": { "coverage": 0 } @@ -4401,19 +4419,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 397, + "total": 335, "by_step": { "validate": 0, "walk": 0, - "reread_arcs": 1, + "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 274, + "shape": 178, "bind": 0, - "where_places": 0, - "materialise": 121, - "answer": 1, - "options": 0 + "where_places": 1, + "materialise": 153, + "answer": 2, + "options": 1 } }, "entity_refs": [], @@ -4932,7 +4950,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -4994,19 +5012,19 @@ } ], "timing_ms": { - "total": 1120, + "total": 1059, "by_step": { "validate": 0, - "walk": 797, - "reread_arcs": 2, + "walk": 769, + "reread_arcs": 1, "detect_edits": 0, "resolve_survivors": 0, - "shape": 9, - "bind": 0, + "shape": 12, + "bind": 1, "where_places": 0, - "materialise": 168, - "answer": 8, - "options": 136 + "materialise": 190, + "answer": 9, + "options": 77 } }, "rendered_sentence": "2 readings. Kenya's annual amount of emissions: carbon dioxide. Air and GHG emissions · 2013 to 2020. Kenya's annual amount of emissions: carbon dioxide. Air and GHG emissions · 2013 to 2020.", @@ -5606,7 +5624,7 @@ "value": { "ref": { "ref": "CarbonDioxide", - "label": "CarbonDioxide" + "label": "Carbon Dioxide (CO2)" }, "value_kind": "enum_value" }, @@ -5686,7 +5704,9 @@ "axis": "what", "segment": "heading", "entity_refs": [], - "states": [] + "states": [ + "how:emittedThing" + ] }, { "kind": "slot", @@ -5845,178 +5865,27 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, "measure_label_axes": [], "measure_axis_values": [], "open_axis_facts": [ - { - "property": "emissionSource", - "varies": true, - "distinct_count": 16, - "undeclared_count": 6, - "values": [ - { - "value": { - "ref": "Agriculture", - "label": "Agriculture" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/4", - "label": "Chemical Evaporation (SCC 4)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/1", - "label": "External Combustion (SCC 1)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/23", - "label": "Industrial Processes - NonPoint Source (SCC 23)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/3", - "label": "Industrial Processes - Point Source (SCC 3)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/2", - "label": "Internal Combustion Engines (SCC 2)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/6", - "label": "MACT Source Categories (SCC 6)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/28", - "label": "Miscellaneous Area Sources (SCC 28)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/22", - "label": "Mobile Sources (SCC 22)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/27", - "label": "Natural Sources (SCC 27)" - }, - "count": 1 - }, - { - "value": { - "ref": "ClimateTrace_OtherAgriculturalSoilEmissions", - "label": "Other Agricultural Soil Emissions" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/24", - "label": "Solvent Utilization (SCC 24)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/21", - "label": "Stationary Source Fuel Combustion (SCC 21)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/25", - "label": "Storage and Transport (SCC 25)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/5", - "label": "Waste Disposal (SCC 5)" - }, - "count": 1 - }, - { - "value": { - "ref": "EPA_SCC/26", - "label": "Waste Disposal, Treatment, and Recovery (SCC 26)" - }, - "count": 1 - } - ] - }, { "property": "emittedThing", - "varies": true, - "distinct_count": 6, - "undeclared_count": 16, + "varies": false, + "value": { + "ref": "CarbonDioxide", + "label": "Carbon Dioxide (CO2)" + }, + "distinct_count": 1, + "undeclared_count": 0, "values": [ { "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" - }, - "count": 1 - }, - { - "value": { - "ref": "CarbonMonoxide", - "label": "CarbonMonoxide" - }, - "count": 1 - }, - { - "value": { - "ref": "GreenhouseGas", - "label": "GreenhouseGas" - }, - "count": 1 - }, - { - "value": { - "ref": "OxidesOfNitrogen", - "label": "Oxides Of Nitrogen (NOX)" - }, - "count": 1 - }, - { - "value": { - "ref": "OxidesOfSulfur", - "label": "OxidesOfSulfur" - }, - "count": 1 - }, - { - "value": { - "ref": "VolatileOrganicCompound", - "label": "Volatile Organic Compound (VOC)" + "label": "Carbon Dioxide (CO2)" }, "count": 1 } @@ -6273,7 +6142,7 @@ "value": { "ref": { "ref": "CarbonDioxide", - "label": "CarbonDioxide" + "label": "Carbon Dioxide (CO2)" }, "value_kind": "enum_value" }, @@ -6353,7 +6222,9 @@ "axis": "what", "segment": "heading", "entity_refs": [], - "states": [] + "states": [ + "how:emittedThing" + ] }, { "kind": "slot", @@ -6489,7 +6360,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, @@ -6498,49 +6369,18 @@ "open_axis_facts": [ { "property": "emittedThing", - "varies": true, - "distinct_count": 6, + "varies": false, + "value": { + "ref": "CarbonDioxide", + "label": "Carbon Dioxide (CO2)" + }, + "distinct_count": 1, "undeclared_count": 0, "values": [ { "value": { "ref": "CarbonDioxide", - "label": "Carbon Dioxide" - }, - "count": 1 - }, - { - "value": { - "ref": "CarbonMonoxide", - "label": "CarbonMonoxide" - }, - "count": 1 - }, - { - "value": { - "ref": "GreenhouseGas", - "label": "GreenhouseGas" - }, - "count": 1 - }, - { - "value": { - "ref": "OxidesOfNitrogen", - "label": "Oxides Of Nitrogen (NOX)" - }, - "count": 1 - }, - { - "value": { - "ref": "OxidesOfSulfur", - "label": "OxidesOfSulfur" - }, - "count": 1 - }, - { - "value": { - "ref": "VolatileOrganicCompound", - "label": "Volatile Organic Compound (VOC)" + "label": "Carbon Dioxide (CO2)" }, "count": 1 } diff --git a/python/client-contract/src/dfo_client_contract/scenarios/resubmit-placeholder-without-restatement.json b/python/client-contract/src/dfo_client_contract/scenarios/resubmit-placeholder-without-restatement.json index eb14cf3..0a15890 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/resubmit-placeholder-without-restatement.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/resubmit-placeholder-without-restatement.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:32:06Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:59Z" }, "call": [ { @@ -36,7 +36,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "8419a37e166c4f1c99132fe35dbe31ce" + "x-dfo-trace-id": "e46fa17bc1484e228c45bc2637a31869" }, "json": { "schema_version": 1, @@ -47,14 +47,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "8419a37e166c4f1c99132fe35dbe31ce", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "e46fa17bc1484e228c45bc2637a31869", "warnings": [], "timing_ms": { - "total": 3120, + "total": 3952, "by_step": { "coverage": 0, - "extract": 1374, + "extract": 1129, "cache_lookup": 0 } }, @@ -62,7 +62,7 @@ "input_tokens": 2241, "output_tokens": 196, "cached_tokens": 843, - "jev_input_tokens": 20184, + "jev_input_tokens": 20185, "jev_output_tokens": 4309 }, "timed_out": false @@ -73,28 +73,28 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 1744, + "total": 2818, "by_step": { - "recall": 2, - "phase_resolve_places": 0, + "recall": 1, + "phase_resolve_places": 1, "phase_check_entities": 0, "shape_arcs": 3, - "shape_probe": 80, - "shape_groups": 5, + "shape_probe": 98, + "shape_groups": 6, "shape_labels": 1, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 178, + "shape": 1214, "phase_prune_and_rank": 0, - "bind": 126, - "phase_bind": 11, - "phase_settle": 7, - "phase_place_bindings": 17, + "bind": 157, + "phase_bind": 24, + "phase_settle": 1, + "phase_place_bindings": 33, "phase_facet_requests": 0, - "materialise": 26, - "phase_materialise": 0, - "answer": 30, - "phase_outcome": 1347 + "materialise": 22, + "phase_materialise": 25, + "answer": 33, + "phase_outcome": 1307 } }, "rendered_sentence": "2 readings. Kenya's deaths rate per capita. World Development Indicators · 2013 to 2022. Kenya's number of deaths. Kenya Census · 2018 to 2022.", @@ -816,7 +816,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1599,6 +1599,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years0", + "label": "Years 0" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -1815,6 +1824,15 @@ "value_kind": "enum_value" } }, + { + "value": { + "ref": { + "ref": "Years85", + "label": "Years 85" + }, + "value_kind": "enum_value" + } + }, { "value": { "ref": { @@ -2350,7 +2368,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -2375,22 +2393,22 @@ "variable_text": "savings", "warnings": [], "timing_ms": { - "total": 1383, + "total": 1496, "by_step": { - "recall": 222, + "recall": 235, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 1, "shape_probe_coverage": 1, - "shape_probe": 181, - "shape_groups": 41, + "shape_probe": 96, + "shape_groups": 908, "shape_labels": 1, - "shape_admission": 0, - "phase_derive_shapes": 0, - "shape": 1000, + "shape_admission": 6, + "phase_derive_shapes": 1, + "shape": 1119, "phase_prune_and_rank": 0, - "bind": 103, - "phase_bind": 58 + "bind": 137, + "phase_bind": 4 } }, "rendered_sentence": "Could not match \"savings\" to a supported indicator. Try a different wording or a related indicator.", @@ -2727,7 +2745,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "00de3c0cb919474893de7c4eb875f9be" + "x-dfo-trace-id": "cf171b52e7e34625bb6dba046e930821" }, "json": { "schema_version": 1, @@ -2736,11 +2754,11 @@ "extract_skipped": true }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "00de3c0cb919474893de7c4eb875f9be", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "cf171b52e7e34625bb6dba046e930821", "warnings": [], "timing_ms": { - "total": 293, + "total": 351, "by_step": { "coverage": 0 } @@ -2753,19 +2771,19 @@ "variable_text": "mortality", "warnings": [], "timing_ms": { - "total": 242, + "total": 283, "by_step": { "validate": 0, "walk": 0, "reread_arcs": 0, "detect_edits": 0, "resolve_survivors": 0, - "shape": 149, - "bind": 0, - "where_places": 1, - "materialise": 90, - "answer": 1, - "options": 1 + "shape": 172, + "bind": 1, + "where_places": 0, + "materialise": 108, + "answer": 2, + "options": 0 } }, "entity_refs": [], @@ -3284,7 +3302,7 @@ "start_year": 2013, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40" + "capability_version": "cap_765c9d2a0c7529b9" }, "confidence": "high", "available_at_place": true, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-body-not-an-object.json b/python/client-contract/src/dfo_client_contract/scenarios/search-body-not-an-object.json index 54acd79..9e2a4fb 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-body-not-an-object.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-body-not-an-object.json @@ -30,7 +30,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": [ { @@ -42,14 +42,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -66,9 +66,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -88,6 +88,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -184,7 +185,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -213,7 +215,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -235,6 +238,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -259,7 +263,7 @@ "error": { "class": "ContractError", "code": "contract_violation", - "trace_id": "304fe92b998b45849ff11a547aadc3e8" + "trace_id": "01cf4a9623c449759f24d249f8783de1" } } ] diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous-twice.json b/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous-twice.json new file mode 100644 index 0000000..1365929 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous-twice.json @@ -0,0 +1,385 @@ +{ + "format_version": 1, + "name": "search-place-ambiguous-twice", + "description": "A flow naming one ambiguous place at both ends answers one no_data part with two entity_ambiguous groups of the same text: each option and slot token names its group by entity_choice, so a menu groups options by entity_choice, never by mentions.", + "origin": { + "kind": "recorded", + "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:14Z" + }, + "call": [ + { + "op": "search", + "question": "aid from Congo to Congo" + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "raw_text", + "query": "aid from Congo to Congo" + } + } + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json", + "x-dfo-trace-id": "14c23c7a7a01430eb3f485e658ae7e51" + }, + "json": { + "schema_version": 1, + "query_echo": { + "entry_path": "raw_text", + "raw_query": "aid from Congo to Congo", + "normalized_query": "aid from Congo to Congo", + "extract_skipped": false + }, + "diagnostics": { + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "14c23c7a7a01430eb3f485e658ae7e51", + "warnings": [], + "timing_ms": { + "total": 1535, + "by_step": { + "coverage": 0, + "extract": 1020, + "cache_lookup": 0 + } + }, + "llm_usage": { + "input_tokens": 2226, + "output_tokens": 106, + "cached_tokens": 840 + }, + "timed_out": false + }, + "kind": "resolution", + "parts": [ + { + "variable_text": "aid", + "warnings": [], + "timing_ms": { + "total": 377, + "by_step": { + "recall": 377, + "phase_resolve_places": 0, + "phase_check_entities": 0 + } + }, + "rendered_sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "sentence_plan": [ + { + "kind": "text", + "text": "\"", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Congo", + "segment": "heading", + "role": "origin", + "entity_choice": 0, + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": "\" (from) and \"", + "segment": "heading", + "entity_refs": [], + "states": [] + }, + { + "kind": "slot", + "text": "Congo", + "axis": "where", + "segment": "heading", + "role": "destination", + "entity_choice": 1, + "entity_refs": [], + "states": [] + }, + { + "kind": "text", + "text": "\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "segment": "heading", + "entity_refs": [], + "states": [] + } + ], + "restatement": { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + }, + "entity_refs": [ + { + "text": "Congo", + "refs": [] + }, + { + "text": "Congo", + "refs": [] + } + ], + "operations": [], + "departures": [], + "assumptions": [], + "status": "no_data", + "no_data": { + "reason": "entity_ambiguous", + "category": "unresolved", + "message": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "entity_terms": [ + "Congo", + "Congo" + ], + "ambiguous_entities": [ + { + "text": "Congo", + "candidates": [ + { + "ref": "country/COD", + "label": "Congo [DRC]", + "kind": "country" + }, + { + "ref": "country/COG", + "label": "Congo [Republic]", + "kind": "country" + } + ] + }, + { + "text": "Congo", + "candidates": [ + { + "ref": "country/COD", + "label": "Congo [DRC]", + "kind": "country" + }, + { + "ref": "country/COG", + "label": "Congo [Republic]", + "kind": "country" + } + ] + } + ], + "source_terms": [], + "ambiguous_sources": [], + "unbound_qualifiers": [], + "options": [ + { + "mentions": [ + "Congo" + ], + "label": "Congo [DRC]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "role_text": "from", + "dcid": "country/COD" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 0 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [Republic]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "role_text": "from", + "dcid": "country/COG" + }, + { + "text": "Congo", + "name": "Republic of the Congo", + "role_text": "to" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 0 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [DRC]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "role_text": "to", + "dcid": "country/COD" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 1 + }, + { + "mentions": [ + "Congo" + ], + "label": "Congo [Republic]", + "data_query": { + "kind": "parsed", + "parts": [ + { + "variable_text": "aid", + "entities": [ + { + "text": "Congo", + "name": "Democratic Republic of the Congo", + "role_text": "from" + }, + { + "text": "Congo", + "role_text": "to", + "dcid": "country/COG" + } + ], + "sources": [], + "operations": [] + } + ] + }, + "entity_choice": 1 + } + ] + } + } + ], + "rendered_sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "sentence_plan": [ + { + "kind": "text", + "text": "\"", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Congo", + "segment": "heading", + "role": "origin", + "entity_choice": 0, + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "\" (from) and \"", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "slot", + "text": "Congo", + "axis": "where", + "segment": "heading", + "role": "destination", + "entity_choice": 1, + "entity_refs": [], + "states": [], + "part_index": 0 + }, + { + "kind": "text", + "text": "\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 0 + } + ], + "status": "no_data" + } + } + } + ], + "expect": [ + { + "answer": { + "kind": "resolution", + "status": "no_data", + "sentence": "\"Congo\" (from) and \"Congo\" (to) each match more than one entity: \"Congo [DRC]\" and \"Congo [Republic]\". Be more specific.", + "complete": false, + "specs": [], + "gaps": [ + { + "part": 0, + "kind": "choice", + "options": 4 + } + ] + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous.json b/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous.json index 7763a30..27b7347 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-place-ambiguous.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:27Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:11Z" }, "call": [ { @@ -31,7 +31,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -42,14 +42,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -66,9 +66,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -88,6 +88,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -184,7 +185,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -213,7 +215,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -235,6 +238,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-proxy-rate-limited.json b/python/client-contract/src/dfo_client_contract/scenarios/search-proxy-rate-limited.json index f930fba..6925a06 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-proxy-rate-limited.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-proxy-rate-limited.json @@ -61,7 +61,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -72,14 +72,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -96,9 +96,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -118,6 +118,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -214,7 +215,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -243,7 +245,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -265,6 +268,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-503.json b/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-503.json index 9603c79..a01b3dc 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-503.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-503.json @@ -60,7 +60,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8" + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1" }, "json": { "schema_version": 1, @@ -71,14 +71,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -95,9 +95,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -117,6 +117,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -213,7 +214,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -242,7 +244,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -264,6 +267,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-past-cap.json b/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-past-cap.json new file mode 100644 index 0000000..c772459 --- /dev/null +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-retry-after-past-cap.json @@ -0,0 +1,57 @@ +{ + "format_version": 1, + "name": "search-retry-after-past-cap", + "description": "A retryable refusal whose Retry-After is past the 60-second cap raises at once, with no wait and no retry: ServerError with its code, service_unavailable, and its trace id.", + "origin": { + "kind": "synthetic", + "derived_from": "search-place-ambiguous", + "edit": "A 503 service_unavailable with retryable true and Retry-After 61 answers the search." + }, + "call": [ + { + "op": "search", + "question": "ODA disbursements from Germany to Congo in 2018" + } + ], + "exchanges": [ + { + "request": { + "method": "POST", + "path": "/search/resolve", + "json": { + "schema_version": 1, + "input": { + "kind": "raw_text", + "query": "ODA disbursements from Germany to Congo in 2018" + } + } + }, + "response": { + "status": 503, + "headers": { + "content-type": "application/problem+json", + "retry-after": "61" + }, + "json": { + "type": "/codes/service_unavailable", + "title": "Service Unavailable", + "status": 503, + "detail": "The service is busy.", + "code": "service_unavailable", + "trace_id": "00000000000000000000000000000000", + "capability_version": "synthetic", + "retryable": true + } + } + } + ], + "expect": [ + { + "error": { + "class": "ServerError", + "code": "service_unavailable", + "trace_id": "00000000000000000000000000000000" + } + } + ] +} diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-server-warning.json b/python/client-contract/src/dfo_client_contract/scenarios/search-server-warning.json index 6424a89..5f5376f 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-server-warning.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-server-warning.json @@ -38,7 +38,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8", + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1", "x-dfo-warning": "Search runs on a reduced index while it is rebuilt." }, "json": { @@ -50,14 +50,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -74,9 +74,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -96,6 +96,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -192,7 +193,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -221,7 +223,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -243,6 +246,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -276,7 +280,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8", + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1", "x-dfo-warning": "Search runs on a reduced index while it is rebuilt.", "sunset": "Sat, 01 May 2027 00:00:00 GMT" }, @@ -289,14 +293,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -313,9 +317,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -335,6 +339,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -431,7 +436,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -460,7 +466,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -482,6 +489,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -515,7 +523,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8", + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1", "x-dfo-warning": "Search answers from a cached release.", "deprecation": "@1777593600", "sunset": "Sat, 01 May 2027 00:00:00 GMT" @@ -529,14 +537,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -553,9 +561,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -575,6 +583,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -671,7 +680,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -700,7 +710,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -722,6 +733,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-sunset-warning.json b/python/client-contract/src/dfo_client_contract/scenarios/search-sunset-warning.json index 8c3b97e..9dc10a7 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-sunset-warning.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-sunset-warning.json @@ -34,7 +34,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8", + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1", "sunset": "Sat, 01 May 2027 00:00:00 GMT" }, "json": { @@ -46,14 +46,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -70,9 +70,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -92,6 +92,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -188,7 +189,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -217,7 +219,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -239,6 +242,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 @@ -272,7 +276,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "304fe92b998b45849ff11a547aadc3e8", + "x-dfo-trace-id": "01cf4a9623c449759f24d249f8783de1", "sunset": "Sat, 01 May 2027 00:00:00 GMT" }, "json": { @@ -284,14 +288,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "304fe92b998b45849ff11a547aadc3e8", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "01cf4a9623c449759f24d249f8783de1", "warnings": [], "timing_ms": { - "total": 1416, + "total": 1162, "by_step": { "coverage": 0, - "extract": 1297, + "extract": 1040, "cache_lookup": 0 } }, @@ -308,9 +312,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 115, + "total": 117, "by_step": { - "recall": 115, + "recall": 117, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -330,6 +334,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -426,7 +431,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -455,7 +461,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -477,6 +484,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [], "part_index": 0 diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-two-parts-place-ambiguous.json b/python/client-contract/src/dfo_client_contract/scenarios/search-two-parts-place-ambiguous.json index 36a4e81..9e48a68 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-two-parts-place-ambiguous.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-two-parts-place-ambiguous.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:28Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:13Z" }, "call": [ { @@ -31,7 +31,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "4b576ea3cd364ddeb664b3b219dffb19" + "x-dfo-trace-id": "5df79c87dd3c43589df5c2921f3c93e3" }, "json": { "schema_version": 1, @@ -42,14 +42,14 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "4b576ea3cd364ddeb664b3b219dffb19", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "5df79c87dd3c43589df5c2921f3c93e3", "warnings": [], "timing_ms": { - "total": 1260, + "total": 1209, "by_step": { "coverage": 0, - "extract": 1255, + "extract": 1202, "cache_lookup": 0 } }, @@ -88,6 +88,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -195,7 +196,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -228,7 +230,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -237,9 +240,9 @@ "variable_text": "ODA disbursements", "warnings": [], "timing_ms": { - "total": 1, + "total": 2, "by_step": { - "recall": 1, + "recall": 2, "phase_resolve_places": 0, "phase_check_entities": 0 } @@ -259,6 +262,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, @@ -366,7 +370,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 }, { "mentions": [ @@ -399,7 +404,8 @@ "operations": [] } ] - } + }, + "entity_choice": 0 } ] } @@ -420,6 +426,7 @@ "axis": "where", "segment": "heading", "role": "destination", + "entity_choice": 0, "entity_refs": [], "states": [] }, diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.json b/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.json index e612f0d..10533af 100644 --- a/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.json +++ b/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.json @@ -5,8 +5,8 @@ "origin": { "kind": "recorded", "base_url": "https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/api", - "dfo_commit": "0ba344270784ae9929a369da8acd00b399dfced8", - "recorded_at": "2026-10-24T16:31:33Z" + "dfo_commit": "d80df9f5fd598b7345255470932d202ee5d49942", + "recorded_at": "2026-10-26T09:23:20Z" }, "call": [ { @@ -38,7 +38,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "50172d3089cd486e929895f6865985bb" + "x-dfo-trace-id": "6cfb1ed8ec3e430a91c10329304d9af4" }, "json": { "schema_version": 1, @@ -49,15 +49,15 @@ "extract_skipped": false }, "diagnostics": { - "engine_build": "0ba344270784ae9929a369da8acd00b399dfced8", - "trace_id": "50172d3089cd486e929895f6865985bb", + "engine_build": "d80df9f5fd598b7345255470932d202ee5d49942", + "trace_id": "6cfb1ed8ec3e430a91c10329304d9af4", "warnings": [], "timing_ms": { - "total": 1979, + "total": 1831, "by_step": { "coverage": 0, - "extract": 1147, - "cache_lookup": 1 + "extract": 1224, + "cache_lookup": 0 } }, "llm_usage": { @@ -82,7 +82,7 @@ { "code": "role_scope_defaulted", "severity": "warn", - "message": "no provider was named; the answer was resolved across every provider with data." + "message": "No provider was named, so the answer covers every provider with data." }, { "code": "whole_total_available", @@ -97,30 +97,30 @@ } ], "timing_ms": { - "total": 829, + "total": 603, "by_step": { - "recall": 300, + "recall": 81, "phase_resolve_places": 0, "phase_check_entities": 0, "shape_arcs": 3, - "shape_probe_coverage": 5, - "shape_probe": 152, - "shape_groups": 4, + "shape_probe_coverage": 11, + "shape_probe": 116, + "shape_groups": 9, "shape_labels": 0, "shape_admission": 1, "phase_derive_shapes": 0, - "shape": 264, + "shape": 227, "phase_prune_and_rank": 0, - "bind": 97, - "phase_bind": 14, - "phase_settle": 9, + "bind": 106, + "phase_bind": 16, + "phase_settle": 8, "phase_place_bindings": 1, "phase_facet_requests": 0, - "materialise": 34, - "phase_materialise": 0, - "answer": 16, - "options": 36, - "phase_outcome": 58 + "materialise": 38, + "phase_materialise": 5, + "answer": 19, + "options": 11, + "phase_outcome": 91 } }, "restatement": { @@ -533,7 +533,7 @@ "label": "OECD Terms and Conditions" } }, - "coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "facets": { "provenance": { "basis": "fallback", @@ -682,7 +682,7 @@ "source": "observed" } ], - "rendered_sentence": "Gross disbursements to Ethiopia from 110 providers for official development assistance in 2022. Current US dollars · OECD DAC2a.", + "rendered_sentence": "Gross disbursements to Ethiopia from 95 providers for official development assistance in 2022. Current US dollars · OECD DAC2a.", "sentence_plan": [ { "kind": "slot", @@ -727,7 +727,7 @@ }, { "kind": "slot", - "text": "110 providers", + "text": "95 providers", "segment": "heading", "role": "origin", "binding_ref": "origin:", @@ -911,7 +911,7 @@ "label": "disbursement" } ], - "provenance": "measure_elected" + "provenance": "binder" }, { "key": { @@ -982,7 +982,7 @@ "start_year": 2022, "end_year": 2022 }, - "capability_version": "cap_9deef72763b45f40", + "capability_version": "cap_765c9d2a0c7529b9", "default_reading": { "applied": true, "removed_stat_vars": [] @@ -1136,7 +1136,7 @@ "role_axis_facts": [ { "role": "origin", - "member_count": 110, + "member_count": 95, "members_available": true } ], @@ -3091,7 +3091,7 @@ } } ], - "rendered_sentence": "Gross disbursements to Ethiopia from 110 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. You asked for a ranking. This shows the amounts behind it, because Search does not compute rankings. Assumed: official development assistance.", + "rendered_sentence": "Gross disbursements to Ethiopia from 95 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. You asked for a ranking. This shows the amounts behind it, because Search does not compute rankings. Assumed: official development assistance.", "sentence_plan": [ { "kind": "slot", @@ -3140,7 +3140,7 @@ }, { "kind": "slot", - "text": "110 providers", + "text": "95 providers", "segment": "heading", "role": "origin", "binding_ref": "origin:", @@ -3393,7 +3393,8 @@ }, "options": { "format": "parquet", - "bundle": false + "bundle": false, + "without_party_totals": true } } }, @@ -3401,7 +3402,7 @@ "status": 200, "headers": { "content-type": "application/json", - "x-dfo-trace-id": "66eb538f1f7943f8a03a5c3500185e5a" + "x-dfo-trace-id": "003f90ea7c3b4c0ea7414ee5a282c0f4" }, "json": { "schema_version": 1, @@ -3461,14 +3462,14 @@ } } }, - "content_version": "cv_534dd4c2bf9bdafc837d51a3ba4bcf3e", + "content_version": "cv_c17d6a131cb4556543b91556e46814e9", "prepared_at": "2026-10-20T16:52:43Z", "coverage": { - "rows": 110, + "rows": 95, "dimensions": [ { "label": "financingProvider", - "count": 110, + "count": 95, "axis": "origin", "noun": { "singular": "provider", @@ -3642,9 +3643,31 @@ "family_member_count": 1, "family_member_digest": "1a36e1a7292b1bfff95a5415de0be74cac7bb033217ed50e79faa8ce51a224c0", "frame_grammar_version": "1.1.0", + "party_totals_left_out": [ + { + "role": "financingProvider", + "entities": [ + "AfricanDevelopmentBankGroup", + "DACCountriesTotal", + "DACEUMembers", + "DACEUMembersAndEUInstitutions", + "DACMultilaterals", + "DACNonDACCountries", + "DACOfficialDonors", + "DACOtherMultilateralOrganisations", + "DACPrivateDonors", + "DACRegionalDevelopmentBanks", + "DACUnitedNations", + "DACWorldBank", + "GroupOfSeven", + "PostcodeLotteryGroup", + "WorldBankGroup" + ] + } + ], "origin_catalogue_build_id": "20261019T174519Z-2e5e8cfd", "served_from": "head", - "confirmed_at": "2026-10-24T16:31:36.102594+00:00" + "confirmed_at": "2026-10-26T09:23:22.526168+00:00" }, "warnings": [], "diagnostics": [ @@ -3656,34 +3679,35 @@ ], "artifacts": [ { - "artifact_digest": "sha256:f0ac1a058c4addb59b1974b73706b12ddc112b409aac006d6a561279f61f0c6a", + "artifact_digest": "sha256:d2ea67ebd67a84915c8e97a7ef4b66733e2d624a1cc5d1c927d6673c8c2f668e", "rendering": { "bundle": false, "format": "parquet", - "layout": 4 + "layout": 4, + "without_party_totals": true }, - "bytes": 10927, + "bytes": 10329, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-23T16:59:51.176407+00:00", + "created_at": "2026-10-24T20:59:40.869991+00:00", "tier": "working", - "url": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_534dd4c2bf9bdafc837d51a3ba4bcf3e/files/sha256:f0ac1a058c4addb59b1974b73706b12ddc112b409aac006d6a561279f61f0c6a", + "url": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_c17d6a131cb4556543b91556e46814e9/files/sha256:d2ea67ebd67a84915c8e97a7ef4b66733e2d624a1cc5d1c927d6673c8c2f668e", "enrichment_stamp": "08ed6c734fc69ea2204a21f4e1dd8b963f14dd1e03a2484c618049ac2e2d93d8" }, { - "artifact_digest": "sha256:5b508be6b46ce8e01315f9c89a1837e17ed7b49d7d2554011cfdba9f32203084", + "artifact_digest": "sha256:d59c03c57a8ea2d1da95b4fb2b1131919e8a13b2003ef51fc1b3801ca46be960", "rendering": { "statistics": "series" }, - "bytes": 10167, + "bytes": 9527, "media_type": "application/vnd.apache.parquet", - "created_at": "2026-10-20T16:52:44.112078+00:00", + "created_at": "2026-10-23T22:01:14.760267+00:00", "tier": "working", - "url": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_534dd4c2bf9bdafc837d51a3ba4bcf3e/files/sha256:5b508be6b46ce8e01315f9c89a1837e17ed7b49d7d2554011cfdba9f32203084" + "url": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_c17d6a131cb4556543b91556e46814e9/files/sha256:d59c03c57a8ea2d1da95b4fb2b1131919e8a13b2003ef51fc1b3801ca46be960" } ], - "series_count": 110, + "series_count": 95, "output_options": { - "capability_version": "dset_cc753cefa9199881", + "capability_version": "dset_76f5fa7244511c58", "columns": [ { "name": "stat_var", @@ -3695,7 +3719,7 @@ "name": "financingProvider", "kind": "role", "droppable": false, - "distinct": 110 + "distinct": 95 }, { "name": "financingRecipient", @@ -3763,7 +3787,7 @@ "state": "partial", "reason": "enrichment_unavailable", "coverage": { - "entities": 110, + "entities": 95, "covered": 40 }, "missing": [ @@ -3771,10 +3795,6 @@ "ref": "AdaptationFund", "label": "Adaptation Fund" }, - { - "ref": "AfricanDevelopmentBankGroup", - "label": "African Development Bank Group" - }, { "ref": "AfricanDevelopmentFund", "label": "African Development Fund" @@ -3804,8 +3824,12 @@ "label": "Conrad N. Hilton Foundation" }, { - "ref": "DACCountriesTotal", - "label": "DAC countries" + "ref": "DavidAndLucilePackardFoundation", + "label": "David and Lucile Packard Foundation" + }, + { + "ref": "DutchPostcodeLottery", + "label": "Dutch Postcode Lottery" } ], "truncated": true @@ -3817,7 +3841,7 @@ "state": "partial", "reason": "enrichment_unavailable", "coverage": { - "entities": 110, + "entities": 95, "covered": 40 }, "missing": [ @@ -3825,10 +3849,6 @@ "ref": "AdaptationFund", "label": "Adaptation Fund" }, - { - "ref": "AfricanDevelopmentBankGroup", - "label": "African Development Bank Group" - }, { "ref": "AfricanDevelopmentFund", "label": "African Development Fund" @@ -3858,8 +3878,12 @@ "label": "Conrad N. Hilton Foundation" }, { - "ref": "DACCountriesTotal", - "label": "DAC countries" + "ref": "DavidAndLucilePackardFoundation", + "label": "David and Lucile Packard Foundation" + }, + { + "ref": "DutchPostcodeLottery", + "label": "Dutch Postcode Lottery" } ], "truncated": true @@ -3870,8 +3894,8 @@ "token": "financingProvider_label", "state": "available", "coverage": { - "entities": 110, - "covered": 110 + "entities": 95, + "covered": 95 }, "missing": [], "truncated": false @@ -3882,8 +3906,8 @@ "token": "financingProvider_type", "state": "available", "coverage": { - "entities": 110, - "covered": 110 + "entities": 95, + "covered": 95 }, "missing": [], "truncated": false @@ -3894,8 +3918,8 @@ "token": "financingProvider_party_kind", "state": "available", "coverage": { - "entities": 110, - "covered": 110 + "entities": 95, + "covered": 95 }, "missing": [], "truncated": false @@ -3906,8 +3930,8 @@ "token": "financingProvider_party_class", "state": "available", "coverage": { - "entities": 110, - "covered": 110 + "entities": 95, + "covered": 95 }, "missing": [], "truncated": false @@ -3918,8 +3942,8 @@ "token": "financingProvider_party_whole", "state": "available", "coverage": { - "entities": 110, - "covered": 110 + "entities": 95, + "covered": 95 }, "missing": [], "truncated": false @@ -4037,10 +4061,10 @@ ] } }, - "series_coverage_stamp": "45bdd65f2fd33ac77a897180047080d33ed9db812afe0cbb887bf22e2ac52690", + "series_coverage_stamp": "e6e717ab8c77e84d010915893ede5809e9ecec67929a07cfafbcc2210edfa1eb", "retention": { "pinned": false, - "held_until": "2026-11-19T16:52:44.112184Z" + "held_until": "2026-11-22T22:01:14.671345Z" } } } @@ -4048,21 +4072,21 @@ { "request": { "method": "GET", - "origin_path": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_534dd4c2bf9bdafc837d51a3ba4bcf3e/files/sha256:f0ac1a058c4addb59b1974b73706b12ddc112b409aac006d6a561279f61f0c6a" + "origin_path": "/api/datasets/spec_7778beed72f8433bb7fde4293acfd175/cv_c17d6a131cb4556543b91556e46814e9/files/sha256:d2ea67ebd67a84915c8e97a7ef4b66733e2d624a1cc5d1c927d6673c8c2f668e" }, "response": { "status": 302, "headers": { "content-type": "text/html", - "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Af0ac1a058c4addb59b1974b73706b12ddc112b409aac006d6a561279f61f0c6a?X-Goog-Signature=redacted", - "x-dfo-trace-id": "cf517ecd99ac4157ab7d54003b5771a5" + "location": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ad2ea67ebd67a84915c8e97a7ef4b66733e2d624a1cc5d1c927d6673c8c2f668e?X-Goog-Signature=redacted", + "x-dfo-trace-id": "014016fd216a48729c10484f342a83a6" } } }, { "request": { "method": "GET", - "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Af0ac1a058c4addb59b1974b73706b12ddc112b409aac006d6a561279f61f0c6a?X-Goog-Signature=redacted" + "url": "https://storage.googleapis.com/dev-fin-observatory-datasets-working/sha256%3Ad2ea67ebd67a84915c8e97a7ef4b66733e2d624a1cc5d1c927d6673c8c2f668e?X-Goog-Signature=redacted" }, "response": { "status": 200, @@ -4078,7 +4102,7 @@ "answer": { "kind": "resolution", "status": "definite", - "sentence": "Gross disbursements to Ethiopia from 110 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. You asked for a ranking. This shows the amounts behind it, because Search does not compute rankings. Assumed: official development assistance.", + "sentence": "Gross disbursements to Ethiopia from 95 providers for official development assistance in 2022. Current US dollars · OECD DAC2a. You asked for a ranking. This shows the amounts behind it, because Search does not compute rankings. Assumed: official development assistance.", "complete": true, "specs": [ "spec_7778beed72f8433bb7fde4293acfd175" @@ -4090,33 +4114,33 @@ "error": { "class": "SearchWarning", "code": null, - "trace_id": "50172d3089cd486e929895f6865985bb" + "trace_id": "6cfb1ed8ec3e430a91c10329304d9af4" } }, { "dataset": { - "rows": 110, + "rows": 95, "columns": [ - "stat_var", - "financingProvider", + "time_period", "financingProvider_label", + "financingRecipient_label", + "value", + "unit_label", + "dac2aAidType_label", + "provenance_label", + "financingProvider", "financingProvider_kind", "financingRecipient", - "financingRecipient_label", "financingRecipient_kind", "dac2aAidType", - "dac2aAidType_label", - "time_period", - "value", "unit", - "unit_label", "currency", "price_basis", "base_year", "measurementMethod", "measurementMethod_label", "provenance", - "provenance_label" + "stat_var" ], "complete": true, "members": [ @@ -4125,8 +4149,8 @@ "aid" ], "spec_id": "spec_7778beed72f8433bb7fde4293acfd175", - "content_version": "cv_534dd4c2bf9bdafc837d51a3ba4bcf3e", - "sentence": "Gross disbursements to Ethiopia from 110 providers for official development assistance in 2022. Current US dollars · OECD DAC2a." + "content_version": "cv_c17d6a131cb4556543b91556e46814e9", + "sentence": "Gross disbursements to Ethiopia from 95 providers for official development assistance in 2022. Current US dollars · OECD DAC2a." } ], "missing": [], diff --git a/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.parquet b/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.parquet index 7f04cba..91ee4cc 100644 Binary files a/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.parquet and b/python/client-contract/src/dfo_client_contract/scenarios/search-warning-top5.parquet differ diff --git a/python/client-contract/src/dfo_client_contract/served/SOURCE.json b/python/client-contract/src/dfo_client_contract/served/SOURCE.json index 4e6aeef..8d5b00b 100644 --- a/python/client-contract/src/dfo_client_contract/served/SOURCE.json +++ b/python/client-contract/src/dfo_client_contract/served/SOURCE.json @@ -1,6 +1,6 @@ { "format_version": 1, "repository": "DevelopmentFinanceObservatory/apis", - "commit": "0ba344270784ae9929a369da8acd00b399dfced8", + "commit": "d80df9f5fd598b7345255470932d202ee5d49942", "captured_from": "dfo_apis.app.create_app() in process" } diff --git a/python/client-contract/src/dfo_client_contract/served/openapi.json b/python/client-contract/src/dfo_client_contract/served/openapi.json index e3ca7a2..c5ab609 100644 --- a/python/client-contract/src/dfo_client_contract/served/openapi.json +++ b/python/client-contract/src/dfo_client_contract/served/openapi.json @@ -3567,25 +3567,26 @@ "kind": { "type": "string", "enum": [ + "name_corrected", "declared_default", "open_role", "place_scope" ], "title": "Kind", - "description": "what the engine filled. 'declared_default' is a slot the request left open that its code system's or shape's declared default fixed. 'open_role' is a flow role no request word asked for, read across every entity on it with data, as one series per member where the role scope default filled it and the shape declares the role, else as the total over them. 'place_scope' is a request that named no place, read across every place with data" + "description": "what the engine filled. 'name_corrected' is an entity the request typed in words that are no listed name or alias of it, such as a misspelling, read as that entity. 'declared_default' is a slot the request left open that its code system's or shape's declared default fixed. 'open_role' is a flow role no request word asked for, read across every entity on it with data, as one series per member where the role scope default filled it and the shape declares the role, else as the total over them. 'place_scope' is a request that named no place, read across every place with data" }, "binding_ref": { "type": "string", "maxLength": 1036, "title": "Binding Ref", - "description": "the identity binding whose edit replaces this assumption: 'how:' for a declared default, 'origin:' or 'where:' for a role or the place scope, as on the sentence's own tokens" + "description": "the identity binding whose edit replaces this assumption: 'how:' for a declared default, 'origin:' or 'where:' for a role, the place scope or the role a corrected name binds, as on the sentence's own tokens" }, "text": { "type": "string", "maxLength": 200, "minLength": 1, "title": "Text", - "description": "the words the sentence's assumptions clause states for this fill: the bound code's full label for a declared default ('official development assistance' for CRSTypeOfFlow10, whose short name is 'ODA'), 'every' and the role's noun in the answer's flow family for an open role ('every provider', 'every creditor'), and 'every country', 'every place' or 'the world as a whole' for the place scope, as its members are typed" + "description": "the words the sentence's assumptions clause states for this fill: the typed words in quotes and the entity's label for a corrected name ('\"Germny\" read as Germany'), the bound code's full label for a declared default ('official development assistance' for CRSTypeOfFlow10, whose short name is 'ODA'), 'every' and the role's noun in the answer's flow family for an open role ('every provider', 'every creditor'), and 'every country', 'every place' or 'the world as a whole' for the place scope, as its members are typed" }, "refs": { "items": { @@ -3595,7 +3596,21 @@ "type": "array", "maxItems": 1, "title": "Refs", - "description": "the value dcid a declared default bound its slot to, as the slot's trace.slot_filters[].refs list it (['CRSTypeOfFlow10'] for official development assistance). An open role and the place scope carry []. A dataset_resubmit echo matches a declared default to its slot by these values" + "description": "the value dcid a declared default bound its slot to, as the slot's trace.slot_filters[].refs list it (['CRSTypeOfFlow10'] for official development assistance), or the entity dcid a corrected name reads (['country/DEU']). An open role and the place scope carry []. A dataset_resubmit echo matches a declared default to its slot, and a corrected name to its role's entity, by these values" + }, + "asked": { + "anyOf": [ + { + "type": "string", + "maxLength": 200, + "minLength": 1 + }, + { + "type": "null" + } + ], + "title": "Asked", + "description": "the words the request typed for a corrected name ('Germny'), which `text` quotes; absent on every other kind. A dataset_resubmit echo restates a corrected name with these words, and drops one that carries none" } }, "additionalProperties": false, @@ -4301,7 +4316,7 @@ }, "type": "array", "title": "Entity Refs", - "description": "the binding of each entity text of `restatement`, once per text in restatement order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" + "description": "the binding of each entity of `restatement`, one per restatement.entities row, in row order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" }, "operations": { "items": { @@ -6115,6 +6130,56 @@ "title": "ConvertedUnitModel", "description": "A converted unit, as the frame's `unit` cell states it." }, + "CountedEntityYearsModel": { + "properties": { + "count": { + "type": "integer", + "title": "Count", + "description": "how many entities there are" + }, + "entities": { + "items": { + "$ref": "#/components/schemas/EntityYearsModel" + }, + "type": "array", + "title": "Entities", + "description": "the first ten of them, sorted" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "count", + "entities" + ], + "title": "CountedEntityYearsModel", + "description": "The entities a conversion left rows out of, each with its years: how many, and the\nfirst ten, sorted (`materialise.codes.CountedEntityYears`). The message states the count\nand names none, so a reader labels the refs." + }, + "CountedRefsModel": { + "properties": { + "count": { + "type": "integer", + "title": "Count", + "description": "how many there are" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Refs", + "description": "the `dcid:` refs of the first five of them, ten for omitted_entities: a group's members in request order, entities sorted" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "count", + "refs" + ], + "title": "CountedRefsModel", + "description": "The things of one kind a warning counts: how many, and the `dcid:` refs of the first\nfive, or ten for a conversion's left-out entities (`materialise.codes.CountedRefs`). The message states the count and names none, so a\nreader labels the refs." + }, "CountedStatVarModel": { "properties": { "stat_var": { @@ -7281,7 +7346,7 @@ }, "type": "array", "title": "Entity Refs", - "description": "the binding of each entity text of `restatement`, once per text in restatement order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" + "description": "the binding of each entity of `restatement`, one per restatement.entities row, in row order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" }, "operations": { "items": { @@ -8244,7 +8309,7 @@ "text": { "type": "string", "title": "Text", - "description": "the entity text, equal to some restatement.entities[i].text" + "description": "the entity text, equal to restatement.entities[i].text, where i is this entry's position in Part.entity_refs" }, "refs": { "items": { @@ -8262,7 +8327,32 @@ "refs" ], "title": "EntityRefs", - "description": "The graph nodes one entity text of a part's restatement binds in the answer." + "description": "The graph nodes one entity row of a part's restatement binds in the answer." + }, + "EntityYearsModel": { + "properties": { + "ref": { + "type": "string", + "title": "Ref", + "description": "the entity's `dcid:` ref" + }, + "years": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Years", + "description": "the years left out, as runs of consecutive years, `1990-2001` or `2004`" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "ref", + "years" + ], + "title": "EntityYearsModel", + "description": "One entity a conversion left rows out of, and the years of those rows\n(`materialise.codes.EntityYears`)." }, "Exclusion": { "properties": { @@ -10799,6 +10889,19 @@ "data_query": { "$ref": "#/components/schemas/ParsedInput", "description": "a one-part parsed input settling this choice only. Post it unchanged as the input of /api/search/resolve; its one part is a dataset_resubmit parsed entry's part. A second ambiguous wording refuses again. Whether data exists for it is not checked" + }, + "entity_choice": { + "anyOf": [ + { + "type": "integer", + "minimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Entity Choice", + "description": "for an entity_ambiguous option, the index into NoData.ambiguous_entities of the group this option settles; null on every other option. Two groups can share one wording ('aid from Congo to Congo' asks 'Congo' for each end), so a client groups an entity menu by this index, never by mentions, and joins it to the sentence slot token with the same entity_choice" } }, "additionalProperties": false, @@ -10894,7 +10997,7 @@ }, "type": "array", "title": "Entity Refs", - "description": "the binding of each entity text of `restatement`, once per text in restatement order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" + "description": "the binding of each entity of `restatement`, one per restatement.entities row, in row order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" }, "operations": { "items": { @@ -12101,7 +12204,21 @@ "maxLength": 128, "minLength": 1, "title": "Text", - "description": "the entity's text as the caller identified it" + "description": "the entity's text as the caller identified it: the request's own words for it, which name resolution reads and a rendered sentence quotes" + }, + "name": { + "anyOf": [ + { + "type": "string", + "maxLength": 128, + "minLength": 1 + }, + { + "type": "null" + } + ], + "title": "Name", + "description": "the entity's name, when its words (`text`) could name more than one entity and no `dcid` pins it: the row's identity, which keys its role and members within the part. Name resolution reads `text` with `name` beside it, as the raw-text door reads a query's words beside the name extraction gave them, so two rows whose words are alike stay two entities. Words that differ from `name` only in case read as `name`, and one `name` posted with two texts that differ beyond case in a part refuses `request_invalid`. Without it, `text` is the identity; a `name` equal to `text` is the field left out, and a `name` beside a `dcid` is invalid. A row no dcid pins whose identity, its `name` or else its `text`, is a dcid another row of the part pins refuses `request_invalid` at the row" }, "role_text": { "anyOf": [ @@ -12128,7 +12245,7 @@ } ], "title": "Dcid", - "description": "an optional caller-pinned dcid for this entity, naming exactly which graph node `text` refers to -- the door a client with a candidate's dcid in hand (e.g. from NoData.ambiguous_entities) posts through, instead of relying on name resolution to disambiguate `text` on its own. When present, this entity's name resolution is skipped: the pinned dcid seeds the entry's pre_resolved map directly, still confirmed against the graph. An unconfirmable dcid (absent from the graph, or present but type-less) refuses invalid_role_ref (422). `text` stays required and is not re-derived from the dcid; it is still what a rendered sentence and any further restatement echo back." + "description": "an optional caller-pinned dcid for this entity, naming exactly which graph node `text` refers to -- the door a client with a candidate's dcid in hand (e.g. from NoData.ambiguous_entities) posts through, instead of relying on name resolution to disambiguate `text` on its own. When present, this entity's name resolution is skipped: the pinned dcid seeds the entry's pre_resolved map directly, still confirmed against the graph. An unconfirmable dcid (absent from the graph, or present but type-less) refuses invalid_role_ref (422). `text` stays required and is not re-derived from the dcid; it is still what a rendered sentence and any further restatement echo back. The dcid is the row's identity, so rows pinning one dcid under different texts are one entity, whose words are the first row's. A row no dcid pins whose `name`, or else `text`, is this dcid refuses request_invalid at that row." }, "members": { "anyOf": [ @@ -12140,7 +12257,7 @@ } ], "title": "Members", - "description": "true asks for the members of the set `text` names, each as its own series: the countries of a continent or UN subregion, or the members of a group. It has no effect on a single place. Text whose wording names members, such as 'EU member states', reads as true without the flag. With a pinned `dcid` it is honoured for a continent or subregion that the membership policy holds or omits, whatever the node's graph type, and refuses request_invalid for a group's own node, whose members a pin cannot supply. One part names a set once: the same `text` posted with both values refuses request_invalid" + "description": "true asks for the members of the set `text` names, each as its own series: the countries of a continent or UN subregion, or the members of a group. It has no effect on a single place. Text whose wording names members, such as 'EU member states', reads as true without the flag. With a pinned `dcid` it is honoured for a continent or subregion that the membership policy holds or omits, whatever the node's graph type, and refuses request_invalid for a group's own node, whose members a pin cannot supply. One part names a set once: the same entity posted with both values refuses request_invalid" } }, "additionalProperties": false, @@ -14155,7 +14272,7 @@ "near_matches" ], "title": "GraphResolveResult", - "description": "One text's outcome. `input` echoes the text; fields a status does not use are `null`.\n\n- `resolved`: `entity`, `matched` and `reading`. A group or region adds `projection`,\n `member_count` and `members`: the list route's link, or the members inline (at most\n 256) when the request expanded them. The three are `null` for a region the membership\n policy does not hold.\n- `ambiguous`: `candidates`, in the resolver's order.\n- `not_found`: `near_matches`, hints for a person; a program should pick none by id.\n- `not_held`: `entity`, the id the resolver named, which does not open on the node route.\n\n`reading` is `members` when the text names a group's members (\"EU countries\"), else\n`entity`." + "description": "One text's outcome. `input` echoes the text; fields a status does not use are `null`.\n\n- `resolved`: `entity`, `matched` and `reading`. A group or region adds `projection`,\n `member_count` and `members`: the list route's link, or the members inline (at most\n 256) when the request expanded them. The three are `null` for a region the membership\n policy does not hold. `near_matches` lists up to 3 other places or organisations of the\n requested kinds that share the name (\"Georgia\" resolves to the country, with the US\n state `geoId/13` here), else `null`. When `kinds` removed the first match, a node of a\n requested kind sharing the name resolves in its place and `near_matches` lists the\n removed match (\"Georgia\" with `kinds: [\"place_other\"]` resolves to `geoId/13`, with\n the country here).\n- `ambiguous`: `candidates`, in the resolver's order. When `kinds` removed the first\n match and up to 3 nodes of a requested kind share the name, `near_matches` lists the\n removed match, else `null`.\n- `not_found`: `near_matches`, hints for a person; a program should pick none by id.\n- `not_held`: `entity`, the id the resolver named, which does not open on the node route.\n\n`reading` is `members` when the text names a group's members (\"EU countries\"), else\n`entity`." }, "ResolvedEntity": { "properties": { @@ -14421,7 +14538,7 @@ }, "type": "array", "title": "Named", - "description": "every code row the span names in the packaged index, any code system; empty when none did" + "description": "every code row the span names in the packaged index, any code system, and every measure value it names that the answer does not carry; empty when none did" }, "slot": { "anyOf": [ @@ -14443,7 +14560,7 @@ "no_vocabulary" ], "title": "Disposition", - "description": "\"bound\" when a final binding covers a named row (or a descendant of one); \"unbound\" when a row was named but no binding covers it; \"no_vocabulary\" when the term names no code at all" + "description": "\"bound\" when a final binding covers a named row (or a descendant of one); \"unbound\" when a code row was named but no binding covers it, or a measure value was named that the answer does not carry; \"no_vocabulary\" when the term names neither" }, "start": { "anyOf": [ @@ -14487,7 +14604,7 @@ } ], "title": "Evidence", - "description": "what read the term: a code label or synonym, the bind model, \"uncovered\" for words no reader covers on the elected shape, unbound beside a recognised code and no_vocabulary beside none when the binder rated them as not stated by its series, \"unavailable\" for a code no table holds, \"unsupported\" for a distinction no binding expresses" + "description": "what read the term: a code label or synonym, the bind model, \"uncovered\" for words no reader covers on the elected shape, unbound beside a recognised code or naming a measure value the answer does not carry, and no_vocabulary beside none when the binder rated them as not stated by its series, \"unavailable\" for a code no table holds, \"unsupported\" for a distinction no binding expresses" }, "operation": { "anyOf": [ @@ -14522,7 +14639,7 @@ "disposition" ], "title": "ResolvedQualifier", - "description": "One narrowing term the request's text named, and what became of it.\n\nThe term's text can come from the variable phrase, from the verbatim\n`query_text`, or from an admitted qualifier tie (see\n`_resolve_query_qualifiers`). `text` is the verbatim span of whichever\nsource named it. `named` is every code row -- any system -- that span\nnames in the packaged index;\nempty when the text names no code at all (a `no_vocabulary` term). `slot`\nis the constraint property dcid of the `slot_filters` entry whose\nbinding consumed the term, or, on an `\"unbound\"` value the answer's shape\nhas a slot for but leaves unmet, that slot's property; `None` otherwise.\n`disposition` is\n`\"bound\"` when some final binding on a named row's code system covers\nit (that row or a descendant), `\"unbound\"` when a row was named but no\nbinding covers it, and `\"no_vocabulary\"` when the term names no code at\nall: an LLM-reported phrase (`bind`'s own `unplaced` output) that no\ntaxonomy in this deployment's index can express, an exclusion or comparison\nphrase (`\"unsupported\"`), or words beside no code the binder rated as not\nstated on the elected shape (`\"uncovered\"`).\n\n`start` and `end` locate the term in the part's variable phrase. `evidence` says what\nread it, and `None` means a code's label or synonym matched the term as the query\nspelled it, the reading every recognised span records:\n\n- `\"label\"` and `\"synonym\"` name a match on a code's label or synonym; no record sets\n them, since a recognised span records `None`.\n- `\"model\"` marks a phrase the bind model read through a code it chose from the\n phrase's candidate menu, a lexical hit of the menu or a descendant of one\n (`phase_rank_bind._model_records`); `named` holds the chosen codes.\n- `\"uncovered\"` marks words no reader covers on the elected shape\n (`phase_rank_bind._coverage_records`). Beside a recognised code they are recorded\n `\"unbound\"` with that code's rows as `named`. Beside no code they are recorded\n `\"no_vocabulary\"` with `named` empty when the binder rates them below\n `SearchSettings.stated_word_bar` as not stated by the shape's series; a word it rates\n at the bar or above, or leaves unrated, reads as stated.\n- `\"unavailable\"` marks a code term the elected shape cannot apply: one naming only\n codes no shape can bind, or, on a part naming a dataset, one no row of that dataset\n holds (`phase_rank_bind._unavailable_qualifiers`), recorded `\"unbound\"`.\n- `\"unsupported\"` marks an exclusion or comparison phrase no binding expresses\n (`phase_rank_bind._unsupported_records`), recorded `\"no_vocabulary\"`.\n\n`operation` and `selected` describe how the term's codes combine with\nanother term's on one code system, and the codes it asked for before that\ncombination simplified." + "description": "One narrowing term the request's text named, and what became of it.\n\nThe term's text can come from the variable phrase, from the verbatim\n`query_text`, or from an admitted qualifier tie (see\n`_resolve_query_qualifiers`). `text` is the verbatim span of whichever\nsource named it. `named` is every code row -- any system -- that span\nnames in the packaged index, and every measure value it names that the\nanswer does not carry (a dcid of the index's measure values, such as\n`amountMobilised`);\nempty when the text names neither (a `no_vocabulary` term). `slot`\nis the constraint property dcid of the `slot_filters` entry whose\nbinding consumed the term, or, on an `\"unbound\"` value the answer's shape\nhas a slot for but leaves unmet, that slot's property; `None` otherwise.\n`disposition` is\n`\"bound\"` when some final binding on a named row's code system covers\nit (that row or a descendant), `\"unbound\"` when a code row was named but\nno binding covers it or a measure value was named that the answer does not\ncarry, and `\"no_vocabulary\"` when the term names neither: an LLM-reported\nphrase (`bind`'s own `unplaced` output) that no taxonomy in this\ndeployment's index can express, an exclusion or comparison phrase\n(`\"unsupported\"`), or words beside no code the binder rated as not stated\non the elected shape (`\"uncovered\"`).\n\n`start` and `end` locate the term in the part's variable phrase. `evidence` says what\nread it, and `None` means a code's label or synonym matched the term as the query\nspelled it, the reading every recognised span records:\n\n- `\"label\"` and `\"synonym\"` name a match on a code's label or synonym; no record sets\n them, since a recognised span records `None`.\n- `\"model\"` marks a phrase the bind model read through a code it chose from the\n phrase's candidate menu, a lexical hit of the menu or a descendant of one\n (`phase_rank_bind._model_records`); `named` holds the chosen codes.\n- `\"uncovered\"` marks words no reader covers on the elected shape\n (`phase_rank_bind._coverage_records`). Beside a recognised code they are recorded\n `\"unbound\"` with that code's rows as `named`. On a shape the measure index describes,\n a word that every name of a measure value's reading holds names that value when no\n span names it and the answer does not carry it, and its record adds the value to\n `named`. Beside no code such a word is recorded `\"unbound\"` naming the value, whatever\n the binder rates it; \"mobilised\" names `amountMobilised` on an amount series. Other\n words beside no code are recorded `\"no_vocabulary\"` with `named` empty when the\n binder rates them below `SearchSettings.stated_word_bar` as not stated by the shape's\n series; a word it rates at the bar or above, or leaves unrated, reads as stated.\n- `\"unavailable\"` marks a code term the elected shape cannot apply: one naming only\n codes no shape can bind, or, on a part naming a dataset, one no row of that dataset\n holds (`phase_rank_bind._unavailable_qualifiers`), recorded `\"unbound\"`.\n- `\"unsupported\"` marks an exclusion or comparison phrase no binding expresses\n (`phase_rank_bind._unsupported_records`), recorded `\"no_vocabulary\"`.\n\n`operation` and `selected` describe how the term's codes combine with\nanother term's on one code system, and the codes it asked for before that\ncombination simplified." }, "ResolvedScope": { "properties": { @@ -15383,7 +15500,7 @@ } ], "title": "Member Count", - "description": "how many entities this axis' open scope resolves to. None when the count could not be established without a read this response chose not to make (e.g. a discovered opposite axis whose own role index was omitted after a cap trip); the fact still publishes in that case, naming the axis as open with an explicit unknown rather than staying silent about it" + "description": "how many entities this axis' open scope resolves to, leaving out the group totals the code index declares beside their members (party_kind total), as the default download leaves them out; a scope of totals alone counts them all. None when the count could not be established without a read this response chose not to make (e.g. a discovered opposite axis whose own role index was omitted after a cap trip); the fact still publishes in that case, naming the axis as open with an explicit unknown rather than staying silent about it" }, "members_available": { "type": "boolean", @@ -15725,7 +15842,20 @@ "type": "null" } ], - "description": "origin/destination when this token renders a directional entity role (the flow's two endpoints read back), or the directional scope an unbound where slot resolved to. entity_refs is the discriminator: non-empty on the entity read-back, empty on the unbound-where-slot fallback. binding_ref is not: an entity read-back carries binding_ref too whenever the shape declares a where slot backing the 'destination' axis (e.g. dev-finance), the same condition under which the unbound-where-slot fallback always carries it, so a present binding_ref value never separates the two. Both kinds join to the Spec.resolved.entity_options entry on the same role. A third token sits on an entity_ambiguous no_data plan, which has no Spec: the ambiguous wording, with the role the request gave it (null when it gave none), no entity_refs and no binding_ref, and text equal to one NoData.ambiguous_entities[i].text on the same part. The client titles that wording's candidate menu by this role and posts the pick's entry of NoData.options." + "description": "origin/destination when this token renders a directional entity role (the flow's two endpoints read back), or the directional scope an unbound where slot resolved to. entity_refs is the discriminator: non-empty on the entity read-back, empty on the unbound-where-slot fallback. binding_ref is not: an entity read-back carries binding_ref too whenever the shape declares a where slot backing the 'destination' axis (e.g. dev-finance), the same condition under which the unbound-where-slot fallback always carries it, so a present binding_ref value never separates the two. Both kinds join to the Spec.resolved.entity_options entry on the same role. A third token sits on an entity_ambiguous no_data plan, which has no Spec: the ambiguous wording, with the role the request gave it (null when it gave none), no entity_refs and no binding_ref, and text equal to one NoData.ambiguous_entities[i].text on the same part, the i its entity_choice names. The client titles that wording's candidate menu by this role and posts the pick's entry of NoData.options." + }, + "entity_choice": { + "anyOf": [ + { + "type": "integer", + "minimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Entity Choice", + "description": "on the slot token an entity_ambiguous no_data plan marks for an ambiguous wording, the index into NoData.ambiguous_entities of the group it asks, on the part its part_index names in an envelope plan. The NoData.options with the same entity_choice form that slot's menu. Null on every other token" }, "binding_ref": { "anyOf": [ @@ -18902,7 +19032,7 @@ }, "type": "array", "title": "Entity Refs", - "description": "the binding of each entity text of `restatement`, once per text in restatement order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" + "description": "the binding of each entity of `restatement`, one per restatement.entities row, in row order, so a client can tell which node a text such as 'G7' bound. A response field kept off `restatement`, which posts back unchanged. The binding is the one the answer reads: for a candidates part, the elected reading's; for a part folded into an earlier part's spec, its own. [] when `restatement` is null or names no entity" }, "operations": { "items": { @@ -19282,7 +19412,7 @@ } ], "title": "Params", - "description": "the counts and labels the message states, for a client to compose its own text. observations_series_truncated: {shown, total, order}, the series shown, the confirmed series behind them, and how the shown series were chosen: 'ranked' (the most recent and largest) or 'fixed' (a selection made before values were read, not ranked by value). elected_shape_no_data: {no_data, elected}, the labels of the readings with no data and of the readings answered instead. un_system_unavailable: {places, variables, variable_labels}, the place dcids the UN front door refused, the variables of the UN reads that failed, each empty when none, and each variable's graph name, in variables order, '' where the graph holds none; a warning without params says the UN search itself was unavailable. whole_total_available: {total, total_label, whole_of}, the dcid and label of the total the source publishes for the side the answer lists member by member, and the classes of party it covers, comma-joined (official, private, residual). Omitted for every other code" + "description": "the counts and labels the message states, for a client to compose its own text. observations_series_truncated: {shown, total, order}, the series shown, the confirmed series behind them, and how the shown series were chosen: 'ranked' (the most recent and largest) or 'fixed' (a selection made before values were read, not ranked by value). elected_shape_no_data: {no_data, elected}, the labels of the readings with no data and of the readings answered instead. un_system_unavailable: {places, variables, variable_labels}, the place dcids the UN front door refused, the variables of the UN reads that failed, each empty when none, and each variable's graph name, in variables order, '' where the graph holds none; a warning without params says the UN search itself was unavailable. whole_total_available: {total, total_label, whole_of}, the dcid and label of the total the source publishes for the side the answer lists member by member, and the classes of party it covers, comma-joined (official, private, residual). currency_unmapped_omitted: {count, entities, entity_labels}, how many entities the conversion left out, the dcids of the first ten, sorted, and each one's name, in entities order, '' where the conversion tables hold none, which is usual here, since an entity no currency names, such as a regional aggregate, is rarely in those tables; its dcid stays in entities. deflator_missing_omitted: the same with entity_years, the years each entity's rows were left out, as runs of consecutive years ('1990-2001, 2004'), in entities order; a warning without params says the loaded tables cannot tell which entities. Omitted for every other code" } }, "additionalProperties": false, @@ -19293,7 +19423,7 @@ "message" ], "title": "Warning", - "description": "A run-health signal. Note: this name shadows the Python builtin Warning.\n\nOnly `observations_series_truncated`, `elected_shape_no_data`, `un_system_unavailable`\nand `whole_total_available` set `params`. A `None` `params` is left out of every dump, so other codes' warnings stay\nexactly `{code, severity, message}` plus `spec_id` where one applies." + "description": "A run-health signal. Note: this name shadows the Python builtin Warning.\n\nOnly `observations_series_truncated`, `elected_shape_no_data`, `un_system_unavailable`,\n`whole_total_available`, `currency_unmapped_omitted` and `deflator_missing_omitted` set\n`params`. A `None` `params` is left out of every dump, so other codes' warnings stay\nexactly `{code, severity, message}` plus `spec_id` where one applies." }, "WarningModel": { "properties": { @@ -19344,6 +19474,95 @@ ], "title": "Stat Var Label", "description": "stat_vars_summed only: the family's label, the string the presentation door gives `stat_var_ref` on a chart (`code_index.labels.family_label` over the spec identity: its measure partition label, then its bound slots' labels); absent when neither has a label" + }, + "member_count": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Member Count", + "description": "coverage_below_scope only, when it names a group member: how many series the group requested" + }, + "off_cadence_members": { + "anyOf": [ + { + "$ref": "#/components/schemas/CountedRefsModel" + }, + { + "type": "null" + } + ], + "description": "coverage_below_scope only: the group members holding observations in the requested years at another frequency than the group serves, which the figures leave out; absent when there are none" + }, + "out_of_window_members": { + "anyOf": [ + { + "$ref": "#/components/schemas/CountedRefsModel" + }, + { + "type": "null" + } + ], + "description": "coverage_below_scope only: the group members holding no observation in the requested years; absent when there are none, and when the served provenance reads an absent member as zero" + }, + "unserved_entities": { + "anyOf": [ + { + "$ref": "#/components/schemas/CountedRefsModel" + }, + { + "type": "null" + } + ], + "description": "coverage_below_scope only: the entities the request binds that served no observation, sorted; absent when there are none" + }, + "requested_years": { + "anyOf": [ + { + "$ref": "#/components/schemas/YearRangeModel" + }, + { + "type": "null" + } + ], + "description": "coverage_below_scope only: the year range the request asked for, when the data covers fewer of its years; absent otherwise" + }, + "served_years": { + "anyOf": [ + { + "$ref": "#/components/schemas/YearRangeModel" + }, + { + "type": "null" + } + ], + "description": "coverage_below_scope only: the years the data covers, both bounds set, beside requested_years; absent otherwise" + }, + "omitted_entities": { + "anyOf": [ + { + "$ref": "#/components/schemas/CountedRefsModel" + }, + { + "type": "null" + } + ], + "description": "currency_unmapped_omitted only: the entities whose rows the conversion left out, sorted" + }, + "omitted_entity_years": { + "anyOf": [ + { + "$ref": "#/components/schemas/CountedEntityYearsModel" + }, + { + "type": "null" + } + ], + "description": "deflator_missing_omitted only: the entities whose rows the conversion left out for a missing deflator, sorted, each with the years of those rows" } }, "additionalProperties": false, @@ -19354,7 +19573,7 @@ "message" ], "title": "WarningModel", - "description": "One manifest warning. Only `stat_vars_summed` sets the three structured fields. A `None`\nfield is left out of every dump, so other codes' warnings, on the wire and in stored version\nbodies, stay exactly `{code, severity, message}`." + "description": "One manifest warning. Only `stat_vars_summed` sets its three structured fields, a\ngroup's `coverage_below_scope` its three member fields, `coverage_below_scope`\n`unserved_entities`, `requested_years` and `served_years`, `currency_unmapped_omitted`\n`omitted_entities`, and\n`deflator_missing_omitted` `omitted_entity_years`. A `None` field is left out of every dump, so other codes' warnings, on\nthe wire and in stored version bodies, stay exactly `{code, severity, message}`." }, "WindowAll": { "properties": { @@ -19561,6 +19780,38 @@ ], "title": "YearRange" }, + "YearRangeModel": { + "properties": { + "start": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Start", + "description": "the first year; absent when the request left it open" + }, + "end": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "End", + "description": "the last year; absent when the request left it open" + } + }, + "additionalProperties": false, + "type": "object", + "title": "YearRangeModel", + "description": "A run of years, both bounds included (`materialise.codes.YearRange`)." + }, "datasets_api__capabilities__Capabilities": { "properties": { "capability_version": { diff --git a/python/client-contract/src/dfo_client_contract/warning-codes.json b/python/client-contract/src/dfo_client_contract/warning-codes.json index 5d55edf..5b9ee2d 100644 --- a/python/client-contract/src/dfo_client_contract/warning-codes.json +++ b/python/client-contract/src/dfo_client_contract/warning-codes.json @@ -26,6 +26,11 @@ "class": "DfoDeprecationWarning", "condition": "An API response carries a Deprecation or Sunset header, once per process per distinct pair of their values." }, + { + "code": "series_left_out", + "class": "SeriesLeftOutWarning", + "condition": "A derived frame left series out for missing data; the client returns the frame." + }, { "code": "server_warning", "class": "DfoWarning", diff --git a/python/client-contract/src/dfo_client_contract/warning-codes.schema.json b/python/client-contract/src/dfo_client_contract/warning-codes.schema.json index 4427722..d639eea 100644 --- a/python/client-contract/src/dfo_client_contract/warning-codes.schema.json +++ b/python/client-contract/src/dfo_client_contract/warning-codes.schema.json @@ -32,7 +32,8 @@ "class": { "enum": [ "DfoWarning", - "DfoDeprecationWarning" + "DfoDeprecationWarning", + "SeriesLeftOutWarning" ] }, "condition": { diff --git a/python/client-contract/tests/test_analysis_rules.py b/python/client-contract/tests/test_analysis_rules.py new file mode 100644 index 0000000..2faae0c --- /dev/null +++ b/python/client-contract/tests/test_analysis_rules.py @@ -0,0 +1,361 @@ +"""Replay the analysis and publication rule vectors through a reference +implementation of each rule. + +The rules are stated in `dfo_client_contract/rules/README.md`. DFO applies +them on the server; a client reads their results and applies only the +output comparison. Each function here is the shortest faithful reading of one +rule, so a vector that disagrees with its rule fails here before the server +or a client replays it. +""" + +from __future__ import annotations + +import hashlib +import math +from typing import Any + +import pytest +import rfc8785 + +from dfo_client_contract import rule + +# The frame columns that name no role (docs/frame-grammar-v1.md). +_NOT_ROLES = frozenset( + { + "stat_var", + "time_period", + "value", + "unit", + "measurementMethod", + "provenance", + "status", + } +) +_PERCENT = "Percent" + + +class RuleError(Exception): + def __init__(self, code: str) -> None: + super().__init__(code) + self.code = code + + +def _vectors(name: str) -> list[dict[str, Any]]: + return rule(name)["vectors"] + + +def _ids(name: str) -> list[str]: + return [vector["name"] for vector in _vectors(name)] + + +def jcs(value: Any) -> str: + """The RFC 8785 serialisation of a JSON value.""" + return rfc8785.dumps(value).decode("utf-8") + + +def _without_recipe(unit: Any) -> Any: + if not isinstance(unit, dict): + return unit + if "numerator" in unit: + return {side: _without_recipe(part) for side, part in unit.items()} + return {name: item for name, item in unit.items() if name != "recipe"} + + +def output_digest(frame: dict[str, Any], variables: dict[str, str]) -> tuple[str, str]: + """The output digest document of a frame, and its token, with `variables` + naming each variable an operation wrote by its `ops_` id.""" + roles = [column for column in frame["columns"] if column not in _NOT_ROLES] + columns = ["stat_var", *roles, "time_period", "value", "unit"] + rows = [] + for row in frame["rows"]: + cells = dict(zip(frame["columns"], row, strict=True)) + cells["unit"] = _without_recipe(cells["unit"]) + stat_var = cells["stat_var"] + cells["stat_var"] = variables.get(stat_var.removeprefix("dcid:"), stat_var) + rows.append([cells[column] for column in columns]) + rows.sort(key=lambda cells: jcs(cells).encode("utf-8")) + document = jcs({"digest_version": "out2", "columns": columns, "rows": rows}) + digest = hashlib.sha256(document.encode("utf-8")).hexdigest() + return document, f"out_{digest[:32]}" + + +def _identity(member: dict[str, Any]) -> tuple[str, ...]: + spec = member["spec"] + return ( + spec["spec_id"], + member["content_version"], + jcs(member["artifact"]["rendering"]), + jcs(spec.get("recipe")), + ) + + +def _output(member: dict[str, Any]) -> tuple[Any, ...]: + if "output_digest" in member: + return ("derived", member["output_digest"]) + rendering = member["artifact"]["rendering"] + cut = rendering.get("without_party_totals", False) + return ("dataset", member["content_version"], cut) + + +def refs_equal(left: dict[str, Any], right: dict[str, Any]) -> bool: + return [_identity(m) for m in left["members"]] == [ + _identity(m) for m in right["members"] + ] + + +def same_data(left: dict[str, Any], right: dict[str, Any]) -> bool: + return [_output(m) for m in left["members"]] == [ + _output(m) for m in right["members"] + ] + + +def _paired( + data: list[dict[str, Any]], other: list[dict[str, Any]], on: dict[str, str] +) -> list[tuple[dict[str, Any], dict[str, Any]]]: + """Each data row with the other row at its paired keys and period.""" + pairs = [] + for row in data: + found = [ + match + for match in other + if match["time_period"] == row["time_period"] + and all(row[mine] == match[theirs] for mine, theirs in on.items()) + ] + assert len(found) <= 1, row + pairs.extend((row, match) for match in found) + return pairs + + +def _growth(vector: dict[str, Any]) -> list[dict[str, Any]]: + """Each member's change over the window. Two rows of one member in one + period of the window refuse with `role_ambiguous`. A member is left out + without a value at both ends (`endpoint_missing`), for a relative change + with a zero or sign-crossing base (`zero_or_sign_crossing_base`), and + for a change that is not finite (`non_finite`).""" + arguments = vector["arguments"] + start, end = arguments["window"] + by = arguments["by"] + relative = arguments.get("change", "relative") == "relative" + groups: dict[tuple[str, ...], dict[str, dict[str, Any]]] = {} + for row in vector["data"]: + period = row["time_period"] + if not start <= period <= end: + continue + periods = groups.setdefault(tuple(row[r] for r in by), {}) + if period in periods: + raise RuleError("role_ambiguous") + periods[period] = row + rows = [] + for key, periods in groups.items(): + if start not in periods or end not in periods: + continue + first, last = periods[start], periods[end] + base, final = first["value"], last["value"] + if relative and (base == 0 or final == 0 or (base < 0) != (final < 0)): + continue + change = (final - base) / base if relative else final - base + if not math.isfinite(change): + continue + rows.append( + { + "stat_var": first["stat_var"], + **dict(zip(by, key, strict=True)), + "time_period": f"{start}/{end}", + "value": change, + "unit": None if relative else first["unit"], + } + ) + return rows + + +def _share_across(vector: dict[str, Any]) -> list[dict[str, Any]]: + by = vector["arguments"]["by"] + totals: dict[str, list[float]] = {} + sums: dict[tuple[Any, ...], list[float]] = {} + for row in vector["data"]: + totals.setdefault(row["time_period"], []).append(row["value"]) + key = (*(row[r] for r in by), row["time_period"]) + sums.setdefault(key, []).append(row["value"]) + return [ + { + **dict(zip(by, key[:-1], strict=True)), + "time_period": key[-1], + "value": math.fsum(values) / math.fsum(totals[key[-1]]) * 100.0, + "unit": _PERCENT, + } + for key, values in sums.items() + ] + + +def operation_output(vector: dict[str, Any]) -> list[dict[str, Any]]: + verb, arguments = vector["verb"], vector["arguments"] + if verb == "growth": + return _growth(vector) + if verb == "share_of" and "other" not in vector: + return _share_across(vector) + on = arguments["on"] + rows = [] + for row, match in _paired(vector["data"], vector["other"], on): + keys = {mine: row[mine] for mine in on} + period = row["time_period"] + if verb == "join": + rows.extend( + { + "stat_var": side["stat_var"], + **keys, + "time_period": period, + "value": side["value"], + "unit": side["unit"], + } + for side in (row, match) + ) + continue + value, unit = { + "share_of": (row["value"] / match["value"] * 100.0, _PERCENT), + "compare": (row["value"] - match["value"], row["unit"]), + "sum_of": (row["value"] + match["value"], row["unit"]), + }[verb] + rows.append({**keys, "time_period": period, "value": value, "unit": unit}) + return rows + + +def _population(population: dict[str, Any]) -> str: + """A population in its one spelling: each binding's values sorted, once.""" + bindings = { + column: sorted(set(values)) + for column, values in population.get("bindings", {}).items() + } + return jcs({"spec_id": population["spec_id"], "bindings": bindings}) + + +def rolling_anchor(vector: dict[str, Any]) -> str: + stated = { + jcs(sorted({_population(p) for p in step["population"]})) + for step in vector["steps"] + } + if len(stated) > 1: + raise RuleError("rolling_populations_differ") + held = [] + for population in vector["steps"][0]["population"]: + bindings = population.get("bindings", {}) + series = [ + set(item["periods"]) + for item in vector["series"] + if item["spec_id"] == population["spec_id"] + and all(item["coordinates"].get(c) in v for c, v in bindings.items()) + ] + if not series: + raise RuleError("rolling_anchor_unresolved") + held.append(series) + periods = {period for series in held for periods in series for period in periods} + for period in sorted(periods, reverse=True): + if all(2 * sum(period in s for s in series) >= len(series) for series in held): + return period + raise RuleError("rolling_anchor_unresolved") + + +def _slots(cards: list[str], held: list[str | None]) -> list[str | None]: + count = max(len(cards), len(held)) + return [cards[i] if i < len(cards) else None for i in range(count)] + + +def _check_token(share: dict[str, Any], step: dict[str, Any]) -> None: + token = step.get("write_token") + if token is None: + raise RuleError("write_token_required") + if token != "held" or not share["has_token"]: + raise RuleError("write_token_invalid") + + +def _state(share: dict[str, Any]) -> dict[str, Any]: + return { + "latest": share["latest"], + "slots": [ + {"slot": number, "card": card} + for number, card in enumerate(share["slots"], start=1) + ], + } + + +def publish(steps: list[dict[str, Any]]) -> list[dict[str, Any]]: + shares: dict[str, dict[str, Any]] = {} + outcomes: list[dict[str, Any]] = [] + for step in steps: + op = step["op"] + if op == "share": + share = shares[step["share"]] = { + "latest": step["latest"], + "has_token": not step.get("tokenless", False), + "slots": _slots(step["cards"], []), + } + outcomes.append({"share": _state(share)}) + continue + share = shares[step["share"]] + try: + if op in ("replace", "unshare"): + _check_token(share, step) + if op == "replace": + if not share["latest"]: + raise RuleError("share_not_replaceable") + share["slots"] = _slots(step["cards"], share["slots"]) + if op == "unshare": + del shares[step["share"]] + outcomes.append({"removed": True}) + continue + except RuleError as error: + outcomes.append({"error": error.code}) + continue + outcomes.append({"share": _state(share)}) + return outcomes + + +def _rows(rows: list[dict[str, Any]]) -> list[str]: + return sorted(jcs(row) for row in rows) + + +@pytest.mark.parametrize("vector", _vectors("output-digest"), ids=_ids("output-digest")) +def test_output_digest(vector: dict[str, Any]) -> None: + assert output_digest(vector["frame"], vector.get("variables", {})) == ( + vector["document"], + vector["digest"], + ) + + +@pytest.mark.parametrize( + "vector", _vectors("output-comparison"), ids=_ids("output-comparison") +) +def test_output_comparison(vector: dict[str, Any]) -> None: + left, right = vector["left"], vector["right"] + assert refs_equal(left, right) is vector["equal"] + assert same_data(left, right) is vector["same_data"] + assert refs_equal(right, left) is vector["equal"] + assert same_data(right, left) is vector["same_data"] + + +@pytest.mark.parametrize( + "vector", _vectors("operation-outputs"), ids=_ids("operation-outputs") +) +def test_operation_outputs(vector: dict[str, Any]) -> None: + if "error" in vector: + with pytest.raises(RuleError) as excinfo: + operation_output(vector) + assert excinfo.value.code == vector["error"] + return + assert _rows(operation_output(vector)) == _rows(vector["rows"]) + + +@pytest.mark.parametrize( + "vector", _vectors("rolling-composition"), ids=_ids("rolling-composition") +) +def test_rolling_composition(vector: dict[str, Any]) -> None: + if "error" in vector: + with pytest.raises(RuleError) as excinfo: + rolling_anchor(vector) + assert excinfo.value.code == vector["error"] + return + assert rolling_anchor(vector) == vector["period"] + + +@pytest.mark.parametrize("vector", _vectors("publication"), ids=_ids("publication")) +def test_publication(vector: dict[str, Any]) -> None: + assert publish(vector["steps"]) == vector["outcomes"] diff --git a/python/client-contract/tests/test_rules.py b/python/client-contract/tests/test_rules.py index 79e438d..72345f2 100644 --- a/python/client-contract/tests/test_rules.py +++ b/python/client-contract/tests/test_rules.py @@ -7,12 +7,15 @@ from __future__ import annotations +import ipaddress +import math import ntpath import posixpath import re import tomllib import zlib from typing import Any +from urllib.parse import urlsplit import pytest @@ -20,15 +23,20 @@ _BASE62 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" _KEY_PREFIXES = ("dfo_k_", "dfo_p_") +_DEFAULT_BASE_URL = "https://developmentfinance.org/api/v1" _TABLE = re.compile(r"\[([A-Za-z0-9_-]+)\]") _ENTRY = re.compile(r'([A-Za-z0-9_-]+)[ \t]*=[ \t]*"([^"\\\x00-\x1f\x7f]*)"') +_MAJOR_MINOR = re.compile(r"([0-9]+)\.([0-9]+)") class RuleError(Exception): - def __init__(self, reason: str, line: int | None = None) -> None: + def __init__( + self, reason: str, line: int | None = None, names: tuple[str, ...] = () + ) -> None: super().__init__(reason) self.reason = reason self.line = line + self.names = names def _vectors(name: str) -> list[dict[str, Any]]: @@ -123,6 +131,45 @@ def look_up_key(vector: dict[str, Any]) -> tuple[str, str | None]: return "credentials_file", profile["key"] +_NON_DECIMAL_IPV4 = re.compile(r"(?:[^:]*\.)?(?:0x[0-9a-f]*|[0-9]+)") + + +_PORT = re.compile(r"(?:\[[^\]]*\]|[^:]*)(?::([0-9]*))?") + + +def is_valid_base_url(url: str) -> bool: + if re.search(r"[^\x21-\x7e]|[?#\\]", url): + return False + parts = urlsplit(url) + if parts.scheme not in ("http", "https") or "@" in parts.netloc: + return False + segments = {s.lower().replace("%2e", ".") for s in parts.path.split("/")} + if not parts.hostname or {".", ".."} & segments: + return False + port = _PORT.fullmatch(parts.netloc) + if port is None or (port[1] is not None and not 1 <= int(port[1] or 0) <= 65535): + return False + host = parts.hostname.removesuffix(".") + if _NON_DECIMAL_IPV4.fullmatch(host): + try: + ipaddress.IPv4Address(host) + except ValueError: + return False + return True + + +def resolve_base_url(vector: dict[str, Any]) -> tuple[str, str]: + if (argument := vector.get("base_url")) is not None: + source, url = "argument", argument + elif from_env := _set(vector["env"], "DFO_BASE_URL"): + source, url = "environment", from_env + else: + source, url = "default", _DEFAULT_BASE_URL + if not is_valid_base_url(url): + raise RuleError("base_url_invalid") + return source, url.rstrip("/") + + def cache_root(vector: dict[str, Any]) -> str: platform, home, env = vector["platform"], vector["home"], vector["env"] path = ntpath if platform == "windows" else posixpath @@ -156,6 +203,156 @@ def cache_entry(vector: dict[str, Any]) -> str: return f"manifests/{vector['spec_id']}/{vector['content_version']}/{key}.json" +def same_value(left: Any, right: Any) -> bool: + """Numbers by value, booleans apart from numbers, NaN as NaN, lists item + by item in order, structs field by field in any order.""" + if isinstance(left, bool) or isinstance(right, bool): + return type(left) is type(right) and left == right + if isinstance(left, float) and isinstance(right, float): + return left == right or (math.isnan(left) and math.isnan(right)) + if isinstance(left, list) and isinstance(right, list): + return len(left) == len(right) and all(map(same_value, left, right)) + if isinstance(left, dict) and isinstance(right, dict): + return left.keys() == right.keys() and all( + same_value(value, right[key]) for key, value in left.items() + ) + return left == right + + +def varies(column: str, rows: list[dict[str, Any]]) -> bool: + return any(not same_value(row.get(column), rows[0].get(column)) for row in rows) + + +def reading_order( + members: list[dict[str, Any]], reading: list[list[str]] +) -> tuple[list[str], list[list[Any]]]: + """Stacks the members, a row holding null in a column its member lacks, + then orders the columns and each member's rows.""" + stacked = [ + {**dict(zip(member["columns"], row, strict=True)), "part": member.get("part")} + for member in members + for row in member["rows"] + ] + served = list(dict.fromkeys(c for member in members for c in member["columns"])) + first = [ + name + for names in reading + if (name := next((n for n in names if n in served), None)) is not None + ] + rest = [c for c in served if c not in first and c != "stat_var"] + varying = {c for c in rest if varies(c, stacked)} + columns = [ + *(["part"] if len(members) > 1 else []), + *first, + *[c for c in rest if c in varying], + *[c for c in rest if c not in varying], + *(["stat_var"] if "stat_var" in served else []), + ] + keys = [c for c in first if c not in ("value", "time_period")] + keys += [c for c in columns if c in varying] + if "stat_var" in served and varies("stat_var", stacked): + keys.append("stat_var") + keys += ["time_period"] if "time_period" in first else [] + nested = { + c for row in stacked for c, v in row.items() if isinstance(v, list | dict) + } + keys = [k for k in keys if k not in nested] + rows: list[list[Any]] = [] + for part in dict.fromkeys(row["part"] for row in stacked): + block = [row for row in stacked if row["part"] == part] + block.sort(key=lambda row: [(row.get(k) is None, row.get(k)) for k in keys]) + rows.extend([row.get(c) for c in columns] for row in block) + return columns, rows + + +def client_identity(client: str, version: str, runtime: dict[str, str]) -> str: + label = runtime["name"] + if label not in ("browser", "other"): + match = _MAJOR_MINOR.match(runtime["version"]) + assert match, runtime + label += f" {match[1]}.{match[2]}" + return f"{client}/{version} ({label})" + + +def check_caller_headers(headers: dict[str, str]) -> str | None: + if any(name.lower() == "x-dfo-client" for name in headers): + return "header_invalid" + return None + + +def spec_roles(identity: dict[str, Any]) -> dict[str, list[str] | None]: + caller = {wire: key for key, wire in rule("role-keys")["keys"].items()} + flow = rule("role-keys")["flow_keys"] + bound = { + caller[role]: dcids for role, dcids in identity.get("entities", {}).items() + } + named = [*bound, *(caller[role] for role in identity.get("open_roles", []))] + if any(key in flow for key in named): + named.extend(flow) + return {key: bound.get(key) for key in dict.fromkeys(named)} + + +def posted_entities( + identity: dict[str, Any], entities: dict[str, list[str] | None] +) -> dict[str, list[str]]: + keys = rule("role-keys")["keys"] + roles = spec_roles(identity) + for key in entities: + if roles and key not in roles: + raise RuleError("invalid_argument", names=(key, *roles)) + if not roles and key in keys.values() and key not in keys: + caller = next(name for name, wire in keys.items() if wire == key) + raise RuleError("invalid_argument", names=(key, caller)) + posted = dict(identity.get("entities", {})) + for key, dcids in entities.items(): + if dcids is None: + posted.pop(keys.get(key, key), None) + else: + posted[keys.get(key, key)] = dcids + return posted + + +def series_left_out(members: list[dict[str, Any]]) -> bool: + """Whether a fetch of derived members with these derived manifests issues + `series_left_out`: once, when any manifest lists a left-out series or says + `left_out_complete` false. A null `left_out` issues nothing.""" + return any( + member["left_out"] or member["left_out_complete"] is False for member in members + ) + + +def selector_query(rendering: dict[str, Any]) -> str: + def written(value: Any) -> str: + return ("true" if value else "false") if isinstance(value, bool) else value + + return "&".join( + f"{key}={written(rendering[key])}" + for key in rule("file-selector")["query_keys"] + if key in rendering + ) + + +# The cells `same_value` judges, the same cases as the validator's tests; +# NaN has no JSON form, so no vector can carry it. +SAME_VALUE_CASES = [ + (1, 1.0, True), + (True, 1, False), + (float("nan"), float("nan"), True), + (None, None, True), + ("1", 1, False), + ([True], [1], False), + ([float("nan")], [float("nan")], True), + ([1], [1.0], True), + ({"a": 1, "b": 2}, {"b": 2, "a": 1.0}, True), + ({"a": True}, {"a": 1}, False), +] + + +@pytest.mark.parametrize(("left", "right", "same"), SAME_VALUE_CASES) +def test_same_value(left: Any, right: Any, same: bool) -> None: + assert same_value(left, right) is same + + @pytest.mark.parametrize("name", RULES) def test_every_rule_has_vectors(name: str) -> None: assert rule(name)["rule"] == name @@ -202,6 +399,16 @@ def test_key_lookup(vector: dict[str, Any]) -> None: assert look_up_key(vector) == (vector["source"], vector["key"]) +@pytest.mark.parametrize("vector", _vectors("base-url"), ids=_ids("base-url")) +def test_base_url(vector: dict[str, Any]) -> None: + if "error" in vector: + with pytest.raises(RuleError) as excinfo: + resolve_base_url(vector) + assert excinfo.value.reason == vector["error"]["reason"] + return + assert resolve_base_url(vector) == (vector["source"], vector["url"]) + + @pytest.mark.parametrize("vector", _vectors("cache-dir"), ids=_ids("cache-dir")) def test_cache_dir(vector: dict[str, Any]) -> None: path = ntpath if vector["platform"] == "windows" else posixpath @@ -213,3 +420,55 @@ def test_cache_dir(vector: dict[str, Any]) -> None: @pytest.mark.parametrize("vector", _vectors("cache-layout"), ids=_ids("cache-layout")) def test_cache_layout(vector: dict[str, Any]) -> None: assert cache_entry(vector) == vector["path"] + + +@pytest.mark.parametrize("vector", _vectors("reading-order"), ids=_ids("reading-order")) +def test_reading_order(vector: dict[str, Any]) -> None: + reading = rule("reading-order")["reading_columns"] + columns, rows = reading_order(vector["members"], reading) + assert (columns, rows) == (vector["columns"], vector["rows"]) + + +@pytest.mark.parametrize( + "vector", _vectors("client-identity"), ids=_ids("client-identity") +) +def test_client_identity(vector: dict[str, Any]) -> None: + if "headers" in vector: + assert check_caller_headers(vector["headers"]) == vector.get("reason") + return + value = client_identity(vector["client"], vector["version"], vector["runtime"]) + assert value == vector["value"] + assert re.fullmatch(rule("client-identity")["pattern"], value) + + +@pytest.mark.parametrize("vector", _vectors("role-keys"), ids=_ids("role-keys")) +def test_role_keys(vector: dict[str, Any]) -> None: + roles = spec_roles(vector["identity"]) + assert list(roles.items()) == list(vector["roles"].items()) + if "error" in vector: + with pytest.raises(RuleError) as excinfo: + posted_entities(vector["identity"], vector["entities"]) + error = excinfo.value + assert (error.reason, list(error.names)) == ( + vector["error"]["reason"], + vector["error"]["names"], + ) + elif "entities" in vector: + posted = posted_entities(vector["identity"], vector["entities"]) + assert posted == vector["posted"] + + +@pytest.mark.parametrize( + "vector", _vectors("series-left-out"), ids=_ids("series-left-out") +) +def test_series_left_out(vector: dict[str, Any]) -> None: + issued = ["series_left_out"] if series_left_out(vector["members"]) else [] + if vector["strict"] and issued: + assert vector.get("raises") == issued[0] + else: + assert vector.get("warnings") == issued + + +@pytest.mark.parametrize("vector", _vectors("file-selector"), ids=_ids("file-selector")) +def test_file_selector(vector: dict[str, Any]) -> None: + assert selector_query(vector["rendering"]) == vector["query"] diff --git a/python/client-contract/uv.lock b/python/client-contract/uv.lock index bee0e2a..3bcd032 100644 --- a/python/client-contract/uv.lock +++ b/python/client-contract/uv.lock @@ -140,6 +140,7 @@ dev = [ { name = "pytest" }, { name = "pytest-cov" }, { name = "referencing" }, + { name = "rfc8785" }, { name = "ruff" }, { name = "ty" }, ] @@ -153,6 +154,7 @@ dev = [ { name = "pytest", specifier = ">=9.1.0" }, { name = "pytest-cov", specifier = ">=7.1.0" }, { name = "referencing", specifier = ">=0.36" }, + { name = "rfc8785", specifier = ">=0.1.4" }, { name = "ruff", specifier = ">=0.15.0" }, { name = "ty", specifier = ">=0.0.59" }, ] @@ -383,6 +385,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, ] +[[package]] +name = "rfc8785" +version = "0.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ef/2f/fa1d2e740c490191b572d33dbca5daa180cb423c24396b856f5886371d8b/rfc8785-0.1.4.tar.gz", hash = "sha256:e545841329fe0eee4f6a3b44e7034343100c12b4ec566dc06ca9735681deb4da", size = 14321, upload-time = "2024-09-27T16:33:31.206Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/78/119878110660b2ad709888c8a1614fce7e2fab39080ab960656dc8605bf6/rfc8785-0.1.4-py3-none-any.whl", hash = "sha256:520d690b448ecf0703691c76e1a34a24ddcd4fc5bc41d589cb7c58ec651bcd48", size = 9240, upload-time = "2024-09-27T16:33:29.683Z" }, +] + [[package]] name = "rpds-py" version = "2026.9.1" diff --git a/python/descriptions/CHANGELOG.md b/python/descriptions/CHANGELOG.md index 3c6e776..6a59d01 100644 --- a/python/descriptions/CHANGELOG.md +++ b/python/descriptions/CHANGELOG.md @@ -136,5 +136,18 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 repeated drawn `ref` and a card drawn from a computed version carrying a measure. The card-revision and live-rendering vectors carry the measure, fixed values and drawn values with graph definitions. +- `operations.json` names each analysis operation's `verb`, the function a + client calls, and its `output`, the shape of the rows it writes; an + operation that takes inputs by role carries `arguments`, the roles a + client's `data` and `other` fill and how `on` pairs them, so `sum` and + `compare` pair by role as `quotient`, `share` and `join` do. `compare` gives + `data` minus `other`, with `data` in the `right` role. The `percent` unit + rule names its unit, `Percent`, in `ref`, and a relative growth has no unit. + `tools/validate_descriptions_vectors.py` checks each verb is listed once and + the arguments fill the op's two roles. +- `sum`, `share` and `compare` list `unmatched_key` under `left_out`, the + members of one side with no partner on the other. The `sum` and `compare` + descriptions cover the paired form: members of `data` and `other` pair by + role through `on`, and a call to `compare` returns `data` minus `other`. [unreleased]: https://github.com/DevelopmentFinanceObservatory/spec-kit/commits/main/python diff --git a/python/descriptions/README.md b/python/descriptions/README.md index 201231d..8bd92d1 100644 --- a/python/descriptions/README.md +++ b/python/descriptions/README.md @@ -118,25 +118,27 @@ The size bound is on the stored list and `check_steps` checks it where the list `operations.json` lists, for each operation, its kind, whether it takes one input, inputs by role or none, its one fixed policy, the rule for the unit it produces, the coverage it records, and the data gaps it can leave out with what each counts. The analysis operations carry the policy names and exclusion reasons the `ai` repo's analysis engine already uses. -| `op` | Kind | Inputs | Policy | Unit | Coverage | Known `left_out` reasons | -| ----------------- | --------------- | -------------------------- | ----------------------------------- | ------------------------------------- | ---------- | -------------------------------------------------------------------------------------- | -| `retrieve` | retrieval | one version | `retrieve_minted_version` | as recorded | | | -| `sum` | analysis | `left`, `right` | `sum_of_intersect_periods` | the inputs' shared unit | | `period_unmatched`, `unit_unbound` | -| `quotient` | analysis | `numerator`, `denominator` | `quotient_of_intersect_periods` | composed | per period | `unmatched_key`, `period_unmatched`, `zero_denominator`, `unit_unbound`, `party_total` | -| `share` | analysis | `numerator`, `denominator` | `share_of_intersect_periods` | per cent | per period | `period_unmatched`, `unit_unbound`, `party_total` | -| `growth` | analysis | one | `growth_endpoint_years` | the input's, or a dimensionless ratio | | `endpoint_missing`, `zero_or_sign_crossing_base`, `non_finite`, `unit_unbound` | -| `rank` | analysis | one | `rank_what_is_present` | the input's, or none for an ordinal | per entity | `party_total`, `unit_unbound` | -| `aggregate` | analysis | one | `aggregate_skip_absent_grain_cells` | the input's | per period | `party_total`, `unit_unbound` | -| `filter` | analysis | one | `filter_row_value_predicates` | the input's | | | -| `compare` | analysis | `left`, `right` | `compare_intersect_periods` | the inputs' shared unit | | `period_unmatched`, `unit_unbound` | -| `join` | analysis | `left`, `right` | `join_inner` | each side its own | | `unmatched_key`, `unit_unbound` | -| `source_step` | source pipeline | none | `source_pipeline_as_recorded` | as recorded | | | -| `choose_source` | mint choice | none | `facet_default_by_coverage` | as recorded | | | -| `sum_variables` | mint choice | none | `sum_family_members` | the summed variables' shared unit | | | -| `convert` | mint choice | none | `convert_by_rate_and_deflator` | converted | | `currency_unmapped`, `deflator_missing` | -| `continue_series` | mint choice | none | `continue_with_source_estimates` | as recorded | | | -| `select` | card selection | one version | `card_selection` | the version's | | | -| `change` | card derivation | one | `change_between_basis_periods` | the input's, or per cent | | | +Each analysis operation also names its `verb`, the function a client calls, and its `output`, the shape of the rows it writes with a sentence on their values. `share` writes per cent, 100 times the ratio, under the unit `Percent`, which its unit's `ref` names. `join` writes long format, one row per side at each key and period, told apart by `stat_var`. `growth` writes one row per member with the window as its period, and a relative change as a fraction with no unit. An operation that takes inputs by role carries `arguments`: the role a client's first reading, `data`, fills, the role its second, `other`, fills, and how `on` pairs their roles. `sum` and `compare` pair by `on` as `quotient`, `share` and `join` do, and each side is one series when `on` is left out. `compare` subtracts the left series from the right, so `data` fills `right` and the client's call gives `data` minus `other`. The client contract's `operation-outputs` rule pins these units and shapes with vectors. + +| `op` | Kind | Inputs | Policy | Unit | Coverage | Known `left_out` reasons | +| ----------------- | --------------- | -------------------------- | ----------------------------------- | ----------------------------------- | ---------- | -------------------------------------------------------------------------------------- | +| `retrieve` | retrieval | one version | `retrieve_minted_version` | as recorded | | | +| `sum` | analysis | `left`, `right` | `sum_of_intersect_periods` | the inputs' shared unit | | `unmatched_key`, `period_unmatched`, `unit_unbound` | +| `quotient` | analysis | `numerator`, `denominator` | `quotient_of_intersect_periods` | composed | per period | `unmatched_key`, `period_unmatched`, `zero_denominator`, `unit_unbound`, `party_total` | +| `share` | analysis | `numerator`, `denominator` | `share_of_intersect_periods` | per cent | per period | `unmatched_key`, `period_unmatched`, `unit_unbound`, `party_total` | +| `growth` | analysis | one | `growth_endpoint_years` | the input's, or none for a fraction | | `endpoint_missing`, `zero_or_sign_crossing_base`, `non_finite`, `unit_unbound` | +| `rank` | analysis | one | `rank_what_is_present` | the input's, or none for an ordinal | per entity | `party_total`, `unit_unbound` | +| `aggregate` | analysis | one | `aggregate_skip_absent_grain_cells` | the input's | per period | `party_total`, `unit_unbound` | +| `filter` | analysis | one | `filter_row_value_predicates` | the input's | | | +| `compare` | analysis | `left`, `right` | `compare_intersect_periods` | the inputs' shared unit | | `unmatched_key`, `period_unmatched`, `unit_unbound` | +| `join` | analysis | `left`, `right` | `join_inner` | each side its own | | `unmatched_key`, `unit_unbound` | +| `source_step` | source pipeline | none | `source_pipeline_as_recorded` | as recorded | | | +| `choose_source` | mint choice | none | `facet_default_by_coverage` | as recorded | | | +| `sum_variables` | mint choice | none | `sum_family_members` | the summed variables' shared unit | | | +| `convert` | mint choice | none | `convert_by_rate_and_deflator` | converted | | `currency_unmapped`, `deflator_missing` | +| `continue_series` | mint choice | none | `continue_with_source_estimates` | as recorded | | | +| `select` | card selection | one version | `card_selection` | the version's | | | +| `change` | card derivation | one | `change_between_basis_periods` | the input's, or per cent | | | ## Descriptions diff --git a/python/descriptions/src/dfo_descriptions/schemas/operations.json b/python/descriptions/src/dfo_descriptions/schemas/operations.json index 4e31ff1..06fb175 100644 --- a/python/descriptions/src/dfo_descriptions/schemas/operations.json +++ b/python/descriptions/src/dfo_descriptions/schemas/operations.json @@ -17,66 +17,115 @@ }, { "op": "sum", + "verb": "sum_of", "kind": "analysis", "input": "roles", - "roles": ["left", "right"], + "roles": [ + "left", + "right" + ], + "arguments": { + "data": "left", + "other": "right", + "on": "Pairs each role of `data` with the role of `other` that holds the same entities, a role to a role, or names the same role on both sides as a list. Left out, each side is one series." + }, "policy": "sum_of_intersect_periods", "new_variable": true, "unit": { "rule": "same_as_input", "text": "The unit both sides share." }, + "output": { + "shape": "per_key_and_period", + "text": "One row for each paired key and period both sides hold, its value `data` plus `other`, its keys named as `data` names them." + }, "coverage": null, "left_out": [ + { + "reason": "unmatched_key", + "of": [ + "entities" + ], + "text": "Members of one side with no partner on the other, so their total is unknown." + }, { "reason": "period_unmatched", - "of": ["periods"], + "of": [ + "periods" + ], "text": "Periods one side reports and the other does not, so their total is unknown." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], - "description": "Adds two single-entity series period by period, over the periods both report." + "description": "Adds two series period by period, over the periods both report. Given `on`, it pairs the members of `data` and `other` by role and adds each pair, leaving out a member with no partner." }, { "op": "quotient", + "verb": "quotient_of", "kind": "analysis", "input": "roles", - "roles": ["numerator", "denominator"], + "roles": [ + "numerator", + "denominator" + ], + "arguments": { + "data": "numerator", + "other": "denominator", + "on": "Pairs each role of `data` with the role of `other` that holds the same entities, a role to a role, or names the same role on both sides as a list." + }, "policy": "quotient_of_intersect_periods", "new_variable": true, "unit": { "rule": "composed", "text": "The numerator's unit per the denominator's." }, + "output": { + "shape": "per_key_and_period", + "text": "One row for each numerator key and period the denominator also holds, its value `data` divided by `other`." + }, "coverage": "period_coverage", "left_out": [ { "reason": "unmatched_key", - "of": ["entities"], + "of": [ + "entities" + ], "text": "Numerator entities with no denominator." }, { "reason": "period_unmatched", - "of": ["entities", "entity_periods"], + "of": [ + "entities", + "entity_periods" + ], "text": "Entities that share no period with their denominator, or values in periods the other side lacks." }, { "reason": "zero_denominator", - "of": ["entities", "entity_periods"], + "of": [ + "entities", + "entity_periods" + ], "text": "Entities, or values, whose denominator sums to zero." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." }, { "reason": "party_total", - "of": ["pooled_entities"], + "of": [ + "pooled_entities" + ], "text": "Flow parties of a pooled role that total other parties of it, left out so nothing counts twice; one entry per pooled role, counting its distinct parties across the step's inputs." } ], @@ -84,30 +133,58 @@ }, { "op": "share", + "verb": "share_of", "kind": "analysis", "input": "roles", - "roles": ["numerator", "denominator"], + "roles": [ + "numerator", + "denominator" + ], + "arguments": { + "data": "numerator", + "other": "denominator", + "on": "Pairs each role of `data` with the role of `other` that holds the same entities, a role to a role, or names the same role on both sides as a list. With no `other`, each member of `by` is divided by the total across every member of `by` in the same period." + }, "policy": "share_of_intersect_periods", "new_variable": true, "unit": { "rule": "percent", - "text": "Per cent." + "ref": "Percent", + "text": "Per cent, under the unit `Percent`." + }, + "output": { + "shape": "per_key_and_period", + "text": "One row for each key and period, its value 100 times `data` divided by `other`." }, "coverage": "period_coverage", "left_out": [ + { + "reason": "unmatched_key", + "of": [ + "entities" + ], + "text": "Members of one side with no partner on the other, so their share is unknown." + }, { "reason": "period_unmatched", - "of": ["periods", "entity_periods"], + "of": [ + "periods", + "entity_periods" + ], "text": "Periods, or values, one side reports and the other does not." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." }, { "reason": "party_total", - "of": ["pooled_entities"], + "of": [ + "pooled_entities" + ], "text": "Flow parties of a pooled role that total other parties of it, left out so nothing counts twice; one entry per pooled role, counting its distinct parties across the step's inputs." } ], @@ -115,6 +192,7 @@ }, { "op": "growth", + "verb": "growth", "kind": "analysis", "input": "single", "roles": [], @@ -122,28 +200,40 @@ "new_variable": false, "unit": { "rule": "same_as_input_or_dimensionless", - "text": "The input's unit for an absolute change, a dimensionless ratio for a relative one." + "text": "The input's unit for an absolute change; none for a relative one, whose value is a fraction." + }, + "output": { + "shape": "per_member_over_window", + "text": "One row for each member holding a value at both ends of the window, its period the window as a span such as `2018/2022`, its value the change: a fraction for a relative change, 0.25 for 25 per cent, and an amount in the input's unit for an absolute one." }, "coverage": null, "left_out": [ { "reason": "endpoint_missing", - "of": ["entities"], + "of": [ + "entities" + ], "text": "Entities with no value in the start or the end period." }, { "reason": "zero_or_sign_crossing_base", - "of": ["entities"], + "of": [ + "entities" + ], "text": "Entities whose base is zero or changes sign, so a relative change is undefined." }, { "reason": "non_finite", - "of": ["entities"], + "of": [ + "entities" + ], "text": "Entities whose change overflows to a non-finite value." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], @@ -151,6 +241,7 @@ }, { "op": "rank", + "verb": "rank", "kind": "analysis", "input": "single", "roles": [], @@ -160,16 +251,24 @@ "rule": "same_as_input_or_none", "text": "The input's unit; none for a per-period ranking without a limit, whose values are rank positions." }, + "output": { + "shape": "ranked_members", + "text": "One row for each ranked member, or each member and period for a ranking per period, its value the measured value, or the rank position for a ranking per period without a limit." + }, "coverage": "entity_coverage", "left_out": [ { "reason": "party_total", - "of": ["pooled_entities"], + "of": [ + "pooled_entities" + ], "text": "Flow parties of a pooled role that total other parties of it, left out so nothing counts twice; one entry per pooled role, counting its distinct parties across the step's inputs." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], @@ -177,6 +276,7 @@ }, { "op": "aggregate", + "verb": "aggregate", "kind": "analysis", "input": "single", "roles": [], @@ -186,16 +286,24 @@ "rule": "same_as_input", "text": "The input's unit." }, + "output": { + "shape": "per_key_and_period", + "text": "One row for each key and period, its value the reduction of the rows it pools." + }, "coverage": "period_coverage", "left_out": [ { "reason": "party_total", - "of": ["pooled_entities"], + "of": [ + "pooled_entities" + ], "text": "Flow parties of a pooled role that total other parties of it, left out so nothing counts twice; one entry per pooled role, counting its distinct parties across the step's inputs." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], @@ -203,6 +311,7 @@ }, { "op": "filter", + "verb": "filter", "kind": "analysis", "input": "single", "roles": [], @@ -212,57 +321,102 @@ "rule": "same_as_input", "text": "The input's unit." }, + "output": { + "shape": "input_rows", + "text": "The input's rows that meet the predicates, unchanged." + }, "coverage": null, "left_out": [], "description": "Keeps the values that meet the caller's predicates. What the predicates drop is the caller's selection and stays in the parameters." }, { "op": "compare", + "verb": "compare", "kind": "analysis", "input": "roles", - "roles": ["left", "right"], + "roles": [ + "left", + "right" + ], + "arguments": { + "data": "right", + "other": "left", + "on": "Pairs each role of `data` with the role of `other` that holds the same entities, a role to a role, or names the same role on both sides as a list. Left out, each side is one series." + }, "policy": "compare_intersect_periods", "new_variable": true, "unit": { "rule": "same_as_input", "text": "The unit both sides share." }, + "output": { + "shape": "per_key_and_period", + "text": "One row for each paired key and period both sides hold, its value `data` minus `other`, its keys named as `data` names them. `data` fills the `right` role, since the step subtracts the left series from the right." + }, "coverage": null, "left_out": [ + { + "reason": "unmatched_key", + "of": [ + "entities" + ], + "text": "Members of one side with no partner on the other." + }, { "reason": "period_unmatched", - "of": ["periods"], + "of": [ + "periods" + ], "text": "Periods one side reports and the other does not." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], - "description": "Subtracts the left series from the right for each period both report, and writes the difference under a variable of its own. The step records in `sides` the entities each side holds." + "description": "Subtracts the left series from the right for each period both report, so a call returns `data` minus `other`, and writes the difference under a variable of its own. Given `on`, it pairs the members of `data` and `other` by role and compares each pair, leaving out a member with no partner. The step records in `sides` the entities each side holds." }, { "op": "join", + "verb": "join", "kind": "analysis", "input": "roles", - "roles": ["left", "right"], + "roles": [ + "left", + "right" + ], + "arguments": { + "data": "left", + "other": "right", + "on": "Pairs each role of `data` with the role of `other` that holds the same entities, a role to a role, or names the same role on both sides as a list. Left out, the roles both readings hold under one name." + }, "policy": "join_inner", "new_variable": false, "unit": { "rule": "per_side", "text": "Each side keeps its own unit." }, + "output": { + "shape": "long_by_stat_var", + "text": "Two rows for each paired key and period both readings hold, one from each side, told apart by `stat_var`, each keeping its own value and unit, the key named as `data` names it." + }, "coverage": null, "left_out": [ { "reason": "unmatched_key", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Keys and periods one side holds and the other does not." }, { "reason": "unit_unbound", - "of": ["entity_periods"], + "of": [ + "entity_periods" + ], "text": "Values that declare no unit." } ], @@ -328,12 +482,17 @@ "left_out": [ { "reason": "currency_unmapped", - "of": ["entities"], + "of": [ + "entities" + ], "text": "Entities no single national currency names, in a conversion into each entity's own currency." }, { "reason": "deflator_missing", - "of": ["entities", "entity_periods"], + "of": [ + "entities", + "entity_periods" + ], "text": "Entities with no deflator in the base year, or values in years the deflator table lacks." } ], diff --git a/python/descriptions/src/dfo_descriptions/schemas/operations.schema.json b/python/descriptions/src/dfo_descriptions/schemas/operations.schema.json index 8dbeb80..e6ec469 100644 --- a/python/descriptions/src/dfo_descriptions/schemas/operations.schema.json +++ b/python/descriptions/src/dfo_descriptions/schemas/operations.schema.json @@ -5,7 +5,9 @@ "description": "The operations a step record can name: for each, its kind, how it takes its inputs, its one fixed policy, whether it writes a new variable, the unit it produces, the coverage it records, and the data gaps it can leave out with what each counts.", "type": "object", "additionalProperties": false, - "required": ["operations"], + "required": [ + "operations" + ], "properties": { "operations": { "type": "array", @@ -35,13 +37,38 @@ "op": { "$ref": "steps.schema.json#/$defs/name" }, + "verb": { + "description": "The analysis function a client calls for this operation, such as `share_of` for `share`.", + "enum": [ + "aggregate", + "compare", + "filter", + "growth", + "join", + "quotient_of", + "rank", + "share_of", + "sum_of" + ] + }, "kind": { "description": "`retrieval` reads a version; `analysis` is one of the analysis engine's operations; `source_pipeline` is a step the source's pipeline applied before the DFO read the data; `mint_choice` is a choice the DFO made while minting a version; `card_selection` is what a card draws from its version; `card_derivation` is a value a card derives from what it draws.", - "enum": ["retrieval", "analysis", "source_pipeline", "mint_choice", "card_selection", "card_derivation"] + "enum": [ + "retrieval", + "analysis", + "source_pipeline", + "mint_choice", + "card_selection", + "card_derivation" + ] }, "input": { "description": "`single`: one `input`. `roles`: `inputs` keyed by the names in `roles`. `none`: the step takes no input, since it records a choice inside the version being made.", - "enum": ["single", "roles", "none"] + "enum": [ + "single", + "roles", + "none" + ] }, "roles": { "type": "array", @@ -50,6 +77,28 @@ "$ref": "steps.schema.json#/$defs/name" } }, + "arguments": { + "description": "How a client's call fills the roles of an analysis operation that takes inputs by role: `data`, its first reading, and `other`, its second, each name the role it fills, and `on` states how the call pairs the two readings' roles.", + "type": "object", + "additionalProperties": false, + "required": [ + "data", + "other", + "on" + ], + "properties": { + "data": { + "$ref": "steps.schema.json#/$defs/name" + }, + "other": { + "$ref": "steps.schema.json#/$defs/name" + }, + "on": { + "type": "string", + "minLength": 1 + } + } + }, "policy": { "$ref": "steps.schema.json#/$defs/name" }, @@ -60,10 +109,13 @@ "unit": { "type": "object", "additionalProperties": false, - "required": ["rule", "text"], + "required": [ + "rule", + "text" + ], "properties": { "rule": { - "description": "`as_recorded`: the unit the version holds. `same_as_input`: the input's unit, which every input must share. `composed`: the numerator's unit per the denominator's. `percent`: per cent. `same_as_input_or_dimensionless`: the input's unit for an absolute change, a dimensionless ratio for a relative one. `per_side`: each side keeps its own unit. `converted`: the unit converted into. `same_as_input_or_percent`: the input's unit for a difference, per cent for a percent change. `same_as_input_or_none`: the input's unit for a ranking of values, none for an ordinal ranking, whose values are rank positions.", + "description": "`as_recorded`: the unit the version holds. `same_as_input`: the input's unit, which every input must share. `composed`: the numerator's unit per the denominator's. `percent`: per cent, 100 times the ratio, under the unit `ref` names. `same_as_input_or_dimensionless`: the input's unit for an absolute change, none for a relative one, whose value is a fraction. `per_side`: each side keeps its own unit. `converted`: the unit converted into. `same_as_input_or_percent`: the input's unit for a difference, per cent for a percent change. `same_as_input_or_none`: the input's unit for a ranking of values, none for an ordinal ranking, whose values are rank positions.", "enum": [ "as_recorded", "same_as_input", @@ -76,6 +128,35 @@ "same_as_input_or_none" ] }, + "ref": { + "description": "The unit's graph reference, for a rule that names one unit: `Percent` for `percent`.", + "type": "string", + "minLength": 1 + }, + "text": { + "type": "string", + "minLength": 1 + } + } + }, + "output": { + "description": "The rows an analysis operation writes. `per_key_and_period`: one row for each key and period. `long_by_stat_var`: one row for each side at each key and period, told apart by `stat_var`. `per_member_over_window`: one row for each member, its period the window as a span. `ranked_members`: one row for each ranked member, or each member and period. `input_rows`: the input's rows that pass.", + "type": "object", + "additionalProperties": false, + "required": [ + "shape", + "text" + ], + "properties": { + "shape": { + "enum": [ + "per_key_and_period", + "long_by_stat_var", + "per_member_over_window", + "ranked_members", + "input_rows" + ] + }, "text": { "type": "string", "minLength": 1 @@ -84,7 +165,11 @@ }, "coverage": { "description": "`period_coverage`: how many entities each period's total covers. `entity_coverage`: how many periods each ranked entity holds. `null`: none.", - "enum": ["period_coverage", "entity_coverage", null] + "enum": [ + "period_coverage", + "entity_coverage", + null + ] }, "left_out": { "description": "The data gaps the operation can leave out. A step may name a reason this list lacks.", @@ -92,7 +177,11 @@ "items": { "type": "object", "additionalProperties": false, - "required": ["reason", "of", "text"], + "required": [ + "reason", + "of", + "text" + ], "properties": { "reason": { "$ref": "steps.schema.json#/$defs/name" @@ -103,7 +192,12 @@ "minItems": 1, "uniqueItems": true, "items": { - "enum": ["entities", "periods", "entity_periods", "pooled_entities"] + "enum": [ + "entities", + "periods", + "entity_periods", + "pooled_entities" + ] } }, "text": { @@ -117,7 +211,102 @@ "type": "string", "minLength": 1 } - } + }, + "allOf": [ + { + "if": { + "properties": { + "kind": { + "const": "analysis" + } + } + }, + "then": { + "required": [ + "verb", + "output" + ] + }, + "else": { + "not": { + "anyOf": [ + { + "required": [ + "verb" + ] + }, + { + "required": [ + "output" + ] + }, + { + "required": [ + "arguments" + ] + } + ] + } + } + }, + { + "if": { + "properties": { + "kind": { + "const": "analysis" + }, + "input": { + "const": "roles" + } + } + }, + "then": { + "required": [ + "arguments" + ] + }, + "else": { + "not": { + "required": [ + "arguments" + ] + } + } + }, + { + "if": { + "properties": { + "unit": { + "properties": { + "rule": { + "const": "percent" + } + } + } + } + }, + "then": { + "properties": { + "unit": { + "required": [ + "ref" + ] + } + } + }, + "else": { + "properties": { + "unit": { + "not": { + "required": [ + "ref" + ] + } + } + } + } + } + ] } } } diff --git a/python/spec-kit/CHANGELOG.md b/python/spec-kit/CHANGELOG.md index c7a8ec0..70e3735 100644 --- a/python/spec-kit/CHANGELOG.md +++ b/python/spec-kit/CHANGELOG.md @@ -64,6 +64,32 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 differ from the published figure. Its params name the total (`total`, `total_label`) and the party classes it covers (`whole_of`: `official`, `private` or `residual`, comma-joined). +- Code `allowance_exhausted` in `registries/codes.json` and its bundled + mirror: a 429 refusal on `datasets.pin`, `presentation.shares` and + `presentation.write` once the caller has spent its daily allowance of + pins, shares or cards. Its problem document says `retryable: false`, and + its `Retry-After` header names the reset. +- Endpoint `datasets.analysis` in the registry schema, with its refusals in + `registries/codes.json` and its bundled mirror: 36 new codes at 422, + `rolling_populations_differ`, `several_members` and the 34 the analysis + engine serves, among them `unit_heterogeneous`, `cross_frame_unit_mismatch`, + `join_output_collision`, `rolling_anchor_unresolved`, `role_ambiguous`, + `compare_no_key_matched` and `sum_of_no_key_matched`; `timeout` at 504; and + 27 existing codes at the status each has on `datasets.mint`, + `datasets.snapshot` or the platform rows, among them `request_invalid` + (400), `snapshot_revoked` (410), `over_ceiling` (422), `upstream_error` + (502) and `service_unavailable` (503). +- Endpoint `presentation.shares` in the registry schema, with its refusals in + `registries/codes.json` and its bundled mirror: `write_token_required` (403) + for a replace or delete without a write token; `write_token_invalid` (403) + for one whose token is not the share's, or that names a share created + without a token; `share_not_replaceable` (409); `payload_too_large` (413) + for a body larger than the share routes accept; and `binding_invalid`, + `schema_invalid`, `constraint_violation`, `resource_exhausted` and + `request_invalid` (400), `not_found` and `frame_unavailable` (404), + `edit_conflict` (409), `internal_error` (500), `release_mismatch` and + `service_unavailable` (503), and `timeout` (504), at the statuses the + components routes give them. ### Changed @@ -74,6 +100,19 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `https://raw.githubusercontent.com/DevelopmentFinanceObservatory/spec-kit/main/`, with each file's path unchanged. The previous prefix named a private repository and did not resolve. +- On `presentation.write`, codes `binding_invalid`, `constraint_violation` + and `resource_exhausted` map at 400, `frame_unavailable` at 404 and + `release_mismatch` at 503, the statuses the Presentation write routes + serve. `presentation.render` keeps 422 for the first two and 409 for the + rest. `frame_unavailable` covers a frame that is not held in a state + Presentation can load, and its definition names each case. A frame whose + stored artifact the store fails to read is also `frame_unavailable`, served + at 503 and retryable; the registry holds one status for each endpoint and + channel, so it records the 404 the write routes serve for every other + case. +- Code `edit_conflict` maps to `presentation.render` at 409, for a render + of a component that has no current revision, and its definition names + that case. - Codes `identity_quarantined` (500), `service_unavailable` (503) and `internal_error` (500) map to `datasets.description` at the statuses they have on `datasets.snapshot`. @@ -82,6 +121,9 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Code `upstream_error` maps to `graph.members` at 502, the status the members route answers when the read that checks whether the graph holds the node fails upstream, as `graph.node` answers for the same read. +- `pin_not_permitted` is retired in `registries/codes.json` and its bundled + mirror: the pin route admits every caller under its allowance, so nothing + raises it. Its `datasets.pin` mapping stays. - `starved_axis_binding_restored` is retired in `registries/codes.json` and its bundled mirror: Search no longer restores a starved constraint axis, so nothing raises it. @@ -210,6 +252,14 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 mirror: a 422 refusal on `search.resolve` for a `dataset_resubmit` whose identity names an open role, when the StatVars the definition elects declare no role on that axis, so the definition cannot give one series per member. +- Codes `requested_option_unavailable` and `conflicting_requested_options` in + `registries/codes.json` and its bundled mirror: 422 refusals on + `search.resolve` for a `dataset_resubmit` that posts `scope_entries`. + `requested_option_unavailable` covers an entry that names an axis the part + does not offer, an option the axis does not list, or an unavailable option. + `conflicting_requested_options` covers entries that cannot all apply: two + options on one axis, an option that conflicts with an entry on another + axis, or entries that rule out every option of an axis. - Optional top-level identity member `open_roles` in `dfo_spec_kit.identity`: a list of flow roles, from `origin` and `destination`, whose every member the request asks for, as distinct from an diff --git a/python/spec-kit/src/dfo_spec_kit/registry/codes.json b/python/spec-kit/src/dfo_spec_kit/registry/codes.json index 0a351ae..6ec8b03 100644 --- a/python/spec-kit/src/dfo_spec_kit/registry/codes.json +++ b/python/spec-kit/src/dfo_spec_kit/registry/codes.json @@ -1,6 +1,32 @@ { "format_version": 1, "entries": [ + { + "code": "allowance_exhausted", + "classification": "refusal", + "definition": "The caller has spent its daily allowance for this kind of write, shares, cards or pins, so the write is refused until the allowance resets, which the response's Retry-After header names. The problem document says retryable false, so a client raises at once and waits for no reset.", + "owning_contract": "platform", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.pin", + "channel": "problem_details", + "http_status": 429 + }, + { + "endpoint": "presentation.write", + "channel": "problem_details", + "http_status": 429 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 429 + } + ], + "lifecycle": "active" + }, { "code": "alternates_unevaluated", "classification": "warning", @@ -47,6 +73,11 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 410 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 410 } ], "lifecycle": "active" @@ -180,12 +211,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 422 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -288,6 +324,11 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 501 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 501 } ], "lifecycle": "active" @@ -429,6 +470,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -475,6 +521,73 @@ ], "lifecycle": "active" }, + { + "code": "compare_no_key_matched", + "classification": "refusal", + "definition": "No member of a compare step's data side pairs with a member of its other side through the step's on mapping, so there is nothing to compare.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "comparison_periods_disjoint", + "classification": "refusal", + "definition": "The two sides of a compare, quotient or sum_of step have no period in common.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "composed_unit_sum_unsupported", + "classification": "refusal", + "definition": "An analysis step adds values whose unit is a quotient, and quotients added together are not a quotient in that unit.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "conflicting_requested_options", + "classification": "refusal", + "definition": "The scope_entries posted for a part cannot all apply: two entries name different options on one axis, an option conflicts with an entry on another axis, or the entries rule out every option of an axis.", + "owning_contract": "search", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "search.resolve", + "doors": [ + "dataset_resubmit" + ], + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "conjunction_cross_shape", "classification": "warning", @@ -562,12 +675,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 422 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -599,6 +717,70 @@ ], "lifecycle": "active" }, + { + "code": "coverage_insufficient", + "classification": "refusal", + "definition": "The grid an aggregate step groups over is too sparse: the share of expected cells with a value is below the floor the step requires.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_facet_mismatch", + "classification": "refusal", + "definition": "The two sides of an analysis step hold one StatVar read under different values of a facet, whose values do not combine.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_unit_mismatch", + "classification": "refusal", + "definition": "The two sides of an analysis step resolve to different units, and combining their values would be wrong.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_unit_unbound", + "classification": "refusal", + "definition": "One side of an analysis step declares no unit while the other declares one, so the sides cannot be combined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "currency_unmapped_omitted", "classification": "warning", @@ -672,6 +854,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -723,6 +910,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -848,6 +1040,38 @@ ], "lifecycle": "active" }, + { + "code": "derived_input_ordinal", + "classification": "refusal", + "definition": "An analysis step reads a derived result whose values are rank positions, and a ratio or growth rate over a rank position has no meaning.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "derived_input_unusable", + "classification": "refusal", + "definition": "An analysis step reads a derived result whose rows hold spans of periods, which a growth step cannot window over.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "direction_discarded", "classification": "warning", @@ -910,7 +1134,7 @@ { "code": "edit_conflict", "classification": "refusal", - "definition": "A revise_component request targets a revision that is no longer the component's current one.", + "definition": "The component's current revision is not the one the request reads or targets: a revise_component request names a revision that is no longer current, or the component has no current revision.", "owning_contract": "presentation", "owner": "platform", "introduced_in": "v1", @@ -919,6 +1143,16 @@ "endpoint": "presentation.write", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.render", + "channel": "problem_details", + "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 409 } ], "lifecycle": "active" @@ -1261,6 +1495,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1316,6 +1555,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1347,6 +1591,22 @@ ], "lifecycle": "active" }, + { + "code": "facet_heterogeneous", + "classification": "refusal", + "definition": "An analysis step reads an input that carries more than one value of a facet, such as the measurement method, and the response lists the candidates.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "facet_not_available", "classification": "refusal", @@ -1403,6 +1663,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1443,6 +1708,38 @@ ], "lifecycle": "active" }, + { + "code": "filter_predicate_type_mismatch", + "classification": "refusal", + "definition": "A filter step names a predicate of a kind, or with a value of a type, that does not fit the column it applies to.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "filter_unit_conversion_unsupported", + "classification": "refusal", + "definition": "A filter step puts a unit predicate on an input whose units are converted, which the step cannot match.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "frame_not_refreshable", "classification": "refusal", @@ -1459,10 +1756,26 @@ ], "lifecycle": "active" }, + { + "code": "frame_too_large", + "classification": "refusal", + "definition": "The result of an analysis step is too large to store, so the step is refused and nothing is stored.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "frame_unavailable", "classification": "refusal", - "definition": "The frame the intent's frame_ref names cannot be read at request time.", + "definition": "The frame a request names is not held in a state Presentation can load: it was never stored or published, it has expired or been revoked, its spec is quarantined, or its stored artifact is missing, fails its digest check, does not decode, or holds more rows than Presentation loads.", "owning_contract": "presentation", "owner": "platform", "introduced_in": "v1", @@ -1470,12 +1783,33 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 404 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 404 + } + ], + "lifecycle": "active" + }, + { + "code": "growth_gap_year", + "classification": "refusal", + "definition": "No entity has a value in both endpoint years of a growth step, so the step has no growth to compute.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1517,6 +1851,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1583,6 +1922,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1664,6 +2008,16 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1710,6 +2064,38 @@ ], "lifecycle": "active" }, + { + "code": "join_no_common_key_columns", + "classification": "refusal", + "definition": "A join step names no key columns and the two sides share no role column to join on.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "join_output_collision", + "classification": "refusal", + "definition": "The two sides of a join step have the same StatVar, unit and facets, so the rows a matched key emits cannot be told apart.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "member_not_resolvable", "classification": "refusal", @@ -1742,6 +2128,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1810,6 +2201,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1869,6 +2265,22 @@ ], "lifecycle": "active" }, + { + "code": "mixed_time_grain", + "classification": "refusal", + "definition": "An input frame holds periods of more than one grain, or a mix of points and spans, so no analysis step can read it, and reading it again gives the same periods.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "multi_recipient_truncated", "classification": "warning", @@ -2033,6 +2445,22 @@ ], "lifecycle": "active" }, + { + "code": "non_finite_result", + "classification": "refusal", + "definition": "An analysis step overflows to a value that is not finite, so repeating the step with the same input reaches the same result.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "not_found", "classification": "refusal", @@ -2055,6 +2483,11 @@ "endpoint": "presentation.render", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -2412,6 +2845,22 @@ ], "lifecycle": "active" }, + { + "code": "op_spec_invalid", + "classification": "refusal", + "definition": "The recipe holds an analysis step whose op-spec is not valid, such as an unknown field, a missing argument or a value out of range.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "open_role_unavailable", "classification": "refusal", @@ -2463,6 +2912,22 @@ ], "lifecycle": "active" }, + { + "code": "operation_uncovered", + "classification": "refusal", + "definition": "An aggregate step names an aggregation the analysis does not support.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "option_coverage_probe_failed", "classification": "warning", @@ -2588,6 +3053,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -2641,6 +3111,22 @@ ], "lifecycle": "active" }, + { + "code": "payload_too_large", + "classification": "refusal", + "definition": "A share's request body is larger than the share routes accept.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 413 + } + ], + "lifecycle": "active" + }, { "code": "pin_not_permitted", "classification": "refusal", @@ -2655,7 +3141,7 @@ "http_status": 403 } ], - "lifecycle": "active" + "lifecycle": "retired" }, { "code": "pivot_not_homogeneous", @@ -2760,6 +3246,22 @@ ], "lifecycle": "active" }, + { + "code": "place_levels_mixed", + "classification": "refusal", + "definition": "A sum step pools parties whose totals contain one another, such as a region and its countries, which would count members twice.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "place_scope_defaulted", "classification": "warning", @@ -2963,6 +3465,54 @@ ], "lifecycle": "active" }, + { + "code": "quotient_of_no_key_matched", + "classification": "refusal", + "definition": "No key of a quotient step's numerator matches a key of its denominator, so no quotient can be computed.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "quotient_of_side_unit_unsupported", + "classification": "refusal", + "definition": "A side of a quotient step cannot be a part of the result's unit: it is a share or growth rate, already a composed unit, or a sum_of result with no unit.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "quotient_of_zero_denominator", + "classification": "refusal", + "definition": "The denominator of a quotient step sums to zero in every matched cell, so the step has no quotient to return.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "rate_coverage_gap", "classification": "refusal", @@ -3006,6 +3556,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3061,6 +3616,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3077,6 +3637,11 @@ "endpoint": "datasets.recipe", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -3132,12 +3697,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 503 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 503 } ], "lifecycle": "active" @@ -3170,6 +3740,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3301,9 +3876,38 @@ "http_status": 400 }, { - "endpoint": "datasets.snapshot", + "endpoint": "datasets.snapshot", + "channel": "problem_details", + "http_status": 400 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 400 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 + } + ], + "lifecycle": "active" + }, + { + "code": "requested_option_unavailable", + "classification": "refusal", + "definition": "A scope_entries entry names an axis the part does not offer, an option the axis does not list, or an option that is unavailable for this part.", + "owning_contract": "search", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "search.resolve", + "doors": [ + "dataset_resubmit" + ], "channel": "problem_details", - "http_status": 400 + "http_status": 422 } ], "lifecycle": "active" @@ -3353,12 +3957,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -3409,6 +4018,22 @@ ], "lifecycle": "active" }, + { + "code": "role_ambiguous", + "classification": "refusal", + "definition": "A column an analysis step names, to group, rank, join or filter on, is not one its input has a grain for.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "role_declaration_unverified", "classification": "warning", @@ -3575,6 +4200,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3597,6 +4227,54 @@ ], "lifecycle": "active" }, + { + "code": "rolling_anchor_unresolved", + "classification": "refusal", + "definition": "The latest period a rolling step counts from is not established, so the step cannot resolve its periods.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "rolling_grain_unsupported", + "classification": "refusal", + "definition": "A rolling step counts the latest period over data whose periods are not calendar years, which it does not support.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "rolling_populations_differ", + "classification": "refusal", + "definition": "The rolling steps of one analysis recipe count the latest period over different populations of series, so the recipe is refused before it runs, whether or not the periods they would settle on agree.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "rollup_not_summable", "classification": "refusal", @@ -3645,6 +4323,11 @@ "endpoint": "presentation.render", "channel": "problem_details", "http_status": 400 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -3822,6 +4505,32 @@ "endpoint": "graph.documentation", "channel": "problem_details", "http_status": 503 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 503 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 503 + } + ], + "lifecycle": "active" + }, + { + "code": "several_members", + "classification": "refusal", + "definition": "An analysis step takes one reading for each input, and an input names a dataset that holds several.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3845,6 +4554,86 @@ ], "lifecycle": "active" }, + { + "code": "share_not_replaceable", + "classification": "refusal", + "definition": "A replace names a snapshot share, whose cards are fixed for good. Only a live share takes new cards.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 409 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_grain_mismatch", + "classification": "refusal", + "definition": "No key of a share step has a period its numerator and denominator both report, so there is no period to divide.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_key_unknown", + "classification": "refusal", + "definition": "The denominator mapping of a share step covers only some of the numerator's key columns, which leaves the pairing of the others undefined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_key_unmatched", + "classification": "refusal", + "definition": "No key of a share step's numerator has a denominator group, so no share can be computed.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_zero_denominator", + "classification": "refusal", + "definition": "The denominator of a share step sums to zero in a period, so the share is undefined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "skill_attachment_kind_unconfirmed", "classification": "warning", @@ -4266,6 +5055,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -4327,6 +5121,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 410 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 410 } ], "lifecycle": "active" @@ -4494,6 +5293,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 409 } ], "lifecycle": "active" @@ -4555,6 +5359,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -4598,6 +5407,22 @@ ], "lifecycle": "retired" }, + { + "code": "stat_var_unresolved", + "classification": "refusal", + "definition": "An analysis step needs one StatVar and its input holds several, or none the step can use. A composite input read with one series per StatVar is the common case.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "stat_vars_summed", "classification": "warning", @@ -4625,6 +5450,22 @@ ], "lifecycle": "active" }, + { + "code": "sum_of_no_key_matched", + "classification": "refusal", + "definition": "No member of a sum_of step's data side pairs with a member of its other side through the step's on mapping, so there is nothing to add.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "time_text_not_parsed", "classification": "warning", @@ -4742,6 +5583,16 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 504 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 504 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 504 } ], "lifecycle": "active" @@ -4792,6 +5643,22 @@ ], "lifecycle": "active" }, + { + "code": "unit_heterogeneous", + "classification": "refusal", + "definition": "An analysis step reads an input that carries more than one unit, or a pinned unit the input does not declare, and the response lists the candidates to pin.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "unit_not_resolved", "classification": "refusal", @@ -4997,6 +5864,11 @@ "endpoint": "graph.members", "channel": "problem_details", "http_status": 502 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 502 } ], "lifecycle": "active" @@ -5038,6 +5910,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 503 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 503 } ], "lifecycle": "active" @@ -5165,6 +6042,38 @@ } ], "lifecycle": "active" + }, + { + "code": "write_token_invalid", + "classification": "refusal", + "definition": "A replace or delete of a share carries a write token other than the share's, or names a share created without one, which no token can change.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 403 + } + ], + "lifecycle": "active" + }, + { + "code": "write_token_required", + "classification": "refusal", + "definition": "A replace or delete of a share carries no write token.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 403 + } + ], + "lifecycle": "active" } ] } diff --git a/registries/codes.json b/registries/codes.json index 0a351ae..6ec8b03 100644 --- a/registries/codes.json +++ b/registries/codes.json @@ -1,6 +1,32 @@ { "format_version": 1, "entries": [ + { + "code": "allowance_exhausted", + "classification": "refusal", + "definition": "The caller has spent its daily allowance for this kind of write, shares, cards or pins, so the write is refused until the allowance resets, which the response's Retry-After header names. The problem document says retryable false, so a client raises at once and waits for no reset.", + "owning_contract": "platform", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.pin", + "channel": "problem_details", + "http_status": 429 + }, + { + "endpoint": "presentation.write", + "channel": "problem_details", + "http_status": 429 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 429 + } + ], + "lifecycle": "active" + }, { "code": "alternates_unevaluated", "classification": "warning", @@ -47,6 +73,11 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 410 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 410 } ], "lifecycle": "active" @@ -180,12 +211,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 422 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -288,6 +324,11 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 501 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 501 } ], "lifecycle": "active" @@ -429,6 +470,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -475,6 +521,73 @@ ], "lifecycle": "active" }, + { + "code": "compare_no_key_matched", + "classification": "refusal", + "definition": "No member of a compare step's data side pairs with a member of its other side through the step's on mapping, so there is nothing to compare.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "comparison_periods_disjoint", + "classification": "refusal", + "definition": "The two sides of a compare, quotient or sum_of step have no period in common.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "composed_unit_sum_unsupported", + "classification": "refusal", + "definition": "An analysis step adds values whose unit is a quotient, and quotients added together are not a quotient in that unit.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "conflicting_requested_options", + "classification": "refusal", + "definition": "The scope_entries posted for a part cannot all apply: two entries name different options on one axis, an option conflicts with an entry on another axis, or the entries rule out every option of an axis.", + "owning_contract": "search", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "search.resolve", + "doors": [ + "dataset_resubmit" + ], + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "conjunction_cross_shape", "classification": "warning", @@ -562,12 +675,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 422 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -599,6 +717,70 @@ ], "lifecycle": "active" }, + { + "code": "coverage_insufficient", + "classification": "refusal", + "definition": "The grid an aggregate step groups over is too sparse: the share of expected cells with a value is below the floor the step requires.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_facet_mismatch", + "classification": "refusal", + "definition": "The two sides of an analysis step hold one StatVar read under different values of a facet, whose values do not combine.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_unit_mismatch", + "classification": "refusal", + "definition": "The two sides of an analysis step resolve to different units, and combining their values would be wrong.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "cross_frame_unit_unbound", + "classification": "refusal", + "definition": "One side of an analysis step declares no unit while the other declares one, so the sides cannot be combined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "currency_unmapped_omitted", "classification": "warning", @@ -672,6 +854,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -723,6 +910,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -848,6 +1040,38 @@ ], "lifecycle": "active" }, + { + "code": "derived_input_ordinal", + "classification": "refusal", + "definition": "An analysis step reads a derived result whose values are rank positions, and a ratio or growth rate over a rank position has no meaning.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "derived_input_unusable", + "classification": "refusal", + "definition": "An analysis step reads a derived result whose rows hold spans of periods, which a growth step cannot window over.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "direction_discarded", "classification": "warning", @@ -910,7 +1134,7 @@ { "code": "edit_conflict", "classification": "refusal", - "definition": "A revise_component request targets a revision that is no longer the component's current one.", + "definition": "The component's current revision is not the one the request reads or targets: a revise_component request names a revision that is no longer current, or the component has no current revision.", "owning_contract": "presentation", "owner": "platform", "introduced_in": "v1", @@ -919,6 +1143,16 @@ "endpoint": "presentation.write", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.render", + "channel": "problem_details", + "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 409 } ], "lifecycle": "active" @@ -1261,6 +1495,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1316,6 +1555,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1347,6 +1591,22 @@ ], "lifecycle": "active" }, + { + "code": "facet_heterogeneous", + "classification": "refusal", + "definition": "An analysis step reads an input that carries more than one value of a facet, such as the measurement method, and the response lists the candidates.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "facet_not_available", "classification": "refusal", @@ -1403,6 +1663,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1443,6 +1708,38 @@ ], "lifecycle": "active" }, + { + "code": "filter_predicate_type_mismatch", + "classification": "refusal", + "definition": "A filter step names a predicate of a kind, or with a value of a type, that does not fit the column it applies to.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "filter_unit_conversion_unsupported", + "classification": "refusal", + "definition": "A filter step puts a unit predicate on an input whose units are converted, which the step cannot match.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "frame_not_refreshable", "classification": "refusal", @@ -1459,10 +1756,26 @@ ], "lifecycle": "active" }, + { + "code": "frame_too_large", + "classification": "refusal", + "definition": "The result of an analysis step is too large to store, so the step is refused and nothing is stored.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "frame_unavailable", "classification": "refusal", - "definition": "The frame the intent's frame_ref names cannot be read at request time.", + "definition": "The frame a request names is not held in a state Presentation can load: it was never stored or published, it has expired or been revoked, its spec is quarantined, or its stored artifact is missing, fails its digest check, does not decode, or holds more rows than Presentation loads.", "owning_contract": "presentation", "owner": "platform", "introduced_in": "v1", @@ -1470,12 +1783,33 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 404 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 404 + } + ], + "lifecycle": "active" + }, + { + "code": "growth_gap_year", + "classification": "refusal", + "definition": "No entity has a value in both endpoint years of a growth step, so the step has no growth to compute.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1517,6 +1851,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1583,6 +1922,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1664,6 +2008,16 @@ "endpoint": "datasets.files", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1710,6 +2064,38 @@ ], "lifecycle": "active" }, + { + "code": "join_no_common_key_columns", + "classification": "refusal", + "definition": "A join step names no key columns and the two sides share no role column to join on.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "join_output_collision", + "classification": "refusal", + "definition": "The two sides of a join step have the same StatVar, unit and facets, so the rows a matched key emits cannot be told apart.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "member_not_resolvable", "classification": "refusal", @@ -1742,6 +2128,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -1810,6 +2201,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 500 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 500 } ], "lifecycle": "active" @@ -1869,6 +2265,22 @@ ], "lifecycle": "active" }, + { + "code": "mixed_time_grain", + "classification": "refusal", + "definition": "An input frame holds periods of more than one grain, or a mix of points and spans, so no analysis step can read it, and reading it again gives the same periods.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "multi_recipient_truncated", "classification": "warning", @@ -2033,6 +2445,22 @@ ], "lifecycle": "active" }, + { + "code": "non_finite_result", + "classification": "refusal", + "definition": "An analysis step overflows to a value that is not finite, so repeating the step with the same input reaches the same result.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "not_found", "classification": "refusal", @@ -2055,6 +2483,11 @@ "endpoint": "presentation.render", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -2412,6 +2845,22 @@ ], "lifecycle": "active" }, + { + "code": "op_spec_invalid", + "classification": "refusal", + "definition": "The recipe holds an analysis step whose op-spec is not valid, such as an unknown field, a missing argument or a value out of range.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "open_role_unavailable", "classification": "refusal", @@ -2463,6 +2912,22 @@ ], "lifecycle": "active" }, + { + "code": "operation_uncovered", + "classification": "refusal", + "definition": "An aggregate step names an aggregation the analysis does not support.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "option_coverage_probe_failed", "classification": "warning", @@ -2588,6 +3053,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -2641,6 +3111,22 @@ ], "lifecycle": "active" }, + { + "code": "payload_too_large", + "classification": "refusal", + "definition": "A share's request body is larger than the share routes accept.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 413 + } + ], + "lifecycle": "active" + }, { "code": "pin_not_permitted", "classification": "refusal", @@ -2655,7 +3141,7 @@ "http_status": 403 } ], - "lifecycle": "active" + "lifecycle": "retired" }, { "code": "pivot_not_homogeneous", @@ -2760,6 +3246,22 @@ ], "lifecycle": "active" }, + { + "code": "place_levels_mixed", + "classification": "refusal", + "definition": "A sum step pools parties whose totals contain one another, such as a region and its countries, which would count members twice.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "place_scope_defaulted", "classification": "warning", @@ -2963,6 +3465,54 @@ ], "lifecycle": "active" }, + { + "code": "quotient_of_no_key_matched", + "classification": "refusal", + "definition": "No key of a quotient step's numerator matches a key of its denominator, so no quotient can be computed.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "quotient_of_side_unit_unsupported", + "classification": "refusal", + "definition": "A side of a quotient step cannot be a part of the result's unit: it is a share or growth rate, already a composed unit, or a sum_of result with no unit.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "quotient_of_zero_denominator", + "classification": "refusal", + "definition": "The denominator of a quotient step sums to zero in every matched cell, so the step has no quotient to return.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "rate_coverage_gap", "classification": "refusal", @@ -3006,6 +3556,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3061,6 +3616,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3077,6 +3637,11 @@ "endpoint": "datasets.recipe", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -3132,12 +3697,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 503 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 503 } ], "lifecycle": "active" @@ -3170,6 +3740,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3301,9 +3876,38 @@ "http_status": 400 }, { - "endpoint": "datasets.snapshot", + "endpoint": "datasets.snapshot", + "channel": "problem_details", + "http_status": 400 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 400 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 + } + ], + "lifecycle": "active" + }, + { + "code": "requested_option_unavailable", + "classification": "refusal", + "definition": "A scope_entries entry names an axis the part does not offer, an option the axis does not list, or an option that is unavailable for this part.", + "owning_contract": "search", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "search.resolve", + "doors": [ + "dataset_resubmit" + ], "channel": "problem_details", - "http_status": 400 + "http_status": 422 } ], "lifecycle": "active" @@ -3353,12 +3957,17 @@ { "endpoint": "presentation.write", "channel": "problem_details", - "http_status": 409 + "http_status": 400 }, { "endpoint": "presentation.render", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -3409,6 +4018,22 @@ ], "lifecycle": "active" }, + { + "code": "role_ambiguous", + "classification": "refusal", + "definition": "A column an analysis step names, to group, rank, join or filter on, is not one its input has a grain for.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "role_declaration_unverified", "classification": "warning", @@ -3575,6 +4200,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 422 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3597,6 +4227,54 @@ ], "lifecycle": "active" }, + { + "code": "rolling_anchor_unresolved", + "classification": "refusal", + "definition": "The latest period a rolling step counts from is not established, so the step cannot resolve its periods.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "rolling_grain_unsupported", + "classification": "refusal", + "definition": "A rolling step counts the latest period over data whose periods are not calendar years, which it does not support.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "rolling_populations_differ", + "classification": "refusal", + "definition": "The rolling steps of one analysis recipe count the latest period over different populations of series, so the recipe is refused before it runs, whether or not the periods they would settle on agree.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "rollup_not_summable", "classification": "refusal", @@ -3645,6 +4323,11 @@ "endpoint": "presentation.render", "channel": "problem_details", "http_status": 400 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 400 } ], "lifecycle": "active" @@ -3822,6 +4505,32 @@ "endpoint": "graph.documentation", "channel": "problem_details", "http_status": 503 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 503 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 503 + } + ], + "lifecycle": "active" + }, + { + "code": "several_members", + "classification": "refusal", + "definition": "An analysis step takes one reading for each input, and an input names a dataset that holds several.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 } ], "lifecycle": "active" @@ -3845,6 +4554,86 @@ ], "lifecycle": "active" }, + { + "code": "share_not_replaceable", + "classification": "refusal", + "definition": "A replace names a snapshot share, whose cards are fixed for good. Only a live share takes new cards.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 409 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_grain_mismatch", + "classification": "refusal", + "definition": "No key of a share step has a period its numerator and denominator both report, so there is no period to divide.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_key_unknown", + "classification": "refusal", + "definition": "The denominator mapping of a share step covers only some of the numerator's key columns, which leaves the pairing of the others undefined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_key_unmatched", + "classification": "refusal", + "definition": "No key of a share step's numerator has a denominator group, so no share can be computed.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, + { + "code": "share_of_zero_denominator", + "classification": "refusal", + "definition": "The denominator of a share step sums to zero in a period, so the share is undefined.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "skill_attachment_kind_unconfirmed", "classification": "warning", @@ -4266,6 +5055,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -4327,6 +5121,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 410 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 410 } ], "lifecycle": "active" @@ -4494,6 +5293,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 409 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 409 } ], "lifecycle": "active" @@ -4555,6 +5359,11 @@ "endpoint": "datasets.description", "channel": "problem_details", "http_status": 404 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 404 } ], "lifecycle": "active" @@ -4598,6 +5407,22 @@ ], "lifecycle": "retired" }, + { + "code": "stat_var_unresolved", + "classification": "refusal", + "definition": "An analysis step needs one StatVar and its input holds several, or none the step can use. A composite input read with one series per StatVar is the common case.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "stat_vars_summed", "classification": "warning", @@ -4625,6 +5450,22 @@ ], "lifecycle": "active" }, + { + "code": "sum_of_no_key_matched", + "classification": "refusal", + "definition": "No member of a sum_of step's data side pairs with a member of its other side through the step's on mapping, so there is nothing to add.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "time_text_not_parsed", "classification": "warning", @@ -4742,6 +5583,16 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 504 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 504 + }, + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 504 } ], "lifecycle": "active" @@ -4792,6 +5643,22 @@ ], "lifecycle": "active" }, + { + "code": "unit_heterogeneous", + "classification": "refusal", + "definition": "An analysis step reads an input that carries more than one unit, or a pinned unit the input does not declare, and the response lists the candidates to pin.", + "owning_contract": "datasets", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 422 + } + ], + "lifecycle": "active" + }, { "code": "unit_not_resolved", "classification": "refusal", @@ -4997,6 +5864,11 @@ "endpoint": "graph.members", "channel": "problem_details", "http_status": 502 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 502 } ], "lifecycle": "active" @@ -5038,6 +5910,11 @@ "endpoint": "datasets.refresh", "channel": "problem_details", "http_status": 503 + }, + { + "endpoint": "datasets.analysis", + "channel": "problem_details", + "http_status": 503 } ], "lifecycle": "active" @@ -5165,6 +6042,38 @@ } ], "lifecycle": "active" + }, + { + "code": "write_token_invalid", + "classification": "refusal", + "definition": "A replace or delete of a share carries a write token other than the share's, or names a share created without one, which no token can change.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 403 + } + ], + "lifecycle": "active" + }, + { + "code": "write_token_required", + "classification": "refusal", + "definition": "A replace or delete of a share carries no write token.", + "owning_contract": "presentation", + "owner": "platform", + "introduced_in": "v1", + "mappings": [ + { + "endpoint": "presentation.shares", + "channel": "problem_details", + "http_status": 403 + } + ], + "lifecycle": "active" } ] } diff --git a/schemas/registry/README.md b/schemas/registry/README.md index 64825ab..df4da75 100644 --- a/schemas/registry/README.md +++ b/schemas/registry/README.md @@ -59,9 +59,9 @@ Retiring a door is a schema change, reviewed as one: dropping a value from `code - **`endpoint`** lists the Search endpoints `search.resolve` (`POST /api/search/resolve`) and `search.options` (`POST /api/search/options`). - The Datasets endpoints are `datasets.register` (`POST /api/datasets/register`), `datasets.mint` (`POST /api/datasets`), `datasets.definition` (`GET /api/datasets/{id}`), `datasets.options` (`GET /api/datasets/{id}/options`, the write-free preview door), `datasets.members` (`POST /api/datasets/{id}/members`, the write-free member read), `datasets.snapshot` (`GET /api/datasets/{id}/{content_version}`, and `GET /api/datasets/{id}/versions`, which pages a dataset's servable versions), `datasets.files` (`GET /api/datasets/{id}/{content_version}/files/{digest}`, which answers a stored artifact with a 302 redirect to its download link, and `GET /api/datasets/{id}/{content_version}/file`, which selects a stored version's file by format, bundle and layout, renders it when it is missing, and answers the same redirect), `datasets.description` (`GET /api/datasets/{id}/{content_version}/description`, a stored version's description, for a `spec_` id with a `cv_` version or an `ops_` id with an `ocv_` version, and `GET .../description/values`, which pages one dimension's values), `datasets.observations` (`GET /api/datasets/{id}/{content_version}/observations`, a stored version's observations), `datasets.pin` (`POST /api/datasets/{id}/{content_version}/pin`), `datasets.publish` (`POST /api/datasets/frames/publish`, the door an analysis service uses to publish an already-computed frame under its own identity pair), `datasets.recipe` (`PUT` and `GET /api/datasets/frames/{spec_id}/{content_version}/recipe`, the recorded computation behind a published frame, and `GET /api/datasets/recipes/replays/{replay_key}`, the frame stored under a replay key), `datasets.refresh` (`POST /api/datasets/frames/refresh`, which rebuilds a dataset frame's rendering on current data), and `datasets.operations` (`GET /api/datasets/operations/{operation_id}`). + The Datasets endpoints are `datasets.register` (`POST /api/datasets/register`), `datasets.mint` (`POST /api/datasets`), `datasets.definition` (`GET /api/datasets/{id}`), `datasets.options` (`GET /api/datasets/{id}/options`, the write-free preview door), `datasets.members` (`POST /api/datasets/{id}/members`, the write-free member read), `datasets.snapshot` (`GET /api/datasets/{id}/{content_version}`, and `GET /api/datasets/{id}/versions`, which pages a dataset's servable versions), `datasets.files` (`GET /api/datasets/{id}/{content_version}/files/{digest}`, which answers a stored artifact with a 302 redirect to its download link, and `GET /api/datasets/{id}/{content_version}/file`, which selects a stored version's file by format, bundle and layout, renders it when it is missing, and answers the same redirect), `datasets.description` (`GET /api/datasets/{id}/{content_version}/description`, a stored version's description, for a `spec_` id with a `cv_` version or an `ops_` id with an `ocv_` version, and `GET .../description/values`, which pages one dimension's values), `datasets.observations` (`GET /api/datasets/{id}/{content_version}/observations`, a stored version's observations), `datasets.pin` (`POST /api/datasets/{id}/{content_version}/pin`), `datasets.publish` (`POST /api/datasets/frames/publish`, the door an analysis service uses to publish an already-computed frame under its own identity pair), `datasets.recipe` (`PUT` and `GET /api/datasets/frames/{spec_id}/{content_version}/recipe`, the recorded computation behind a published frame, and `GET /api/datasets/recipes/replays/{replay_key}`, the frame stored under a replay key), `datasets.refresh` (`POST /api/datasets/frames/refresh`, which rebuilds a dataset frame's rendering on current data), `datasets.operations` (`GET /api/datasets/operations/{operation_id}`), and `datasets.analysis` (`POST /api/datasets/analysis`, which runs an analysis recipe tree and answers its derived frame, and `POST /api/datasets/analysis/anchor`, which resolves a recipe's latest period and stores nothing). - The Presentation endpoints are `presentation.write` (`POST /api/presentation/components`, `PATCH /api/presentation/components/{component_id}`) and `presentation.render` (`POST /api/presentation/components/{component_id}/revisions/{revision}/render` and `GET .../render/data`, which draw a stored revision on a named target frame and mint no revision). Search's and Datasets' `schema`, `capabilities`, and `healthz` routes and Presentation's read routes carry no permanent codes. + The Presentation endpoints are `presentation.write` (`POST /api/presentation/components`, `PATCH /api/presentation/components/{component_id}`) `presentation.render` (`POST /api/presentation/components/{component_id}/revisions/{revision}/render` and `GET .../render/data`, which draw a stored revision on a named target frame and mint no revision), and `presentation.shares` (the routes under `/api/presentation/shares` that create, replace and delete a share). Search's and Datasets' `schema`, `capabilities`, and `healthz` routes and Presentation's read routes carry no permanent codes. The Graph endpoints are `graph.node` (`GET /api/graph/node/{dcid}`), `graph.nodes` (`POST /api/graph/nodes`, a batch of node reads), `graph.members` (`GET /api/graph/node/{dcid}/members`, one page of a node's list), `graph.roots` (`GET /api/graph/roots`), `graph.documentation` (`GET /api/graph/documentation/{dcid}`, a source's document), `graph.sources` (`GET /api/graph/sources`), and `graph.resolve` (`POST /api/graph/resolve`). `graph.node` and `graph.documentation` carry `node_not_held`, `request_invalid`, `service_unavailable`, and `upstream_error`; `graph.nodes` and `graph.resolve` carry `request_invalid`, `unsupported_schema_version`, `service_unavailable`, and `upstream_error`; `graph.members` carries `node_not_held`, `request_invalid`, `service_unavailable`, and `upstream_error`, the last when the read that checks whether the graph holds the node fails upstream; and `graph.roots` carries `service_unavailable`. A node the graph does not hold answers inside a `graph.nodes` batch as a `not_held` entry, so that endpoint carries no `node_not_held`. Graph's `sources` and `capabilities` routes carry no permanent codes. diff --git a/schemas/registry/code-registry.schema.json b/schemas/registry/code-registry.schema.json index 5c46e24..23765d4 100644 --- a/schemas/registry/code-registry.schema.json +++ b/schemas/registry/code-registry.schema.json @@ -101,6 +101,7 @@ "search.options", "presentation.write", "presentation.render", + "presentation.shares", "datasets.register", "datasets.mint", "datasets.definition", @@ -115,6 +116,7 @@ "datasets.recipe", "datasets.refresh", "datasets.operations", + "datasets.analysis", "graph.node", "graph.nodes", "graph.members", @@ -243,6 +245,7 @@ "enum": [ "presentation.write", "presentation.render", + "presentation.shares", "datasets.register", "datasets.mint", "datasets.definition", @@ -257,6 +260,7 @@ "datasets.recipe", "datasets.refresh", "datasets.operations", + "datasets.analysis", "graph.node", "graph.nodes", "graph.members", diff --git a/tools/README.md b/tools/README.md index 6e4fb59..b2d73e8 100644 --- a/tools/README.md +++ b/tools/README.md @@ -38,13 +38,13 @@ uv run tools/validate_provenance_docs_vectors.py uv run tools/validate_descriptions_vectors.py ``` -`validate_client_contract.py` validates the client contract under `python/client-contract/src/dfo_client_contract/` against its schemas, then checks what they cannot express: every `strict-fields.json` pointer resolves in `served/openapi.json` and stays required and non-nullable, or keeps its enum values; `actions.json` holds one row, in code order, for each active refusal code in `registries/codes.json` that reaches a problem document or a no_data reason, with a gap kind exactly for the no_data reasons; `client-codes.json` lists each code a client raises itself once, in code order, none of them a code in `actions.json` or the registry, each with a class a scenario names and a retryable flag; `proxy-codes.json` and `warning-codes.json` list their codes once, in code order, apart from every other table, and the warning codes are those the client rules issue; `served/openapi.json` holds the Search, Datasets, Graph and Presentation surfaces and nothing outside them but the platform's `/codes` route; each recorded scenario's requests match a served route and its JSON bodies validate against that route's schemas, where a required member stays required when it admits null; every problem document in a scenario, synthetic ones and failed operations' errors included, carries an active code registered at its status or a code `proxy-codes.json` lists at its status, and one with a proxy code is `application/problem+json` with the common members, its own status and `retryable` true; no 202 in a scenario reads `done`; each recorded scenario names `served/SOURCE.json`'s commit; each scenario's `expect` equals the outcomes its steps reach over its exchanges, with refusals' error classes and gap kinds from `actions.json`, each error the client raises itself carrying its code and class from `client-codes.json`, a repeated refused request counted a retry only where the retry rules allow one, a record that stops retrying while a retry remains refused, and the waits before retries and the warnings response headers issue derived; each POST carries its body, each build posts the body its spec, narrowing and layout give, each Search request posts the whole body its steps derive, each poll names the operation the client holds, and each signed download is the URL its redirect names; each body file a record names is a bare file name in `scenarios/` and no symlink; every absolute URL a record holds is a signed URL on the storage host with the redacted query, or a `location` under the base URL; and each `DatasetRef` example gets the verdict it states. It does not import `dfo_client_contract`, whose `tests/test_rules.py` replays the rule vectors. Also a PEP 723 script: +`validate_client_contract.py` validates the client contract under `python/client-contract/src/dfo_client_contract/` against its schemas, then checks what they cannot express: every `strict-fields.json` pointer resolves in `served/openapi.json` and stays required and non-nullable, or keeps its enum values; `actions.json` holds one row, in code order, for each active refusal code in `registries/codes.json` that reaches a problem document or a no_data reason, with a gap kind exactly for the no_data reasons; `client-codes.json` lists each code a client raises itself once, in code order, none of them a code in `actions.json` or the registry, each with a class a scenario names and a retryable flag; `proxy-codes.json` and `warning-codes.json` list their codes once, in code order, apart from every other table, and the warning codes are those the client rules issue; `served/openapi.json` holds the Search, Datasets, Graph and Presentation surfaces and nothing outside them but the platform's `/codes` route; each recorded scenario's requests match a served route and its JSON bodies validate against that route's schemas, where a required member stays required when it admits null; every problem document in a scenario, synthetic ones and failed operations' errors included, carries an active code registered at its status or a code `proxy-codes.json` lists at its status, and one with a proxy code is `application/problem+json` with the common members, its own status and `retryable` true; no 202 in a scenario reads `done`; each recorded scenario names `served/SOURCE.json`'s commit; each scenario's `expect` equals the outcomes its steps reach over its exchanges, with refusals' error classes and gap kinds from `actions.json`, each error the client raises itself carrying its code and class from `client-codes.json`, a repeated refused request counted a retry only where the retry rules allow one, a record that stops retrying while a retry remains refused, and the waits before retries and the warnings response headers issue derived; each POST carries its body, each build posts the body its spec, narrowing, layout and, for a fetch or a mint, party totals give, each dataset's columns are the reading-order rule's order of the downloaded Parquet files' columns, each Search request posts the whole body its steps derive, each poll names the operation the client holds, and each signed download is the URL its redirect names; each body file a record names is a bare file name in `scenarios/` and no symlink; every absolute URL a record holds is a signed URL on the storage host with the redacted query, or a `location` under the base URL; each `DatasetRef` example gets the verdict it states, each reading-order vector orders its columns and rows as the rule does, and each file-selector vector's query is the one the rule builds. It does not import `dfo_client_contract`, whose `tests/test_rules.py` replays the rule vectors. Also a PEP 723 script: ```bash uv run tools/validate_client_contract.py ``` -`capture_client_contract.py` refreshes the client contract from dfo-apis. `openapi --dfo-apis PATH` builds the composed app in that checkout's environment, in process, and writes `served/openapi.json` and `served/SOURCE.json`, refusing a checkout with uncommitted changes or untracked files and an environment out of step with its `uv.lock`; `record` runs the contract's scenarios against a base URL, dev by default, writes the records with every absolute URL's query redacted and a local app's signed URLs written as storage URLs, refusing a record that still names any other host, derives the synthetic ones and validates the result, and `record --offline` derives the outcomes again from the records with no request. A recorded scenario holds only exchanges the server made. `record` refuses a recorded scenario that a canned exchange answered, and a synthetic derivation makes no request. Nothing in a record marks an exchange as canned, so `validate_client_contract.py` cannot check this rule and the capture enforces it. It reads and writes a binary body only through a bare file name beside its record, and refuses a name that is absolute or leaves that directory, or a symlink, before it reads or writes anything. It imports its outcome rules from `validate_client_contract.py`. `python/client-contract/README.md` has the recipe. A PEP 723 script whose tests run without network or pyarrow: +`capture_client_contract.py` refreshes the client contract from dfo-apis. `openapi --dfo-apis PATH` builds the composed app in that checkout's environment, in process, and writes `served/openapi.json` and `served/SOURCE.json`, refusing a checkout with uncommitted changes or untracked files and an environment out of step with its `uv.lock`; `record` runs the contract's scenarios against a base URL, dev by default, writes the records with every absolute URL's query redacted and a local app's signed URLs written as storage URLs, refusing a record that still names any other host, derives the synthetic ones and validates the result, and `record --offline` derives the outcomes again from the records with no request. A recorded scenario holds only exchanges the server made. `record` refuses a recorded scenario that a canned exchange answered, and a synthetic derivation makes no request. Nothing in a record marks an exchange as canned, so `validate_client_contract.py` cannot check this rule and the capture enforces it. It reads and writes a binary body only through a bare file name beside its record, and refuses a name that is absolute or leaves that directory, or a symlink, before it reads or writes anything. It imports its outcome rules from `validate_client_contract.py`. `python/client-contract/README.md` has the recipe. A PEP 723 script whose tests run without network: ```bash uv run tools/capture_client_contract.py record diff --git a/tools/capture_client_contract.py b/tools/capture_client_contract.py index b232ab8..c069777 100644 --- a/tools/capture_client_contract.py +++ b/tools/capture_client_contract.py @@ -38,24 +38,23 @@ replaced by `X-Goog-Signature=redacted`. A local app's blob URL becomes the storage URL a deployment signs. A record holding any other absolute URL, outside a `location` under the base URL, is refused. The long builds need -two specs the server has not built, so a local recording starts on a fresh -datasets database. `--scenarios-dir` writes outside the package, -unvalidated. The run ends by validating the whole contract with +two specs the server has not built, and the run picks each from the first +start year whose spec the server has not built, so each record moves both +start years up. `--scenarios-dir` writes outside the package, unvalidated. +The run ends by validating the whole contract with `tools/validate_client_contract.py`, and fails when it reports a problem, when the binary bodies pass 5 MB, or when the records and bodies together pass `SCENARIOS_BUDGET`, 16 MB. The outcome rules, error classes, gap kinds and long-build pacing come from `tools/validate_client_contract.py`, so a record's `expect` is written -by the code that checks it. pyarrow loads only when a Parquet body is read, so -the tests run without it. +by the code that checks it. """ from __future__ import annotations import argparse import copy -import io import json import os import re @@ -81,6 +80,7 @@ POLL_BACKOFF_CAP, POLL_BACKOFF_FIRST, POLL_PATH, + RETRY_AFTER_MAX_S, SCENARIOS_BUDGET, SIGNED_FETCH_STATUS, STORAGE_ORIGIN, @@ -89,6 +89,7 @@ Answer, ClientError, LongBuild, + ServedTable, absolute_url_problems, answer_from_body, answer_gaps, @@ -105,19 +106,23 @@ digest_of, download_target, failed_operation, + fetch_rendering, gap_kinds, header_warnings, held_spec, + load_reading_columns, + load_selector_keys, may_retry, newest_plain, owns_body_file, part_label, pick_artifact, posted_scope, + presented_columns, problem_body, read_from, refusal_error, - rendering_for, + resumed_rendering, resubmitted_sentence, retry_wait, scenarios_size, @@ -126,6 +131,7 @@ search_warns, selector_path, sentence_entry, + served_table, shared_cacheable, stored_version, surface_problems, @@ -142,8 +148,10 @@ REPOSITORY = "DevelopmentFinanceObservatory/apis" BINARY_BUDGET = 5_000_000 -# The rendering both clients ask for, bare long Parquet. -RENDERING = {"format": "parquet", "bundle": False} +# The rendering a fetch asks for by default, bare long Parquet with the party +# totals of open party roles left out, and so the rendering a ref it writes +# records. +RENDERING = fetch_rendering(None) # The response headers a client reads, and so the ones a record keeps, beside # a `cache-control` that marks the response shared-cacheable, which excuses it # from carrying `x-dfo-trace-id`. @@ -560,6 +568,12 @@ class Member: rows: int columns: list[str] rendering: dict[str, Any] = field(default_factory=lambda: dict(RENDERING)) + # The value of each column that holds one value in every row. + same: Mapping[str, Any] = field(default_factory=dict) + + @property + def table(self) -> ServedTable: + return ServedTable(tuple(self.columns), self.rows, self.same) @property def spec_id(self) -> str: @@ -577,13 +591,15 @@ def ref(self) -> dict[str, Any]: def dataset_outcome( members: Sequence[Member], + reading: Sequence[Sequence[str]], *, missing: Sequence[Mapping[str, Any]] = (), search_warning: bool | None = None, ) -> dict[str, Any]: """A dataset outcome. Several members stack into one long table with a `part` column first, holding each member's `part_label`, then every - member's columns in member order.""" + member's columns in member order. The columns are in the order the + reading-order rule gives.""" summaries = [ { "parts": member.parts, @@ -594,11 +610,10 @@ def dataset_outcome( for member in members ] dataset: dict[str, Any] = {"rows": sum(member.rows for member in members)} - if len(members) == 1: - dataset["columns"] = list(members[0].columns) - else: - columns = dict.fromkeys(c for m in members for c in m.columns) - dataset["columns"] = ["part", *columns] + dataset["columns"] = presented_columns( + [member.table for member in members], reading + ) + if len(members) > 1: dataset["part_rows"] = [ {"part": part_label(summary), "rows": member.rows} for summary, member in zip(summaries, members, strict=True) @@ -607,13 +622,6 @@ def dataset_outcome( return {"dataset": {**dataset, **summary}} -def parquet_shape(data: bytes) -> tuple[int, list[str]]: - import pyarrow.parquet as pq # noqa: PLC0415 - optional, so the tests run without it - - parquet = pq.ParquetFile(io.BytesIO(data)) - return parquet.metadata.num_rows, list(parquet.schema_arrow.names) - - def _reached[T](results: list[T], step: str) -> T: """The result a step produced. A step that refused cannot be acted on.""" if not results: @@ -635,25 +643,64 @@ class Minted: """The operation a mint started or a resume read: its resource, the headers that came with it, the build request a failed operation posts again (None for a resumed one), the spec the client holds, and the - rendering it downloads.""" + rendering the build posted, or for a resumed operation None and the + layout whose rendering `resumed_rendering` reads from the done + manifest.""" resource: dict[str, Any] headers: dict[str, str] request: dict[str, Any] | None spec: Spec | None - rendering: dict[str, Any] + rendering: dict[str, Any] | None + layout: str | None = None def _step_options( - step: dict[str, Any], *, layout: str | None, strict: bool + step: dict[str, Any], + *, + layout: str | None, + strict: bool, + party_totals: bool = False, ) -> dict[str, Any]: if layout is not None: step["layout"] = layout if strict: step["strict"] = True + if party_totals: + step["party_totals"] = True return step +@dataclass(frozen=True, slots=True) +class ClientRules: + """The rule tables a client reads from `rules/`: the reading-order rule's + reading slots and the file-selector rule's query keys.""" + + reading: Sequence[Sequence[str]] + selector_keys: Sequence[str] + + +def load_client_rules(package_dir: Path) -> ClientRules: + """The rule tables under `package_dir`, read before any request. + + Raises: + SystemExit: A rule file is missing or does not hold its table. + """ + reading = load_reading_columns(package_dir) + if reading is None: + raise SystemExit( + f"{package_dir / 'rules' / 'reading-order.json'} is missing or holds no " + "reading_columns, so no dataset's columns can be ordered" + ) + selector_keys = load_selector_keys(package_dir) + if selector_keys is None: + raise SystemExit( + f"{package_dir / 'rules' / 'file-selector.json'} is missing or holds no " + "query_keys, so no file selector query can be built" + ) + return ClientRules(reading, selector_keys) + + @dataclass class Session: """A client over a recorder: each step appends to `call` and `expect`.""" @@ -661,7 +708,8 @@ class Session: recorder: Recorder actions: Mapping[str, Mapping[str, Any]] client_codes: Mapping[str, Mapping[str, Any]] - read_shape: Callable[[bytes], tuple[int, list[str]]] = parquet_shape + rules: ClientRules + read_shape: Callable[[bytes], ServedTable] = served_table sleep: Callable[[float], None] = time.sleep call: list[dict[str, Any]] = field(default_factory=list) expect: list[dict[str, Any]] = field(default_factory=list) @@ -702,7 +750,7 @@ def _retried( `may_retry` allows, waiting `retry_wait` before each retry.""" response = send() answers = [_read(response)] - while may_retry(request, answers): + while may_retry(request, answers, self.actions): wait = retry_wait(response.headers, len(answers) - 1) self.retry_waits.append(wait) self.sleep(wait) @@ -765,11 +813,13 @@ def fetch_answer( partial: bool = False, strict: bool = False, layout: str | None = None, + party_totals: bool = False, ) -> None: - """Builds the answer's readings. A declined or catalogue answer, a gap - outside a partial fetch, a partial fetch with no reading, a strict - fetch whose readings depart from the question, and the frame layout - with several readings each raise before any request.""" + """Builds the answer's readings, leaving out party totals unless + `party_totals`. A declined or catalogue answer, a gap outside a + partial fetch, a partial fetch with no reading, a strict fetch whose + readings depart from the question, and the frame layout with several + readings each raise before any request.""" step: dict[str, Any] = {"op": "fetch"} if partial: step["partial"] = True @@ -787,16 +837,21 @@ def run() -> dict[str, Any]: ClientError(search_warning_code(answer), error="SearchWarning"), answer.trace_id, ) - rendering = rendering_for(layout) + rendering = fetch_rendering(layout, party_totals=party_totals) if "layout" in rendering and len(readings) > 1: raise ClientError("invalid_argument") members = [ self.build(parts, spec, None, rendering, strict=strict) for parts, spec in readings ] - return dataset_outcome(members, missing=gaps, search_warning=warns) + return dataset_outcome( + members, self.rules.reading, missing=gaps, search_warning=warns + ) - self._step(_step_options(step, layout=layout, strict=strict), run) + options = _step_options( + step, layout=layout, strict=strict, party_totals=party_totals + ) + self._step(options, run) def fetch_spec( self, @@ -807,10 +862,11 @@ def fetch_spec( facets: Mapping[str, str] | None = None, layout: str | None = None, strict: bool = False, + party_totals: bool = False, ) -> list[Member]: - """Builds `spec` narrowed by `window`, `entities` and `facets`. A - role in `entities` replaces the spec's binding of it, or drops it - when null.""" + """Builds `spec` narrowed by `window`, `entities` and `facets`, + leaving out party totals unless `party_totals`. A role in `entities` + replaces the spec's binding of it, or drops it when null.""" step: dict[str, Any] = {"op": "fetch", "spec": spec.step()} if window is not None: step["window"] = list(window) if isinstance(window, tuple) else window @@ -825,18 +881,23 @@ def fetch_spec( built: list[Member] = [] def run() -> dict[str, Any]: - rendering = rendering_for(layout) + rendering = fetch_rendering(layout, party_totals=party_totals) built.append(self.build([], spec, scope, rendering, strict=strict)) - return dataset_outcome(built) + return dataset_outcome(built, self.rules.reading) - self._step(_step_options(step, layout=layout, strict=strict), run) + options = _step_options( + step, layout=layout, strict=strict, party_totals=party_totals + ) + self._step(options, run) return built def fetch_ref(self, ref: dict[str, Any]) -> None: """The exact versions a ref names, through the lookup order.""" def run() -> dict[str, Any]: - return dataset_outcome([self._stored(member) for member in ref["members"]]) + return dataset_outcome( + [self._stored(member) for member in ref["members"]], self.rules.reading + ) self._step({"op": "fetch", "ref": ref}, run) @@ -851,11 +912,13 @@ def run() -> dict[str, Any]: self._step({"op": "newest_version", "spec": spec.step()}, run) return found[0] if found else None - def mint(self, spec: Spec, *, layout: str | None = None) -> str: + def mint( + self, spec: Spec, *, layout: str | None = None, party_totals: bool = False + ) -> str: operation: list[str] = [] def run() -> dict[str, Any]: - rendering = rendering_for(layout) + rendering = fetch_rendering(layout, party_totals=party_totals) body = _build_body(spec, None, rendering) response = self._api("POST", "/datasets", body) resource = self._acted(response) @@ -868,7 +931,10 @@ def run() -> dict[str, Any]: self._step( _step_options( - {"op": "mint", "spec": spec.step()}, layout=layout, strict=False + {"op": "mint", "spec": spec.step()}, + layout=layout, + strict=False, + party_totals=party_totals, ), run, ) @@ -881,9 +947,7 @@ def run() -> dict[str, Any]: path = POLL_PATH.format(operation_id=operation_id) response = self._api("GET", path) resource = self._acted(response) - self.minted = Minted( - resource, response.headers, None, None, rendering_for(layout) - ) + self.minted = Minted(resource, response.headers, None, None, None, layout) return _operation_outcome(resource) step = {"op": "operation", "operation_id": operation_id} @@ -897,8 +961,9 @@ def result(self) -> None: def run() -> dict[str, Any]: manifest = self._wait(minted.resource, minted.headers, minted.request) - member = self._member([], minted.spec, manifest, minted.rendering, False) - return dataset_outcome([member]) + rendering = minted.rendering or resumed_rendering(minted.layout, manifest) + member = self._member([], minted.spec, manifest, rendering, False) + return dataset_outcome([member], self.rules.reading) self._step({"op": "result"}, run) @@ -946,16 +1011,17 @@ def _member( held = None if spec is None else spec.step() sentence = self._identity_sentence(parts, definition, held) digest = artifact["artifact_digest"] - rows, columns = self.read_shape(self._download(route, digest)) + table = self.read_shape(self._download(route, digest)) return Member( parts, definition, manifest["content_version"], sentence, digest, - rows, - columns, + table.rows, + list(table.columns), dict(rendering), + table.same, ) def _identity_sentence( @@ -1055,16 +1121,17 @@ def stored_file( data, refusal = self._stored_file(member, manifest) if data is None: return None, refusal - rows, columns = self.read_shape(data) + table = self.read_shape(data) stored = Member( member["parts"], spec["definition"], version, member["sentence"], member["artifact"]["digest"], - rows, - columns, + table.rows, + list(table.columns), dict(rendering), + table.same, ) return stored, None @@ -1102,6 +1169,7 @@ def _stored_file( member["spec"]["spec_id"], member["content_version"], member["artifact"]["rendering"], + self.rules.selector_keys, ) try: return self._download(path, digest, origin=False), None @@ -1182,6 +1250,7 @@ def _build_body( FOLDED = "ODA disbursements for health and for education from Germany to Kenya, and GDP of Kenya" PLACE = "ODA disbursements from Germany to Congo in 2018" TWO_PLACES = "Kenya's GDP, and ODA disbursements from Germany to Congo" +PLACE_TWICE = "aid from Congo to Congo" DECLINED = "should donors give more aid to Kenya?" CATALOGUE = "what data do you have?" TOP5 = "top 5 donors to Ethiopia in 2022" @@ -1203,21 +1272,35 @@ def _build_body( CANDIDATES_BESIDE_ASSUMED = ( "mortality in Kenya and aid flows to Kenya from 2013 to 2022" ) -# Two specs just above the 100,000-row sync budget, one for the mint and one -# for the operation another process started: a version the server has built -# answers inline, so each long build needs a spec the server has not built. -# A flow side the question leaves unnamed reads as its source's whole total, so -# both sides name groups read as their members: about 3,700 donor and -# recipient pairs of one variable over 60 years. A variable broken down by -# sector would list thousands of variables in the done manifest. -ASYNC = ( - "ODA disbursements from the DAC countries to the G77 countries from 1963 to 2024" -) -ASYNC_RESUMED = ( - "ODA disbursements from the OECD countries to the G77 countries from 1966 to 2024" +# Two specs just above the 100,000-row sync budget, the DAC donors' for the +# mint and the OECD donors' for the operation another process started: a +# version the server has built answers inline, so each long build needs a spec +# the server has not built, and the run takes the first start year in +# LONG_BUILD_YEARS whose spec it has not. Each record builds both for good, so +# the next record starts the scan at the year after the one LONG_BUILD_RECORDS +# holds, and a later start year estimates fewer rows, so the scan stops at the +# first year below the band. A flow side the question leaves unnamed +# reads as its source's whole total, so both sides name groups read as their +# members: about 3,700 donor and recipient pairs of one variable over some 60 +# years. A variable broken down by sector would list thousands of variables in +# the done manifest. +LONG_BUILD = ( + "ODA disbursements from the {donors} countries to the G77 countries" + " from {year} to 2024" ) +LONG_BUILD_YEARS = range(1960, 2001) +# The record holding each donor group's last long build. +LONG_BUILD_RECORDS = {"DAC": "mint-async-build", "OECD": "operation-resume"} NARROW_WINDOW = (2020, 2023) REBOUND_ORIGIN = "country/FRA" +# The DAC donors' flow to the G77, narrowed to one recipient and year with its +# origin open: the DAC2a table lists the donor group totals beside their +# members, about 110 rows with them and 95 without. +PARTY_TOTALS_FLOW = ( + "ODA disbursements from the DAC countries to the G77 countries from 1967 to 2024" +) +PARTY_TOTALS_WINDOW = (2022, 2022) +PARTY_TOTALS_ENTITIES = {"origin": None, "destination": ["country/KEN"]} SYNC_ROWS = 100_000 @@ -1229,7 +1312,10 @@ class Context: transport: Transport actions: Mapping[str, Mapping[str, Any]] client_codes: Mapping[str, Mapping[str, Any]] - read_shape: Callable[[bytes], tuple[int, list[str]]] = parquet_shape + rules: ClientRules + read_shape: Callable[[bytes], ServedTable] = served_table + # The first start year `async_spec` tries for each donor group. + long_build_starts: Mapping[str, int] = field(default_factory=dict) _cache: dict[str, Any] = field(default_factory=dict) def session(self, name: str, **recorder: Any) -> Session: @@ -1237,6 +1323,7 @@ def session(self, name: str, **recorder: Any) -> Session: Recorder(name, self.base_url, self.transport, **recorder), self.actions, self.client_codes, + self.rules, read_shape=self.read_shape, ) @@ -1283,22 +1370,61 @@ def definite_ref(self) -> dict[str, Any]: self._cache["ref"] = {"format_version": 1, "members": [member.ref()]} return self._cache["ref"] - def async_spec(self, question: str = ASYNC) -> Spec: - """The spec `question` answers, whose estimate is just above the sync - budget.""" - (part,) = self.definite_answer(question).parts - rows = part["spec"]["resolved"].get("download", {}).get("rows") or 0 - require( - SYNC_ROWS < rows < 2 * SYNC_ROWS, - f"{rows} rows is just above the sync budget", - ) - return _part_spec(part["spec"]) + def async_spec(self, donors: str) -> Spec: + """The spec the LONG_BUILD question for `donors` answers at the first + start year in LONG_BUILD_YEARS, from `long_build_starts`, whose answer + is one definite part, whose estimate is just above the sync budget, + and whose spec no earlier year or other donors answered and the server + has not built. A year whose estimate is at or below the budget ends + the scan, since later years estimate fewer rows. + + Raises: + ShapeError: No year gives such a spec. + """ + picks: dict[str, Spec] = self._cache.setdefault("long builds", {}) + if donors not in picks: + session = self.session("prerequisite") + seen = {spec.spec_id for spec in picks.values()} + first = max(self.long_build_starts.get(donors, 0), LONG_BUILD_YEARS[0]) + for year in range(first, LONG_BUILD_YEARS[-1] + 1): + answer = session.search(LONG_BUILD.format(donors=donors, year=year)) + if answer.status != "definite" or len(answer.parts) != 1: + continue + (part,) = answer.parts + rows = part["spec"]["resolved"].get("download", {}).get("rows") + if rows is None: + continue + spec = _part_spec(part["spec"]) + if rows <= SYNC_ROWS: + raise ShapeError( + f"the {donors} donors have no long build left: from {year} " + f"the estimate, {rows:,} rows, is within the {SYNC_ROWS:,}-row " + "sync budget; change LONG_BUILD to a question with more rows" + ) + # A start year before the source's first year clips to the same + # spec as the years after it, so a spec already seen is built or + # is the other donors' pick. + if rows >= 2 * SYNC_ROWS or spec.spec_id in seen: + continue + seen.add(spec.spec_id) + # `_newest` and not `newest_version`, which reads a refused + # versions request as no version. + if session._newest(spec.spec_id) is None: + picks[donors] = spec + break + else: + raise ShapeError( + f"no start year from {first} to" + f" {LONG_BUILD_YEARS[-1]} gives the {donors} donors a spec" + " just above the sync budget that the server has not built" + ) + return picks[donors] def running_operation(self) -> str: """The id of a long build another process started.""" if "operation" not in self._cache: self._cache["operation"] = self.session("prerequisite").mint( - self.async_spec(ASYNC_RESUMED) + self.async_spec("OECD"), party_totals=True ) return self._cache["operation"] @@ -1388,6 +1514,36 @@ def run(s: Session, ctx: Context) -> None: return run +def _fetch_spec_party_totals(*, keep: bool) -> Callable[[Session, Context], None]: + """Fetches the PARTY_TOTALS_FLOW spec narrowed to PARTY_TOTALS_ENTITIES in + PARTY_TOTALS_WINDOW, keeping party totals when `keep`, and checks + against a fetch with the other setting that leaving them out drops rows + and answers a version of its own.""" + + def run(s: Session, ctx: Context) -> None: + spec = ctx.definite_spec(PARTY_TOTALS_FLOW) + + def fetched(session: Session, party_totals: bool) -> Member: + (member,) = session.fetch_spec( + spec, + window=PARTY_TOTALS_WINDOW, + entities=PARTY_TOTALS_ENTITIES, + party_totals=party_totals, + ) + return member + + member = fetched(s, keep) + other = fetched(ctx.session("prerequisite"), not keep) + kept, cut = (member, other) if keep else (other, member) + require(kept.rows > cut.rows, "leaving out party totals drops rows") + require( + kept.content_version != cut.content_version, + "the version without party totals is a version of its own", + ) + + return run + + def _fetch_spec_grouped( *, window: tuple[int, int] | None = None, @@ -1440,7 +1596,7 @@ def _fetch_ref_unknown_version(s: Session, ctx: Context) -> None: def _mint_async(s: Session, ctx: Context) -> None: - s.mint(ctx.async_spec(ASYNC)) + s.mint(ctx.async_spec("DAC"), party_totals=True) s.result() @@ -1482,6 +1638,35 @@ def _place_answer(s: Session, question: str, parts: int) -> Answer: return answer +def _search_place_twice(s: Session, ctx: Context) -> None: + """One entity_ambiguous part asking "Congo" for each end of the flow: two + groups with one text, each option and slot token naming its group.""" + answer = s.search(PLACE_TWICE) + require(len(answer.parts) == 1, "one part") + (part,) = answer.parts + no_data = part.get("no_data", {}) + require(no_data.get("reason") == "entity_ambiguous", "an entity_ambiguous part") + require( + [group["text"] for group in no_data.get("ambiguous_entities", [])] + == ["Congo", "Congo"], + 'two ambiguous groups, each "Congo"', + ) + require( + [(o["entity_choice"], o["label"]) for o in no_data.get("options", [])] + == [(choice, label) for choice in (0, 1) for label in PLACE_LABELS], + f"each group offers {PLACE_LABELS} under its own entity_choice", + ) + require( + [ + token["entity_choice"] + for token in part.get("sentence_plan") or [] + if token.get("entity_choice") is not None + ] + == [0, 1], + "one slot token per group", + ) + + def _search_place(question: str, parts: int) -> Callable[[Session, Context], None]: def run(s: Session, ctx: Context) -> None: _place_answer(s, question, parts) @@ -1677,6 +1862,21 @@ def _resubmit_placeholder(s: Session, ctx: Context) -> None: "a new spec_id bound to destination alone.", _fetch_spec_entities({"origin": None}), ), + Recorded( + "fetch-spec-party-totals-left-out", + "Fetching a spec of DAC2a ODA to Kenya in 2022 with its origin open posts " + "without_party_totals: true in the build's options, as every fetch does by default, so " + "the build leaves out the donor group totals beside their members and answers a " + "version of its own, with fewer rows than fetch-spec-party-totals-kept.", + _fetch_spec_party_totals(keep=False), + ), + Recorded( + "fetch-spec-party-totals-kept", + "Fetching the spec of fetch-spec-party-totals-left-out with party_totals=True posts no " + "without_party_totals, so the build keeps the donor group totals and answers the plain " + "version, with more rows.", + _fetch_spec_party_totals(keep=True), + ), Recorded( "fetch-spec-group-narrowed", "Fetching a spec whose destination is the least developed countries group with " @@ -1770,6 +1970,13 @@ def _resubmit_placeholder(s: Session, ctx: Context) -> None: "parts, each an entity_ambiguous choice gap with one option per country.", _search_place(TWO_PLACES, 2), ), + Recorded( + "search-place-ambiguous-twice", + "A flow naming one ambiguous place at both ends answers one no_data part with two " + "entity_ambiguous groups of the same text: each option and slot token names its group " + "by entity_choice, so a menu groups options by entity_choice, never by mentions.", + _search_place_twice, + ), Recorded( "choose-place-one-part", 'Choosing "Congo [DRC]" by label in a one-part answer posts that option\'s data_query ' @@ -1978,6 +2185,18 @@ def _retry_after_503(parent: dict[str, Any], ctx: Context) -> Derived: return _refused_first(parent, 0, refused) +def _retry_after_past_cap(parent: dict[str, Any], ctx: Context) -> Derived: + """A retryable 503 whose Retry-After is one second past RETRY_AFTER_MAX_S + answers the search, which raises it unretried.""" + first = parent["exchanges"][0] + headers = {"retry-after": str(int(RETRY_AFTER_MAX_S) + 1)} + refused = _refusal(first["request"], 503, _BUSY, headers) + error = refusal_error( + 503, _BUSY, ctx.actions, refused["response"]["headers"] + ).outcome(ctx.client_codes) + return Derived(copy.deepcopy(parent["call"]), [refused], [error]) + + def _proxy_429(parent: dict[str, Any], ctx: Context) -> Derived: first = parent["exchanges"][0] refused = _refusal( @@ -2339,6 +2558,7 @@ def _expired_file_selector(parent: dict[str, Any], ctx: Context) -> Derived: member["spec"]["spec_id"], member["content_version"], member["artifact"]["rendering"], + ctx.rules.selector_keys, ) selector = { "request": {"method": "GET", "path": path}, @@ -2359,8 +2579,9 @@ def _poll_index(exchanges: Sequence[dict[str, Any]]) -> int: def _async_fetch( change: Callable[[list[dict[str, Any]], dict[str, Any]], None] | None = None, ) -> Callable[[dict[str, Any], Context], Derived]: - """The parent's mint and result as one fetch of its spec, which makes the - same exchanges. `change` edits the exchanges and the fetch's outcome.""" + """The parent's mint and result as one fetch of its spec that keeps + party totals, which makes the same exchanges, since a mint posts the + plain rendering. `change` edits the exchanges and the fetch's outcome.""" def derive(parent: dict[str, Any], ctx: Context) -> Derived: mint, result = parent["call"] @@ -2375,9 +2596,12 @@ def derive(parent: dict[str, Any], ctx: Context) -> Derived: outcome = copy.deepcopy(parent["expect"][1]) if change is not None: change(exchanges, outcome["dataset"]["long_builds"][0]) - return Derived( - [{"op": "fetch", "spec": copy.deepcopy(mint["spec"])}], exchanges, [outcome] - ) + fetch = { + "op": "fetch", + "spec": copy.deepcopy(mint["spec"]), + "party_totals": True, + } + return Derived([fetch], exchanges, [outcome]) return derive @@ -2399,25 +2623,32 @@ def _accepted_running(exchanges: list[dict[str, Any]], build: dict[str, Any]) -> exchanges[0]["response"]["json"]["operation"]["status"] = "running" -# The retry_after_ms each poll that finds the build running reads in -# fetch-spec-poll-pacing-changes. The values go up, up again, then down. +# The retry_after_ms the polls that find the build running read in +# fetch-spec-poll-pacing-changes, in turn. The values go up, up again, then down. _CHANGING_PACING_MS = (2500, 4000, 1000) def _pacing_changes(exchanges: list[dict[str, Any]], build: dict[str, Any]) -> None: """Each poll that finds the build running reads its own retry_after_ms from _CHANGING_PACING_MS, and the wait before the next poll is that - poll's value.""" - running = [ - e["response"]["json"]["operation"] - for e in exchanges[_poll_index(exchanges) :] + poll's value. The recording's running polls past that many are left + out, since how many a build takes depends on the server's load.""" + polls = [ + i + for i, e in enumerate(exchanges) if e["request"].get("path", "").startswith("/datasets/operations/") and e["response"]["json"]["operation"]["status"] == "running" ] require( - len(running) == len(_CHANGING_PACING_MS), + len(polls) >= len(_CHANGING_PACING_MS), f"{len(_CHANGING_PACING_MS)} polls find the build running", ) + for index in reversed(polls[len(_CHANGING_PACING_MS) :]): + del exchanges[index] + running = [ + exchanges[i]["response"]["json"]["operation"] + for i in polls[: len(_CHANGING_PACING_MS)] + ] for operation, ms in zip(running, _CHANGING_PACING_MS, strict=True): operation["retry_after_ms"] = ms build["poll_waits"][1:] = [ms / 1000 for ms in _CHANGING_PACING_MS] @@ -2661,6 +2892,7 @@ def _artifact_unlisted(parent: dict[str, Any], ctx: Context) -> Derived: member["spec"]["spec_id"], member["content_version"], member["artifact"]["rendering"], + ctx.rules.selector_keys, ) exchanges[index]["request"] = {"method": "GET", "path": path} return _same(parent, exchanges) @@ -2790,10 +3022,10 @@ def derive(parent: dict[str, Any], ctx: Context) -> Derived: return derive -# The x-dfo-trace-id header the Search responses of `answer-timed-out` and -# `choose-refused-shared-label` carry. The first's answer_incomplete carries -# it, and the second's choice_invalid, which describes the caller's choice, -# does not. +# The x-dfo-trace-id header the Search responses of `answer-timed-out`, +# `answer-timed-out-keeps-reasons` and `choose-refused-shared-label` carry. The +# first two's answer_incomplete carries it, and the third's choice_invalid, +# which describes the caller's choice, does not. ANSWER_TRACE_ID = "5" * 32 @@ -2801,6 +3033,49 @@ def _timed_out(body: dict[str, Any]) -> None: body["diagnostics"]["timed_out"] = True +# The no_data message dfo-apis renders for the reason `timed_out`. +TIMED_OUT_MESSAGE = ( + "Search reached its time limit before answering this part. " + "Try again, or ask a narrower question." +) + + +def _part_timed_out(body: dict[str, Any]) -> None: + """Part 1 is the part Search left unresolved at its time limit: reason + timed_out and the message that reason renders, which is also its sentence + and its whole heading in the answer's composed sentence.""" + part = body["parts"][1] + part["no_data"]["reason"] = "timed_out" + part["no_data"]["message"] = TIMED_OUT_MESSAGE + part["rendered_sentence"] = TIMED_OUT_MESSAGE + (token,) = part["sentence_plan"] + token["text"] = TIMED_OUT_MESSAGE + plan = body["sentence_plan"] + indexes = [i for i, tok in enumerate(plan) if tok.get("part_index") == 1] + first, last = indexes[0], indexes[-1] + require(indexes == list(range(first, last + 1)), "part 1's tokens are adjacent") + heading = "".join(tok["text"] for tok in plan[first : last + 1]) + require(body["rendered_sentence"].endswith(heading), "part 1's tokens end it") + plan[first : last + 1] = [ + { + "kind": "text", + "text": TIMED_OUT_MESSAGE, + "segment": "heading", + "entity_refs": [], + "states": [], + "part_index": 1, + } + ] + body["rendered_sentence"] = ( + body["rendered_sentence"][: -len(heading)] + TIMED_OUT_MESSAGE + ) + + +def _timed_out_part(body: dict[str, Any]) -> None: + _timed_out(body) + _part_timed_out(body) + + def _unsupported(body: dict[str, Any]) -> None: part = body["parts"][1] del part["no_data"] @@ -3464,6 +3739,14 @@ def _empty_narrowing(parent: dict[str, Any], ctx: Context) -> Derived: "search then reaches the same answer; the retry spends one of the two default retries.", _retry_after_503, ), + Synthetic( + "search-retry-after-past-cap", + "search-place-ambiguous", + "A 503 service_unavailable with retryable true and Retry-After 61 answers the search.", + "A retryable refusal whose Retry-After is past the 60-second cap raises at once, with " + "no wait and no retry: ServerError with its code, service_unavailable, and its trace id.", + _retry_after_past_cap, + ), Synthetic( "search-proxy-rate-limited", "search-place-ambiguous", @@ -3639,7 +3922,8 @@ def _empty_narrowing(parent: dict[str, Any], ctx: Context) -> Derived: Synthetic( "fetch-spec-async-build", "mint-async-build", - "The mint and result steps become one fetch of the spec, which makes the same exchanges.", + "The mint and result steps become one fetch of the spec with party_totals, which " + "makes the same exchanges.", "Fetching a spec whose build answers 202 polls the operation: it waits the 202's " "retry_after_ms, then each poll's, backing off from 1 to 8 seconds when a poll gives " "none, and downloads the done manifest's file.", @@ -4118,11 +4402,23 @@ def _empty_narrowing(parent: dict[str, Any], ctx: Context) -> Derived: Synthetic( "answer-timed-out", "answer-mixed-fetch-raises", - "The answer's diagnostics.timed_out becomes true, and its response carries " + "The answer's diagnostics.timed_out becomes true, part 1's no_data reason becomes " + "timed_out with the message that reason renders, and its response carries " f"x-dfo-trace-id {ANSWER_TRACE_ID}.", - "A no_data part of an answer whose diagnostics say Search timed out is a timed_out gap, " - "whatever its reason, and fetching the answer raises IncompleteAnswerError, which " - "describes the answer and carries its response's trace id.", + "A no_data part with reason timed_out, the reason Search gives a part it left " + "unresolved at its time limit, is a timed_out gap, and fetching the answer raises " + "IncompleteAnswerError, which describes the answer and carries its response's trace " + "id.", + _answer_edit(_timed_out_part, {"x-dfo-trace-id": ANSWER_TRACE_ID}), + ), + Synthetic( + "answer-timed-out-keeps-reasons", + "answer-mixed-fetch-raises", + "The answer's diagnostics.timed_out becomes true, part 1 keeps its reason " + f"variable_not_resolved, and its response carries x-dfo-trace-id {ANSWER_TRACE_ID}.", + "A no_data part's gap kind comes from its own reason, whatever diagnostics.timed_out " + "says. A part Search resolved before its time limit keeps its reason's gap kind, and " + "fetching the answer raises IncompleteAnswerError.", _answer_edit(_timed_out, {"x-dfo-trace-id": ANSWER_TRACE_ID}), ), Synthetic( @@ -4207,6 +4503,15 @@ def _empty_narrowing(parent: dict[str, Any], ctx: Context) -> Derived: "request.", _choice_refused("Congo [Nowhere]"), ), + Synthetic( + "choose-refused-label-in-two-groups", + "search-place-ambiguous-twice", + 'A choice of "Congo [DRC]" follows the search.', + "Both entity_choice groups offer an option labelled Congo [DRC], so the label " + "matches two options and the choice raises InvalidRequestError with no request. " + "A choice in such a part names its option by index.", + _choice_refused("Congo [DRC]"), + ), Synthetic( "choose-candidate-by-label", "answer-candidates-choose-fetch", @@ -4452,6 +4757,7 @@ def drive(session: Session, call: Sequence[dict[str, Any]]) -> None: facets=step.get("facets"), layout=step.get("layout"), strict=step.get("strict", False), + party_totals=step.get("party_totals", False), ) case {"op": "fetch", "ref": ref}: session.fetch_ref(ref) @@ -4465,11 +4771,13 @@ def drive(session: Session, call: Sequence[dict[str, Any]]) -> None: partial=step.get("partial", False), strict=step.get("strict", False), layout=step.get("layout"), + party_totals=step.get("party_totals", False), ) case {"op": "mint", "spec": spec}: session.mint( _step_spec(spec), layout=step.get("layout"), + party_totals=step.get("party_totals", False), ) case {"op": "operation", "operation_id": operation_id}: session.resume(operation_id, layout=step.get("layout")) @@ -4489,6 +4797,7 @@ def rederive_recorded(name: str, ctx: Context, directory: Path) -> None: Recorder(name, served.base_url, served), ctx.actions, ctx.client_codes, + ctx.rules, read_shape=ctx.read_shape, sleep=lambda _: None, ) @@ -4508,6 +4817,37 @@ def rederive_recorded(name: str, ctx: Context, directory: Path) -> None: ) +def recorded_start_year(record: Mapping[str, Any]) -> int | None: + """The start year of the first window a record's spec definitions + request: a step's spec, then a manifest an exchange answers.""" + definitions = [ + step["spec"]["definition"] for step in record["call"] if "spec" in step + ] + for exchange in record["exchanges"]: + body = exchange["response"].get("json") + manifest = body.get("manifest", body) if isinstance(body, dict) else None + if isinstance(manifest, dict) and "definition" in manifest: + definitions.append(manifest["definition"]) + for definition in definitions: + window = definition.get("identity", {}).get("requested_window") or {} + if "start_year" in window: + return window["start_year"] + return None + + +def long_build_starts(directory: Path) -> dict[str, int]: + """The first start year to try for each donor group: the year after the + one its LONG_BUILD_RECORDS record built, which the server has built.""" + starts = {} + for donors, name in LONG_BUILD_RECORDS.items(): + path = directory / f"{name}.json" + if path.exists(): + year = recorded_start_year(json.loads(path.read_text(encoding="utf-8"))) + if year is not None: + starts[donors] = year + 1 + return starts + + def run_record(args: argparse.Namespace) -> int: package_dir: Path = args.package_dir directory: Path = args.scenarios_dir or package_dir / "scenarios" @@ -4521,7 +4861,18 @@ def run_record(args: argparse.Namespace) -> int: _offline if args.offline else urllib_transport, load_actions(package_dir), load_client_codes(package_dir), + load_client_rules(package_dir), + long_build_starts=long_build_starts(directory), ) + if not args.offline: + # A long build's spec is the scan most likely to fail, so it runs + # before any scenario is recorded. + for donors, name in LONG_BUILD_RECORDS.items(): + if name in chosen: + try: + ctx.async_spec(donors) + except ShapeError as exc: + raise SystemExit(f"{name}: {exc}") from exc count = 0 for definition in RECORDED: if definition.name not in chosen: diff --git a/tools/tests/test_capture_client_contract.py b/tools/tests/test_capture_client_contract.py index f51bc5d..ba71bdf 100644 --- a/tools/tests/test_capture_client_contract.py +++ b/tools/tests/test_capture_client_contract.py @@ -2,12 +2,12 @@ The recorder's client reproduces each packaged record from the record's own steps and responses, every derived record follows from its parent, and the -writers keep the kit's JSON formatting. No test reaches the network or needs -pyarrow: a fake transport serves the recorded responses, and the Parquet shape -reader is replaced, so rows and columns are the one part of a dataset outcome -these tests leave to the recording. +writers keep the kit's JSON formatting. No test reaches the network: a fake +transport serves the recorded responses, and the Parquet shape reader is +replaced, so rows and columns are the one part of a dataset outcome these +tests leave to the recording. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q """ from __future__ import annotations @@ -32,6 +32,7 @@ SCENARIOS = capture.PACKAGE_DIR / "scenarios" ACTIONS = capture.load_actions(capture.PACKAGE_DIR) CLIENT_CODES = capture.load_client_codes(capture.PACKAGE_DIR) +RULES = capture.load_client_rules(capture.PACKAGE_DIR) def _load(name: str) -> dict[str, Any]: @@ -46,8 +47,8 @@ def _records(kind: str) -> list[str]: ) -def _shape(data: bytes) -> tuple[int, list[str]]: - return len(data), ["column"] +def _shape(data: bytes) -> capture.ServedTable: + return capture.ServedTable(("column",), len(data), {}) def _without_shapes(expect: list[dict[str, Any]]) -> list[dict[str, Any]]: @@ -75,7 +76,7 @@ def test_the_recorder_reproduces_each_record_from_its_steps(name: str) -> None: served = capture.Served(record, SCENARIOS) recorder = capture.Recorder(name, served.base_url, served) session = capture.Session( - recorder, ACTIONS, CLIENT_CODES, read_shape=_shape, sleep=lambda _: None + recorder, ACTIONS, CLIENT_CODES, RULES, read_shape=_shape, sleep=lambda _: None ) capture.drive(session, record["call"]) assert served.pending == [], "the client skipped a recorded exchange" @@ -236,7 +237,11 @@ def test_each_derived_record_follows_from_its_parent(name: str) -> None: definition = next(d for d in capture.SYNTHETIC if d.name == name) parent = _load(record["origin"]["derived_from"]) ctx = capture.Context( - capture.DEV_BASE_URL, capture.Served(parent, SCENARIOS), ACTIONS, CLIENT_CODES + capture.DEV_BASE_URL, + capture.Served(parent, SCENARIOS), + ACTIONS, + CLIENT_CODES, + RULES, ) derived = definition.derive(copy.deepcopy(parent), ctx) assert record["origin"]["edit"] == definition.edit @@ -253,7 +258,12 @@ def test_an_offline_rederivation_rewrites_only_the_expect(tmp_path: Path) -> Non stale["expect"][1]["error"]["code"] = None capture.write_record(tmp_path, stale, {}) ctx = capture.Context( - capture.DEV_BASE_URL, capture._offline, ACTIONS, CLIENT_CODES, read_shape=_shape + capture.DEV_BASE_URL, + capture._offline, + ACTIONS, + CLIENT_CODES, + RULES, + read_shape=_shape, ) capture.rederive_recorded(name, ctx, tmp_path) fresh = (tmp_path / f"{name}.json").read_text(encoding="utf-8") @@ -287,7 +297,12 @@ def test_an_offline_rederivation_refuses_a_body_file_outside_its_directory( exchange["response"]["body_file"] = body_file record.write_text(json.dumps(stale), encoding="utf-8") ctx = capture.Context( - capture.DEV_BASE_URL, capture._offline, ACTIONS, CLIENT_CODES, read_shape=_shape + capture.DEV_BASE_URL, + capture._offline, + ACTIONS, + CLIENT_CODES, + RULES, + read_shape=_shape, ) with pytest.raises(SystemExit) as refused: capture.rederive_recorded(name, ctx, scenarios) @@ -342,7 +357,7 @@ def transport(method: str, url: str, **kwargs: Any) -> Any: raise AssertionError(f"{method} {url} reached the network") definition = capture.Recorded("s", "None.", run) - ctx = capture.Context(capture.DEV_BASE_URL, transport, ACTIONS, CLIENT_CODES) + ctx = capture.Context(capture.DEV_BASE_URL, transport, ACTIONS, CLIENT_CODES, RULES) with pytest.raises(SystemExit, match="1 were canned"): capture.record_recorded(definition, ctx, tmp_path) assert list(tmp_path.iterdir()) == [] @@ -358,7 +373,7 @@ def transport(method: str, url: str, **kwargs: Any) -> Any: raise AssertionError(f"{method} {url} reached the network") definition = capture.Synthetic("s", "parent", "None.", "None.", derive) - ctx = capture.Context(capture.DEV_BASE_URL, transport, ACTIONS, CLIENT_CODES) + ctx = capture.Context(capture.DEV_BASE_URL, transport, ACTIONS, CLIENT_CODES, RULES) with pytest.raises(SystemExit, match="makes no request"): capture.record_synthetic(definition, ctx, tmp_path) @@ -379,22 +394,24 @@ def test_a_renamed_strict_field_ends_the_call_at_the_build() -> None: @pytest.mark.parametrize( - ("timed_out", "status", "expected"), + ("timed_out", "status", "reason", "expected"), [ - (False, "no_data", "not_understood"), - (True, "no_data", "timed_out"), - (False, "unsupported", "unsupported"), - (False, "deferred", "unknown"), + (False, "no_data", "variable_not_resolved", "not_understood"), + (True, "no_data", "variable_not_resolved", "not_understood"), + (False, "no_data", "timed_out", "timed_out"), + (False, "unsupported", "variable_not_resolved", "unsupported"), + (False, "deferred", "variable_not_resolved", "unknown"), ], ) def test_a_part_without_a_reading_is_a_gap_of_its_kind( - timed_out: bool, status: str, expected: str + timed_out: bool, status: str, reason: str, expected: str ) -> None: body = copy.deepcopy( _load("answer-mixed-fetch-raises")["exchanges"][0]["response"]["json"] ) body["diagnostics"]["timed_out"] = timed_out body["parts"][1]["status"] = status + body["parts"][1]["no_data"]["reason"] = reason gaps = capture.answer_gaps( capture.answer_from_body(body), capture.gap_kinds(ACTIONS) ) @@ -402,24 +419,39 @@ def test_a_part_without_a_reading_is_a_gap_of_its_kind( def test_several_members_stack_with_a_part_column_first() -> None: + """The stacked columns follow `part` in the reading-order rule's order, + the reading columns first and `stat_var` last.""" member = capture.Member( - [], {"spec_id": "spec_a"}, "cv_a", "A.", "sha256:a", 2, ["x", "y"] + [], + {"spec_id": "spec_a"}, + "cv_a", + "A.", + "sha256:a", + 2, + ["stat_var", "x", "value"], ) other = capture.Member( [], {"spec_id": "spec_b"}, "cv_b", "B.", "sha256:b", 3, ["y", "z"] ) unnamed = capture.Member( - ["p", "q"], {"spec_id": "spec_b"}, "cv_b", "B.", "sha256:b", 3, ["y", "z"] + ["p", "q"], + {"spec_id": "spec_b"}, + "cv_b", + "B.", + "sha256:b", + 3, + ["value", "time_period"], ) - dataset = capture.dataset_outcome([member, unnamed])["dataset"] - assert dataset["columns"] == ["part", "x", "y", "z"] + dataset = capture.dataset_outcome([member, unnamed], RULES.reading)["dataset"] + assert dataset["columns"] == ["part", "time_period", "value", "x", "stat_var"] assert dataset["rows"] == 5 assert dataset["part_rows"] == [ {"part": "A.", "rows": 2}, {"part": "p; q", "rows": 3}, ] - assert "part_rows" not in capture.dataset_outcome([other])["dataset"] - assert capture.dataset_outcome([member])["dataset"]["columns"] == ["x", "y"] + assert "part_rows" not in capture.dataset_outcome([other], RULES.reading)["dataset"] + columns = capture.dataset_outcome([member], RULES.reading)["dataset"]["columns"] + assert columns == ["value", "x", "stat_var"] @pytest.mark.parametrize( @@ -534,3 +566,176 @@ def fake_run(args: list[str], **kwargs: Any) -> Any: assert {a for a in command if a.startswith("--extra=")} == { f"--extra={e}" for e in ("eval", "api", "engine", "datasets", "graph") } + + +@pytest.mark.parametrize("rule", ["reading-order", "file-selector"]) +def test_a_record_run_without_a_rule_table_stops_before_any_request( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, rule: str +) -> None: + """The rule tables load from --package-dir when the run starts, so a + broken one costs no live call.""" + for name in ("actions.json", "client-codes.json"): + (tmp_path / name).write_bytes((capture.PACKAGE_DIR / name).read_bytes()) + (tmp_path / "rules").mkdir() + for path in (capture.PACKAGE_DIR / "rules").glob("*.json"): + if path.stem != rule: + (tmp_path / "rules" / path.name).write_bytes(path.read_bytes()) + + def transport(method: str, url: str, **kwargs: Any) -> Any: + raise AssertionError(f"{method} {url} reached the network") + + monkeypatch.setattr(capture, "urllib_transport", transport) + with pytest.raises(SystemExit, match=f"{rule}.json is missing"): + capture.main( + ["--package-dir", str(tmp_path), "record", "--only", "answer-declined"] + ) + + +def _long_build_ctx( + monkeypatch: pytest.MonkeyPatch, + answers: Mapping[tuple[str, int], tuple[str, int | None]], + built: set[str], + starts: Mapping[str, int] | None = None, +) -> tuple[Any, list[str], list[str]]: + """A context whose searches answer `answers`, (donors, year) to spec id + and row estimate, and whose versions route lists a version of each spec + in `built`, with the questions searched and the spec ids probed.""" + questions: list[str] = [] + probed: list[str] = [] + + def search(self: Any, question: str) -> Any: + questions.append(question) + for (donors, year), (spec_id, rows) in answers.items(): + if question == capture.LONG_BUILD.format(donors=donors, year=year): + spec = { + "spec_id": spec_id, + "definition": {"spec_id": spec_id}, + "resolved": {"download": {"rows": rows}}, + } + return capture.Answer({}, [{"spec": spec}], None, "definite") + return capture.Answer({}, [], None, "declined") + + def newest(self: Any, spec_id: str) -> str | None: + probed.append(spec_id) + return "cv_built" if spec_id in built else None + + monkeypatch.setattr(capture.Session, "search", search) + monkeypatch.setattr(capture.Session, "_newest", newest) + ctx = capture.Context( + capture.DEV_BASE_URL, + capture._offline, + ACTIONS, + CLIENT_CODES, + RULES, + long_build_starts=starts or {}, + ) + return ctx, questions, probed + + +def test_a_long_build_takes_the_first_unbuilt_spec_in_the_band( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A year outside the row band, a built spec and a spec an earlier year + answered are passed over, and each pick is searched for once.""" + in_band = capture.SYNC_ROWS + 1 + ctx, questions, probed = _long_build_ctx( + monkeypatch, + { + ("DAC", 1960): ("spec_large", 2 * capture.SYNC_ROWS), + ("DAC", 1961): ("spec_built", in_band), + ("DAC", 1962): ("spec_built", in_band), + ("DAC", 1963): ("spec_dac", in_band), + ("OECD", 1960): ("spec_dac", in_band), + ("OECD", 1961): ("spec_oecd", in_band), + }, + {"spec_built"}, + ) + assert ctx.async_spec("DAC").spec_id == "spec_dac" + assert ctx.async_spec("OECD").spec_id == "spec_oecd" + assert ctx.async_spec("DAC").spec_id == "spec_dac" + assert probed == ["spec_built", "spec_dac", "spec_oecd"] + assert len(questions) == 6 + + +def test_a_long_build_with_no_unbuilt_year_is_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(capture, "LONG_BUILD_YEARS", range(1960, 1962)) + in_band = capture.SYNC_ROWS + 1 + ctx, _, _ = _long_build_ctx( + monkeypatch, + {("DAC", 1960): ("spec_a", in_band), ("DAC", 1961): ("spec_b", in_band)}, + {"spec_a", "spec_b"}, + ) + with pytest.raises(capture.ShapeError, match="from 1960 to 1961"): + ctx.async_spec("DAC") + + +def test_a_long_build_scan_stops_at_the_first_year_within_the_budget( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A later start year estimates fewer rows, so none after it can be in + the band, and the scan stops without searching them.""" + ctx, questions, _ = _long_build_ctx( + monkeypatch, + { + ("DAC", 1960): ("spec_a", capture.SYNC_ROWS), + ("DAC", 1961): ("spec_b", capture.SYNC_ROWS + 1), + }, + set(), + ) + with pytest.raises(capture.ShapeError, match="no long build left: from 1960"): + ctx.async_spec("DAC") + assert len(questions) == 1 + + +def test_a_long_build_scan_starts_after_the_recorded_year( + monkeypatch: pytest.MonkeyPatch, +) -> None: + in_band = capture.SYNC_ROWS + 1 + ctx, questions, _ = _long_build_ctx( + monkeypatch, + {("DAC", 1960): ("spec_a", in_band), ("DAC", 1968): ("spec_b", in_band)}, + set(), + starts={"DAC": 1968}, + ) + assert ctx.async_spec("DAC").spec_id == "spec_b" + assert questions == [capture.LONG_BUILD.format(donors="DAC", year=1968)] + + +def test_the_long_build_scans_start_after_the_recorded_picks() -> None: + """The records hold the DAC pick at 1967 and the OECD pick at 1968.""" + assert capture.long_build_starts(SCENARIOS) == {"DAC": 1968, "OECD": 1969} + + +def test_a_long_build_scan_skips_a_year_without_an_estimate( + monkeypatch: pytest.MonkeyPatch, +) -> None: + ctx, _, _ = _long_build_ctx( + monkeypatch, + { + ("DAC", 1960): ("spec_a", None), + ("DAC", 1961): ("spec_b", capture.SYNC_ROWS + 1), + }, + set(), + ) + assert ctx.async_spec("DAC").spec_id == "spec_b" + + +def test_the_running_operation_keeps_party_totals( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The operation another process starts posts the plain rendering, which + the recorded resume and result read.""" + ctx, _, _ = _long_build_ctx( + monkeypatch, {("OECD", 1960): ("spec_oecd", capture.SYNC_ROWS + 1)}, set() + ) + minted: list[dict[str, Any]] = [] + + def mint(self: Any, spec: Any, **options: Any) -> str: + minted.append(options) + return "op_x" + + monkeypatch.setattr(capture.Session, "mint", mint) + assert ctx.running_operation() == "op_x" + assert minted == [{"party_totals": True}] diff --git a/tools/tests/test_validate_client_contract.py b/tools/tests/test_validate_client_contract.py index 89a9859..700a78e 100644 --- a/tools/tests/test_validate_client_contract.py +++ b/tools/tests/test_validate_client_contract.py @@ -11,7 +11,7 @@ bodies included, which takes about a third of a second. The copy hard-links the package's files, so it writes no data where the file system allows. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q """ from __future__ import annotations @@ -21,12 +21,15 @@ import json import os import shutil +import struct import sys from collections.abc import Callable from pathlib import Path from typing import Any from urllib.parse import urlsplit +import pyarrow as pa +import pyarrow.parquet as pq import pytest _MODULE_PATH = Path(__file__).resolve().parent.parent / "validate_client_contract.py" @@ -93,6 +96,17 @@ def _edit(path: Path, change: Callable[[Any], None]) -> None: path.write_text(json.dumps(document), encoding="utf-8") +def _comparison_member(document: Any, name: str, side: str, index: int) -> Any: + """A member of the output-comparison vector `name` on `side`.""" + vector = next(vector for vector in document["vectors"] if vector["name"] == name) + return vector[side]["members"][index] + + +def _digest_vector(document: Any, name: str) -> Any: + """The output-digest vector `name`.""" + return next(vector for vector in document["vectors"] if vector["name"] == name) + + def test_the_real_contract_is_valid(package_dir: Path) -> None: assert validator.validate() == [] assert validator.main() == 0 @@ -156,6 +170,64 @@ def _part_rows(change: Callable[[Any], None]) -> Callable[[Any], None]: return _set_outcome(1, lambda o: change(o["dataset"])) +def _swap_columns(first: str, second: str) -> Callable[[Any], None]: + def change(outcome: Any) -> None: + columns = outcome["dataset"]["columns"] + i, j = columns.index(first), columns.index(second) + columns[i], columns[j] = columns[j], columns[i] + + return change + + +def _selector_query_with_cut(doc: Any) -> None: + """The file selector request asks for the ref member's whole rendering, + `without_party_totals` included, which the server refuses.""" + exchange = next( + e for e in doc["exchanges"] if "/file?" in e["request"].get("path", "") + ) + exchange["request"]["path"] += "&without_party_totals=true" + + +def _time_period_written_as_text(vector: Any) -> None: + """The second member serves its years as text, a valid Arrow column of + its own that mixes types with the first member's in the stacked table.""" + member = vector["members"][1] + at = member["columns"].index("time_period") + for row in member["rows"]: + row[at] = str(row[at]) + + +def _vector(name: str, change: Callable[[Any], None]) -> Callable[[Any], None]: + return lambda doc: change(next(v for v in doc["vectors"] if v["name"] == name)) + + +def _allowance_exhausted(doc: Any) -> None: + """The first search's refusal becomes a 429 `allowance_exhausted`, which + a client raises unretried whatever its document's `retryable` says.""" + response = doc["exchanges"][0]["response"] + response["status"] = 429 + response["json"].update( + code="allowance_exhausted", status=429, type="/codes/allowance_exhausted" + ) + + +def _rename_series_left_out(doc: Any) -> None: + vectors = doc["$defs"]["series_left_out"]["properties"]["vectors"] + vectors["items"]["properties"]["warnings"]["items"]["enum"] = ["series_omitted"] + + +def _served_order_after_reading_columns(holder: Any) -> None: + """Moves the varying co-operation modality pair back to its served place, + just before `crsPurpose`, an order that ignores whether a column + varies.""" + columns = holder["dataset"]["columns"] if "dataset" in holder else holder["columns"] + varying = ["crsCooperationModality", "crsCooperationModality_label"] + for name in varying: + columns.remove(name) + at = columns.index("crsPurpose") + columns[at:at] = varying + + def _set_step(index: int, **fields: Any) -> Callable[[Any], None]: return lambda doc: doc["call"][index].update(fields) @@ -715,6 +787,11 @@ def change(doc: Any) -> None: _first_gap(kind="not_understood"), "step 0 expects", ), + ( + "scenarios/answer-timed-out-keeps-reasons.json", + _first_gap(kind="timed_out"), + "step 0 expects", + ), ( "scenarios/answer-unsupported-part.json", _first_gap(kind="unknown"), @@ -887,7 +964,7 @@ def change(doc: Any) -> None: lambda doc: doc["ref"]["members"][0].update( content_version="cv_" + "0" * 32 ), - "member 0 content_version is not fetch-spec-known-sentence's", + "member 0 content_version is not fetch-spec-party-totals-kept's", ), ( "refs/ref-one-member.json", @@ -899,6 +976,119 @@ def change(doc: Any) -> None: lambda doc: doc.pop("scenario"), "names no scenario that serves its members", ), + ( + "refs/ref-answer-member.json", + lambda doc: doc["ref"]["members"][0].update(parts=[]), + "refs/ref-answer-member.json: member 0 parts is not answer-definite-fetch's", + ), + ( + "refs/ref-without-party-totals.json", + lambda doc: doc["ref"]["members"][0]["artifact"]["rendering"].pop( + "without_party_totals" + ), + "refs/ref-without-party-totals.json: member 0 records rendering", + ), + ( + "refs/ref-derived-member.json", + lambda doc: doc["ref"]["members"][0]["roots"][0].update( + content_version="cv_" + "0" * 32 + ), + "member 0 root 0 is no member fetch-spec-known-sentence serves", + ), + ( + "refs/ref-derived-member.json", + lambda doc: doc["ref"]["members"][0]["roots"][0]["rendering"].pop( + "without_party_totals" + ), + "refs/ref-derived-member.json: member 0 root 0 records rendering", + ), + ( + "refs/ref-derived-member.json", + lambda doc: doc["ref"]["members"][0]["roots"][0]["rendering"].pop("layout"), + "refs/ref-derived-member.json: member 0 root 0 records rendering", + ), + ( + "refs/ref-derived-member.json", + lambda doc: doc["ref"]["members"][0]["resolved_periods"][0].update( + period="2019" + ), + "share replay_key but differ in resolved_periods", + ), + ( + "rules/output-comparison.json", + lambda doc: _comparison_member( + doc, "member-count-differs", "right", 1 + ).update( + content_version=_comparison_member( + doc, "member-count-differs", "right", 0 + )["content_version"] + ), + "share content_version but differ in spec_id", + ), + ( + "rules/output-comparison.json", + lambda doc: _comparison_member( + doc, "derived-file-format-differs", "right", 0 + )["artifact"].update( + digest=_comparison_member( + doc, "derived-file-format-differs", "left", 0 + )["artifact"]["digest"] + ), + "share artifact_digest but differ in artifact_format", + ), + ( + "rules/output-comparison.json", + lambda doc: _comparison_member(doc, "same-derived-ref", "right", 0)[ + "roots" + ][0]["rendering"].update( + groupby={"by": ["time_period"], "op": "sum", "levels": {}} + ), + "share content_version but differ in roots", + ), + ( + "rules/output-digest.json", + lambda doc: doc["vectors"][0]["frame"]["rows"][0].__setitem__( + 0, "dcid:Count_Person" + ), + "vector 'two-rows' document is not its frame's", + ), + ( + "rules/output-digest.json", + lambda doc: doc["vectors"][0].update(digest=doc["vectors"][1]["digest"]), + "vector 'two-rows' digest is not its document's", + ), + ( + "rules/output-digest.json", + lambda doc: next( + vector + for vector in doc["vectors"] + if vector["name"] == "quotient-of-sides-swapped" + ).update( + variables=next( + vector + for vector in doc["vectors"] + if vector["name"] == "quotient-of-variable-named" + )["variables"] + ), + "vector 'quotient-of-sides-swapped' document is not its frame's", + ), + ( + "rules/output-digest.json", + lambda doc: _digest_vector(doc, "quotient-of-a-sum-of")[ + "name_documents" + ].pop("mv_ae90cefe711d4addb439f873f2010dba"), + "mv_ae90cefe711d4addb439f873f2010dba has no name document", + ), + ( + "rules/output-digest.json", + lambda doc: _digest_vector(doc, "quotient-of-variable-named")[ + "name_documents" + ].update( + mv_d355e59c50ff8cf07e86a4da12b64297='{"frame_ref":["dcid:Amount_ODA"],' + '"verb":"quotient_of"}' + ), + "mv_d355e59c50ff8cf07e86a4da12b64297 is not its document's", + ), ( "rules/key-format.json", lambda doc: doc["vectors"].append(doc["vectors"][0]), @@ -1244,6 +1434,11 @@ def change(doc: Any) -> None: _drop_action("route_deprecated"), "no row for 'route_deprecated'", ), + ( + "warning-codes.json", + _drop_action("series_left_out"), + "no row for 'series_left_out'", + ), ( "warning-codes.json", _add_action( @@ -1648,11 +1843,6 @@ def change(doc: Any) -> None: _mentions_a_string, "the steps never make exchanges 1 to 1", ), - ( - "scenarios/answer-timed-out.json", - _set_body(0, lambda body: body.update(diagnostics="timed out")), - "step 0 expects", - ), ( "scenarios/answer-candidates-choose-fetch.json", _set_body( @@ -1663,6 +1853,206 @@ def change(doc: Any) -> None: ), "the step follows no answer", ), + ( + _FETCH, + lambda doc: doc["exchanges"][0]["request"]["json"]["options"].pop( + "without_party_totals" + ), + f"{_FETCH}: the exchanges do not fit the steps: _UnfitError('exchange 0 posts", + ), + ( + "scenarios/fetch-spec-party-totals-kept.json", + lambda doc: doc["call"][0].pop("party_totals"), + ( + "scenarios/fetch-spec-party-totals-kept.json: the exchanges do not fit " + "the steps: _UnfitError('exchange 0 posts" + ), + ), + ( + "scenarios/mint-async-build.json", + lambda doc: doc["call"][0].pop("party_totals"), + ( + "scenarios/mint-async-build.json: the exchanges do not fit the " + "steps: _UnfitError('exchange 0 posts" + ), + ), + ( + _FETCH, + _set_outcome(0, lambda o: o["dataset"]["columns"].reverse()), + f"{_FETCH}: step 0: columns", + ), + ( + _FOLDED, + _part_rows(lambda d: d["columns"].append(d["columns"].pop(1))), + f"{_FOLDED}: step 1: columns", + ), + ( + "scenarios/fetch-ref.json", + _set_outcome(0, _swap_columns("currency", "base_year")), + "scenarios/fetch-ref.json: step 0: columns", + ), + ( + "scenarios/fetch-ref.json", + _set_outcome(0, lambda o: o["dataset"]["columns"].remove("price_basis")), + "scenarios/fetch-ref.json: step 0: columns", + ), + ( + "scenarios/answer-definite-fetch.json", + _set_outcome(1, _served_order_after_reading_columns), + "scenarios/answer-definite-fetch.json: step 1: columns", + ), + ( + "rules/reading-order.json", + _vector( + "varying-column-follows-reading-columns", + _served_order_after_reading_columns, + ), + ( + "vector 'varying-column-follows-reading-columns' orders its " + "columns or rows otherwise" + ), + ), + ( + "rules/reading-order.json", + _vector("stack-crs-and-dac2a", lambda v: v["columns"].remove("part")), + "vector 'stack-crs-and-dac2a' orders its columns or rows otherwise", + ), + ( + "rules/reading-order.json", + _vector( + "ties-keep-served-order", + lambda v: v["rows"].insert(1, v["rows"].pop(2)), + ), + "vector 'ties-keep-served-order' orders its columns or rows otherwise", + ), + ( + "rules/reading-order.json", + _vector( + "varying-column-follows-reading-columns", + lambda v: v["rows"].insert(1, v["rows"].pop(2)), + ), + ( + "vector 'varying-column-follows-reading-columns' orders its " + "columns or rows otherwise" + ), + ), + ( + "rules/reading-order.json", + _vector("null-sorts-last", lambda v: v["rows"].insert(0, v["rows"].pop())), + "vector 'null-sorts-last' orders its columns or rows otherwise", + ), + ( + "rules/reading-order.json", + lambda doc: doc.pop("reading_columns"), + ( + "rules/reading-order.json: missing or unreadable, or holds no " + "reading_columns" + ), + ), + ( + "rules/operation-outputs.json", + _vector( + "growth-as-a-fraction", + lambda v: v["arguments"].update(window=["2018", "2018"]), + ), + "vector 'growth-as-a-fraction' has a window that does not start before", + ), + ( + "rules/rules.schema.json", + _rename_series_left_out, + "warning-codes.json: no row for 'series_omitted'", + ), + ( + "refs/ref-derived-member.json", + lambda doc: doc["ref"]["members"][0].update(parts=["ODA to Kenya"]), + "refs/ref-derived-member.json: ref fails the schema at members/0/parts", + ), + ( + "scenarios/fetch-ref.json", + lambda doc: doc["call"][0]["ref"].update(format_version=2), + "scenarios/fetch-ref.json: schema: call/0/ref/format_version: 1 was", + ), + ( + "scenarios/search-retry-after-past-cap.json", + lambda doc: doc["exchanges"][0]["response"]["headers"].update( + {"retry-after": "60"} + ), + "the record holds no retry of POST /search/resolve", + ), + ( + "scenarios/search-retry-after-503.json", + _allowance_exhausted, + "scenarios/search-retry-after-503.json: the steps never make exchanges", + ), + ( + "rules/reading-order.json", + _vector( + "ties-keep-served-order", lambda v: v["members"][0]["rows"][0].pop() + ), + "vector 'ties-keep-served-order' member 0 has a row of", + ), + ( + "rules/reading-order.json", + _vector( + "nested-cells-compare-by-value", + lambda v: v["members"][0]["rows"][1].__setitem__(4, [1]), + ), + "vector 'nested-cells-compare-by-value' member 0 column 'flags' is no " + "Arrow column", + ), + ( + "rules/reading-order.json", + _vector( + "nested-cells-compare-by-value", + lambda v: v["members"][0]["rows"][1].__setitem__(2, {"a": "x", "b": 2}), + ), + "vector 'nested-cells-compare-by-value' member 0 column 'meta' is no " + "Arrow column", + ), + ( + "rules/reading-order.json", + _vector("stack-crs-and-dac2a", _time_period_written_as_text), + "vector 'stack-crs-and-dac2a' mixes JSON types in column", + ), + ( + "scenarios/fetch-ref-expired-file-selector.json", + _selector_query_with_cut, + ( + "scenarios/fetch-ref-expired-file-selector.json: the exchanges do " + "not fit the steps" + ), + ), + ( + "scenarios/fetch-ref-artifact-unlisted.json", + _selector_query_with_cut, + ( + "scenarios/fetch-ref-artifact-unlisted.json: the exchanges do not " + "fit the steps" + ), + ), + ( + "rules/file-selector.json", + _vector( + "long-cut", + lambda v: v.update( + query="format=parquet&bundle=false&without_party_totals=true" + ), + ), + "vector 'long-cut' gives a query other than selector_query's", + ), + ( + "rules/file-selector.json", + _vector( + "keys-in-listed-order", + lambda v: v.update(query="layout=frame&bundle=true&format=csv"), + ), + "vector 'keys-in-listed-order' gives a query other than selector_query's", + ), + ( + "rules/file-selector.json", + lambda doc: doc.pop("query_keys"), + "rules/file-selector.json: missing or unreadable, or holds no query_keys", + ), ], ids=[ "strict-pointer-unresolved", @@ -1703,6 +2093,7 @@ def change(doc: Any) -> None: "strict-fetch-made-lenient", "warn-notice-raises-on-strict", "expect-wrong-timed-out-gap", + "expect-timed-out-gap-over-the-reason", "expect-wrong-unsupported-gap", "known-kind-raises-nothing", "partial-fetch-nothing-answered", @@ -1739,6 +2130,20 @@ def change(doc: Any) -> None: "ref-content-version-stale", "ref-member-not-served", "ref-without-scenario", + "ref-parts-not-served", + "ref-rendering-not-posted", + "ref-derived-root-not-served", + "ref-derived-root-rendering-not-posted", + "ref-derived-root-not-in-frame-layout", + "ref-derived-replay-with-other-periods", + "derived-version-with-other-op-spec", + "derived-file-in-two-formats", + "derived-version-with-other-root-rendering", + "output-digest-document-not-frames", + "output-digest-not-documents", + "output-digest-variable-not-named", + "output-digest-inner-name-without-document", + "output-digest-name-not-its-documents", "rule-vector-repeated", "client-code-without-condition", "client-codes-unsorted", @@ -1804,6 +2209,7 @@ def change(doc: Any) -> None: "proxy-code-is-a-registry-code", "warning-code-ref-digest-changed-missing", "warning-code-missing", + "warning-code-series-left-out-missing", "warning-code-no-rule-issues", "warning-code-is-a-client-code", "location-localhost-alias", @@ -1900,8 +2306,36 @@ def change(doc: Any) -> None: "no-data-options-null-counted", "bound-entities-as-objects-compared", "mentions-as-a-string-matched", - "diagnostics-not-an-object-read", "candidate-spec-not-an-object-read", + "party-totals-left-in-by-default", + "party-totals-kept-without-the-option", + "mint-party-totals-left-in-by-default", + "columns-out-of-reading-order", + "stacked-columns-out-of-reading-order", + "served-order-of-other-columns-swapped", + "served-column-dropped", + "constant-columns-before-varying-ones", + "reading-order-vector-in-served-order", + "reading-order-stacked-vector-without-part", + "reading-order-ties-reordered", + "reading-order-series-interleaved-by-year", + "reading-order-null-sorted-first", + "reading-order-without-reading-columns", + "growth-window-that-does-not-advance", + "series-left-out-warning-renamed-in-its-rule", + "derived-member-with-parts", + "fetch-of-a-format-2-ref", + "retry-after-at-the-cap-waited", + "allowance-exhausted-not-retried", + "reading-order-vector-row-ragged", + "reading-order-vector-list-items-mix-types", + "reading-order-vector-struct-field-mixes-types", + "reading-order-vector-column-mixes-types", + "selector-query-carries-the-cut", + "unlisted-artifact-selector-query-carries-the-cut", + "file-selector-vector-carries-the-cut", + "file-selector-vector-out-of-key-order", + "file-selector-without-query-keys", ], ) def test_a_broken_contract_is_reported( @@ -1915,6 +2349,132 @@ def test_a_broken_contract_is_reported( assert any(expected in problem for problem in problems), problems +@pytest.mark.parametrize( + ("left", "right", "same"), + # The same cases as python/client-contract/tests/test_rules.py. + [ + (1, 1.0, True), + (True, 1, False), + (float("nan"), float("nan"), True), + (None, None, True), + ("1", 1, False), + ([True], [1], False), + ([float("nan")], [float("nan")], True), + ([1], [1.0], True), + ({"a": 1, "b": 2}, {"b": 2, "a": 1.0}, True), + ({"a": True}, {"a": 1}, False), + ], +) +def test_cells_are_the_same_value_as_json_values( + left: Any, right: Any, same: bool +) -> None: + assert validator.same_value(left, right) is same + table = validator.table_from_rows(["c"], [[left], [right]]) + assert ("c" in table.same) is same + + +def _parquet(table: pa.Table) -> bytes: + sink = pa.BufferOutputStream() + pq.write_table(table, sink) + return sink.getvalue().to_pybytes() + + +def _nan(payload: int) -> float: + return struct.unpack(" None: + """Cells `same_value` holds the same, which Arrow's `unique` tells apart, + are one value on both paths.""" + rows = [[2021 - i, "x", cell] for i, cell in enumerate(cells)] + columns = ["time_period", "label", "amount"] + vector = validator.table_from_rows(columns, rows) + served = validator.served_table( + _parquet( + pa.table({name: [row[i] for row in rows] for i, name in enumerate(columns)}) + ) + ) + assert "amount" in served.same + assert validator.column_groups([served], []) == validator.column_groups( + [vector], [] + ) + + +def test_a_served_table_reads_list_struct_and_nan_columns() -> None: + """Columns `unique` cannot hash are judged cell by cell, and a NaN column + holds one value, in one table and over stacked ones.""" + data = _parquet( + pa.table( + { + "listed": [[1, 2], [1, 2]], + "struct": [{"a": 1}, {"a": 2}], + "nan": [float("nan"), float("nan")], + } + ) + ) + table = validator.served_table(data) + assert set(table.same) == {"listed", "nan"} + assert table.nested == {"listed", "struct"} + groups = validator.column_groups([table], []) + assert groups.varying == ("struct",) and groups.sort_keys == [] + assert not validator._varies("nan", [table, table]) + assert validator._varies("struct", [table]) + + +def test_a_spec_built_in_two_renderings_keys_each_by_its_file() -> None: + """A ref member's rendering is checked against the build that answered + its file, so each of a spec's two builds keeps its own rendering.""" + cut = {"format": "parquet", "bundle": False, "without_party_totals": True} + plain = {"format": "parquet", "bundle": False} + + def build(options: dict[str, Any], digest: str) -> dict[str, Any]: + artifact = {"artifact_digest": digest, "rendering": options} + return { + "request": { + "method": "POST", + "path": "/datasets", + "json": {"options": options}, + }, + "response": { + "status": 200, + "json": {"definition": {"spec_id": "spec_a"}, "artifacts": [artifact]}, + }, + } + + scenario = {"exchanges": [build(cut, "sha256:a"), build(plain, "sha256:b")]} + posted = validator.posted_renderings(scenario) + assert posted == {"sha256:a": [cut], "sha256:b": [plain]} + + +def test_a_selector_query_writes_only_booleans_in_lower_case() -> None: + query = validator.selector_query( + {"format": "CSV", "bundle": True}, ["format", "bundle"] + ) + assert query == "format=CSV&bundle=true" + + +def test_a_resumed_result_takes_the_rendering_its_build_posted() -> None: + cut = {"format": "parquet", "bundle": False, "without_party_totals": True} + manifest = {"artifacts": [{"artifact_digest": "sha256:a", "rendering": cut}]} + assert validator.resumed_rendering(None, manifest) == cut + manifest["artifacts"][0]["rendering"] = {"format": "parquet", "bundle": False} + assert validator.resumed_rendering(None, manifest) == validator.rendering_for(None) + + +@pytest.mark.parametrize("rule", ["reading-order", "file-selector"]) +def test_a_missing_rule_table_is_reported(package_dir: Path, rule: str) -> None: + (package_dir / "rules" / f"{rule}.json").unlink() + assert any( + problem.startswith(f"rules/{rule}.json: missing") + for problem in validator.validate() + ) + + def test_a_record_that_holds_only_under_its_recorded_host_is_reported( package_dir: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -1949,6 +2509,36 @@ def change(doc: Any) -> None: assert not [p for p in problems if _CODELESS in p], problems +_DEGRADED = {"code": "engine_degraded", "severity": "degraded", "message": "slow"} + + +@pytest.mark.parametrize( + ("diagnostics", "warns"), + [ + ("timed out", False), + ([_DEGRADED], False), + ({"warnings": [_DEGRADED]}, True), + ], + ids=["a-string", "a-list-of-warnings", "an-object-as-control"], +) +def test_diagnostics_that_is_not_an_object_holds_no_warnings( + package_dir: Path, diagnostics: Any, warns: bool +) -> None: + """The served schema rejects the body, which is a separate problem. The + replay reads the answer, and a fetch that expects no SearchWarning holds + when diagnostics is a string or a list, even a list of warnings, and + differs from the expectation when it is an object whose warnings read.""" + target = "scenarios/answer-definite-fetch.json" + _edit( + package_dir / target, + _set_body(0, lambda body: body.update(diagnostics=diagnostics)), + ) + replayed = [ + p for p in validator.validate() if target in p and "response body:" not in p + ] + assert bool(replayed) is warns, replayed + + def test_one_record_that_crashes_its_derivation_is_its_own_problem( package_dir: Path, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/tools/tests/test_validate_descriptions_vectors.py b/tools/tests/test_validate_descriptions_vectors.py index 4cf4e6b..f41ef47 100644 --- a/tools/tests/test_validate_descriptions_vectors.py +++ b/tools/tests/test_validate_descriptions_vectors.py @@ -5,7 +5,7 @@ does not get the schema verdict it states, and an operation list that breaks a rule the operation schema cannot express, must each be reported. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q """ from __future__ import annotations @@ -124,7 +124,25 @@ def test_an_operation_listed_twice_is_reported(package_dir: Path) -> None: document["operations"].append(document["operations"][1]) path.write_text(json.dumps(document)) - assert validator.validate() == ["operations.json: op 'sum' is listed twice"] + assert validator.validate() == [ + "operations.json: op 'sum' is listed twice", + "operations.json: verb 'sum_of' is listed twice", + ] + + +def test_arguments_outside_the_roles_are_reported(package_dir: Path) -> None: + path = package_dir / "schemas" / "operations.json" + document = json.loads(path.read_text()) + compare = next(e for e in document["operations"] if e["op"] == "compare") + compare["arguments"]["data"] = "numerator" + path.write_text(json.dumps(document)) + + assert validator.validate() == [ + ( + "operations.json: compare: arguments' data and other must fill its " + "two roles ['left', 'right']" + ) + ] def test_a_missing_analysis_operation_is_reported(package_dir: Path) -> None: diff --git a/tools/tests/test_validate_provenance_docs_vectors.py b/tools/tests/test_validate_provenance_docs_vectors.py index 929de7b..567e7bb 100644 --- a/tools/tests/test_validate_provenance_docs_vectors.py +++ b/tools/tests/test_validate_provenance_docs_vectors.py @@ -4,7 +4,7 @@ express, so those checks must read it without raising, and the validator must report every diagnostic and exit 1. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q """ from __future__ import annotations diff --git a/tools/tests/test_validate_registry.py b/tools/tests/test_validate_registry.py index 4d504f2..2255e44 100644 --- a/tools/tests/test_validate_registry.py +++ b/tools/tests/test_validate_registry.py @@ -14,7 +14,7 @@ Also tests check_entries' precondition for that matching: an entry's `mappings` may not repeat an `(endpoint, channel)` pair. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q (importing tools/validate_registry.py imports jsonschema at module scope, so a plain interpreter needs it on the path; `--with` provisions it for the one invocation without a project-level dependency). diff --git a/tools/tests/test_validate_unit_registry.py b/tools/tests/test_validate_unit_registry.py index dd71b87..0cc8097 100644 --- a/tools/tests/test_validate_unit_registry.py +++ b/tools/tests/test_validate_unit_registry.py @@ -1,7 +1,7 @@ """Tests for tools/validate_unit_registry.py: each mutation of the real registry is one rule the schema or the row checks must refuse. -Run with: uv run --with jsonschema --with pytest pytest tools/tests -q +Run with: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q """ from __future__ import annotations diff --git a/tools/validate_client_contract.py b/tools/validate_client_contract.py index 604ba3b..9f1502f 100644 --- a/tools/validate_client_contract.py +++ b/tools/validate_client_contract.py @@ -2,6 +2,7 @@ # requires-python = ">=3.12" # dependencies = [ # "jsonschema>=4.23", +# "pyarrow>=15", # ] # /// """Validate the client contract under @@ -35,8 +36,8 @@ - `proxy-codes.json` and `warning-codes.json` list codes in order, each once, none of them a code another table or `registries/codes.json` holds, and the warning codes are those the client rules issue: `ref_digest_changed`, - `route_deprecated`, `server_warning`, and one `key_` per key-format - reason. + `route_deprecated`, `server_warning`, one `key_` per key-format + reason, and each warning the series-left-out rule's vectors name. - Each scenario's file name matches its `name`, names are unique, `expect` holds one outcome per step, a step that acts on an earlier result follows one, a synthetic scenario derives from a recorded one, and every body file @@ -60,14 +61,17 @@ `client-codes.json`, as `expected_outcomes` derives them, the waits before each poll included, under the record's base URL and again under REPLAY_ORIGIN with the recorded origin a response names moved there. A - dataset's rows, columns and per-part row counts are left to the Parquet - bodies, and each download hashes to its artifact's digest or the outcome - says it does not. Each Search request a + dataset's columns are the downloaded files' columns, read with pyarrow + by `served_table` and ordered by `presented_columns`, which puts the + columns whose values vary, as `same_value` judges cells, before those + that hold one value, its rows and + per-part row counts are left to the Parquet bodies, and each + download hashes to its artifact's digest or the outcome says it does not. Each Search request a record holds posts the input the steps derive: the question, a choice's data_query or dataset_resubmit entries, or a member's identity with the entity groups of the roles its build left bound as they were, and each build posts the body `build_body` gives for the spec the step builds, its - narrowing and its layout, and each Search request the whole body + narrowing, its layout and, for a fetch or a mint, its party totals, and each Search request the whole body `search_body` gives. A POST carries its body, a refused request repeated is a retry only as `may_retry` allows, a record that ends a run of retries while `may_retry` holds does not fit, each poll names the operation the @@ -82,13 +86,28 @@ SCENARIOS_BUDGET bytes. - Each ref example's file name matches its `name` and gets the verdict it states: from `dataset-ref.schema.json`, then from the rules that each - member's definition carries its own `spec_id` and that members hold + dataset member's definition carries its own `spec_id` and that members hold distinct `spec_id`s. A ref example whose ref holds members names the - recorded `scenario` that serves them, and each member's `spec_id`, + recorded `scenario` that serves them, and each member's `spec_id`, `parts`, `content_version` and `sentence` are a dataset member's of that scenario, and its artifact digest is the digest of a file the scenario downloads. + Each root of a derived member is a dataset member's `spec_id` and + `content_version` of that scenario. Each member of an example the schema + accepts records the rendering the scenario's build of its spec posted, and + each root of a derived one that rendering in the frame layout. +- The derived members of the ref examples the schema accepts and of the + output-comparison vectors agree wherever DERIVED_IDENTITY_RULES says one + identity fixes another, so one `ocv_` names one op-spec and one output. - Each rule file under `rules/` validates against `rules.schema.json`, names - the rule its file name names, and names each vector once. + the rule its file name names, and names each vector once, each + reading-order vector's `columns` and `rows` are what `presented_rows` + gives, each file-selector vector's `query` is what `selector_query` + gives, each output-digest vector's document holds its frame's rows, each + operation's variable read as the name its `variables` give it, and + its digest hashes that document, each minted name hashes its name + document and each name it or a name document holds has one, and each + growth vector's window starts before it ends. A missing or unreadable + `rules/reading-order.json` or `rules/file-selector.json` is a problem. It does not import `dfo_client_contract`, whose tests replay the rule vectors. `tools/capture_client_contract.py` imports the outcome @@ -99,16 +118,19 @@ import hashlib import json +import math import re import sys -from collections.abc import Callable, Iterator, Mapping, Sequence +from collections.abc import Callable, Collection, Iterator, Mapping, Sequence from dataclasses import dataclass, field, replace -from functools import partial +from functools import lru_cache, partial from pathlib import Path from typing import Any from urllib.parse import quote, urljoin, urlsplit, urlunsplit import jsonschema +import pyarrow as pa +import pyarrow.parquet as pq from referencing import Registry, Resource from referencing.jsonschema import DRAFT202012 @@ -490,18 +512,20 @@ def _check_other_codes( def _check_warning_codes( document: dict[str, Any], taken: Mapping[str, set[str]], - key_reasons: Sequence[str], + rules_schema: Mapping[str, Any], ) -> list[str]: """As `_check_other_codes`, and the codes are those a client's rules - issue: `ref_digest_changed`, each header warning, and one `key_` - warning per key-format reason.""" + issue: `ref_digest_changed`, each header warning, one `key_` + warning per key-format reason, and each warning the series-left-out + rule issues.""" codes = [row["code"] for row in document["codes"]] problems = _check_other_codes("warning-codes.json", codes, taken) wanted = { REF_DIGEST_CHANGED, ROUTE_DEPRECATED, SERVER_WARNING, - *(f"key_{r}" for r in key_reasons), + *(f"key_{r}" for r in key_format_reasons(rules_schema)), + *series_left_out_warnings(rules_schema), } problems.extend( f"warning-codes.json: no row for {code!r}, a warning a client issues" @@ -520,6 +544,13 @@ def key_format_reasons(rules_schema: Mapping[str, Any]) -> list[str]: return vectors["items"]["properties"]["reason"]["enum"] +def series_left_out_warnings(rules_schema: Mapping[str, Any]) -> set[str]: + """The warnings the series-left-out rule issues, or raises under strict.""" + vectors = rules_schema["$defs"]["series_left_out"]["properties"]["vectors"] + members = vectors["items"]["properties"] + return {*members["warnings"]["items"]["enum"], *members["raises"]["enum"]} + + # The class of every error the key-lookup rule raises. KEY_LOOKUP_CLASS = "AuthenticationError" @@ -760,6 +791,7 @@ def _check_proxy_problem(label: str, response: Mapping[str, Any]) -> list[str]: "take_newest_version": "VersionUnavailableError", "log_in": "AuthenticationError", "request_scope": "PermissionDeniedError", + "stop": "PermissionDeniedError", "wait": "RateLimitError", "wait_for_reset": "QuotaExceededError", "report": "ServerError", @@ -781,22 +813,33 @@ def _check_proxy_problem(label: str, response: Mapping[str, Any]) -> list[str]: RETRYABLE_STATUSES = frozenset({408, 429, 500, 502, 503, 504}) # The retries a client makes by default after a retryable refusal. MAX_RETRIES = 2 +# The longest `Retry-After`, in seconds, a client waits before it tries again. +# A refusal that asks for longer raises at once. +RETRY_AFTER_MAX_S = 60.0 +# The action of the refusals a client never retries, whatever their document +# and status say: a spent allowance clears only when it resets. +NOT_RETRIED_ACTION = "wait_for_reset" # The status that cuts a synchronous build off at the server's limit. BUILD_CUT_OFF = 504 def may_retry( - request: Mapping[str, Any], responses: Sequence[Mapping[str, Any]] + request: Mapping[str, Any], + responses: Sequence[Mapping[str, Any]], + actions: Mapping[str, Mapping[str, Any]], ) -> bool: """Whether a client sends `request` again after `responses`, the answers - to it so far in order: the last is a retryable refusal, the retries made - stay under MAX_RETRIES, and for a build, no earlier answer was a 504 - cut-off, after which the build is posted once more only. So a build - refused 504 then 503 raises the 503, and one refused 503 then 504 is - posted once more after the 504.""" + to it so far in order: the last is a refusal `tries_again` allows, the + retries made stay under MAX_RETRIES, and for a build, no earlier answer + was a 504 cut-off, after which the build is posted once more only. So a + build refused 504 then 503 raises the 503, and one refused 503 then 504 + is posted once more after the 504.""" *earlier, response = responses status = response["status"] - if status < 400 or not retryable(status, problem_body(response)): + headers = response.get("headers", {}) + if status < 400 or not tries_again( + status, problem_body(response), headers, actions + ): return False if len(earlier) >= MAX_RETRIES: return False @@ -841,14 +884,32 @@ def error_class( return STATUS_CLASSES.get(status, fallback) -def retryable(status: int, body: Any) -> bool: - """Whether a client retries a refusal: as its problem document's - `retryable` says, and otherwise by its status.""" +def retryable(status: int, body: Any, actions: Mapping[str, Mapping[str, Any]]) -> bool: + """Whether a client retries a refusal: never when its code's action is + NOT_RETRIED_ACTION, else as its problem document's `retryable` says, and + otherwise by its status.""" + row = actions.get(refusal_code(body) or "") + if row is not None and row["action"] == NOT_RETRIED_ACTION: + return False if isinstance(body, dict) and isinstance(body.get("retryable"), bool): return body["retryable"] return status in RETRYABLE_STATUSES +def tries_again( + status: int, + body: Any, + headers: Mapping[str, str], + actions: Mapping[str, Mapping[str, Any]], +) -> bool: + """Whether a client sends a refused request again, a poll included: the + refusal is `retryable` and its `Retry-After`, when it carries one, is at + most RETRY_AFTER_MAX_S.""" + wait = retry_after(headers) + over_cap = wait is not None and wait > RETRY_AFTER_MAX_S + return retryable(status, body, actions) and not over_cap + + def retry_after(headers: Mapping[str, str]) -> float | None: """The seconds a `Retry-After` header asks for, or None.""" try: @@ -1530,11 +1591,8 @@ def search_warning_code(answer: Answer) -> str | None: def answer_gaps(answer: Answer, kinds: Mapping[str, str]) -> list[dict[str, Any]]: """Each part without a reading. A candidates part is a choice among its - specs, a no_data part takes its reason's gap kind, or `timed_out` when the - answer timed out, an unsupported part is an `unsupported` gap, and a part - of unknown status is an `unknown` gap.""" - diagnostics = read_object(answer.body, "diagnostics") - timed_out = diagnostics is not None and diagnostics.get("timed_out") is True + specs, a no_data part takes its reason's gap kind, an unsupported part is + an `unsupported` gap, and a part of unknown status is an `unknown` gap.""" gaps = [] for index, part in enumerate(answer.parts): match part.get("status"): @@ -1544,11 +1602,7 @@ def answer_gaps(answer: Answer, kinds: Mapping[str, str]) -> list[dict[str, Any] kind, options = "choice", len(part["candidates"]["specs"]) case "no_data": no_data = part["no_data"] - kind = ( - "timed_out" - if timed_out - else kinds.get(no_data["reason"], "no_data") - ) + kind = kinds.get(no_data["reason"], "no_data") options = len(read_objects(no_data, "options")) case "unsupported": kind, options = "unsupported", 0 @@ -1773,6 +1827,15 @@ class _UnfitError(Exception): "long": {"format": "parquet", "bundle": False}, "frame": {"format": "parquet", "bundle": False, "layout": "frame"}, } +# The rendering member a fetch posts as true unless the caller keeps party +# totals, which leaves out the group totals of the identity's open party roles. +WITHOUT_PARTY_TOTALS = "without_party_totals" +# The column a client presents last, the column a stacked dataset puts +# first, the measure no row sorts by, and the sort key that comes last. +STAT_VAR = "stat_var" +PART = "part" +VALUE = "value" +TIME_PERIOD = "time_period" # The snapshot refusal after which a stored version's lookup builds it again. # The version exists, but its file has expired. Any other refusal ends it. FILE_EXPIRED = "artifact_expired" @@ -1788,10 +1851,261 @@ class _UnfitError(Exception): def rendering_for(layout: str | None) -> dict[str, Any]: - """The build request's `options` for a fetch's `layout`, long by default.""" + """The rendering for `layout`, long by default, with party totals + kept.""" return dict(LAYOUT_RENDERINGS[layout or "long"]) +def fetch_rendering( + layout: str | None, *, party_totals: bool = False +) -> dict[str, Any]: + """The build request's `options` for a fetch of an answer or a spec, or + a mint: the layout's rendering, with `without_party_totals` unless the + caller keeps party totals.""" + rendering = rendering_for(layout) + if not party_totals: + rendering[WITHOUT_PARTY_TOTALS] = True + return rendering + + +def resumed_rendering( + layout: str | None, manifest: Mapping[str, Any] +) -> dict[str, Any]: + """The rendering a resumed operation's build posted, as its done + manifest records it: the layout's rendering, with `without_party_totals` + when the artifact for that rendering carries the cut.""" + rendering = rendering_for(layout) + artifact = pick_artifact(manifest, rendering) + found = artifact.get("rendering") if artifact is not None else None + if isinstance(found, dict) and found.get(WITHOUT_PARTY_TOTALS) is True: + rendering[WITHOUT_PARTY_TOTALS] = True + return rendering + + +@dataclass(frozen=True, slots=True) +class ServedTable: + """A served table as the reading-order rule reads it: its columns in the + order the server wrote them, its row count, and the value of each column + that holds one value in every row. A column `same` leaves out varies. + `nested` names the columns holding lists or structs, which no row is + sorted by.""" + + columns: tuple[str, ...] + rows: int + same: Mapping[str, Any] + nested: frozenset[str] = frozenset() + + +def same_value(left: Any, right: Any) -> bool: + """Whether two cells hold the same value, as the reading-order rule + judges it: numbers by value, so 1 and 1.0 are the same, booleans apart + from numbers, NaN the same as NaN, null the same as null, lists item by + item in order, and structs field by field in any order.""" + if isinstance(left, bool) or isinstance(right, bool): + return type(left) is type(right) and left == right + if isinstance(left, float) and isinstance(right, float): + return left == right or (math.isnan(left) and math.isnan(right)) + if isinstance(left, list) and isinstance(right, list): + return len(left) == len(right) and all(map(same_value, left, right)) + if isinstance(left, dict) and isinstance(right, dict): + return left.keys() == right.keys() and all( + same_value(value, right[key]) for key, value in left.items() + ) + return left == right + + +def _one_value(values: Sequence[Any]) -> bool: + """Whether `values` hold one value as `same_value` judges it.""" + return all(same_value(value, values[0]) for value in values[1:]) + + +def table_from_rows( + columns: Sequence[str], rows: Sequence[Sequence[Any]] +) -> ServedTable: + """The `ServedTable` of a table given as rows, as a vector gives one.""" + same = { + name: rows[0][index] + for index, name in enumerate(columns) + if rows and _one_value([row[index] for row in rows]) + } + nested = frozenset( + name + for index, name in enumerate(columns) + if any(isinstance(row[index], list | dict) for row in rows) + ) + return ServedTable(tuple(columns), len(rows), same, nested) + + +@lru_cache(maxsize=64) +def served_table(data: bytes) -> ServedTable: + """The `ServedTable` of a downloaded Parquet file. Each scenario is + replayed twice and each test validates the whole contract, so the + tables are read once per file content. + + Raises: + _UnfitError: The file is not a Parquet file. + """ + try: + table = pq.read_table(pa.BufferReader(data)) + except (pa.ArrowException, OSError) as exc: + raise _UnfitError( + f"a downloaded file is unreadable: not Parquet: {exc}" + ) from exc + nested = frozenset( + field.name for field in table.schema if pa.types.is_nested(field.type) + ) + # Every column is judged by `same_value`, as `table_from_rows` judges a + # vector's, so a vector and its Parquet file group their columns alike. + same = {} + for name in table.column_names if table.num_rows else (): + values = table.column(name).to_pylist() + if _one_value(values): + same[name] = values[0] + return ServedTable(tuple(table.column_names), table.num_rows, same, nested) + + +def filled_slots( + columns: Collection[str], reading: Sequence[Sequence[str]] +) -> list[str]: + """The reading slots `columns` fill, in slot order: each slot's first name + the columns hold.""" + return [ + name + for names in reading + if (name := next((n for n in names if n in columns), None)) is not None + ] + + +def _varies(name: str, tables: Sequence[ServedTable]) -> bool: + """Whether column `name` holds more than one value over the rows of + `tables` stacked, a table without the column holding null in it.""" + values = [] + for table in tables: + if not table.rows: + continue + if name in table.columns and name not in table.same: + return True + values.append(table.same.get(name)) + return not _one_value(values) + + +@dataclass(frozen=True, slots=True) +class ColumnGroups: + """A dataset's columns in the reading-order rule's groups: `part` when + there are several members, the filled reading slots in slot order, the + other columns whose values differ between the stacked rows and those + that hold one value in every row, each in served order, and `stat_var` + with whether it varies.""" + + part: tuple[str, ...] + first: tuple[str, ...] + varying: tuple[str, ...] + same: tuple[str, ...] + last: tuple[str, ...] + last_varies: bool + nested: frozenset[str] = frozenset() + + @property + def columns(self) -> list[str]: + return [*self.part, *self.first, *self.varying, *self.same, *self.last] + + @property + def sort_keys(self) -> list[str]: + """The columns a client sorts each member's rows by: the filled + reading slots in slot order without `value`, a measure, and without + the time period, then the varying columns in presented order, + `stat_var` among them when it varies, then the time period, so each + series' years read together. A column of lists or structs is no + key.""" + keys = [c for c in self.first if c not in (VALUE, TIME_PERIOD)] + keys += [*self.varying, *(self.last if self.last_varies else ())] + keys += [TIME_PERIOD] * (TIME_PERIOD in self.first) + return [key for key in keys if key not in self.nested] + + +def column_groups( + tables: Sequence[ServedTable], reading: Sequence[Sequence[str]] +) -> ColumnGroups: + """The reading-order rule's column groups for a dataset whose members + serve `tables`. The served order gathers the members' columns in member + order, each once.""" + gathered = list(dict.fromkeys(c for table in tables for c in table.columns)) + first = filled_slots(gathered, reading) + rest = [c for c in gathered if c not in first and c != STAT_VAR] + varying = [c for c in rest if _varies(c, tables)] + return ColumnGroups( + part=(PART,) if len(tables) > 1 else (), + first=tuple(first), + varying=tuple(varying), + same=tuple(c for c in rest if c not in varying), + last=(STAT_VAR,) if STAT_VAR in gathered else (), + last_varies=STAT_VAR in gathered and _varies(STAT_VAR, tables), + nested=frozenset().union(*(table.nested for table in tables)), + ) + + +def presented_columns( + tables: Sequence[ServedTable], reading: Sequence[Sequence[str]] +) -> list[str]: + """The columns a client presents for a dataset whose members serve + `tables`, as `column_groups` orders them.""" + return column_groups(tables, reading).columns + + +def presented_rows( + members: Sequence[tuple[str | None, Sequence[str], Sequence[Sequence[Any]]]], + reading: Sequence[Sequence[str]], +) -> tuple[list[str], list[list[Any]]]: + """The columns and rows a client presents for members given as their + `part` value, served columns and served rows. Each member's rows are + sorted by `ColumnGroups.sort_keys`, ascending with nulls last and ties in served + order, and the members follow in member order. A row of a stacked + dataset holds null in a column its member lacks.""" + tables = [table_from_rows(columns, rows) for _, columns, rows in members] + groups = column_groups(tables, reading) + columns, keys = groups.columns, groups.sort_keys + presented = [] + for part, served, rows in members: + at = {name: index for index, name in enumerate(served)} + cells = [ + {**{name: row[index] for name, index in at.items()}, PART: part} + for row in rows + ] + ordered = sorted( + cells, + key=lambda cell: [(cell.get(key) is None, cell.get(key)) for key in keys], + ) + presented.extend([cell.get(name) for name in columns] for cell in ordered) + return columns, presented + + +def load_reading_columns(package_dir: Path) -> list[list[str]] | None: + """The reading slots `rules/reading-order.json` lists, or None when the + file does not hold them, which the validator reports.""" + try: + reading = _load(package_dir / "rules" / "reading-order.json")["reading_columns"] + except (OSError, ValueError, KeyError, TypeError): + return None + if isinstance(reading, list) and all( + isinstance(names, list) and all(isinstance(n, str) for n in names) + for names in reading + ): + return reading + return None + + +def load_selector_keys(package_dir: Path) -> list[str] | None: + """The query keys `rules/file-selector.json` lists, or None when the file + does not hold them, which the validator reports.""" + try: + keys = _load(package_dir / "rules" / "file-selector.json")["query_keys"] + except (OSError, ValueError, KeyError, TypeError): + return None + if isinstance(keys, list) and keys and all(isinstance(k, str) for k in keys): + return keys + return None + + def posted_scope(step: Mapping[str, Any]) -> dict[str, Any] | None: """The `input.scope` a fetch of a spec posts for the step's narrowing, or None when it narrows nothing. The server's @@ -1976,16 +2290,32 @@ def digest_of(data: bytes) -> str: return "sha256:" + hashlib.sha256(data).hexdigest() +def selector_query(rendering: Mapping[str, Any], keys: Sequence[str]) -> str: + """The file selector's query for `rendering`, as the file-selector rule + builds it: each of `keys` the rendering holds, in that order.""" + return "&".join( + f"{key}={_query_value(rendering[key])}" for key in keys if key in rendering + ) + + +def _query_value(value: Any) -> str: + """A query value, a boolean written `true` or `false`.""" + if isinstance(value, bool): + return "true" if value else "false" + return str(value) + + def selector_path( - spec_id: str, content_version: str, rendering: Mapping[str, Any] + spec_id: str, + content_version: str, + rendering: Mapping[str, Any], + keys: Sequence[str], ) -> str: - """The file selector's path for a stored version, its rendering as the - query.""" - query = "&".join( - f"{name}={str(value).lower() if isinstance(value, bool) else value}" - for name, value in rendering.items() + """The file selector's path for a stored version in `rendering`, its + query from `selector_query`.""" + return ( + f"/datasets/{spec_id}/{content_version}/file?{selector_query(rendering, keys)}" ) - return f"/datasets/{spec_id}/{content_version}/file?{query}" # The storage host a signed download names, the only host outside the API a @@ -2115,11 +2445,11 @@ def failed_operation( error=error_class(500, OPERATION_FAILED, actions), trace_id=trace_id(None, headers), ) - return error, retryable(500, None) + return error, retryable(500, None, actions) status = problem.get("status") status = status if isinstance(status, int) else 500 error = refusal_error(status, problem, actions, headers) - return error, retryable(status, problem) + return error, retryable(status, problem, actions) class LongBuild: @@ -2185,9 +2515,9 @@ def polled(self, status: int, headers: Mapping[str, str], body: Any) -> bool: error, again = failed_operation(body, self.actions, headers) else: error = refusal_error(status, body, self.actions, headers) - again = retryable(status, body) + again = retryable(status, body, self.actions) if error.code != OPERATION_FAILED: - if not again: + if not tries_again(status, body, headers, self.actions): raise error if status != 429: if self.retries >= MAX_RETRIES: @@ -2217,13 +2547,16 @@ class _Minted: """An operation a mint started or a resume read, for the result step: its resource, the headers that came with it, the build request a failed operation resubmits (None for a resumed one), the spec the client holds, - and the rendering it downloads.""" + and the rendering the build posted, or for a resumed operation None and + the layout whose rendering `resumed_rendering` reads from the done + manifest.""" resource: dict[str, Any] headers: dict[str, str] request: dict[str, Any] | None spec: dict[str, Any] | None - rendering: dict[str, Any] + rendering: dict[str, Any] | None + layout: str | None = None @dataclass @@ -2238,6 +2571,8 @@ class _Replay: actions: Mapping[str, Mapping[str, Any]] client_codes: Mapping[str, Mapping[str, Any]] body_dir: Path + reading: Sequence[Sequence[str]] | None + selector_keys: Sequence[str] | None cursor: int = 0 answer: Answer | None = None minted: _Minted | None = None @@ -2284,7 +2619,7 @@ def call( retries and self.cursor < len(self.exchanges) and self.exchanges[self.cursor]["request"] == request - and may_retry(request, answers) + and may_retry(request, answers, self.actions) ): headers = exchange["response"].get("headers", {}) self.retry_waits.append(retry_wait(headers, len(answers) - 1)) @@ -2292,7 +2627,7 @@ def call( self.cursor += 1 self.took(exchange) answers.append(exchange["response"]) - if retries and may_retry(request, answers): + if retries and may_retry(request, answers, self.actions): raise _UnfitError( f"exchange {self.cursor - 1} is a retryable refusal with retries " f"left, and the record holds no retry of {method} {target}" @@ -2358,8 +2693,8 @@ def body(self, response: dict[str, Any]) -> bytes: return json.dumps(response["json"]).encode() return b"" - def download(self, target: str, digest: str, *, origin: bool) -> None: - """A file through its files route, named by an `origin_path`, as + def download(self, target: str, digest: str, *, origin: bool) -> bytes: + """The bytes of a file through its files route, named by an `origin_path`, as `artifact_route` places an artifact's `url`, when `origin`, else by a path under the base URL. A 302 names the signed URL in its `location`, which the client fetches once where `download_target` @@ -2395,8 +2730,9 @@ def download(self, target: str, digest: str, *, origin: bool) -> None: continue else: raise ClientError("unexpected_response") - if digest_of(self.body(file)) == digest: - return + data = self.body(file) + if digest_of(data) == digest: + return data failure = ClientError("digest_mismatch") raise failure @@ -2507,12 +2843,13 @@ def member( sentence = None if spec is None else spec.get("sentence") if spec is None or built != spec["spec_id"] or sentence is None: sentence = self.sentence(parts, manifest["definition"], spec) - self.download(route, artifact["artifact_digest"], origin=True) + data = self.download(route, artifact["artifact_digest"], origin=True) return { "parts": parts, "spec_id": built, "content_version": manifest["content_version"], "sentence": sentence, + "served": served_table(data), } def sentence( @@ -2547,8 +2884,10 @@ def stored_version(self, member: dict[str, Any]) -> dict[str, Any]: def stored_file( manifest: dict[str, Any], ) -> tuple[dict[str, Any] | None, ClientError | None]: - served, refusal = self.stored_file(member, manifest) - return (stored if served else None), refusal + data, refusal = self.stored_file(member, manifest) + if data is None: + return None, refusal + return {**stored, "served": served_table(data)}, None def rebuilt(manifest: dict[str, Any]) -> dict[str, Any]: held = {**spec, "sentence": member["sentence"]} @@ -2568,9 +2907,9 @@ def rebuilt(manifest: dict[str, Any]) -> dict[str, Any]: def stored_file( self, member: dict[str, Any], manifest: dict[str, Any] - ) -> tuple[bool, ClientError | None]: - """Whether the files route at the stored digest, or else the file - selector, served the stored file, and the files route's refusal when + ) -> tuple[bytes | None, ClientError | None]: + """The stored file the files route at the stored digest, or else the + file selector, served, or None, and the files route's refusal when it refused. A selector refusal outside FILE_GONE, and a files route one, raise. A selector file of another digest is not the stored one.""" digest = member["artifact"]["digest"] @@ -2581,27 +2920,27 @@ def stored_file( if artifact is not None: try: route = artifact_route(self.base_url, artifact["url"]) - self.download(route, digest, origin=True) + return self.download(route, digest, origin=True), None except ClientError as error: if error.code not in FILE_GONE: raise refusal = error - else: - return True, None + if self.selector_keys is None: + raise _UnfitError("rules/file-selector.json lists no query_keys") path = selector_path( member["spec"]["spec_id"], member["content_version"], member["artifact"]["rendering"], + self.selector_keys, ) try: - self.download(path, digest, origin=False) + return self.download(path, digest, origin=False), None except ClientError as error: if error.code == "digest_mismatch": - return False, refusal + return None, refusal if error.code in FILE_GONE: - return False, refusal or error + return None, refusal or error raise - return True, None def newest(self, spec_id: str) -> str | None: """The newest stored version, as `newest_plain` reads the versions @@ -2685,7 +3024,9 @@ def _fetch(self, step: dict[str, Any]) -> dict[str, Any]: ClientError(search_warning_code(answer), error="SearchWarning"), answer.trace_id, ) - rendering = rendering_for(step.get("layout")) + rendering = fetch_rendering( + step.get("layout"), party_totals=step.get("party_totals", False) + ) if "layout" in rendering and len(readings) > 1: raise ClientError("invalid_argument") members = [ @@ -2697,21 +3038,25 @@ def _fetch(self, step: dict[str, Any]) -> dict[str, Any]: ) for parts, spec in readings ] - return {"dataset": dataset_summary(members, gaps, search_warning=warns)} + return self.dataset(members, gaps, search_warning=warns) def _fetch_spec(self, step: dict[str, Any]) -> dict[str, Any]: - rendering = rendering_for(step.get("layout")) + rendering = fetch_rendering( + step.get("layout"), party_totals=step.get("party_totals", False) + ) manifest = self.build(step["spec"], rendering, posted_scope(step)) strict = step.get("strict", False) member = self.member([], step["spec"], manifest, rendering, strict) - return {"dataset": dataset_summary([member], [])} + return self.dataset([member], []) def _fetch_ref(self, step: dict[str, Any]) -> dict[str, Any]: members = [self.stored_version(member) for member in step["ref"]["members"]] - return {"dataset": dataset_summary(members, [])} + return self.dataset(members, []) def _mint(self, step: dict[str, Any]) -> dict[str, Any]: - rendering = rendering_for(step.get("layout")) + rendering = fetch_rendering( + step.get("layout"), party_totals=step.get("party_totals", False) + ) exchange = self.posted_build(step["spec"], rendering, None) if exchange["response"]["status"] == 200: raise _UnfitError( @@ -2728,8 +3073,7 @@ def _operation(self, step: dict[str, Any]) -> dict[str, Any]: exchange = self.call("GET", POLL_PATH.format(operation_id=step["operation_id"])) resource = self.acted_on(exchange, _OPERATION) headers = exchange["response"].get("headers", {}) - rendering = rendering_for(step.get("layout")) - self.minted = _Minted(resource, headers, None, None, rendering) + self.minted = _Minted(resource, headers, None, None, None, step.get("layout")) return _operation_outcome(resource) def _newest_version(self, step: dict[str, Any]) -> dict[str, Any]: @@ -2742,8 +3086,27 @@ def _result(self, step: dict[str, Any]) -> dict[str, Any]: raise _UnfitError("result follows no mint or operation") minted = self.minted manifest = self.long_build(minted.resource, minted.headers, minted.request) - member = self.member([], minted.spec, manifest, minted.rendering, False) - return {"dataset": dataset_summary([member], [])} + rendering = minted.rendering or resumed_rendering(minted.layout, manifest) + member = self.member([], minted.spec, manifest, rendering, False) + return self.dataset([member], []) + + def dataset( + self, + members: Sequence[Mapping[str, Any]], + gaps: Sequence[Mapping[str, Any]], + *, + search_warning: bool | None = None, + ) -> dict[str, Any]: + """A dataset outcome as `dataset_summary` gives it, with the columns a + client presents: the members' served columns ordered by the + reading-order rule, as `presented_columns` gives them. Without the + rule's reading slots, which the validator reports, the outcome gives + no columns.""" + summary = dataset_summary(members, gaps, search_warning=search_warning) + if self.reading is None: + return {"dataset": summary} + columns = presented_columns([m["served"] for m in members], self.reading) + return {"dataset": {"columns": columns, **summary}} def _operation_outcome(resource: dict[str, Any]) -> dict[str, Any]: @@ -2780,13 +3143,15 @@ def expected_outcomes( actions: Mapping[str, Mapping[str, Any]], client_codes: Mapping[str, Mapping[str, Any]], body_dir: Path, + reading: Sequence[Sequence[str]] | None, + selector_keys: Sequence[str] | None, ) -> list[dict[str, Any]]: """The outcome each of `record`'s steps reaches over its exchanges, its binary bodies read from `body_dir`, over `actions` for refusals' error classes and gap kinds, and over `client_codes` for the class of each error the client raises itself. A - dataset outcome holds what `dataset_summary` gives, without rows and - columns. + dataset outcome holds what `dataset_summary` gives and the columns the + `reading` slots order from the downloaded files, without rows. Raises: ValueError: The exchanges do not fit the steps, or deriving an @@ -2801,6 +3166,8 @@ def expected_outcomes( actions, client_codes, body_dir, + reading, + selector_keys, ) try: outcomes = [replay.step(step) for step in record["call"]] @@ -2970,9 +3337,10 @@ def _check_expect( derived: list[dict[str, Any]], ) -> list[str]: """Each outcome in `expect` against the one its exchanges give. A dataset - outcome's rows, columns and per-part row counts are the Parquet bodies' - and are not derived, except that several members put the `part` column - first and list each member's `part` value, rows summing to the dataset's.""" + outcome's columns are derived from the downloaded files, and its rows + and per-part row counts are not, except that several members put the + `part` column first and list each member's `part` value, rows summing + to the dataset's.""" problems = [] for index, (stated, wanted) in enumerate( zip(record["expect"], derived, strict=False) @@ -2987,10 +3355,22 @@ def _check_expect( if k not in {"rows", "columns", "part_rows"} } } + wanted = { + "dataset": { + k: v for k, v in wanted["dataset"].items() if k != "columns" + } + } problems.extend( f"scenarios/{file_name}: step {index}: {problem}" for problem in _part_problems(dataset, wanted["dataset"]["members"]) ) + columns = derived[index]["dataset"].get("columns") + if columns is not None and dataset["columns"] != columns: + problems.append( + f"scenarios/{file_name}: step {index}: columns " + f"{json.dumps(dataset['columns'])[:300]} are not the reading-order " + f"rule's order of the served columns, {json.dumps(columns)[:300]}" + ) if compared != wanted: problems.append( f"scenarios/{file_name}: step {index} expects " @@ -3042,6 +3422,8 @@ def _check_scenarios( client_codes: Mapping[str, Mapping[str, Any]], proxy_codes: Mapping[str, Mapping[str, Any]], served_commit: str | None, + reading: Sequence[Sequence[str]] | None, + selector_keys: Sequence[str] | None, ) -> list[str]: directory = PACKAGE_DIR / "scenarios" paths = sorted(directory.glob("*.json")) @@ -3138,6 +3520,8 @@ def _check_scenarios( actions=actions, client_codes=client_codes, body_dir=directory, + reading=reading, + selector_keys=selector_keys, ) try: derived = replay(record, base_url=base_url) @@ -3260,7 +3644,7 @@ def _ref_rule(ref: dict[str, Any]) -> str | None: seen: set[str] = set() for member in ref["members"]: spec = member["spec"] - if spec["definition"]["spec_id"] != spec["spec_id"]: + if "definition" in spec and spec["definition"]["spec_id"] != spec["spec_id"]: return "definition_spec_id_mismatch" if spec["spec_id"] in seen: return "repeated_spec_id" @@ -3268,12 +3652,19 @@ def _ref_rule(ref: dict[str, Any]) -> str | None: return None +def _derived(member: Any) -> bool: + """Whether a ref member is a derived member, which names its roots.""" + return isinstance(member, dict) and "roots" in member + + def _served_by(label: str, record: dict[str, Any]) -> list[str]: """Where a ref example's members differ from the members its `scenario` - serves: a `spec_id` no dataset member of it has, another + serves: a `spec_id` no dataset member of it has, other `parts`, another `content_version` or `sentence`, or an artifact digest that is not the - digest of a file the scenario downloads. A ref with no members names no - scenario.""" + digest of a file the scenario downloads. A derived member's roots are + checked instead, each a dataset member's `spec_id` and `content_version`, + since no scenario serves the derived member itself. A ref with no members + names no scenario.""" ref = record.get("ref") members = ref.get("members") if isinstance(ref, dict) else None if not isinstance(members, list) or not members: @@ -3300,6 +3691,14 @@ def _served_by(label: str, record: dict[str, Any]) -> list[str]: digests.add(digest_of(path.read_bytes())) problems = [] for index, member in enumerate(members): + if _derived(member): + problems.extend( + f"{label}: member {index} root {position} is no member {name} serves" + for position, root in enumerate(read_objects(member, "roots")) + if (found := served.get(root.get("spec_id"))) is None + or found.get("content_version") != root.get("content_version") + ) + continue spec = member.get("spec") if isinstance(member, dict) else None found = served.get(spec.get("spec_id") if isinstance(spec, dict) else None) if found is None: @@ -3307,7 +3706,7 @@ def _served_by(label: str, record: dict[str, Any]) -> list[str]: continue problems.extend( f"{label}: member {index} {key} is not {name}'s {found.get(key)!r}" - for key in ("content_version", "sentence") + for key in ("parts", "content_version", "sentence") if member.get(key) != found.get(key) ) artifact = member.get("artifact") @@ -3319,6 +3718,123 @@ def _served_by(label: str, record: dict[str, Any]) -> list[str]: return problems +def _answered_builds(scenario: Mapping[str, Any]) -> Iterator[tuple[Any, Any]]: + """Each build in `scenario` that answered a manifest, as the manifest and + the `options` the build posted: a manifest's own, or for a 202 the + manifest the poll of its operation answered done, in build order.""" + by_operation: dict[str, Any] = {} + for exchange in read_objects(scenario, "exchanges"): + request = read_object(exchange, "request") or {} + body = read_object(read_object(exchange, "response"), "json") or {} + manifest = read_object(body, "manifest") if "manifest" in body else body + operation_id = read_string(read_object(body, "operation"), "operation_id") + if (request.get("method"), request.get("path")) == _BUILD: + options = (read_object(request, "json") or {}).get("options") + if operation_id is not None: + by_operation[operation_id] = options + elif operation_id in by_operation and "manifest" in body: + options = by_operation[operation_id] + else: + continue + yield manifest, options + + +def posted_renderings(scenario: Mapping[str, Any]) -> dict[str, list[Any]]: + """The renderings the builds in `scenario` posted as their `options`, by + the digest of the artifact each build's manifest lists for them. Builds + that answered the same file list each rendering in build order.""" + renderings: dict[str, list[Any]] = {} + for manifest, options in _answered_builds(scenario): + digest = _artifact_digest(manifest, options) + if digest is not None: + renderings.setdefault(digest, []).append(options) + return renderings + + +def _spec_renderings(scenario: Mapping[str, Any]) -> dict[str, list[Any]]: + """The renderings the builds in `scenario` posted as their `options`, by + the spec_id each build's manifest answered, in build order.""" + renderings: dict[str, list[Any]] = {} + for manifest, options in _answered_builds(scenario): + spec_id = read_string(read_object(manifest, "definition"), "spec_id") + if spec_id is not None: + renderings.setdefault(spec_id, []).append(options) + return renderings + + +def _artifact_digest(manifest: Any, options: Any) -> str | None: + """The digest of the artifact `manifest` lists for the posted `options`, + or None when either does not read.""" + if not isinstance(options, dict) or not {"format", "bundle"} <= options.keys(): + return None + artifacts = read_objects(manifest, "artifacts") + artifact = pick_artifact({"artifacts": artifacts}, options) + return read_string(artifact, "artifact_digest") + + +def _root_rendering(posted: Any, rendering: Any) -> Any: + """The rendering a derived member's root records for a build that posted + `posted`: DFO reads an analysis root in the frame layout, and a root read + summed also records the `groupby` DFO rendered it with, which no recorded + build posts.""" + if not isinstance(posted, dict): + return posted + wanted = {**posted, "layout": "frame"} + if isinstance(rendering, dict) and "groupby" in rendering: + wanted.setdefault("groupby", rendering["groupby"]) + return wanted + + +def _posted_by(label: str, record: dict[str, Any]) -> list[str]: + """Where a ref example's member records a rendering other than the one + the build that answered its file in its `scenario` posted, since a ref + records the rendering its fetch posted, or a derived member's root + records one no build of its spec there posted, read as `_root_rendering` + gives. A root's `groupby` is the schema's alone to check.""" + ref = record.get("ref") + members = ref.get("members") if isinstance(ref, dict) else None + name = record.get("scenario") + path = PACKAGE_DIR / "scenarios" / f"{name}.json" + if not isinstance(members, list) or not isinstance(name, str) or not path.is_file(): + return [] # `_served_by` reports a ref that names no scenario + scenario = _load(path) + posted = posted_renderings(scenario) + by_spec = _spec_renderings(scenario) + problems = [] + for index, member in enumerate(members): + if _derived(member): + for position, root in enumerate(read_objects(member, "roots")): + where = f"member {index} root {position}" + spec_id = read_string(root, "spec_id") + rendering = read_object(root, "rendering") + if spec_id not in by_spec: + problems.append( + f"{label}: {where}'s spec is built by no {name} build" + ) + continue + wanted = [_root_rendering(o, rendering) for o in by_spec[spec_id]] + if rendering not in wanted: + problems.append( + f"{label}: {where} records rendering {json.dumps(rendering)}, " + f"but {name}'s builds give " + f"{' and '.join(map(json.dumps, wanted))}" + ) + continue + artifact = read_object(member, "artifact") + digest = read_string(artifact, "digest") + rendering = read_object(artifact, "rendering") + if digest not in posted: + problems.append( + f"{label}: member {index}'s file is answered by no {name} build" + ) + elif rendering not in posted[digest]: + problems.append( + f"{label}: member {index} records rendering {json.dumps(rendering)}, " + f"but {name} posted {' and '.join(map(json.dumps, posted[digest]))}" + ) + return problems + + def _check_refs(validator: jsonschema.Draft202012Validator) -> list[str]: paths = sorted((PACKAGE_DIR / "refs").glob("*.json")) problems = [] if paths else ["refs: no ref examples found"] @@ -3341,6 +3857,7 @@ def _check_refs(validator: jsonschema.Draft202012Validator) -> list[str]: if valid: problems.append(f"{label}: the schema accepts a ref it should reject") continue + problems.extend(_posted_by(label, record)) if not valid: first = next(validator.iter_errors(record.get("ref"))) problems.append(f"{label}: ref fails the schema at {_where(first)}") @@ -3357,9 +3874,273 @@ def _check_refs(validator: jsonschema.Draft202012Validator) -> list[str]: return problems +# --- derived identities ----------------------------------------------------- + +# What fixes what among a derived member's identities. Two members that agree +# on every field of a key agree on every field it determines: `ocv_` hashes +# the `ops_` id, the root versions and the execution identity, and names one +# output; `rpk_` hashes the normal form, the root versions and the execution +# identities, so one replay computes one op-spec, version and output; the +# normal form, the root versions and the execution identity give one `rpk_`; +# and one file holds one output in one format. `roots` holds each root's +# version with the rendering it was read with, since a root read summed +# under a groupby is another input. +DERIVED_IDENTITY_RULES: tuple[tuple[tuple[str, ...], tuple[str, ...]], ...] = ( + ( + ("content_version",), + ("spec_id", "roots", "execution_identity", "output_digest"), + ), + ( + ("replay_key",), + ( + "normal_form_hash", + "roots", + "execution_identity", + "spec_id", + "content_version", + "output_digest", + "resolved_periods", + ), + ), + (("normal_form_hash", "roots", "execution_identity"), ("replay_key",)), + (("artifact_digest",), ("artifact_format", "output_digest")), +) + + +def derived_identities(member: Mapping[str, Any]) -> dict[str, str | None]: + """The identities of a derived member that `DERIVED_IDENTITY_RULES` read, + each as text, None where the member lacks it.""" + spec = read_object(member, "spec") + recipe = read_object(spec, "recipe") + artifact = read_object(member, "artifact") + roots = [ + [ + read_string(root, "spec_id"), + read_string(root, "content_version"), + read_object(root, "rendering"), + ] + for root in read_objects(member, "roots") + ] + periods = member.get("resolved_periods") + return { + "spec_id": read_string(spec, "spec_id"), + "normal_form_hash": read_string(recipe, "normal_form_hash"), + "replay_key": read_string(recipe, "replay_key"), + "content_version": read_string(member, "content_version"), + "output_digest": read_string(member, "output_digest"), + "roots": json.dumps(roots, sort_keys=True), + "execution_identity": read_string(member, "execution_identity"), + "resolved_periods": None + if periods is None + else json.dumps(periods, sort_keys=True), + "artifact_digest": read_string(artifact, "digest"), + "artifact_format": read_string(read_object(artifact, "rendering"), "format"), + } + + +def derived_identity_problems( + members: Sequence[tuple[str, Mapping[str, Any]]], +) -> list[str]: + """Where two labelled derived members agree on a key of + `DERIVED_IDENTITY_RULES` but differ in a field it determines.""" + identities = [(label, derived_identities(member)) for label, member in members] + problems = [] + for key, determined in DERIVED_IDENTITY_RULES: + first: dict[tuple[str | None, ...], tuple[str, dict[str, str | None]]] = {} + for label, facts in identities: + values = tuple(facts[name] for name in key) + if None in values: + continue + if values not in first: + first[values] = (label, facts) + continue + other_label, other = first[values] + problems.extend( + f"{label} and {other_label} share {' and '.join(key)} " + f"but differ in {name}" + for name in determined + if facts[name] != other[name] + ) + return problems + + +def _check_derived_identities() -> list[str]: + """Every derived member of a ref example the schema accepts and of the + output-comparison vectors, checked together, since the examples name one + computation in several places.""" + members: list[tuple[str, Mapping[str, Any]]] = [] + for path in sorted((PACKAGE_DIR / "refs").glob("*.json")): + record = _load(path) + if not isinstance(record, dict) or read_object(record, "error") == { + "by": "schema" + }: + continue + members.extend( + (f"refs/{path.name} member {index}", member) + for index, member in enumerate( + read_objects(read_object(record, "ref"), "members") + ) + if _derived(member) + ) + comparison = _load(PACKAGE_DIR / "rules" / "output-comparison.json") + for vector in read_objects(comparison, "vectors"): + for side in ("left", "right"): + members.extend( + ( + f"rules/output-comparison.json {vector.get('name')} {side} member {index}", + member, + ) + for index, member in enumerate( + read_objects(read_object(vector, side), "members") + ) + if _derived(member) + ) + return derived_identity_problems(members) + + # --- rules ------------------------------------------------------------------ +# The frame columns that name no role (docs/frame-grammar-v1.md). +FRAME_FIXED_COLUMNS = frozenset( + { + "stat_var", + "time_period", + "value", + "unit", + "measurementMethod", + "provenance", + "status", + } +) + + +def _without_recipe(unit: Any) -> Any: + """A unit cell with `recipe` dropped from every unit-request object.""" + if not isinstance(unit, dict): + return unit + if "numerator" in unit or "denominator" in unit: + return {side: _without_recipe(part) for side, part in unit.items()} + return {name: item for name, item in unit.items() if name != "recipe"} + + +def output_digest_problems(label: str, vector: Mapping[str, Any]) -> list[str]: + """Where an output-digest vector's `document` is not its frame's, read as + JSON, or its `digest` is not the document's: the columns `stat_var`, the + role columns, `time_period`, `value` and `unit`, each frame row once in + those columns with no unit `recipe` and a `stat_var` of `dcid:` and an + `ops_` id its `variables` names read as that name, digest version `out2`, + and `out_` and 32 hex digits of the document's SHA-256. The JCS bytes + themselves are the reference implementation's to check.""" + frame = vector["frame"] + variables = vector.get("variables", {}) + roles = [name for name in frame["columns"] if name not in FRAME_FIXED_COLUMNS] + columns = ["stat_var", *roles, "time_period", "value", "unit"] + rows = [] + for row in frame["rows"]: + cells = dict(zip(frame["columns"], row, strict=True)) + cells["unit"] = _without_recipe(cells.get("unit")) + stat_var = cells["stat_var"] + cells["stat_var"] = variables.get(stat_var.removeprefix("dcid:"), stat_var) + rows.append([cells.get(name) for name in columns]) + document = json.loads(vector["document"]) + problems = [] + wanted = {"digest_version": "out2", "columns": columns} + unmatched = list(document.get("rows", [])) + for row in rows: + if row in unmatched: + unmatched.remove(row) + else: + unmatched.append(None) + if {key: document.get(key) for key in wanted} != wanted or unmatched: + problems.append( + f"{label}: vector {vector['name']!r} document is not its frame's" + ) + hashed = "out_" + hashlib.sha256(vector["document"].encode()).hexdigest()[:32] + if vector["digest"] != hashed: + problems.append( + f"{label}: vector {vector['name']!r} digest is not its document's, {hashed}" + ) + return problems + name_document_problems(label, vector) + + +def name_document_problems(label: str, vector: Mapping[str, Any]) -> list[str]: + """Where an output-digest vector's `name_documents` misname a minted + variable: a name that is not `mv_` and 32 hex digits of its document's + SHA-256, or a name in `variables` or in a document's slots with no + document of its own. So a nested step's document holds its input's + `mv_` name, which has its own document.""" + documents = vector.get("name_documents", {}) + problems = [ + f"{label}: vector {vector['name']!r} name {name} is not its document's" + for name, text in documents.items() + if name != "mv_" + hashlib.sha256(text.encode()).hexdigest()[:32] + ] + named = set(vector.get("variables", {}).values()) + for name, text in documents.items(): + try: + document = json.loads(text) + except ValueError: + document = None + slots = ( + [items for slot, items in document.items() if slot != "verb"] + if isinstance(document, dict) and isinstance(document.get("verb"), str) + else None + ) + if slots is None or not all(isinstance(items, list) for items in slots): + problems.append( + f"{label}: vector {vector['name']!r} name {name} document is not " + f"a verb and its slots" + ) + continue + named.update( + held + for items in slots + for held in items + if isinstance(held, str) and held.startswith("mv_") + ) + problems.extend( + f"{label}: vector {vector['name']!r} name {name} has no name document" + for name in sorted(named - set(documents)) + ) + return problems + + +def _json_type(value: Any) -> str: + if isinstance(value, bool): + return "boolean" + return "number" if isinstance(value, int | float) else type(value).__name__ + + +def _unorderable(members: Sequence[Mapping[str, Any]]) -> str | None: + """Why a reading-order vector's members cannot be ordered, or None: a + row whose cells do not match its member's columns, a member's column + whose cells cannot form one Arrow column, as a served Parquet file holds + them, or a column whose cells, nulls aside, mix JSON types across the + stacked members, which the rule gives no order.""" + types: dict[str, set[str]] = {} + for index, member in enumerate(members): + columns = member["columns"] + for row in member["rows"]: + if len(row) != len(columns): + return ( + f"member {index} has a row of {len(row)} cells for " + f"{len(columns)} columns" + ) + for name, cell in zip(columns, row, strict=True): + if cell is not None: + types.setdefault(name, set()).add(_json_type(cell)) + for position, name in enumerate(columns): + try: + pa.array([row[position] for row in member["rows"]]) + except pa.ArrowException as exc: + return f"member {index} column {name!r} is no Arrow column: {exc}" + mixed = sorted(name for name, found in types.items() if len(found) > 1) + if mixed: + return f"mixes JSON types in column {mixed[0]!r}" + return None + + def _check_rules(validator: jsonschema.Draft202012Validator) -> list[str]: paths = sorted( p @@ -3381,6 +4162,41 @@ def _check_rules(validator: jsonschema.Draft202012Validator) -> list[str]: f"{label}: vector {name!r} repeats" for name in sorted({n for n in names if names.count(n) > 1}) ) + problems.extend( + f"{label}: vector {vector['name']!r} has a window that does not " + f"start before it ends" + for vector in document["vectors"] + if document["rule"] == "operation-outputs" + and (window := vector["arguments"].get("window")) is not None + and int(window[0]) >= int(window[1]) + ) + if document["rule"] == "output-digest": + for vector in document["vectors"]: + problems.extend(output_digest_problems(label, vector)) + if document["rule"] == "reading-order": + for vector in document["vectors"]: + unfit = _unorderable(vector["members"]) + if unfit is not None: + problems.append(f"{label}: vector {vector['name']!r} {unfit}") + elif presented_rows( + [ + (member.get("part"), member["columns"], member["rows"]) + for member in vector["members"] + ], + document["reading_columns"], + ) != (vector["columns"], vector["rows"]): + problems.append( + f"{label}: vector {vector['name']!r} orders its columns or " + f"rows otherwise than presented_rows" + ) + if document["rule"] == "file-selector": + problems.extend( + f"{label}: vector {vector['name']!r} gives a query other than " + f"selector_query's" + for vector in document["vectors"] + if selector_query(vector["rendering"], document["query_keys"]) + != vector["query"] + ) return problems @@ -3455,10 +4271,23 @@ def validate() -> list[str]: or _check_warning_codes( warning_codes, {**taken, "proxy-codes.json": {row["code"] for row in proxy_rows}}, - key_format_reasons(rules_schema), + rules_schema, ) ) + reading = load_reading_columns(PACKAGE_DIR) + if reading is None: + problems.append( + "rules/reading-order.json: missing or unreadable, or holds no " + "reading_columns list of column-name lists, so no dataset's columns " + "are checked" + ) + selector_keys = load_selector_keys(PACKAGE_DIR) + if selector_keys is None: + problems.append( + "rules/file-selector.json: missing or unreadable, or holds no " + "query_keys list of names, so no file selector request is checked" + ) problems.extend( _check_scenarios( validators["scenario.schema.json"], @@ -3469,6 +4298,8 @@ def validate() -> list[str]: client_codes={row["code"]: row for row in client_rows}, proxy_codes={row["code"]: row for row in proxy_rows}, served_commit=source.get("commit") if isinstance(source, dict) else None, + reading=reading, + selector_keys=selector_keys, ) ) size = scenarios_size(PACKAGE_DIR / "scenarios") @@ -3478,6 +4309,7 @@ def validate() -> list[str]: ) problems.extend(_check_refs(validators["dataset-ref.schema.json"])) problems.extend(_check_rules(validators["rules/rules.schema.json"])) + problems.extend(_check_derived_identities()) return problems diff --git a/tools/validate_descriptions_vectors.py b/tools/validate_descriptions_vectors.py index f2532a4..4200b7f 100644 --- a/tools/validate_descriptions_vectors.py +++ b/tools/validate_descriptions_vectors.py @@ -12,8 +12,9 @@ Checks that the three schemas are valid draft 2020-12, that `operations.json` validates against `operations.schema.json` and names each op once, the nine -analysis operations exactly, each reason once per op, and roles exactly when -an op takes them. Checks every vector against `descriptions-vector.schema.json` +analysis operations exactly, each reason once per op, roles exactly when an +op takes them, each verb once, and a client call's `data` and `other` filling +the two roles of the op they name. Checks every vector against `descriptions-vector.schema.json` and its input against the schema it names: an accepted vector and a vector the structure rules reject must validate, and a vector the schema rejects must not. Then what the record schema cannot express: each file name matches its `id`, @@ -122,6 +123,19 @@ def _check_operations(validator: jsonschema.Draft202012Validator) -> list[str]: f"operations.json: {entry['op']}: roles must be listed exactly when " "input is 'roles'" ) + arguments = entry.get("arguments") + if arguments and sorted((arguments["data"], arguments["other"])) != sorted( + entry["roles"] + ): + problems.append( + f"operations.json: {entry['op']}: arguments' data and other must " + f"fill its two roles {entry['roles']}" + ) + verbs = [entry["verb"] for entry in entries if "verb" in entry] + problems.extend( + f"operations.json: verb {verb!r} is listed twice" + for verb in sorted({v for v in verbs if verbs.count(v) > 1}) + ) return problems