From 80ffbf4424ca9042a2bc8786f36d2990d5c782fd Mon Sep 17 00:00:00 2001 From: Jorge Rivera Date: Thu, 15 Oct 2026 16:25:01 +0200 Subject: [PATCH] docs: add MIT licence and rewrite docs for public readers --- CONTRIBUTING.md | 62 ++++++++++++++++++++++++++++ LICENSE | 22 ++++++++++ README.md | 98 +++++++++++++++++++++++++++------------------ docs/development.md | 31 ++++++++++++++ 4 files changed, 173 insertions(+), 40 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 LICENSE create mode 100644 docs/development.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..576963f --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing + +Issues and pull requests are welcome. The ONE Data team maintains this repository. For a large change, such as a new contract, a schema change, or a new package, open an issue first so the design can be discussed before you write it. + +Please follow the [Code of Conduct](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/CODE_OF_CONDUCT.md). Report security problems through GitHub private vulnerability reporting (the "Report a vulnerability" button under the Security tab, see the [security policy](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SECURITY.md)), and keep them out of public issues. For questions, see [SUPPORT.md](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SUPPORT.md). + +## Setup + +Everything runs locally and needs no access to ONE's Google Cloud project. Install [uv](https://docs.astral.sh/uv/) and clone the repository. Each package under `python/` is its own uv project with a committed `uv.lock`, so work from inside the package directory: + +```bash +cd python/spec-kit +uv sync +``` + +The six packages are `spec-kit`, `frame-grammar`, `dfo-spec-kit-components`, `provenance-docs`, `descriptions`, and `client-contract`. Python 3.12 or later is required, and CI also runs 3.13. + +## Checks before a pull request + +Run these in each package you changed. They are the commands CI runs. + +```bash +uv run pytest +uv run ruff check . +uv run ruff format --check . +uv run ty check src/ +``` + +`` is `dfo_spec_kit`, `dfo_frame_grammar`, `dfo_spec_kit_components`, `dfo_provenance_docs`, `dfo_descriptions`, or `dfo_client_contract`. `uv run ruff format .` rewrites files to the formatter's style. Each package also has a `.pre-commit-config.yaml`. `pre-commit install` runs the same linters, plus codespell, actionlint, and mdformat, on each commit. + +### Conformance vectors and registries + +The validators under `tools/` check the registries and vector suites without importing the packages, so they test the fixtures independently of the implementations. Run them from the repository root. Each is a self-contained script and uv installs its dependencies. + +```bash +uv run tools/validate_registry.py +uv run tools/validate_unit_registry.py +uv run tools/validate_vectors.py +uv run tools/validate_frame_vectors.py +uv run tools/validate_unit_vectors.py +uv run tools/validate_provenance_docs_vectors.py +uv run tools/validate_descriptions_vectors.py +uv run tools/validate_client_contract.py +uv run --with jsonschema --with pytest pytest tools/tests -q +``` + +The last command tests the validators themselves. [tools/README.md](tools/README.md) says what each validator checks. Each package's tests drive its implementation against the same vectors or fixtures, and `uv run pytest` runs them. + +A change to `registries/codes.json` or `registries/units.json` also needs the packaged mirror under `python/spec-kit/src/dfo_spec_kit/` to match byte for byte. The validators fail if the two differ. Edit the file under `registries/` and copy it to the mirror. [registries/README.md](registries/README.md) has the format. + +## Branches and commits + +Open pull requests against `dev`, the default branch. CI runs on pull requests and pushes to `dev` and `main`, and skips draft pull requests until they are marked ready for review. Commit messages follow Conventional Commits, for example `ci: run spec-kit CI on dev and main`. + +## What a pull request needs + +- Passing CI, meaning the lint, test, and validator jobs in `.github/workflows/ci.yml`. +- Tests for new behaviour. A change to a contract updates its vectors in the same pull request. +- Updated docs. A change to a package's public behaviour adds an entry under `Unreleased` in that package's `CHANGELOG.md`. +- Compliance with [policies/](policies/README.md), which says which change to which artifact needs which version bump. + +Maintainer notes, such as how the packages depend on one another and how releases are checked, are in [docs/development.md](docs/development.md). diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..4b7a716 --- /dev/null +++ b/LICENSE @@ -0,0 +1,22 @@ + +MIT License + +Copyright (c) 2026 ONE Campaign + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index d290c0e..670e102 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,54 @@ -# dfo-spec-kit +# DFO spec kit -The executable form of the platform's shared contracts. It exists so the consumers of these contracts, the Search API in `apis`, the harness and Analysis MCP in `ai`, the website, and the docs, share one source of truth instead of hand-maintained mirrors. This repo is not the spec registry nor the op-spec schema. +The spec kit holds the shared data contracts of the Development Finance Observatory as JSON schemas, registries, conformance vectors, and Python packages that implement them. Services that must agree on the same bytes, ids, and document formats import them from here instead of keeping their own copies. -The target design (repository shape, artifact families, versioning, governance, and delivery sequence) lives in the kit design, maintained outside this repo. This README only orients. That document is authoritative. +It is part of the [Development Finance Observatory](https://github.com/DevelopmentFinanceObservatory), an initiative of [ONE Data](https://data.one.org) built on [Google Data Commons](https://datacommons.org). The org profile has the map of all repositories. The services that consume the kit are `apis` (the Search API), `ai` (the harness and Analysis MCP), `website`, and `docs`. -## Two schema families +This repository is under active development. Interfaces, schemas, and data may change without notice. -The kit carries two schema families. +## What the kit is for -The **dataset-spec family** (spec models, identity projection, JCS canonicalisation, SHA-256 hashing) is Python-first. The behaviour lives in Python, and JSON Schema is exported from the pydantic models. +Each contract answers one question that two or more services must answer identically. -The **component-spec family** (the presentation catalogue, composition schema, operation union) is TypeScript-first. Zod v4 is the author and validator, and generated artifacts (TS prop types, Pydantic models, tool metadata) are frozen and checked for drift. Each family is authored in the language where its behaviour lives. Committed JSON Schema is the neutral frozen centre between them, held together by cross-language conformance fixtures. +- Which dataset is this request asking for? A dataset spec is reduced to a canonical form and hashed into a `spec_id`, so every service mints the same id for the same request. +- Has this table of observations changed? A materialised frame is serialised to canonical bytes and hashed into a `content_version`. +- How do units compare? A unit ref reads as a typed value (money with a currency and price basis, a percent, a count), and refuses comparisons that would mix price bases. +- What does a card or chart look like on the wire? A catalogue of JSON schemas describes each presentation component. +- How is a result documented? Step records and descriptions say what was done to a result, and provenance documents carry transformations, changelogs, and methodology. +- What do the DFO clients expect from the API? The client contract holds the served OpenAPI document, the response members the clients act on, the next step for each refusal code, and recorded scenarios the clients replay. The API's CI checks its app against the same files. -## Status +Every contract ships with conformance vectors, language-neutral fixtures that any implementation, in any language, can run against. Where prose and vectors disagree, the vectors win. -Six Python distributions exist so far. `python/spec-kit/` (`dfo-spec-kit`, import `dfo_spec_kit`): `identity/` implements the spec1 identity scheme (role-slot projection, JCS canonicalisation, SHA-256 `spec_id` minting), conformance-tested against the vector suite. `units/` reads a unit ref into a typed value (`Money`, `UnpricedMoney`, `Percent`, `Count` or `Other`) from `registries/units.json`, conformance-tested against `vectors/units/`. Money whose ref leaves prices unstated cannot be compared until `complete_prices` applies the StatVar's measurement qualifier, and constant-price money with no base year cannot be compared at all. `models/` has no code yet; model extraction comes next. `python/frame-grammar/` (`dfo-spec-kit-frame-grammar`, import `dfo_frame_grammar`): the frame grammar's executable form -- canonical frame bytes and `content_version` minting for a materialised observation table -- conformance-tested against its own vector suite, and depending on `dfo-spec-kit` for the identity canonicalisation primitives it reuses. `python/dfo-spec-kit-components/` (`dfo-spec-kit-components`, import `dfo_spec_kit_components`): the presentation contract bundle as package data, holding the envelope schema, the per-member component schemas, the write-side intent grammar, and their conformance fixtures. Generated Pydantic models for the catalogue land in this same package once the catalogue freezes. `python/provenance-docs/` (`dfo-spec-kit-provenance-docs`, import `dfo_provenance_docs`): the provenance documentation grammar, which decodes the escaped Markdown documents a Provenance node carries and parses the transformations, changelog, methodology and notes templates, conformance-tested against its own vector suite. `python/descriptions/` (`dfo-spec-kit-descriptions`, import `dfo_descriptions`): the step records and descriptions contract, holding the step schema, the operation list, the description schema and a check of the structure rules a step list must meet beyond its schema, conformance-tested against its own vector suite. `python/client-contract/` (`dfo-spec-kit-client-contract`, import `dfo_client_contract`): the contract the DFO Python and TypeScript clients are built to, as package data: the served OpenAPI document of the composed dfo-apis app, the response members the clients act on, the caller's next step for each refusal code, the code of each error a client raises itself, the scenario and `DatasetRef` formats with their records, and the client rules with their vectors. dfo-apis CI checks its app against the same files. +## Packages -The Python distributions live at `python/spec-kit/`, `python/frame-grammar/`, `python/dfo-spec-kit-components/`, `python/provenance-docs/`, `python/descriptions/`, and `python/client-contract/`, each its own installable unit with its own version and dependency surface. A TypeScript package (`ts/`) opens only when the component catalogue freeze lands. +Six Python distributions live under `python/`. Each has its own version, dependencies, tests, and README. All need Python 3.12 or later. -`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit` as an intra-repo path source rather than a git dependency, so the two packages develop together in this repo: a change to identity canonicalisation and a change here that depends on it land in the same commit. `dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions` as an intra-repo path source, because the envelope's `description` references `description.schema.json` by its `$id`. A consumer pins both distributions at one revision. Its catalogue bundle is static data plus one accessor function, `bundle_dir()`. `dfo-spec-kit-provenance-docs` also stands alone, depending on `markdown-it-py` only. `dfo-spec-kit-descriptions` depends on `rfc8785` only. `dfo-spec-kit-client-contract` depends on the standard library only. +| Distribution | Import name | What it does | +| ------------------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `dfo-spec-kit` | `dfo_spec_kit` | Dataset-spec identity (canonical form, `spec_id`) and typed units read from `registries/units.json`. | +| `dfo-spec-kit-frame-grammar` | `dfo_frame_grammar` | Canonical frame bytes and `content_version` for a materialised observation table. | +| `dfo-spec-kit-components` | `dfo_spec_kit_components` | The presentation catalogue: envelope and component schemas, the write-side intent grammar, and fixtures, as package data. | +| `dfo-spec-kit-provenance-docs` | `dfo_provenance_docs` | Decoder and parsers for the Markdown documents a Provenance node carries. | +| `dfo-spec-kit-descriptions` | `dfo_descriptions` | Step and description schemas, the operation list, and a check of the rules a step list must meet. | +| `dfo-spec-kit-client-contract` | `dfo_client_contract` | The contract the DFO clients are built to: served OpenAPI document, strict fields, refusal actions, scenarios, rules. | -Publishing is undecided: no registry workflow exists for any package. Until that's resolved, install a Python package as a git dependency pinned to one commit, e.g. +`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit`. `dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions`. The other four have no dependency inside the kit. + +## What's inside + +```text +docs/ normative specifications: identity-v1.md, frame-grammar-v1.md, and maintainer notes +policies/ compatibility policy per artifact family +python/ the six packages +registries/ codes.json (warning and refusal codes) and units.json (unit refs) +schemas/ JSON Schema for the registries and the vector suites +tools/ validators for the registries, vector suites, and client contract +vectors/ cross-language conformance vectors for identity and units +``` + +## Getting started + +No package is published to PyPI. Install from git, pinned to one commit. Replace `` with a commit SHA from this repository. ```bash uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/spec-kit" @@ -31,7 +59,7 @@ uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subd uv add "git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/client-contract" ``` -A package's metadata names its intra-repo dependency by distribution name alone, and no index serves these distributions. uv reads the path source in `[tool.uv.sources]` and resolves the dependency from the same checkout, so `uv add` of `frame-grammar` or `dfo-spec-kit-components` alone installs `dfo-spec-kit` or `dfo-spec-kit-descriptions` at that commit. Every other installer reads the bare name and fails, so name both distributions at the same commit: +A package declares its kit dependency by name only. uv resolves it from the same commit through a path source in the package's `[tool.uv.sources]`, so `uv add` of `frame-grammar` also installs `dfo-spec-kit`. Other installers read the bare name and fail, so with pip name both distributions at the same commit: ```bash pip install \ @@ -39,7 +67,7 @@ pip install \ "dfo-spec-kit-descriptions @ git+https://github.com/DevelopmentFinanceObservatory/spec-kit@#subdirectory=python/descriptions" ``` -A uv project pins each kit distribution it imports in its own `[tool.uv.sources]`, every entry at the same `rev`, as `apis` pins `dfo-spec-kit-components` and `dfo-spec-kit-frame-grammar`: +A uv project that imports more than one kit distribution pins each in its own `[tool.uv.sources]`, every entry at the same `rev`: ```toml [tool.uv.sources] @@ -47,30 +75,20 @@ dfo-spec-kit-components = { git = "https://github.com/DevelopmentFinanceObservat dfo-spec-kit-descriptions = { git = "https://github.com/DevelopmentFinanceObservatory/spec-kit.git", rev = "", subdirectory = "python/descriptions" } ``` -## Layout +The kit needs no access to ONE's Google Cloud project. Everything here installs, runs, and tests locally. -``` -dfo-spec-kit/ -├── docs/frame-grammar-v1.md # the normative frame grammar / content_version specification -├── docs/identity-v1.md # the normative identity binding-state denotation specification -├── schemas/registry/ # JSON Schema for the code registry -├── schemas/spec/ # frozen JSON Schema exported from the pydantic models -├── schemas/vectors/ # record schemas + docs for the identity and unit vector suites -├── registries/codes.json # the seeded warning/refusal code registry -├── registries/units.json # the unit refs dfo_spec_kit.units types, its currency prefixes and codes -├── vectors/identity/ # cross-language identity conformance fixtures -├── vectors/units/ # cross-language typed-unit conformance fixtures -├── python/spec-kit/ # the dfo-spec-kit package (import dfo_spec_kit) -├── python/frame-grammar/ # the dfo-spec-kit-frame-grammar package (import dfo_frame_grammar), -│ # its own frame grammar vector suite shipped as package data -├── python/dfo-spec-kit-components/ # the dfo-spec-kit-components package (import dfo_spec_kit_components), -│ # the presentation catalogue bundle shipped as package data -├── python/provenance-docs/ # the dfo-spec-kit-provenance-docs package (import dfo_provenance_docs), -│ # the provenance documentation grammar and its vector suite as package data -├── python/descriptions/ # the dfo-spec-kit-descriptions package (import dfo_descriptions), -│ # the step and description schemas, the operation list and their vector suite as package data -├── python/client-contract/ # the dfo-spec-kit-client-contract package (import dfo_client_contract), -│ # the served OpenAPI document, strict fields, caller actions, scenarios, ref format and client rules as package data -├── policies/ # compatibility policy per artifact family -└── tools/ # codegen + drift-check scripts, including the registry and vector validators -``` +## Documentation + +- [docs/identity-v1.md](docs/identity-v1.md) specifies what a spec identity's bindings denote. +- [docs/frame-grammar-v1.md](docs/frame-grammar-v1.md) specifies the frame byte grammar and `content_version`. +- Each package README under `python/` documents that package's API and vector suite. +- [policies/](policies/README.md) states which change to which artifact needs which version bump. +- [docs/development.md](docs/development.md) holds maintainer notes. + +## Contributing + +Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). + +## Licence + +MIT. See [LICENSE](LICENSE). diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..5f2ba4c --- /dev/null +++ b/docs/development.md @@ -0,0 +1,31 @@ +# Maintainer notes + +Background on how the repository fits together. Contributors do not need it to open a pull request (see [CONTRIBUTING.md](../CONTRIBUTING.md)). + +## Two schema families + +The kit carries two schema families, each authored in the language where its behaviour lives. Committed JSON Schema is the neutral centre between them, held together by cross-language conformance vectors. + +The dataset-spec family (spec models, identity projection, JCS canonicalisation, SHA-256 hashing) is Python-first. The behaviour lives in `python/spec-kit`, and `schemas/spec/` is the place for JSON Schema exported from its pydantic models. It holds only a README today, because `dfo_spec_kit.models` has no code yet. + +The component-spec family (the presentation catalogue, composition schema, operation union) is TypeScript-first: Zod v4 authors and validates, and generated artifacts (TypeScript prop types, Pydantic models, tool metadata) are checked for drift. The family's artifact in this repository is the catalogue bundle in `python/dfo-spec-kit-components/`. The repository has no TypeScript package. + +[policies/dataset-spec.toml](../policies/dataset-spec.toml) and [policies/component-spec.toml](../policies/component-spec.toml) state the compatibility rules for each family. + +## Dependencies between packages + +`dfo-spec-kit-frame-grammar` depends on `dfo-spec-kit` through a path source (`../spec-kit`, editable) in its `[tool.uv.sources]`. The two packages develop together here, so a change to identity canonicalisation and a change in the frame grammar that depends on it land in the same commit. + +`dfo-spec-kit-components` depends on `dfo-spec-kit-descriptions` the same way, because the envelope's `description` references `description.schema.json` by its `$id`. The catalogue bundle is static data plus one accessor function, `bundle_dir()`. + +`dfo-spec-kit-provenance-docs` depends on `markdown-it-py` only. `dfo-spec-kit-descriptions` depends on `rfc8785` only. `dfo-spec-kit-client-contract` depends on the standard library only. + +A package's published metadata names its kit dependency by distribution name alone, and no index serves these distributions. uv reads the path source and resolves the dependency from the same checkout. Every other installer reads the bare name and fails, which is why the README shows pip installs that name both distributions at one commit. A consumer pins both at one revision. + +## Releases + +No registry workflow exists for any package, so releases are git revisions. The compatibility policies define "released" as reachable from the most recent git tag matching `v*`. `tools/validate_registry.py` diffs the working registry against that tag to enforce that published code entries do not change, apart from the deletion exception named in the script and growth of a published entry's `mappings`. This repository has no tag reachable from `dev`, so the check is skipped and the `registry` CI job fetches full history so that it can run once one exists. + +## Ownership + +`CODEOWNERS` lists a maintainer for `/schemas/`, `/registries/`, `/vectors/`, and `/python/`.