Make pondie's schema checks pass on main

extraction-readme.md

@@ -110,7 +110,7 @@ whether the vocabulary is short a value. Twenty-six fields are open, including
 `Analysis.spatial_scope`, `Measure.family`, `Task.stimulus_modality`, and `Task.response_modality`.
 
 Which a field is, is storage's decision and not one extraction may revisit;
-`check_extraction_to_storage_map.py` fails if the projection opens or closes a vocabulary.
+`pondie.schema.checks.extraction_to_storage_map` fails if the projection opens or closes a vocabulary.
 
 ### Missingness has one encoding, and it is `not_reported`
 
@@ -154,7 +154,7 @@ This rule does **not** change a value that names something the source reported:
 
 ### Evidence invariants
 
-Enforced by [review/validate_record.py](review/validate_record.py):
+Enforced by `pondie.extraction.record.validate`:
 
 - `extraction_status: not_reported` ⇒ omit `value`, and `evidence.status` must be
   `not_applicable`. A field recorded this way therefore carries **no span**: there is no
@@ -224,7 +224,7 @@ composite of subscales still names the instrument the numbers came from. What th
 *was* goes in `source_definition`, and there it is not optional: `FactorLevel.timepoints` names
 the occasions a factor *compares*, a derived column has no levels to hang them on, so this is
 the only place saying which occasions it spanned. A derived column with neither slot filled is
-an unattributed number, and `validate_record.py` says so.
+an unattributed number, and `pondie.extraction.record.validate` says so.
 
 The region's own facts go on the `Region`: `name` in the source's wording,
 `definition_method` for how it was delimited — `same_study_analysis` where it came from this
@@ -642,7 +642,7 @@ Extraction carries storage's own vocabularies, so no normalization step is neede
 arrives is already one storage accepts.
 
 Facts the storage schema needs that no extraction field supplies. These are exactly the
-storage fields marked `in_subset: [deterministic]`, and `check_extraction_to_storage_map.py`
+storage fields marked `in_subset: [deterministic]`, and `pondie.schema.checks.extraction_to_storage_map`
 asserts that correspondence in both directions, so this table cannot quietly fall behind:
 
 | Storage field | How the mapper fills it |
@@ -676,11 +676,9 @@ extraction fields had nowhere to land and a section of the map called `absorbed_
 recorded where each one went instead.
 
 That is gone. The extraction schema is generated from the storage schema by
-`gen_extraction_schema.py`, so an extraction field with no storage counterpart is now
+`pondie.schema.generate`, so an extraction field with no storage counterpart is now
 impossible by construction rather than merely documented, and
-`check_extraction_to_storage_map.py` asserts the correspondence in both directions.
-[test_map_coverage.py](test_map_coverage.py) exercises each failure mode against mutated
-copies of the real schemas, because a completeness check that cannot fail is worth nothing.
+`pondie.schema.checks.extraction_to_storage_map` asserts the correspondence in both directions.
 
 The shapes that used to need absorbing are listed as candidates at the end of
 [extraction-deviations.yaml](src/study_schema/linkml/extraction-deviations.yaml). They were not obviously wrong --

schema-tutorial.md

@@ -196,7 +196,7 @@ something — see §2.4. **`unique_keys`** says a combination may not repeat wit
 | Hand-edited? | Never — re-running the mapper regenerates it | Never — produce a new version or a separate correction record |
 | Size | 52 classes, 57 vocabularies, 365 fields, across 13 modules | Generated: the same classes plus the evidence types |
 
-The extraction schema is **generated** from storage by `gen_extraction_schema.py`, so the two cannot
+The extraction schema is **generated** from storage by `pondie.schema.generate`, so the two cannot
 drift. The whole of the projection:
 
 - fields marked `deterministic` are dropped (code fills them, so there is nothing to read off a page);
@@ -208,7 +208,7 @@ drift. The whole of the projection:
   record, rather than a list of wrappers;
 - an inlined child is projected too; a reference stays a reference and resolves through `local_id`;
 - `minimum_value`, `rules` and `unique_keys` are **dropped**, because they constrain a scalar and the
-  wrapper is in the way. Those move into `review/validate_record.py` (chapter 12).
+  wrapper is in the way. Those move into `pondie.extraction.record.validate` and `pondie/extraction/record/rules.py` (chapter 12).
 
 You fill the extraction schema. You read either. Everything this tutorial says about a field is true
 of both, because they are the same field.
@@ -289,7 +289,7 @@ low-priority and inside it.
 `deterministic` is what an API, a parser, or a derivation supplies: bibliographic metadata, PubMed
 publication types, minted identifiers, `Table.coordinate_space`, NeuroVault maps. `model_extracted`
 is everything read off the page. Every field is in exactly one of the two, and
-`check_field_provenance.py` fails if not; identifiers and type designators are in neither by
+`pondie.schema.checks.field_provenance` fails if not; identifiers and type designators are in neither by
 definition.
 
 A derivation need not come from outside the record. `ConnectivityEdge.directionality` is derived from
@@ -1505,7 +1505,7 @@ not change shape because the question changed; the cells do.
 
 > **A value the article does not report takes `extraction_status: not_reported`.**
 
-No vocabulary offers an `unstated` member, and `check_schema.py` fails the build if one appears —
+No vocabulary offers an `unstated` member, and `pondie.schema.checks.linkml` fails the build if one appears —
 silence recorded two ways is silence a query finds half of.
 
 Nothing is lost by that rule, because a `not_reported` wrapper is still **present**: everything the
@@ -1534,7 +1534,7 @@ and an extractor is asked for nine. `EffectKind` is bound by no field at all (§
 37 vocabularies come to 26 + 10 + 1.
 
 Whether a field is open or closed is storage's decision, and the projection may not revisit it:
-`check_extraction_to_storage_map.py` fails if extraction opens a closed vocabulary or closes an open
+`pondie.schema.checks.extraction_to_storage_map` fails if extraction opens a closed vocabulary or closes an open
 one. The second direction is the quiet one — closing an open vocabulary works fine and costs you the
 signal that a value is missing.
 
@@ -1635,7 +1635,7 @@ change if a different contrast came off the same model?
 
 ### 12.3 What the validator checks
 
-`review/validate_record.py` runs three tiers over a finished extraction record. Errors mean the record
+`pondie.extraction.record.validate` (with the rules in `pondie/extraction/record/rules.py`) runs three tiers over a finished extraction record. Errors mean the record
 is wrong; warnings mean a reviewer has to look.
 
 **Tier 1 — conformance (errors).** Types, required slots, closed vocabularies, the evidence
@@ -1815,15 +1815,14 @@ model, `FactorLevel.level` within its term, `(term, level)` within one `Effect`,
 **The checks.**
 
 ```bash
-python3 check_schema.py                        # both schemas load; no rule names a missing slot
-python3 check_field_provenance.py --strict     # every field says how it gets filled
-python3 check_extraction_to_storage_map.py     # the projection has not drifted
-python3 gen_extraction_schema.py --check       # the committed extraction tree is current
+python3 -m pondie.schema.checks.linkml                    # both schemas load; no rule names a missing slot
+python3 -m pondie.schema.checks.field_provenance --strict # every field says how it gets filled
+python3 -m pondie.schema.checks.extraction_to_storage_map # the projection has not drifted
+python3 -m pondie.schema.generate --check              # the committed extraction tree is current
 python3 -m pytest                              # everything the review layer stands on
-python3 review/validate_record.py --record <id>.extraction.json --text <text>
+python3 -m pondie.extraction.record.validate --record <id>.extraction.json --text <text>
 ```
 
 **Worked records.** [representing-models.md §5](representing-models.md#5-worked-models) has twelve,
-each with the sentences from the paper it came from and a referent record. Three complete extraction
-records live under `review/examples/`, each with a corrections note saying what the extractor got
-wrong and why.
+each with the sentences from the paper it came from and a referent record. A complete extraction
+record is `examples/run/records/x7Ex4mpleA01.extraction.json`.

src/study_schema/linkml/extraction-to-storage.map.yaml

@@ -4,7 +4,7 @@
 #
 # THE MAP IS AN IDENTITY MAP
 # --------------------------
-# The extraction schema is generated from the storage schema by gen_extraction_schema.py,
+# The extraction schema is generated from the storage schema by pondie.schema.generate,
 # so the two have the same classes with the same slots in the same nesting. A storage
 # field is therefore populated from the extraction field of the same name on the same
 # class, and the mapper needs no instruction to do it. What the mapper does uniformly:
@@ -32,7 +32,7 @@
 #   fact copied from the parsed paper (`copy`, naming `source_artifact` and
 #   `source_field` in neuroimaging-paper-parse), a fact an API states, an identifier the
 #   pipeline mints, a value computed from an ingestion artifact. They are exactly the storage fields marked
-#   `in_subset: [deterministic]`, and check_extraction_to_storage_map.py asserts that
+#   `in_subset: [deterministic]`, and pondie.schema.checks.extraction_to_storage_map asserts that
 #   correspondence in both directions.
 #
 # free_text_normalizations
@@ -81,6 +81,15 @@ derivations:
     transform: generate
   Assessment.id:
     transform: generate
+  CategoryDistribution.category_normalized:
+    transform: derive
+    source_representation: sibling_field
+    source_field: CategoryDistribution.category
+    notes: >-
+      `pondie.normalization.sex_distribution` and `handedness_distribution` fold the
+      study's wording in `category` onto the answer set of the distribution the entry
+      belongs to. A distribution with no normalizer (gender, race, ethnicity) leaves it
+      unset; unset is not UNKNOWN, which says a normalizer ran and found no match.
   Condition.id:
     transform: generate
   ConnectivityEdge.directionality:
@@ -103,10 +112,41 @@ derivations:
       coherence: undirected
       structural_covariance: undirected
       multivariate: undirected
+  CoordinateSet.id:
+    transform: derive
+    source_representation: ingestion_artifact
+    source_field: ParsedAnalysis.key
+    notes: >-
+      The `analysis#<h>` key the coordinate parse gave the analysis this set was read from
+      (neuroimaging-paper-parse), the key `Analysis.source_table_analysis` names. It hashes
+      the set's name, space and points, so nothing here is extracted or minted.
+  CoordinateSet.table:
+    transform: derive
+    source_representation: ingestion_artifact
+    source_field: ParsedAnalysis.table_id
+    notes: >-
+      The Table whose `Table.id` is the parse analysis's `table_id`. Unset when the set was
+      read from the text or a figure, where `table_id` is absent.
+  CoordinateSet.text_excerpt:
+    transform: derive
+    source_representation: ingestion_artifact
+    source_field: ParsedAnalysis.text_spans
+    notes: >-
+      The paper's text over the analysis's `text_spans`, the sentences its coordinates were
+      read from. Unset for a set read from a table.
   Device.id:
     transform: generate
   Region.id:
     transform: generate
+  Group.age_unit_normalized:
+    transform: derive
+    source_representation: sibling_field
+    source_field: Group.age_unit
+    notes: >-
+      `pondie.normalization.age_unit` folds the source's wording in `age_unit` onto the
+      AgeUnit vocabulary, UNKNOWN where the wording does not say. `age_unit` keeps the
+      wording, and its own entry under free_text_normalizations tidies spelling for
+      queries; this field is the comparable one.
   Group.id:
     transform: generate
     notes: UUID assigned by mapper; extraction local_id stored in audit log.

src/study_schema/linkml/neuroimaging-study-extraction.yaml

@@ -11,7 +11,7 @@ name: neuroimaging-study-extraction
 title: Neuroimaging Study Extraction Schema
 description: |-
   Immutable, evidence-first structured output from extracting a neuroimaging publication.
-  This schema is generated from neuroimaging-study-storage by gen_extraction_schema.py. It has the same classes with the same slots in the same nesting; what differs is that every source-derived value is wrapped. An ExtractedValue records whether a value was found (extraction_status), the value itself when it was, how it relates to the source (value_source), why there is none when there is not (unreported_reason), and the exact supporting spans (evidence). A slot needing a scalar uses a typed subtype -- ExtractedString, ExtractedInteger, ExtractedNumber, ExtractedBoolean. Storage's vocabularies come across whole, each field keeping the range storage gives it: a closed enum takes a permissible value and nothing else, and an open one takes a permissible value or, when none fits, the source's own wording. Identifiers, type designators, and pipeline provenance use native LinkML types, because nothing in the paper supplies them.
+  This schema is generated from neuroimaging-study-storage by pondie.schema.generate. It has the same classes with the same slots in the same nesting; what differs is that every source-derived value is wrapped. An ExtractedValue records whether a value was found (extraction_status), the value itself when it was, how it relates to the source (value_source), why there is none when there is not (unreported_reason), and the exact supporting spans (evidence). A slot needing a scalar uses a typed subtype -- ExtractedString, ExtractedInteger, ExtractedNumber, ExtractedBoolean. Storage's vocabularies come across whole, each field keeping the range storage gives it: a closed enum takes a permissible value and nothing else, and an open one takes a permissible value or, when none fits, the source's own wording. Identifiers, type designators, and pipeline provenance use native LinkML types, because nothing in the paper supplies them.
   Evidence is present when one or more independently sufficient EvidenceSets were found, or not_found when a value was extracted but no span supports it. For a not_reported attribute, omit value and set evidence.status to not_applicable. Entities refer to each other by document-local local_id; ingestion resolves them and mints the storage identifiers. Do not manually correct an extraction record: create a new extraction version or a separate correction record.
   Rules this schema cannot state -- which papers are skipped before extraction, how a multivalued extracted fact is shaped, which invariants the structural validator enforces, and what a paper may report that no field holds -- are in extraction-readme.md. Everything this schema does that is not a projection of storage is in extraction-deviations.yaml.
 license: CC0-1.0

src/study_schema/linkml/neuroimaging-study-extraction/acquisition.yaml

@@ -98,14 +98,14 @@ classes:
     attributes:
       stable_id:
         range: string
-        pattern: '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
+        pattern: ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$
         description: >-
           UUID minted by the store the first time this entity appears, never by an extractor,
           and kept across re-runs so a later extraction matches the same entity.
       local_id:
-        identifier: true
         range: string
         required: true
+        identifier: true
       acquisition_type:
         range: string
         required: true
@@ -289,14 +289,14 @@ classes:
     attributes:
       stable_id:
         range: string
-        pattern: '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
+        pattern: ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$
         description: >-
           UUID minted by the store the first time this entity appears, never by an extractor,
           and kept across re-runs so a later extraction matches the same entity.
       local_id:
-        identifier: true
         range: string
         required: true
+        identifier: true
         description: Stable identifier used by Acquisition foreign-key references.
       manufacturer:
         range: ExtractedString