Skip to content

Unify viral reference data and proteome APIs - #123

Merged
iskandr merged 18 commits into
mainfrom
metadata/gdc-genbank-summaries
Oct 25, 2026
Merged

iskandr merged 18 commits into
mainfrom
metadata/gdc-genbank-summaries

Conversation

@iskandr

@iskandr iskandr commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds on-demand viral reference downloads and gives ORFdata one consistent API
for collections, species, sources and protein inputs without exposing storage
machinery as domain concepts.

  • ViralCollection describes one published collection. The same object drives
    discovery, inventory reads, source verification, downloads and offline opens;
    adding a species or source is registry data rather than another code path.
  • collection.inventory() and collection.source_summary(source) return pandas
    frames with declared columns. The primary source is the default everywhere.
  • download_viral_references(), open_viral_references() and
    select_viral_references() work for every collection and return the same
    ViralReferenceFiles interface.
  • list_data() -> tuple[DataStatus, ...] discovers bundled, downloadable and
    buildable data without implying that every item is a download.
  • A Proteome contains ProteinSequence values in .proteins.
    ProteinSet is the general sequence-only input for a selected component;
    with_protein_set() records donor/transgene use and with_viral() adds a
    sourced host association. ncORFs with coordinates and evidence remain
    ORFCatalog records and produce the same ProteinSequence output.
  • Checked-TSV declarations, source checks, fetchers and cached source documents
    are internal implementation details rather than package-root vocabulary.

Downstream producer contract

The open Varcode, Isovar and Vaxrank requests are covered directly:

  • exporter/wrapper names are provenance, not source identity; direct and
    wrapped copies merge once and retain all paths in exporters
  • an external protein may retain taxon_id=None; admitting it to a known host
    requires compatibility="allow_unknown" and does not rewrite the source fact
  • producer version, provider, export paths, exact reference, source checksum,
    ORF category and true/false/unknown boundaries appear in safely escaped FASTA
  • exact-sequence FASTA deduplication uses a neutral group header and maps back
    to every contributing source protein
  • a selected donor/transgene set retains its own taxon/reference/checksum and a
    separate sourced assertion for the target proteome

Proteome snapshots use schema v5. Loading v1-v4 remains supported; v4 producer
IDs and graph links are normalized away from exporter-specific identity while
retaining the exporter path.

Viral reference data

orfdata download viral --species human installs gdc-viruses, the 200-record
GDC collection, in identifier-compatible genome, transcriptome and proteome
layers. viral_genome() opens the collection or a selected subset as a
PyEnsembl genome for Varcode, Isovar, Topiary and Vaxrank.

Every fetched source record must reproduce the pinned source identifier,
sequence length, MD5, SHA-256 and coding-feature count. GDC's HPV16 contig is
therefore taken from PaVE's HPV16REF, whose sequence reproduces the published
GDC MD5, rather than the different GenBank accession printed beside it. A test
registers a mouse collection and downloads it end to end without a new loader.

Named YAML subsets keep panel membership, host association and tumour mechanism
separate. Optional alternate annotations apply only to an identical molecule.
transcripts.tsv records initiation and recoding, and
viral_annotation_survey() reports annotation completeness without a built-in
gene list.

Validation

  • ./test.sh: 826 passed
  • ./lint.sh: passed
  • ./docs.sh: passed
  • python -m build and python -m twine check: passed
  • focused producer identity, unknown-taxon, generic ProteinSet, legacy v4
    migration and FASTA safety tests: passed

@iskandr iskandr changed the title Record resolved GDC viral GenBank accession metadata Add on-demand viral reference downloads Oct 22, 2026
ORFdata packaged three tables beside its code and read each one with its own
loader: a collection's inventory, an NCBI summary table for its primary
source, and an alternate-reference ledger for its second. Three formats, three
checksum checks, three builders — and a fourth source, panel or species meant
a fourth of each.

There is now one of each. A PackagedTable is a tab-separated file with a named
schema and a pinned SHA-256, read and written in orfdata.reference_table; a
collection declares the tables it reads instead of naming resources and
checksums in module constants. Every source is described by the same ledger
(orfdata.reference_ledger.v2): per entry, the identifier that source resolved
it to, the molecule's length, MD5 and SHA-256, its coding-feature count, its
NCBI taxon and its own description. The GenBank summary table becomes
gdc-grch38-d1-vd1-genbank.tsv and the alternate ledger becomes
gdc-grch38-d1-vd1-pave.tsv, both built by one tool from one pass over the
inventory.

Making the sources symmetric made the downloads stricter: a primary record is
now checked against its own ledger the way an alternate always was, so a
record NCBI no longer serves as pinned fails loudly rather than being written.
SourceUnavailable distinguishes a refusal the server states from a record that
cannot be read, so unavailable and inconsistent are recorded from what
happened rather than matched out of an error message. The GenBank reader keeps
a record's DEFINITION and its source feature's taxon, so a source record
describes itself and the parsed-record class no longer shares a name with the
cross-source SourceRecord.

Inventory queries name their collection, by collection name or panel name, and
the published inventory's URL, size and checksum moved to the registry beside
it. A new test registers a second collection for another species from two
packaged tables and downloads it end to end, with no new code path.

Rebuilt and verified against the live sources: the GenBank ledger reproduces
all 200 accession versions, lengths, taxa and titles of the table it replaces
and every sequence checksum of a real download; the PaVE ledger reproduces its
predecessor in every shared column, 187 published and 2 unavailable. A real
build is unchanged at 200 genomes and 1,758 coding transcripts with a
byte-identical genome FASTA, and --alternate-annotations still gives 1,993.

815 tests pass; the inventory builder reproduces its packaged file byte for
byte through the new writer.

Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
read_ledger builds a fresh mapping over bytes that are already read, checked
and cached by the packaged table, so a caller is free to keep or discard what
it gets back.

Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
The HPV16 catalogue checked its own bytes the way the reference tables used
to, so that check now lives once in reference_table.packaged_bytes and both
go through it: nothing packaged is read before its SHA-256 is the pinned one.

Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
Every packaged reference table — a collection's inventory and one ledger per
source — now reads as a pandas DataFrame instead of a tuple of per-row
dataclasses. The row classes existed only because there was nothing to hold a
table; with pandas they are a layer between bytes and callers that earns
nothing, so ViralPanelEntry, LedgerEntry and ViralContext are gone and a row
of a table is a row of a table.

What the tables are for is unchanged, and so is what reading one refuses:
the pinned SHA-256, the declared schema and the declared columns are still
checked before any row is handed over. A table's own header lines now travel
on frame.attrs. LedgerAnchor is Anchor, and the methods its rows used to
carry are plain functions over a row: published, same_molecule, verify_record.
ReferenceSource.ledger_entries is build_ledger and also returns a frame.

On-disk formats and checksums are untouched: all three packaged tables
round-trip byte for byte through the new reader and writer, and a full build
against NCBI and PaVE reproduces the previous genome FASTA SHA-256
(1673b35f4988ec9d176b94491d04342d639742b0dbc9f966cd61dcd6ab4f88d1), its 200
genomes and 1,758 transcripts, and every source and initiation count.

pandas 3 is a new runtime dependency, which is how the rest of OpenVax
already carries tabular data.

Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
Each frame ORFdata hands back now declares its column set in code —
COVERAGE_COLUMNS, CONTEXT_COLUMNS, FRAME_COLUMNS — documents every column's
dtype and meaning in the module docstring and in docs/viral.md, and is held
to that set by a test, so the documentation cannot drift from the frame.

PackagedTable says the same thing generically: a read table's columns are
exactly the ones its declaration names, in that order, every one a str except
those its `numeric` names, which read as nullable Int64 so a blank stays
blank. The ledger's index is named (INDEX) and FRAME_COLUMNS is its shape,
which is now shared by packaged, freshly built and empty ledgers alike.

Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
@iskandr iskandr changed the title Add on-demand viral reference downloads Unify viral reference data and proteome APIs Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant