Repository navigation
Unify viral reference data and proteome APIs - #123
Merged
Merged
Conversation
ORFdata packaged three tables beside its code and read each one with its own loader: a collection's inventory, an NCBI summary table for its primary source, and an alternate-reference ledger for its second. Three formats, three checksum checks, three builders — and a fourth source, panel or species meant a fourth of each. There is now one of each. A PackagedTable is a tab-separated file with a named schema and a pinned SHA-256, read and written in orfdata.reference_table; a collection declares the tables it reads instead of naming resources and checksums in module constants. Every source is described by the same ledger (orfdata.reference_ledger.v2): per entry, the identifier that source resolved it to, the molecule's length, MD5 and SHA-256, its coding-feature count, its NCBI taxon and its own description. The GenBank summary table becomes gdc-grch38-d1-vd1-genbank.tsv and the alternate ledger becomes gdc-grch38-d1-vd1-pave.tsv, both built by one tool from one pass over the inventory. Making the sources symmetric made the downloads stricter: a primary record is now checked against its own ledger the way an alternate always was, so a record NCBI no longer serves as pinned fails loudly rather than being written. SourceUnavailable distinguishes a refusal the server states from a record that cannot be read, so unavailable and inconsistent are recorded from what happened rather than matched out of an error message. The GenBank reader keeps a record's DEFINITION and its source feature's taxon, so a source record describes itself and the parsed-record class no longer shares a name with the cross-source SourceRecord. Inventory queries name their collection, by collection name or panel name, and the published inventory's URL, size and checksum moved to the registry beside it. A new test registers a second collection for another species from two packaged tables and downloads it end to end, with no new code path. Rebuilt and verified against the live sources: the GenBank ledger reproduces all 200 accession versions, lengths, taxa and titles of the table it replaces and every sequence checksum of a real download; the PaVE ledger reproduces its predecessor in every shared column, 187 published and 2 unavailable. A real build is unchanged at 200 genomes and 1,758 coding transcripts with a byte-identical genome FASTA, and --alternate-annotations still gives 1,993. 815 tests pass; the inventory builder reproduces its packaged file byte for byte through the new writer. Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
read_ledger builds a fresh mapping over bytes that are already read, checked and cached by the packaged table, so a caller is free to keep or discard what it gets back. Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
The HPV16 catalogue checked its own bytes the way the reference tables used to, so that check now lives once in reference_table.packaged_bytes and both go through it: nothing packaged is read before its SHA-256 is the pinned one. Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
Every packaged reference table — a collection's inventory and one ledger per source — now reads as a pandas DataFrame instead of a tuple of per-row dataclasses. The row classes existed only because there was nothing to hold a table; with pandas they are a layer between bytes and callers that earns nothing, so ViralPanelEntry, LedgerEntry and ViralContext are gone and a row of a table is a row of a table. What the tables are for is unchanged, and so is what reading one refuses: the pinned SHA-256, the declared schema and the declared columns are still checked before any row is handed over. A table's own header lines now travel on frame.attrs. LedgerAnchor is Anchor, and the methods its rows used to carry are plain functions over a row: published, same_molecule, verify_record. ReferenceSource.ledger_entries is build_ledger and also returns a frame. On-disk formats and checksums are untouched: all three packaged tables round-trip byte for byte through the new reader and writer, and a full build against NCBI and PaVE reproduces the previous genome FASTA SHA-256 (1673b35f4988ec9d176b94491d04342d639742b0dbc9f966cd61dcd6ab4f88d1), its 200 genomes and 1,758 transcripts, and every source and initiation count. pandas 3 is a new runtime dependency, which is how the rest of OpenVax already carries tabular data. Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
Each frame ORFdata hands back now declares its column set in code — COVERAGE_COLUMNS, CONTEXT_COLUMNS, FRAME_COLUMNS — documents every column's dtype and meaning in the module docstring and in docs/viral.md, and is held to that set by a test, so the documentation cannot drift from the frame. PackagedTable says the same thing generically: a read table's columns are exactly the ones its declaration names, in that order, every one a str except those its `numeric` names, which read as nullable Int64 so a blank stays blank. The ledger's index is named (INDEX) and FRAME_COLUMNS is its shape, which is now shared by packaged, freshly built and empty ledgers alike. Claude-Session: https://claude.ai/code/session_01JfMjwFvskXFZXQjVAV7vgM
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds on-demand viral reference downloads and gives ORFdata one consistent API
for collections, species, sources and protein inputs without exposing storage
machinery as domain concepts.
ViralCollectiondescribes one published collection. The same object drivesdiscovery, inventory reads, source verification, downloads and offline opens;
adding a species or source is registry data rather than another code path.
collection.inventory()andcollection.source_summary(source)return pandasframes with declared columns. The primary source is the default everywhere.
download_viral_references(),open_viral_references()andselect_viral_references()work for every collection and return the sameViralReferenceFilesinterface.list_data() -> tuple[DataStatus, ...]discovers bundled, downloadable andbuildable data without implying that every item is a download.
ProteomecontainsProteinSequencevalues in.proteins.ProteinSetis the general sequence-only input for a selected component;with_protein_set()records donor/transgene use andwith_viral()adds asourced host association. ncORFs with coordinates and evidence remain
ORFCatalogrecords and produce the sameProteinSequenceoutput.are internal implementation details rather than package-root vocabulary.
Downstream producer contract
The open Varcode, Isovar and Vaxrank requests are covered directly:
wrapped copies merge once and retain all paths in
exporterstaxon_id=None; admitting it to a known hostrequires
compatibility="allow_unknown"and does not rewrite the source factORF category and true/false/unknown boundaries appear in safely escaped FASTA
to every contributing source protein
separate sourced assertion for the target proteome
Proteome snapshots use schema v5. Loading v1-v4 remains supported; v4 producer
IDs and graph links are normalized away from exporter-specific identity while
retaining the exporter path.
Viral reference data
orfdata download viral --species humaninstallsgdc-viruses, the 200-recordGDC collection, in identifier-compatible genome, transcriptome and proteome
layers.
viral_genome()opens the collection or a selected subset as aPyEnsembl genome for Varcode, Isovar, Topiary and Vaxrank.
Every fetched source record must reproduce the pinned source identifier,
sequence length, MD5, SHA-256 and coding-feature count. GDC's HPV16 contig is
therefore taken from PaVE's
HPV16REF, whose sequence reproduces the publishedGDC MD5, rather than the different GenBank accession printed beside it. A test
registers a mouse collection and downloads it end to end without a new loader.
Named YAML subsets keep panel membership, host association and tumour mechanism
separate. Optional alternate annotations apply only to an identical molecule.
transcripts.tsvrecords initiation and recoding, andviral_annotation_survey()reports annotation completeness without a built-ingene list.
Validation
./test.sh: 826 passed./lint.sh: passed./docs.sh: passedpython -m buildandpython -m twine check: passedProteinSet, legacy v4migration and FASTA safety tests: passed