improv — image provenance

improv is a shared data platform for scientific imaging instruments. It stores, organizes, and queries images and their associated scientific products — regardless of instrument type or scale.

Every image accumulates an append-only provenance log: geolocation, segmentation outputs, classifier scores, human annotations, oceanographic context. Records are never deleted or overwritten. A classifier can be re-run years later and its outputs attach to the same images alongside the original run.

Architecture

improv has three storage layers:

• Object store — raw image bytes and binary products (segmentation masks, etc.), keyed by image ID
• Columnar store — queryable image metadata, provenance records, and plugin index tables (DuckDB+Parquet or VAST DB)
• OLTP database — mutable organizing metadata: instruments, samples, datasets, ingest tasks (PostgreSQL or SQLite)

A plugin system extends provenance handling. Plugins register via dependency injection — the core never imports them at load time. Each plugin handles a specific provenance kind and optionally maintains an index table for fast querying. Plugins are generic and parameterized (geolocation, sample context, machine classification); instrument-specific presets live under improv.plugins.ifcb (IFCB morphometric features, IFCB CNN classification), pinning a kind/index_table onto a generic plugin.

Access patterns

• By time and instrument
• By spatial bounding box (lat/lon/depth)
• By named dataset (defined as time spans)
• By sample (for discrete-sample instruments)
• By provenance kind

REST API

improv exposes a FastAPI service with endpoints for image data and metadata, provenance, instruments, samples, datasets, and ingest task tracking. Classifier support adds taxonomy registration/lookup and two decode paths: a stateless decode (caller supplies a vector) and a decoded read that fetches an image's classification provenance and resolves each record against its own model_version. A thin HTTP client (improv.client.ImprovClient) is provided for ingest scripts that need OLTP access without direct database credentials, including taxonomy registration.

Authentication

The service authenticates with a single shared bearer token, supplied as IMPROV_API_TOKEN. It is required — create_app refuses to start without it rather than silently serving an unprotected API.

export IMPROV_API_TOKEN='...'   # generate with: python -c 'import secrets; print(secrets.token_urlsafe(32))'

Callers pass it as Authorization: Bearer <token>. Missing or invalid token → 401; a valid token lacking the required scope → 403.

Protection is declared per endpoint by scope, not inferred from the HTTP verb:

Surface	Auth
All writes (image ingest, provenance ingest, instrument/sample/dataset/taxonomy registration, ingest-task create/update/delete)	write scope
Ingest-task read, classifier decode	read scope
Image bytes, blobs, metadata, image/sample search, provenance reads, dataset and instrument lookup, taxonomy lookup	open — no token

Reads of image and provenance data are deliberately open. Deployments needing read authentication should put the service behind a reverse proxy, or add require_scope(READ) to the read routes.

Token handling is swappable. improv.api.auth defines a TokenVerifier protocol whose only implementation today is StaticTokenVerifier; moving to per-client tokens with real per-token scopes means adding a DB-backed verifier and changing the single construction site in create_app — routes are untouched.

ImprovClient reads IMPROV_API_TOKEN from the environment by default, or takes it explicitly:

ImprovClient("http://improv.example:8000", token="...")

Ingest architecture

Batch producers (ingest pipelines, classifiers) use a hybrid approach:

• OLTP operations (register instruments, samples, ingest tasks, classifier taxonomies) go through the REST API
• High-volume writes (image metadata, provenance, index tables, image bytes) go directly to the columnar store and object store

This avoids coupling ingest scripts to the database while keeping high-throughput writes off the HTTP path. Low-volume producers that prefer not to depend on the columnar store directly can post small provenance batches over REST (POST /images/provenance/batch, one instrument per batch).

Idempotency

The provenance log is append-only, so writes are never overwritten — idempotency is achieved by appending, then deduplicating at read time. This works identically on every backend (VAST DB, DuckDB+Parquet, and any future one), because append is the only operation they all share.

• A provenance record is identified by (image_id, kind, source, data_hash), where data_hash is a canonical RFC 8785 (JCS) hash of the data payload, stamped server-side. Re-posting a byte-identical record re-appends a row that collapses to one on read; a genuinely different payload (e.g. a new model_version) hashes differently and is retained. Canonicalization normalizes key order and number formatting (1.0 == 1) across producers in different languages, and rejects NaN/Infinity.
• Index records are deterministic projections deduplicated on their full column tuple.

Client contract: a record's timestamp is the event time (when the image was collected, or the classifier result produced) — a property of the observed fact, captured once. It is not the time of the HTTP request. Retries must resend the identical record, so put real event time in timestamp, never wall-clock-at-send; otherwise each attempt looks distinct and will not deduplicate. (The server separately records its own write time.)

Install

pip install .            # base — columnar store, object store, models, client
pip install '.[db]'      # adds SQLAlchemy for direct OLTP access
pip install '.[service]' # adds FastAPI, CLI, migrations, S3 object store
pip install '.[vastdb]'  # adds the VAST DB columnar backend

Running the service

The service is configured entirely from the environment. improv.asgi:app builds it by calling load_config() and create_app().

# Neither schema is created at startup. Run both before serving.
improv db upgrade     # OLTP schema — owned by Alembic
improv schema sync    # columnar tables and plugin index tables

uvicorn improv.asgi:app --host 0.0.0.0 --port 8000 --workers 4 \
        --proxy-headers --forwarded-allow-ips=<reverse-proxy-ip>

--proxy-headers matters whenever a reverse proxy terminates the connection; without it the client IP and scheme the app sees are the proxy's.

improv schema sync reads the same IMPROV_* variables the service does, so it provisions exactly the store the workers will open — including an index table for every plugin named in IMPROV_PLUGINS. Both commands are idempotent and exit non-zero on failure. Re-run improv schema sync after adding a plugin to IMPROV_PLUGINS, or that plugin's index table will not exist and its writes will fail.

Startup declares these tables — the store needs each table's schema and partition keys in memory — then checks they exist and refuses to start if any are missing, naming them. Forgetting to sync is a boot failure, not a surprise at the first write.

Startup never creates them. Creation is a single-process operation: with --workers 4 the lifespan runs once per worker concurrently, and no columnar backend offers an atomic create-if-absent, so several workers can each observe an absent table and each try to create it. (VAST DB's SDK takes a fail_if_exists flag, but it gates a client-side existence check before an unconditional create RPC — the flag never reaches the server.) This is the same discipline Alembic already imposes on the OLTP side. The existence check is a read, so it carries no such risk — but it does mean the columnar store must be reachable for a worker to boot.

GET /healthz is open and shallow — it touches neither Postgres nor the columnar store, so it reports process liveness rather than dependency health.

Environment variables

Columnar store

Variable	Notes
IMPROV_DB_BACKEND	duckdb (default) or vastdb
IMPROV_DB_ROOT	duckdb: local path or s3://bucket/prefix — required
IMPROV_S3_ENDPOINT	duckdb on S3: bare host:port, e.g. vast-s3.whoi.edu:9000
IMPROV_S3_ACCESS_KEY, IMPROV_S3_SECRET_KEY	duckdb on S3
IMPROV_S3_USE_SSL	set false for an http-only endpoint
IMPROV_DUCKDB_THREADS	DuckDB thread count; default is all cores
IMPROV_VASTDB_ENDPOINT	vastdb — required
IMPROV_VASTDB_ACCESS_KEY, IMPROV_VASTDB_SECRET_KEY	vastdb — required
IMPROV_VASTDB_BUCKET, IMPROV_VASTDB_SCHEMA	vastdb — required

OLTP database

Variable	Notes
IMPROV_DATABASE_URL	SQLAlchemy URL, e.g. postgresql+psycopg2://user:pass@host/improv. Required to serve — the service refuses to start without it rather than falling back to a throwaway in-memory database

Plugins and parsers

Comma-separated names, instantiated in order. An unknown name fails startup: a plugin that is configured but not registered would leave its index table missing and its provenance kind unindexed, which otherwise only shows up much later as empty query results.

Variable	Available values
IMPROV_PLUGINS	geolocation, sample_context, blob, machine_classification, ifcb_features, ifcb_cnn_classification
IMPROV_PARSERS	ifcb

Object store — at most one; S3 wins if both are configured.

Variable	Notes
IMPROV_OBJECT_BUCKET	bucket for image bytes and blobs; selects the S3 backend
IMPROV_OBJECT_S3_ENDPOINT	full URL with scheme, e.g. https://vast-s3.whoi.edu — note this differs from IMPROV_S3_ENDPOINT, which DuckDB and PyArrow want as a bare host:port
IMPROV_OBJECT_S3_ACCESS_KEY, IMPROV_OBJECT_S3_SECRET_KEY
IMPROV_OBJECT_S3_CA_BUNDLE	CA bundle path for an internally-signed endpoint
IMPROV_STORAGE_PATH	local/NFS path, selects HashdirStore instead

Auth

Variable	Notes
IMPROV_API_TOKEN	Required. See Authentication above

Docker

Two containers — the uvicorn API and Postgres — with the reverse proxy outside Docker. .env is the entire configuration mechanism: the application reads os.environ directly, with no dotenv support of its own.

cp .env.example .env && chmod 600 .env   # then fill it in
docker compose up -d --build

Bring-up order is enforced rather than assumed: db must pass pg_isready, then the one-shot migrate (improv db upgrade) and schema-sync (improv schema sync) services both run to completion, then api starts. If either exits non-zero, api never starts.

Nothing self-creates schema — Alembic owns the OLTP side, improv schema sync owns the columnar side — so a schema change is deployed by rebuilding and running up again, which re-runs both one-shots as no-ops when already at head. Note that docker compose restart api does not re-run them: after changing IMPROV_PLUGINS, use docker compose up -d.

The API publishes on 127.0.0.1:8000 only. Postgres publishes no port at all (docker compose exec db psql -U improv to reach it).

Dependencies

Package	Role
amplify-db-utils	Columnar storage (DuckDB+Parquet / VAST DB)
amplify-storage-utils	Object storage (HashdirStore / S3)
pydantic	Models and validation
pyarrow	Columnar data exchange
rfc8785	Canonical (JCS) hashing of provenance payloads for idempotency
httpx	Thin ingest client
fastapi, sqlalchemy, alembic	Service extras