diff --git a/docs/plans/2026-10-13-python-sdk-tracer-findings.md b/docs/plans/2026-10-13-python-sdk-tracer-findings.md new file mode 100644 index 000000000..b57cb043d --- /dev/null +++ b/docs/plans/2026-10-13-python-sdk-tracer-findings.md @@ -0,0 +1,298 @@ +# Python SDK tracer — findings from the first attempt + +> **Superseded on `next` (2026-10-20).** This is a snapshot of `main` on +> 13 October 2026. The engine has since moved on the `next` branch: a +> record crosses as a frame (`ValueFrame`), not an `FfiValue` tree; Stack +> Encrypt's EQL types are v4 only (`public.eql_v4_text_eq`), not the v3 +> domains this tracer stored into; most of the 51 EQL v4 types are +> producible (40 on 2026-10-20; `languages/golang/encrypt/eql/eql_gen.go` +> on `next` says which), not only `TextEq`; and the new kinds (`decimal`, +> `date`, `timestamp`) have landed. Read the engine sections as history. The +> Python and toolchain section (pyo3 0.29, maturin-only `extension-module`, +> libpython on the loader path, `Pending` as `IntoFuture`, width narrowing, +> the psycopg plumbing) still holds. The restart plan reads +> `packages/stack-encrypt/docs/language-integration.md` first; that guide +> is PR #1287, open against `next` as this header is written, and this +> document merges after it. + +**Status:** a proto-prototype, stopped by decision on 2026-10-13 after 7 of 9 +tasks, to be restarted fresh. The branch is kept as a reference under +`prototype/python-sdk-tracer`. **Do not merge it.** Read this file, then +the Python SDK discovery ticket, then start again from `next`. + +The code is disposable. What follows is not: it is what the code proved, +and most of it would take days to rediscover. + +## What was proven, live + +Against ZeroKMS (through the `stash auth login` profile) and a Postgres with +EQL v3.0.6 applied — 53 tests green, 3 Rust unit tests green: + +1. `import stackencrypt` from an `abi3-py39` wheel built by maturin over + `stack-encrypt`, in a fourth Cargo workspace at `languages/python`. +2. `Plan.of(User).context("users").build()` reads `Annotated` hints into the + engine's data plan, fails closed at `build()` on an undeclared field + (import time when that call sits at module level — by convention, not by + construction; see "Gaps against the discovery ticket's wording"), and the engine's + own `plan_with` validates the rest with no credentials. +3. `Cipher(plan).encrypt(models)` seals a batch in **one** ZeroKMS request; + an empty batch makes none; `decrypt` returns the model type; `query` + returns a term. +4. **Byte parity with Rust**, produced at test time: the shell opens what + Rust sealed, Rust opens what the shell sealed, and the equality term is + byte-identical. +5. **The whole six-step path through PostgreSQL**: insert into + `public.eql_v3_text_eq`, equality query through `eql_v3.query_text_eq`, + read back, decrypt. This is the "TextEq proven end-to-end in PostgreSQL" + that the catalog's storage-only `Text` domain records as its own blocker. + +Not done: CI (Task 8) and the scorecard (Task 9). + +## Findings about the engine and the catalog + +These are facts about `main` as of 2026-10-13, independent of any Python +design choice. + +- **`stack-encrypt`'s `dynamic` feature is the entire binding surface, and + it is off by default.** `stack_encrypt::dynamic::record` — the data plan, + `FfiValue`, `plan_with`, `encrypt_with`, `decrypt_with`, `query` — does not + exist without it. `eql-bindings`' own `stack-encrypt` feature takes the + crate with `default-features = false` and does **not** enable it; the + binding crate must. `http` (credential discovery) is a default and should + be named explicitly so it cannot be dropped silently. +- **The catalog produces exactly one EQL type: `TextEq`.** Every other entry + resolves `Unproducible`, with four recorded reasons: 42 types because + "how this family encodes a plaintext for the stack-encrypt producer + profile is not specified; only the text family's encoding is"; `TextMatch` + and the `Ope` types because "the engine derives match and OPE terms, but + no EQL type is built from them yet"; the ORE types because "EQL stores a + block ORE term and the engine derives a CLLW ORE term; the two are + different algorithms"; `Text` because it "follows once TextEq is proven + end-to-end in PostgreSQL". (Superseded: both are producible on `next`; + see open question 1.) **Consequence for the discovery ticket:** decision 9 lists + `equality`, `match`, `ore` as the engine features the four tasks need, and + backlog item 5 is "`match` and `ore` queries". Neither is a Python change; + both wait on the engine. Amend the ticket before anyone builds against it. +- **The engine-side design for that gap already exists:** "Value encodings + for new EQL kinds" (7 Oct 2026, the same day as the discovery ticket's + decisions; an internal design draft, since landed on `next` as + `vitaminc-ore` and the new kinds, so the engine source is now the + reference). What it established at the time: + - The 42 non-text families are tracked in **#1130** and are "blocked on + encoding, not on code". A kind must exist in **three** encodings owned + by three crates — the ciphertext tag (`vitaminc-aead-value`), the + equality domain (`vitaminc-prf`), the order encoding (`cllw-ore` / + `ore-rs`) — and today they disagree about which types exist (`i16` is in + two of three, so it cannot be a kind). + - The ORE refusal is **#1132**: EQL v3 stores **block ORE** for the + `_ord_ore` domains and **CLLW OPE** for `_ord` / `_ord_ope`; Stack + Encrypt derives CLLW ORE and CLLW OPE and no block ORE. That is why + `TextOrdOre` is refused although text is a kind. + - The proposal: a `vitaminc-ore` crate with one order-preserving plaintext + encoding (adopting `orderable-bytes`) and pluggable schemes (block ORE, + CLLW ORE, CLLW OPE); new kinds `Int16`, `Date`, `Timestamp`, `Decimal` + with new frozen tags from `0x0C`; a conformance test that fails unless + every kind has all three encodings. + - It does **not** cover `TextMatch`, whose blocker is different ("the + engine derives match terms, but no EQL type is built from them yet"). + So `match` is a smaller, separate, unticketed piece of work; `ore` is + this proposal. + - **It reaches Python directly.** The kind table is "a cross-language + commitment", and every new kind means a codec "in each binding": when + `Int16` / `Date` / `Timestamp` / `Decimal` land, the shell's + `convert.rs` grows an arm each, the width markers gain four names, and + `Date` versus `Timestamp` is the same declaration ambiguity it already + flags for Go's `time.Time`. The restart's declaration shape should + leave room for that rather than hard-code today's nine scalar kinds. +- **The engine core does more than the catalog.** Through the bare-outputs + path (`"outputs": ["c", "match"]`) it seals any kind and derives `eq`, + `match`, `ore`, `ope`. What is `TextEq`-only is the *framing* into an EQL + v3 column type, which is `eql-bindings`' catalog, not the engine. +- **`"type"` is required on every field with a term output** (#1096, now on + `main`), and `admits` decides which kinds an index accepts: `Equality` + takes the integer kinds, `string`, `bytes`; `Match` takes `string` only; + `Ore`/`Ope` take anything but arrays and objects. Kind names are + vitaminc's lowercase vocabulary (`string`, `int64`, `float64`, `bool`…), + parsed with `ValueKind: FromStr`; use that parser, never a table. +- **A stored record is a `CipherText` tree, not bytes.** Per row a `Map`; + per field a `Map` keyed by output: a target field's EQL JSON rides as a + passthrough `FfiValue::Bytes` under `"eql"`, a passthrough field's value + as a passthrough `FfiValue` under `"passthrough"`, a ciphertext as a + sealed subtree under `"c"`. The first two extract directly to column + values; the third needs a tree codec (the Go host has one). The tracer + refused bare-`"c"` fields by name and lost nothing. +- **Ciphertexts are never byte-equal** (fresh IV per seal). "The bytes + agree" can only mean the deterministic equality term; the ciphertext is + proven by cross-opening. +- **The EQL identifier `i` is stamped from the plan's context** at encrypt + time — `{"t": "", "c": ""}` — and is + authenticated into the ciphertext. `.context("")` and the SQL + table should be the same word. Postgres does not check it. +- **Credentials:** `StackCipher::new()` → `AutoStrategy`: `CS_CLIENT_ACCESS_KEY` + + `CS_WORKSPACE_CRN` (`auto_strategy.rs:120`), else the device session in + `~/.cipherstash//auth.json` (`:129-133`); the client + key likewise from `CS_CLIENT_ID`/`CS_CLIENT_KEY` or `secretkey.json`. A + logged-in developer machine needs nothing set. **Any committed bytes + keyed to a workspace keyset (equality terms) pass only in that + workspace.** Produce the Rust side of a parity check at test time instead. +- **`TextEq` needs nothing from Postgres that does not already exist**, which + is part of why it could ship first. Equality lives entirely in `hm`, the + HMAC; the ciphertext `c` is sealed with a fresh IV, so two stored values + for the same plaintext are *different* JSON and an index on the column + itself never matches. The index EQL v3 is built for is a functional one on + the extracted term — + `create index … using hash (eql_v3.eq_term(col))` (or `btree`: + the term's type rides on `bytea`'s default operator classes, so both work + and only `=` means anything) — and the bundle marks the extractor + `IMMUTABLE` and inlinable so the domain's `=` operator folds into the + query and the planner engages that index "structurally". The tracer + created **no** index on the encrypted column; its equality queries were + sequential scans. The ORE / OPE domains are the ones that need EQL's own + operator classes, because their terms do not sort as bytes. + +## Findings about Python and the toolchain + +- **pyo3 0.29, not 0.23**, for CPython 3.14. `abi3-py39` gives one wheel + per platform and builds against an interpreter newer than pyo3 knows. + Renames since the plan was written: `downcast` → `cast`, `allow_threads` + → `detach`, `PyObject` → `Py`. +- **`extension-module` must be maturin-only** (`[tool.maturin].features`), + not in `Cargo.toml`. On by default it stops `cargo test` and `cargo build` + linking (`Undefined symbols … _Py*`); maturin supplies + `-undefined dynamic_lookup`. `cargo check` is the compile check, + `maturin develop` the link check. +- **Anything that links pyo3 outside a wheel needs libpython on the loader + path** at run time (`DYLD_FALLBACK_LIBRARY_PATH` / `LD_LIBRARY_PATH` = + `sysconfig.get_config_var("LIBDIR")`), including `cargo test` and a + parity binary. CI must set it. +- **`Pending` is `IntoFuture`, not `Future`.** A sync shell is one Tokio + runtime, `into_future()` *before* `py.detach`, then `block_on`. There is + no second engine entry for sync, which keeps G6 reachable when async + comes. +- **Python's `int` has no width** and `34`/`34.0` derive different ORE + terms, so the annotation carries one (`Int64`, `Text`, …) and the shell + narrows exactly — stricter than `dynamic::read`, which converts lossless + widths and is right only for query literals. Check `bool` before `int`. + `None` encodes to `FfiValue::Null` and is refused by the declared kind. +- **The width markers should be a type, not a bare string**, or an unrelated + `Annotated[str, "a note"]` reads as a declaration. +- **psycopg plumbing** that touches no bytes: send a stored EQL value as + text with `::jsonb` in the SQL (`Jsonb(str)` wraps a JSON *string* and + fails the domain check); read it back as `email::text` and `.encode()` + (psycopg otherwise returns a `dict`). Postgres reorders keys; read by + name, never by position. +- **Environment quirks on this machine:** cargo and pip need the sandbox + off (shared `~/.cargo` writes; the proxy's TLS breaks pip and `gh`); + the repo's node scripts need node ≥ 20.11 (`import.meta.dirname`) — the + shell's is 20.3.1, mise has 24.11.1; `ghcr.io/cipherstash/postgres-eql` + needs a fresh `docker login ghcr.io`; the asdf `psql` shim has no + version, so install EQL through the container's own `psql`. + +## Design decisions made along the way (provisional — reviewers decide) + +- Targets resolve by **(kind, index)**, not by EQL name: Python writes + `{"kind": "string", "index": "eq"}` and the engine answers `TextEq` plus + both SQL domains. No EQL name is written in Python. An explicit + `EQL(name=...)` escape hatch exists. +- A **record is a dict of field name → column value** (EQL JSON as `bytes`, + passthrough as itself). It is what SQL wants, and no record ever needs a + serialised form. +- The **SQLAlchemy layer should be a separate PyPI distribution**, pure + Python, depending on the core wheel — the user's call on 2026-10-13; + not yet on the ticket. Backlog item 2 covers the call site, not the + shipping unit, so this is unowned. +- The live-test gate is "does the engine construct", not "are `CS_*` set". + +## Gaps against the discovery ticket's wording, found on re-reading + +Two places where what was built is narrower than what the ticket says, and +the findings above would otherwise imply the ticket was met. + +- **"The engine checks the declaration when Python defines the class" + (week-1 step 2) — not quite.** The check runs at `build()`, one call + later, wherever the caller puts that call. + `Plan.of(User).context(…).build()` at module scope runs at import time + **by convention**, because the call sits at module scope; nothing fires + when the `class` statement finishes. A class decorator — `@stash.model("users")` above `@dataclass` + — would run `build()` at class definition **by construction**, hold the + `PlanSpec` on the class, and let the call site be `Cipher(User)`. It is a + second way in (G6), so it has to earn its place: it carries the context, + which is per-model and which `Annotated` cannot express, and it reaches + the moment the ticket names. The stronger form makes it *the* way and + retires `Plan.of` from the public surface. Costs: decorator order + (`@dataclass(slots=True)` returns a new class, so a plan set below it is + lost; a transforming base such as Pydantic's or SQLAlchemy's declarative + may rewrite the class after it was read), and committing to "the model + class owns its plan". A plain `@dataclass` below the decorator is + **fine** — a class's annotations exist before `@dataclass` runs, so an + earlier claim that the wrong order "sees no fields" was mistaken. Backlog + item 1. +- **The model contract is not "a dataclass", and should not become one.** + psycopg imposes nothing — it sees tuples and parameters. The shell relies + on exactly three things: annotated fields readable through + `get_type_hints(cls, include_extras=True)`; attribute access (`getattr`) + on the way in; and a **keyword-argument constructor** matching the field + names on the way out (`model(**row)` in `decrypt`). Dataclasses meet that + with the least ceremony, which is the only reason the tracer used them. + So do `attrs`, Pydantic, `NamedTuple`, and SQLAlchemy declarative models + (default constructor takes kwargs) — the first user's actual classes. + Restricting to `@dataclass` would exclude them while enforcing more than + the code needs. State the contract explicitly (a `Protocol`), and let the + SQLAlchemy layer supply a base that meets it. `is_dataclass` is the cheap + guard for decorator order, not the right one. + +## Defects found in the plan (`2026-10-08-python-sdk-tracer.md`) + +All ruled on and ledgered; listed so the restart's plan does not repeat them: +missing `dynamic`/`http` features; pyo3 0.23; `cargo build` as a check; +`"Utf8Str"`-style kind names; `FfiValue::String(Vec)` (it is +`Utf8String`); an invented `Value::kind_of`; a committed workspace-keyed +fixture; "Python encrypted" wording (Python never encrypts — the check is +the engine *through the shell* against the engine bare); the record as +`list[bytes]`; `decrypt_with`'s fifth argument (`expected: Option