From e4e4823533c232e0120e325c15c7db453a1648b6 Mon Sep 17 00:00:00 2001 From: Yuji Yokoo Date: Mon, 2 Nov 2026 21:04:44 +0900 Subject: [PATCH 1/5] feat(python): decrypt rows back into models (#1390) Cipher.decrypt takes the rows psycopg read with dict_row and returns instances of the declared class, built with its constructor. A sealed column may be EQL JSON as a str, as UTF-8 bytes, or as a dict. Other columns are ignored. Every row is checked before any key request: its shape, its declared columns, its EQL JSON, and the table and column the JSON's "i" names. Each passthrough value is checked by exact type, as on encrypt. A refused row raises the new DecryptError or InvalidValueError, and the call returns no instance. The core crate decrypts with the engine's fake key source in its tests, so the round trip is tested without ZeroKMS. The engine does not say which field a kind mismatch was in. So decrypt_kind_mismatch names the field only when the plan has one sealed field. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- languages/python/README.md | 19 +- languages/python/crates/core/Cargo.toml | 10 +- languages/python/crates/core/src/client.rs | 5 + languages/python/crates/core/src/decrypt.rs | 699 ++++++++++++++++++ languages/python/crates/core/src/encrypt.rs | 47 +- languages/python/crates/core/src/lib.rs | 4 +- languages/python/crates/core/src/values.rs | 58 +- languages/python/crates/engine/src/decrypt.rs | 239 ++++++ languages/python/crates/engine/src/encrypt.rs | 4 +- languages/python/crates/engine/src/errors.rs | 7 +- languages/python/crates/engine/src/lib.rs | 2 + .../python/python/stackencrypt/__init__.py | 2 + .../python/python/stackencrypt/_cipher.py | 33 +- .../python/python/stackencrypt/_engine.pyi | 5 + .../python/python/stackencrypt/_errors.py | 13 + languages/python/tests/test_decrypt.py | 152 ++++ languages/python/tests/test_encrypt_live.py | 146 +++- 18 files changed, 1395 insertions(+), 52 deletions(-) create mode 100644 languages/python/crates/core/src/decrypt.rs create mode 100644 languages/python/crates/engine/src/decrypt.rs create mode 100644 languages/python/tests/test_decrypt.py diff --git a/AGENTS.md b/AGENTS.md index e4ef21c32..197730ef2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,7 +93,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates". - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do. - `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. The stack-encrypt guest has a second build with the EQL types (`mise run wasm:guest:build:eql`, cargo feature `eql`, linking `eql-bindings` by path; `wasm:guest:build:eql:deterministic` is its test build), embedded by `encrypt/eql` and registered on import, so a program whose generated code names an EQL type runs it (ADR-0007, amended). `encrypt/eql`'s types and `Types` table are generated from the EQL catalog by `eql-codegen` (`eql_gen.go`, written by `mise run types:generate` in `packages/eql` and drift-gated by `types:check` and `cargo test -p eql-codegen`). Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. Releases are cut by hand with `.github/workflows/release-golang.yml` (`gh workflow run release-golang.yml -f version=X.Y.Z[-pre] --ref `): because the guests are gitignored on main and Go fetches a version as the files committed at its tag, the workflow builds them, commits them on a release-only commit no branch points at, tags it `languages/golang/vX.Y.Z` and pushes the tag alone, then attaches the EQL v4 SQL bundle to the GitHub release. The ref must carry the EQL v4 bundle, or the build stops at "Build the EQL SQL bundles". `scripts/go-release-version.mjs` refuses, before anything is built, every version Go would refuse (non-canonical semver, `+build` metadata, a v2+ major on a module path with no `/vN` suffix, a version shaped like a pseudo-version), a final version dispatched from anywhere but `main`, and a final version while `languages/golang/go.mod` pins a `github.com/cipherstash/vitaminc/…` module at a pseudo-version (vitaminc's Go bindings have no tags, and a TEMPORARY pin on an unreleased commit must be gone before the module is tagged); a pre-release may come from a branch and may carry such a pin. The build job also refuses a version the Go proxy already lists in `@v/list`, which catches a tag deleted after the proxy fetched it; it never asks the proxy for the version itself before the tag exists, because the proxy would cache "not found" for up to 30 minutes and `verify` would fail after the push. The build checks the `Cargo.lock` of the EQL workspace and both guest workspaces with `--locked` first, since none of those builds passes it. It runs in a job with no write token and hands the guests to a `publish` job that holds `contents: write` and runs only git and gh. `publish` fetches the commit with plain unauthenticated git rather than `actions/checkout`, whose `token` input defaults to the job's writable token, and refuses a release commit whose tree is not the one the build tested. `publish` takes the paths it commits from the workflow's fixed `GUESTS` list, never from the artifact, and refuses a `guests.sha256` that is not exactly one SHA-256 line for each of those three guests: the build job runs third-party code after writing that manifest, so a path read from it could put any file on the tag (`scripts/__tests__/go-release-artifact.test.mjs`). A version is immutable once the Go proxy has fetched it, so a tag is never moved or deleted: if `publish` or `verify` fails after the tag is pushed, use "Re-run failed jobs" on the same run, which keeps a tag that run pushed and creates or completes the release; a new dispatch of the same version is refused. Never tag a main commit by hand: `go get` would fetch a module with no guests. -- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. `crates/core` compiles the Go guest's EQL resolver, `languages/golang/encrypt/guest/src/targets.rs`, in place through `#[path]`. So both SDKs run one resolver, and a change to that file is a change to both. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. A `live` job runs `mise run test:live`: `encrypt` against ZeroKMS with the repository's `CS_*` secrets, and the rows inserted into a `postgres:17` service with the EQL v4 bundle built from the tree. It skips on a fork's pull request, and the task fails on any skipped test. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, attrs, Pydantic, SQLAlchemy, and psycopg. The middle three are for the declaration tests, and psycopg is for the live tests. None has an upper bound, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: Python ADR-0001 under `languages/python/docs/adr/`, the spec in #1341. +- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. `crates/core` compiles the Go guest's EQL resolver, `languages/golang/encrypt/guest/src/targets.rs`, in place through `#[path]`. So both SDKs run one resolver, and a change to that file is a change to both. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. A `live` job runs `mise run test:live`: `encrypt` and `decrypt` against ZeroKMS with the repository's `CS_*` secrets, and the rows inserted into and read back from a `postgres:17` service with the EQL v4 bundle built from the tree. It skips on a fork's pull request, and the task fails on any skipped test. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, attrs, Pydantic, SQLAlchemy, and psycopg. The middle three are for the declaration tests, and psycopg is for the live tests. None has an upper bound, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: Python ADR-0001 under `languages/python/docs/adr/`, the spec in #1341. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). diff --git a/languages/python/README.md b/languages/python/README.md index 4edb2d0ba..161c30cf5 100644 --- a/languages/python/README.md +++ b/languages/python/README.md @@ -2,7 +2,7 @@ `stackencrypt` is the Python SDK for CipherStash Stack. It encrypts the fields of a Python model into values you store in PostgreSQL with EQL v4. EQL is CipherStash's set of PostgreSQL types for encrypted values. -This is a stub. The package is not released. Today it can encrypt a model. Decrypting and querying come in later releases. +This is a stub. The package is not released. Today it can encrypt a model and decrypt it again. Querying comes in a later release. ## Before you start @@ -65,6 +65,23 @@ with psycopg.connect(DATABASE_URL) as conn: `encrypt` checks every value before it sends anything. A value of the wrong type, such as `True` for an `int` field, raises `stackencrypt.InvalidValueError`. The error names the field. It never contains the value. +## Read the rows back + +Read the rows with psycopg's `dict_row`, then decrypt them with the same `Cipher`: + +```python +from psycopg.rows import dict_row + +with psycopg.connect(DATABASE_URL) as conn: + with conn.cursor(row_factory=dict_row) as cur: + cur.execute("SELECT email, age FROM users") + users = cipher.decrypt(cur.fetchall()) # a list of User +``` + +`decrypt` builds whole `User` instances, so the `SELECT` must name every declared column, passthrough columns included. A row without one raises `stackencrypt.DecryptError` with the code `decrypt_missing_column`. Other columns in the row are ignored. + +`decrypt` checks every row before it sends anything. A value stored under another table or column raises `DecryptError` with the code `decrypt_foreign_row`. A passthrough value of another type than the field declares raises `InvalidValueError`. When any row fails, `decrypt` returns no instances. + ## Run the tests From `languages/python`: diff --git a/languages/python/crates/core/Cargo.toml b/languages/python/crates/core/Cargo.toml index a4f475ac0..03d9c887b 100644 --- a/languages/python/crates/core/Cargo.toml +++ b/languages/python/crates/core/Cargo.toml @@ -25,8 +25,10 @@ eql = [] eql-bindings = { path = "../../../../packages/eql/crates/eql-bindings", features = ["targets"] } # For the engine error's diagnostic code. miette = "7.5.0" -# For the plan's frame: the fields in declaration order. -serde = "1" +# For the plan's frame: the fields in declaration order. With derive, for +# the `i` of a stored EQL value, read before any key request. +serde = { version = "1", features = ["derive"] } +serde_json = "1" stack-auth = { workspace = true } stack-encrypt = { workspace = true } stack-kms = { workspace = true } @@ -38,5 +40,5 @@ tokio = { version = "1", features = ["rt-multi-thread"] } zerokms-protocol = "0.12.31" [dev-dependencies] -# The shared resolver file's own tests use it. -serde_json = "1" +# The engine's in-memory key source, for the round trip with no ZeroKMS. +stack-kms = { workspace = true, features = ["test-support"] } diff --git a/languages/python/crates/core/src/client.rs b/languages/python/crates/core/src/client.rs index 6a774bfec..b3eedf07d 100644 --- a/languages/python/crates/core/src/client.rs +++ b/languages/python/crates/core/src/client.rs @@ -87,6 +87,11 @@ impl Client { self.cipher.default_keyset() } + /// The cipher itself. + pub fn cipher(&self) -> &StackCipher { + &self.cipher + } + /// Run `future` to completion on the shared runtime, blocking this /// thread. The caller releases the GIL around this. pub fn block_on(&self, future: F) -> F::Output { diff --git a/languages/python/crates/core/src/decrypt.rs b/languages/python/crates/core/src/decrypt.rs new file mode 100644 index 000000000..f75a21401 --- /dev/null +++ b/languages/python/crates/core/src/decrypt.rs @@ -0,0 +1,699 @@ +//! `decrypt`: stored rows into the engine's record tree, the engine's record +//! decryption under a model's plan, and the walk of the opened frame into +//! one row of values per stored row. +//! +//! The extension reads each row's columns with the GIL held and hands them +//! to a [`Row`]: an EQL column as its JSON text, a passthrough column as a +//! value already checked by exact type. Everything a row can answer is +//! checked there, before any key request (the spec in #1341, "Errors"): the +//! JSON parses, and its `i` names this table and column. Then the engine +//! opens the whole batch under one pending, and [`decrypt`] reads the opened +//! frame back as a [`Scalar`] per field. + +use std::future::Future; + +use stack_encrypt::dynamic::plaintext::{ + Error as SinkError, FixedType, Logical, MapSink, Physical, SeqSink, Sink, SourceError, +}; +use stack_encrypt::dynamic::record::{self, EQL_KEY}; +use stack_encrypt::dynamic::{Output, Scope, ValueFrame}; +use stack_encrypt::kms::{DataKeySource, IndexKeySource}; +use stack_encrypt::{CipherText, Encrypt, PassthroughFrame, StackCipher, StackCipherText}; + +use crate::client; +use crate::encrypt::{Field, ModelPlan}; +use crate::plan; +use crate::refusal::Refusal; +use crate::targets; +use crate::values::{self, Misfit, Scalar, Value}; + +/// The stored rows of one `decrypt` call, as the engine's record tree. +#[derive(Default)] +pub struct Rows { + rows: Vec, +} + +impl Rows { + /// No rows yet, with room for `capacity`. + pub fn with_capacity(capacity: usize) -> Self { + Self { + rows: Vec::with_capacity(capacity), + } + } + + /// How many rows were added. + pub fn len(&self) -> usize { + self.rows.len() + } + + /// Whether no row was added. + pub fn is_empty(&self) -> bool { + self.rows.is_empty() + } + + /// Add one row, its columns added in any order. + pub fn push(&mut self, row: Row) { + self.rows.push(CipherText::Map(row.fields)); + } +} + +/// One stored row, built a column at a time. +#[derive(Default)] +pub struct Row { + fields: Vec<(String, StackCipherText)>, +} + +impl Row { + /// An empty row. + pub fn new() -> Self { + Self::default() + } + + /// Add `field`'s EQL column, given as its JSON text. + /// + /// # Errors + /// + /// `decrypt_malformed` when the text is not a JSON object with an `i` + /// naming a table and a column; `decrypt_foreign_row` when that `i` + /// names another table or column than `field` of `plan`. Both name the + /// field. + pub fn eql(&mut self, plan: &ModelPlan, field: &Field, json: &[u8]) -> Result<(), Refusal> { + let (table, column) = identifier(json).ok_or_else(|| { + refusal( + "decrypt_malformed", + field, + "the column value is not an EQL value: JSON text of an object with an \ + \"i\" naming its table and column." + .to_owned(), + ) + })?; + if table != plan.table() || column != field.name { + return Err(refusal( + "decrypt_foreign_row", + field, + format!( + "the column value was stored under another table or column, not {:?}.{:?}. \ + Check the query selects this model's table and column.", + plan.table(), + field.name + ), + )); + } + let frame = ValueFrame::of(Carried(Value::Bytes(json))) + .map_err(|_| refusal("stack_encrypt_unknown", field, "bytes".to_owned()))?; + self.add(field, EQL_KEY, frame); + Ok(()) + } + + /// Add `field`'s passthrough column, a value the extension has already + /// checked by exact type against the field's annotation. + /// + /// # Errors + /// + /// The [`Misfit`] when the value is outside what the engine carries, + /// such as an integer wider than 128 bits. + pub fn passthrough(&mut self, field: &Field, value: Value<'_>) -> Result<(), Misfit> { + values::scalar(None, value)?; + // A value that fits always writes. + let frame = ValueFrame::of(Carried(value)).map_err(|_| Misfit::OutOfRange)?; + self.add(field, Output::Passthrough.key(), frame); + Ok(()) + } + + fn add(&mut self, field: &Field, key: &str, frame: ValueFrame) { + let node = CipherText::Passthrough(PassthroughFrame(frame.risky_ref().to_vec())); + self.fields.push(( + field.name.clone(), + CipherText::Map(vec![(key.to_owned(), node)]), + )); + } +} + +/// The `i.t` and `i.c` of an EQL value's JSON text, or `None` when it is not +/// JSON text of an object with both. +fn identifier(json: &[u8]) -> Option<(String, String)> { + #[derive(serde::Deserialize)] + struct Stored { + i: Identifier, + } + #[derive(serde::Deserialize)] + struct Identifier { + t: String, + c: String, + } + let stored: Stored = serde_json::from_slice(json).ok()?; + Some((stored.i.t, stored.i.c)) +} + +fn refusal(code: &str, field: &Field, message: String) -> Refusal { + Refusal { + code: code.to_owned(), + message: format!("field {:?}: {message}", field.name), + field: Some(field.name.clone()), + reason: None, + } +} + +/// Decrypt `rows` under `plan` on the shared client. Returns one row of +/// values per stored row, in [`ModelPlan::fields`] order. +/// +/// Builds the shared client on the first call in the process. The caller +/// has released the GIL. +/// +/// # Errors +/// +/// The client's refusal, or the engine's for the decryption: +/// `decrypt_kind_mismatch` for a value stored as another kind than the +/// field's. +pub fn decrypt(plan: &ModelPlan, rows: Rows) -> Result>, Refusal> { + let client = client::shared()?; + client.block_on(decrypt_with(client.cipher(), plan, rows)) +} + +/// [`decrypt`] on `cipher`, under its default keyset: a value sealed under +/// another keyset is refused before any key is requested. +/// +/// # Errors +/// +/// As [`decrypt`]. +pub fn decrypt_with<'a, K>( + cipher: &'a StackCipher, + plan: &'a ModelPlan, + rows: Rows, +) -> impl Future>, Refusal>> + 'a +where + K: DataKeySource + IndexKeySource + Sync + 'static, +{ + let count = rows.len(); + async move { + let scope = Scope::Keyset(cipher.default_keyset()); + let tree = CipherText::Sequence(rows.rows); + let pending = record::decrypt_with(scope, tree, plan.plan(), None, &targets::resolver()) + .map_err(|error| opening(plan, &plan::refusal(&error)))?; + let frame = pending + .await + .map_err(|error| opening(plan, &Refusal::of(&error)))?; + read_rows(plan, &frame, count) + } +} + +/// An engine refusal of the decryption, with the SDK's code where the spec +/// gives one. +/// +/// `stack_encrypt_target_stored` comes before any key request: the JSON has +/// an `i` but is not the field's EQL type. `stack_encrypt_opened_type` comes +/// after: the value opened, but was sealed as another type. The engine does +/// not say which field that was. So the field is named only when the plan +/// has one sealed field. +fn opening(plan: &ModelPlan, refused: &Refusal) -> Refusal { + let code = match refused.code.as_str() { + "stack_encrypt_target_stored" => "decrypt_malformed", + "stack_encrypt_opened_type" => "decrypt_kind_mismatch", + _ => return refused.clone(), + }; + let mut sealed = plan.fields().iter().filter(|field| field.eql.is_some()); + let field = match (refused.field.clone(), sealed.next(), sealed.next()) { + (Some(field), _, _) => Some(field), + (None, Some(only), None) => Some(only.name.clone()), + _ => None, + }; + Refusal { + code: code.to_owned(), + message: match &field { + Some(name) if refused.field.is_none() => format!("field {name:?}: {}", refused.message), + _ => refused.message.clone(), + }, + field, + reason: refused.reason.clone(), + } +} + +fn read_rows( + plan: &ModelPlan, + frame: &ValueFrame, + count: usize, +) -> Result>, Refusal> { + let mut rows = Vec::with_capacity(count); + frame + .visit(Opened { + fields: plan.fields(), + rows: &mut rows, + }) + .map_err(|ReadError(what)| engine_fault(what))?; + if rows.len() != count { + return Err(engine_fault( + "the output has another number of rows than the input", + )); + } + Ok(rows) +} + +fn engine_fault(what: &str) -> Refusal { + Refusal { + code: "stack_encrypt_unknown".to_owned(), + message: format!( + "the engine's opened record does not have the shape the SDK reads: {what}. \ + Report this message as a stackencrypt bug." + ), + field: None, + reason: None, + } +} + +/// A value the engine reads as it is: what [`Row`] stores. +#[derive(Clone, Copy)] +struct Carried<'a>(Value<'a>); + +impl Encrypt for Carried<'_> { + fn visit(self, sink: S) -> Result { + values::write(sink, None, self.0) + } +} + +#[derive(Debug)] +struct ReadError(&'static str); + +impl SinkError for ReadError { + fn source(_: SourceError) -> Self { + ReadError("the frame could not be walked") + } +} + +fn not_expected(what: &'static str) -> Result { + Err(ReadError(what)) +} + +/// The opened frame: a sequence of maps, one per stored row. +struct Opened<'a> { + fields: &'a [Field], + rows: &'a mut Vec>, +} + +impl<'a> Sink for Opened<'a> { + type Ok = (); + type Error = ReadError; + type Seq = Self; + type Map = Nothing; + + fn scalar(self, _: Option, _: Physical<'_>) -> Result<(), ReadError> { + not_expected("a scalar in place of the rows") + } + + fn typed_null(self, _: FixedType) -> Result<(), ReadError> { + not_expected("a null in place of the rows") + } + + fn seq(self, _: Option) -> Result { + Ok(self) + } + + fn map(self, _: Option) -> Result { + not_expected("one record in place of the rows") + } + + fn passthrough(self, _: P) -> Result<(), ReadError> { + not_expected("a passthrough in place of the rows") + } +} + +impl SeqSink for Opened<'_> { + type Ok = (); + type Error = ReadError; + type Element<'s> + = OpenedRow<'s> + where + Self: 's; + + fn element(&mut self) -> Result, ReadError> { + Ok(OpenedRow { + fields: self.fields, + rows: self.rows, + }) + } + + fn end(self) -> Result<(), ReadError> { + Ok(()) + } +} + +/// One opened row: a map of the plan's fields. +struct OpenedRow<'a> { + fields: &'a [Field], + rows: &'a mut Vec>, +} + +impl<'a> Sink for OpenedRow<'a> { + type Ok = (); + type Error = ReadError; + type Seq = Nothing; + type Map = OpenedFields<'a>; + + fn scalar(self, _: Option, _: Physical<'_>) -> Result<(), ReadError> { + not_expected("a scalar in place of a row") + } + + fn typed_null(self, _: FixedType) -> Result<(), ReadError> { + not_expected("a null in place of a row") + } + + fn seq(self, _: Option) -> Result { + not_expected("a sequence in place of a row") + } + + fn map(self, _: Option) -> Result, ReadError> { + Ok(OpenedFields { + values: vec![None; self.fields.len()], + fields: self.fields, + rows: self.rows, + }) + } + + fn passthrough(self, _: P) -> Result<(), ReadError> { + not_expected("a passthrough in place of a row") + } +} + +struct OpenedFields<'a> { + fields: &'a [Field], + values: Vec>, + rows: &'a mut Vec>, +} + +impl MapSink for OpenedFields<'_> { + type Ok = (); + type Error = ReadError; + type Entry<'s> + = OpenedValue<'s> + where + Self: 's; + + fn entry(&mut self, key: &str) -> Result, ReadError> { + let at = self + .fields + .iter() + .position(|field| field.name == key) + .ok_or(ReadError("a field the plan does not have"))?; + Ok(OpenedValue { + slot: &mut self.values[at], + }) + } + + fn end(self) -> Result<(), ReadError> { + let row = self + .values + .into_iter() + .collect::>>() + .ok_or(ReadError("a row without one of the plan's fields"))?; + self.rows.push(row); + Ok(()) + } +} + +/// One opened field: one scalar. +struct OpenedValue<'a> { + slot: &'a mut Option, +} + +impl Sink for OpenedValue<'_> { + type Ok = (); + type Error = ReadError; + type Seq = Nothing; + type Map = Nothing; + + fn scalar(self, ty: Option, value: Physical<'_>) -> Result<(), ReadError> { + let scalar = values::scalar_of(ty, value).map_err(ReadError)?; + if self.slot.replace(scalar).is_some() { + return not_expected("a field twice in one row"); + } + Ok(()) + } + + fn typed_null(self, _: FixedType) -> Result<(), ReadError> { + *self.slot = Some(Scalar::Null); + Ok(()) + } + + fn seq(self, _: Option) -> Result { + not_expected("a sequence in place of a value") + } + + fn map(self, _: Option) -> Result { + not_expected("a map in place of a value") + } + + fn passthrough(self, _: P) -> Result<(), ReadError> { + not_expected("a nested passthrough") + } +} + +/// A container sink that is never built. +enum Nothing {} + +impl SeqSink for Nothing { + type Ok = (); + type Error = ReadError; + type Element<'s> = OpenedValue<'s>; + + fn element(&mut self) -> Result, ReadError> { + match *self {} + } + + fn end(self) -> Result<(), ReadError> { + match self {} + } +} + +impl MapSink for Nothing { + type Ok = (); + type Error = ReadError; + type Entry<'s> = OpenedValue<'s>; + + fn entry(&mut self, _: &str) -> Result, ReadError> { + match *self {} + } + + fn end(self) -> Result<(), ReadError> { + match self {} + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::encrypt::{encrypt_with, Column}; + use crate::values::Kind; + use stack_kms::FakeDataKeySource; + + fn run(future: F) -> F::Output { + tokio::runtime::Builder::new_current_thread() + .build() + .unwrap() + .block_on(future) + } + + fn cipher() -> StackCipher { + run(StackCipher::builder().kms(FakeDataKeySource::new()).init()).unwrap() + } + + fn sealed(name: &str, eql: &str, kind: Kind) -> Field { + Field { + name: name.to_owned(), + eql: Some((eql.to_owned(), kind)), + } + } + + fn passthrough(name: &str) -> Field { + Field { + name: name.to_owned(), + eql: None, + } + } + + fn users() -> ModelPlan { + ModelPlan::new( + "users", + vec![sealed("email", "TextEq", Kind::String), passthrough("age")], + ) + .unwrap() + } + + /// Rows of `(field, kind, value)` as the input frame `encrypt` takes. + struct Source<'a>(Vec, Value<'a>)>>); + + impl Encrypt for Source<'_> { + fn visit(self, sink: S) -> Result { + let mut rows = sink.seq(Some(self.0.len()))?; + for row in self.0 { + let mut map = rows.element()?.map(Some(row.len()))?; + for (name, kind, value) in row { + let entry = map.entry(name)?; + values::write(entry, kind, value)?; + } + map.end()?; + } + rows.end() + } + } + + /// Encrypt one `users` row per email, with `age` 30 + its index. + fn stored( + cipher: &StackCipher, + plan: &ModelPlan, + emails: &[&str], + ) -> Vec> { + let rows = emails + .iter() + .enumerate() + .map(|(at, email)| { + vec![ + ("email", Some(Kind::String), Value::Text(email)), + ("age", None, Value::Int(30 + at as i128)), + ] + }) + .collect(); + let source = ValueFrame::of(Source(rows)).unwrap(); + run(encrypt_with( + &cipher.default_keyset(), + plan, + source, + emails.len(), + )) + .unwrap() + } + + /// The stored columns back as one [`Row`] each, as the extension adds them. + fn rows_of(plan: &ModelPlan, stored: &[Vec]) -> Rows { + let mut rows = Rows::with_capacity(stored.len()); + for columns in stored { + let mut row = Row::new(); + for (field, column) in plan.fields().iter().zip(columns) { + match column { + Column::Eql(json) => row.eql(plan, field, json.as_bytes()).unwrap(), + Column::Passthrough(Scalar::Int(i)) => { + row.passthrough(field, Value::Int(*i)).unwrap() + } + Column::Passthrough(other) => panic!("not an age: {other:?}"), + } + } + rows.push(row); + } + rows + } + + fn eql(columns: &[Column]) -> &str { + match &columns[0] { + Column::Eql(json) => json, + other => panic!("not EQL: {other:?}"), + } + } + + #[test] + fn a_batch_round_trips_through_encrypt_and_decrypt() { + let cipher = cipher(); + let plan = users(); + let stored = stored(&cipher, &plan, &["a@example.com", "b@example.com"]); + + let opened = run(decrypt_with(&cipher, &plan, rows_of(&plan, &stored))).unwrap(); + + assert_eq!( + opened, + vec![ + vec![Scalar::Text("a@example.com".to_owned()), Scalar::Int(30)], + vec![Scalar::Text("b@example.com".to_owned()), Scalar::Int(31)], + ] + ); + } + + #[test] + fn a_value_from_another_table_is_a_foreign_row() { + let cipher = cipher(); + let stored = stored(&cipher, &users(), &["a@example.com"]); + let accounts = ModelPlan::new( + "accounts", + vec![sealed("email", "TextEq", Kind::String), passthrough("age")], + ) + .unwrap(); + + let refused = Row::new() + .eql(&accounts, &accounts.fields()[0], eql(&stored[0]).as_bytes()) + .unwrap_err(); + + assert_eq!(refused.code, "decrypt_foreign_row"); + assert_eq!(refused.field.as_deref(), Some("email")); + } + + #[test] + fn a_value_from_another_column_is_a_foreign_row() { + let cipher = cipher(); + let stored = stored(&cipher, &users(), &["a@example.com"]); + let plan = ModelPlan::new("users", vec![sealed("login", "TextEq", Kind::String)]).unwrap(); + + let refused = Row::new() + .eql(&plan, &plan.fields()[0], eql(&stored[0]).as_bytes()) + .unwrap_err(); + + assert_eq!(refused.code, "decrypt_foreign_row"); + assert_eq!(refused.field.as_deref(), Some("login")); + } + + #[test] + fn text_that_is_not_an_eql_value_is_malformed() { + let cipher = cipher(); + let plan = users(); + let stored = stored(&cipher, &plan, &["a@example.com"]); + let json = eql(&stored[0]); + let truncated = &json.as_bytes()[..json.len() / 2]; + + for bytes in [truncated, b"[]", b"{}", b"{\"i\": 1}", b"\xff"] { + let refused = Row::new().eql(&plan, &plan.fields()[0], bytes).unwrap_err(); + assert_eq!(refused.code, "decrypt_malformed", "{bytes:?}"); + assert_eq!(refused.field.as_deref(), Some("email")); + } + } + + #[test] + fn an_eql_value_with_only_an_identifier_is_refused_before_any_key_request() { + let cipher = cipher(); + let plan = users(); + let mut row = Row::new(); + row.eql( + &plan, + &plan.fields()[0], + br#"{"i": {"t": "users", "c": "email"}}"#, + ) + .unwrap(); + row.passthrough(&plan.fields()[1], Value::Int(1)).unwrap(); + let mut rows = Rows::default(); + rows.push(row); + + let refused = run(decrypt_with(&cipher, &plan, rows)).unwrap_err(); + + assert_eq!(refused.code, "decrypt_malformed", "{refused:?}"); + assert_eq!(refused.field.as_deref(), Some("email")); + } + + #[test] + fn a_value_stored_as_another_kind_is_a_kind_mismatch() { + let cipher = cipher(); + let numbers = + ModelPlan::new("users", vec![sealed("email", "IntegerEq", Kind::Int32)]).unwrap(); + let source = ValueFrame::of(Source(vec![vec![( + "email", + Some(Kind::Int32), + Value::Int(7), + )]])) + .unwrap(); + let stored = run(encrypt_with(&cipher.default_keyset(), &numbers, source, 1)).unwrap(); + let text = ModelPlan::new("users", vec![sealed("email", "TextEq", Kind::String)]).unwrap(); + let mut row = Row::new(); + row.eql(&text, &text.fields()[0], eql(&stored[0]).as_bytes()) + .unwrap(); + let mut rows = Rows::default(); + rows.push(row); + + let refused = run(decrypt_with(&cipher, &text, rows)).unwrap_err(); + + assert_eq!(refused.code, "decrypt_kind_mismatch", "{refused:?}"); + assert_eq!(refused.field.as_deref(), Some("email")); + } +} diff --git a/languages/python/crates/core/src/encrypt.rs b/languages/python/crates/core/src/encrypt.rs index ac40cab34..06edac6d0 100644 --- a/languages/python/crates/core/src/encrypt.rs +++ b/languages/python/crates/core/src/encrypt.rs @@ -12,7 +12,8 @@ use stack_encrypt::dynamic::record::{self, Plan, EQL_KEY}; use stack_encrypt::dynamic::{self, Output, ValueFrame}; -use stack_encrypt::{CipherText, Encrypt, StackCipherText}; +use stack_encrypt::kms::{DataKeySource, IndexKeySource}; +use stack_encrypt::{CipherText, Encrypt, KeysetCipher, StackCipherText}; use crate::client; use crate::plan; @@ -31,6 +32,7 @@ pub struct Field { /// A model's plan, built once per `Cipher` and read on every call. pub struct ModelPlan { + table: String, plan: Plan, fields: Vec, } @@ -55,7 +57,11 @@ impl ModelPlan { }) .collect(); let plan = plan::build(table, &columns)?; - Ok(Self { plan, fields }) + Ok(Self { + table: table.to_owned(), + plan, + fields, + }) } /// The fields, in plan order. The extension writes each model's values @@ -63,6 +69,16 @@ impl ModelPlan { pub fn fields(&self) -> &[Field] { &self.fields } + + /// The table the model is stored in. + pub fn table(&self) -> &str { + &self.table + } + + /// The engine's plan. + pub(crate) fn plan(&self) -> &Plan { + &self.plan + } } /// One column of an encrypted row. @@ -89,12 +105,27 @@ pub fn encrypt( rows: usize, ) -> Result>, Refusal> { let client = client::shared()?; - let tree = client.block_on(async { - let keyset = client.keyset(); - let pending = record::encrypt_with(&keyset, source, &plan.plan, &targets::resolver()) - .map_err(|error| plan::refusal(&error))?; - pending.await.map_err(|error| Refusal::of(&error)) - })?; + client.block_on(encrypt_with(&client.keyset(), plan, source, rows)) +} + +/// [`encrypt`] under `keyset`. +/// +/// # Errors +/// +/// The engine's refusal for the encryption. +pub async fn encrypt_with( + keyset: &KeysetCipher<'_, K>, + plan: &ModelPlan, + source: ValueFrame, + rows: usize, +) -> Result>, Refusal> +where + K: DataKeySource + IndexKeySource + Sync + 'static, +{ + let tree = record::encrypt_with(keyset, source, &plan.plan, &targets::resolver()) + .map_err(|error| plan::refusal(&error))? + .await + .map_err(|error| Refusal::of(&error))?; read_rows(plan, tree, rows) } diff --git a/languages/python/crates/core/src/lib.rs b/languages/python/crates/core/src/lib.rs index 31a17d9de..8be71d7ce 100644 --- a/languages/python/crates/core/src/lib.rs +++ b/languages/python/crates/core/src/lib.rs @@ -4,12 +4,14 @@ //! It holds the engine commit the build recorded (`build.rs`), the engine's //! check of a model's plan ([`plan`]), the credential lookup //! ([`credentials`]), the one engine client a process shares ([`client`]), -//! the value rules ([`values`]) and `encrypt` itself ([`encrypt`]). Each +//! the value rules ([`values`]), `encrypt` itself ([`encrypt`]) and +//! `decrypt` ([`decrypt`]). Each //! reports an engine error as a [`refusal::Refusal`]. The Python SDK spec //! (#1341, "Layout") says what else belongs here. pub mod client; pub mod credentials; +pub mod decrypt; pub mod encrypt; pub mod plan; pub mod refusal; diff --git a/languages/python/crates/core/src/values.rs b/languages/python/crates/core/src/values.rs index b4727d483..0841a9c2c 100644 --- a/languages/python/crates/core/src/values.rs +++ b/languages/python/crates/core/src/values.rs @@ -245,6 +245,38 @@ pub fn read(bytes: &[u8]) -> Result { .map_err(|ReadError(what)| malformed(what)) } +/// An opened scalar as a host value. +/// +/// # Errors +/// +/// What the value is, when the SDK has no host value for it: an integer +/// wider than 128 bits, a date or timestamp outside the range, or a logical +/// type the SDK does not read. +pub fn scalar_of(ty: Option, value: Physical<'_>) -> Result { + Ok(match (ty, value) { + (None, Physical::Null) => Scalar::Null, + (None, Physical::Bool(b)) => Scalar::Bool(b), + (None, Physical::Int(i)) => Scalar::Int(i.to_i128().ok_or("an integer too wide")?), + (None, Physical::Float(Float::F32(f))) => Scalar::Float(f64::from(f)), + (None, Physical::Float(Float::F64(f))) => Scalar::Float(f), + (None, Physical::Text(s)) => Scalar::Text(s.to_owned()), + (None, Physical::Bytes(b)) => Scalar::Bytes(b.to_vec()), + (Some(Logical::Date), Physical::Int(days)) => Scalar::Date( + days.to_i128() + .and_then(|d| i32::try_from(d).ok()) + .ok_or("a date outside the range")?, + ), + (Some(Logical::Timestamp(unit)), Physical::Int(count)) => Scalar::Timestamp( + kinds::timestamp_nanos(unit, count).ok_or("a timestamp outside the range")?, + ), + (Some(Logical::Decimal { scale }), Physical::Int(mantissa)) => Scalar::Decimal { + mantissa: mantissa.to_i128().ok_or("a decimal too wide")?, + scale, + }, + _ => return Err("a logical type the SDK does not read"), + }) +} + fn malformed(what: &str) -> Refusal { Refusal { code: "stack_encrypt_unknown".to_owned(), @@ -272,31 +304,7 @@ impl Sink for Reader { type Map = Never; fn scalar(self, ty: Option, value: Physical<'_>) -> Result { - Ok(match (ty, value) { - (None, Physical::Null) => Scalar::Null, - (None, Physical::Bool(b)) => Scalar::Bool(b), - (None, Physical::Int(i)) => { - Scalar::Int(i.to_i128().ok_or(ReadError("an integer too wide"))?) - } - (None, Physical::Float(Float::F32(f))) => Scalar::Float(f64::from(f)), - (None, Physical::Float(Float::F64(f))) => Scalar::Float(f), - (None, Physical::Text(s)) => Scalar::Text(s.to_owned()), - (None, Physical::Bytes(b)) => Scalar::Bytes(b.to_vec()), - (Some(Logical::Date), Physical::Int(days)) => Scalar::Date( - days.to_i128() - .and_then(|d| i32::try_from(d).ok()) - .ok_or(ReadError("a date outside the range"))?, - ), - (Some(Logical::Timestamp(unit)), Physical::Int(count)) => Scalar::Timestamp( - kinds::timestamp_nanos(unit, count) - .ok_or(ReadError("a timestamp outside the range"))?, - ), - (Some(Logical::Decimal { scale }), Physical::Int(mantissa)) => Scalar::Decimal { - mantissa: mantissa.to_i128().ok_or(ReadError("a decimal too wide"))?, - scale, - }, - _ => return Err(ReadError("a logical type the SDK does not read")), - }) + scalar_of(ty, value).map_err(ReadError) } fn typed_null(self, _: FixedType) -> Result { diff --git a/languages/python/crates/engine/src/decrypt.rs b/languages/python/crates/engine/src/decrypt.rs new file mode 100644 index 000000000..863ee7d5c --- /dev/null +++ b/languages/python/crates/engine/src/decrypt.rs @@ -0,0 +1,239 @@ +//! `decrypt` at the boundary: the rows psycopg returned into the engine's +//! stored record, and the opened values into instances of the model class. +//! +//! Every row is read and checked with the GIL held, before any key request +//! (the spec in #1341, "Errors"): it is a mapping, it has every plan field's +//! column, each EQL column is EQL JSON stored under this table and column, +//! and each passthrough value has the field's exact type. Then the GIL is +//! released and the engine opens the whole list. The instances are built +//! only after every value has been turned into a Python object, so a call +//! that fails returns no instance. + +use pyo3::prelude::*; +use pyo3::sync::PyOnceLock; +use pyo3::types::{PyBytes, PyDict, PyList, PyString}; +use stackencrypt_core::decrypt::{self as core, Row, Rows}; +use stackencrypt_core::encrypt::Field; +use stackencrypt_core::refusal::Refusal; +use stackencrypt_core::values::Scalar; + +use crate::encrypt::Plan; +use crate::errors::raise; +use crate::values::{read, to_python, PyKind}; + +/// Decrypt `rows` under `plan` into instances of `cls`, one per row, built +/// with the plan's fields as keyword arguments. +/// +/// An empty list returns an empty list and touches nothing. +#[pyfunction] +pub fn decrypt<'py>( + py: Python<'py>, + plan: &Plan, + cls: &Bound<'py, PyAny>, + rows: Vec>, +) -> PyResult> { + if rows.is_empty() { + return Ok(PyList::empty(py)); + } + let mut stored = Rows::with_capacity(rows.len()); + for (at, row) in rows.iter().enumerate() { + stored.push(stored_row(plan, row, at)?); + } + let opened = py + .detach(|| core::decrypt(&plan.plan, stored)) + .map_err(|refused| raise(py, refused))?; + // Every value first, then every instance: a value Python cannot hold + // fails the call before any constructor runs. + let mut arguments = Vec::with_capacity(opened.len()); + for values in opened { + let kwargs = PyDict::new(py); + for (field, value) in plan.plan.fields().iter().zip(values) { + kwargs.set_item(&field.name, python_value(py, field, value)?)?; + } + arguments.push(kwargs); + } + let out = PyList::empty(py); + for kwargs in arguments { + // The class's own constructor: an error it raises propagates as it is. + out.append(cls.call((), Some(&kwargs))?)?; + } + Ok(out) +} + +/// One row as the engine's stored row, checked. +fn stored_row(plan: &Plan, row: &Bound<'_, PyAny>, at: usize) -> PyResult { + let py = row.py(); + if !row.is_instance(mapping(py)?)? { + return Err(raise( + py, + Refusal { + code: "decrypt_row_shape".to_owned(), + message: format!( + "row {at} is a {}, not a mapping of column names to values. Read the \ + rows with psycopg's dict_row: cursor(row_factory=dict_row).", + type_name(row) + ), + field: None, + reason: None, + }, + )); + } + let mut stored = Row::new(); + for (field, kind) in plan.plan.fields().iter().zip(&plan.kinds) { + if !row.contains(&field.name)? { + return Err(raise( + py, + Refusal { + code: "decrypt_missing_column".to_owned(), + message: format!( + "row {at} has no column {:?}. Select every declared column of the \ + model, passthrough columns included.", + field.name + ), + field: Some(field.name.clone()), + reason: None, + }, + )); + } + let value = row.get_item(&field.name)?; + let in_row = |refused: Refusal| { + raise( + py, + Refusal { + message: format!("row {at}, {}", refused.message), + ..refused + }, + ) + }; + match &field.eql { + Some(_) => stored + .eql(&plan.plan, field, &eql_text(field, &value)?) + .map_err(in_row)?, + None => passthrough(&mut stored, field, *kind, &value).map_err(in_row)?, + } + } + Ok(stored) +} + +/// An EQL column's JSON text, as UTF-8 bytes: from a `str`, from `bytes` +/// that are UTF-8, or from a `dict` the driver already parsed. +fn eql_text(field: &Field, value: &Bound<'_, PyAny>) -> PyResult> { + let py = value.py(); + let malformed = |what: String| { + raise( + py, + Refusal { + code: "decrypt_malformed".to_owned(), + message: format!("field {:?}: the column value {what}.", field.name), + field: Some(field.name.clone()), + reason: None, + }, + ) + }; + if let Ok(text) = value.cast::() { + return Ok(text.to_cow()?.as_bytes().to_vec()); + } + if let Ok(bytes) = value.cast::() { + let bytes = bytes.as_bytes(); + return match std::str::from_utf8(bytes) { + Ok(_) => Ok(bytes.to_vec()), + Err(_) => Err(malformed("is bytes that are not UTF-8".to_owned())), + }; + } + if value.cast::().is_ok() { + return match dumps(py)?.call1((value,)) { + Ok(text) => Ok(text.cast::()?.to_cow()?.as_bytes().to_vec()), + Err(_) => Err(malformed("is a dict that is not JSON".to_owned())), + }; + } + if value.is_none() { + return Err(malformed( + "is None, a SQL NULL, not an EQL value".to_owned(), + )); + } + Err(malformed(format!( + "is a {}, not the str, bytes or dict of an EQL value", + type_name(value) + ))) +} + +/// A passthrough column's value, checked by exact type as on `encrypt`. +fn passthrough( + stored: &mut Row, + field: &Field, + kind: PyKind, + value: &Bound<'_, PyAny>, +) -> Result<(), Refusal> { + let named = |refused: Refusal| Refusal { + message: format!("field {:?}: {}", field.name, refused.message), + field: Some(field.name.clone()), + reason: Some("passthrough".to_owned()), + ..refused + }; + let read = match read(value, kind, "passthrough") { + Ok(Ok(read)) => read, + Ok(Err(refused)) => return Err(named(refused)), + Err(error) => { + return Err(named(Refusal { + code: "decrypt_malformed".to_owned(), + message: format!("the value could not be read: {error}"), + field: None, + reason: None, + })) + } + }; + stored.passthrough(field, read.value()).map_err(|misfit| { + named(Refusal { + code: misfit.code().to_owned(), + message: "the value is outside what the engine stores.".to_owned(), + field: None, + reason: None, + }) + }) +} + +/// An opened value as the Python object of its field. +fn python_value(py: Python<'_>, field: &Field, value: Scalar) -> PyResult> { + let ranged = matches!(value, Scalar::Date(_) | Scalar::Timestamp(_)); + to_python(py, value).map_err(|error| { + if !ranged { + return error; + } + raise( + py, + Refusal { + code: "decrypt_not_representable".to_owned(), + message: format!( + "field {:?}: the opened value is outside the range Python's datetime \ + holds.", + field.name + ), + field: Some(field.name.clone()), + reason: None, + }, + ) + }) +} + +fn mapping(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>> { + static MAPPING: PyOnceLock> = PyOnceLock::new(); + let mapping = MAPPING.get_or_try_init(py, || -> PyResult> { + Ok(py.import("collections.abc")?.getattr("Mapping")?.unbind()) + })?; + Ok(mapping.bind(py)) +} + +fn dumps(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>> { + static DUMPS: PyOnceLock> = PyOnceLock::new(); + let dumps = DUMPS.get_or_try_init(py, || -> PyResult> { + Ok(py.import("json")?.getattr("dumps")?.unbind()) + })?; + Ok(dumps.bind(py)) +} + +fn type_name(value: &Bound<'_, PyAny>) -> String { + value.get_type().name().map_or_else( + |_| "value of an unknown type".to_owned(), + |name| name.to_string(), + ) +} diff --git a/languages/python/crates/engine/src/encrypt.rs b/languages/python/crates/engine/src/encrypt.rs index fc30750c5..8084b431b 100644 --- a/languages/python/crates/engine/src/encrypt.rs +++ b/languages/python/crates/engine/src/encrypt.rs @@ -27,9 +27,9 @@ type FieldSpec<'py> = (String, Option, Option, Bound<'py, PyType /// A model's plan, built once by `Cipher` and passed to every call. #[pyclass(frozen, module = "stackencrypt._engine")] pub struct Plan { - plan: ModelPlan, + pub(crate) plan: ModelPlan, /// The Python type each field's value must have, in plan order. - kinds: Vec, + pub(crate) kinds: Vec, } /// Build the plan for a model stored in `table`. diff --git a/languages/python/crates/engine/src/errors.rs b/languages/python/crates/engine/src/errors.rs index 58f120786..7c5ecdd6d 100644 --- a/languages/python/crates/engine/src/errors.rs +++ b/languages/python/crates/engine/src/errors.rs @@ -3,8 +3,9 @@ //! The exception classes live in `stackencrypt._errors`, in Python, so that //! the pure-Python checks and the extension raise the same ones. The class //! is chosen by the code's prefix, as the spec's "Errors" section assigns -//! them: `value_*` is `InvalidValueError`, `stack_auth_*` is -//! `AuthenticationError`, and anything else is the base `StashError`. +//! them: `value_*` is `InvalidValueError`, `decrypt_*` is `DecryptError`, +//! `stack_auth_*` is `AuthenticationError`, and anything else is the base +//! `StashError`. use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; @@ -26,6 +27,8 @@ pub fn raise(py: Python<'_>, refused: Refusal) -> PyErr { fn exception(py: Python<'_>, refused: &Refusal) -> PyResult { let class = if refused.code.starts_with("value_") { "InvalidValueError" + } else if refused.code.starts_with("decrypt_") { + "DecryptError" } else if refused.code.starts_with("stack_auth_") { "AuthenticationError" } else { diff --git a/languages/python/crates/engine/src/lib.rs b/languages/python/crates/engine/src/lib.rs index e9d2d008f..64c525190 100644 --- a/languages/python/crates/engine/src/lib.rs +++ b/languages/python/crates/engine/src/lib.rs @@ -3,6 +3,7 @@ //! Deliberately thin: this crate only converts between Python and Rust at //! the boundary. Logic belongs in `stackencrypt-core` and `stack-encrypt`. +mod decrypt; mod encrypt; mod errors; mod values; @@ -94,6 +95,7 @@ fn _engine(module: &Bound<'_, PyModule>) -> PyResult<()> { module.add_class::()?; module.add_function(wrap_pyfunction!(encrypt::build_plan, module)?)?; module.add_function(wrap_pyfunction!(encrypt::encrypt, module)?)?; + module.add_function(wrap_pyfunction!(decrypt::decrypt, module)?)?; module.add_function(wrap_pyfunction!(encrypt::client_builds, module)?)?; module.add_function(wrap_pyfunction!(encrypt::request_count, module)?)?; #[cfg(feature = "gate-probe")] diff --git a/languages/python/python/stackencrypt/__init__.py b/languages/python/python/stackencrypt/__init__.py index b306957ed..c85c6c767 100644 --- a/languages/python/python/stackencrypt/__init__.py +++ b/languages/python/python/stackencrypt/__init__.py @@ -15,6 +15,7 @@ from stackencrypt._errors import ( AuthenticationError, DeclarationError, + DecryptError, InvalidValueError, StashError, ) @@ -29,6 +30,7 @@ "AuthenticationError", "Cipher", "DeclarationError", + "DecryptError", "Ignore", "InvalidValueError", "Passthrough", diff --git a/languages/python/python/stackencrypt/_cipher.py b/languages/python/python/stackencrypt/_cipher.py index 76f2b1b69..e9bd4f5de 100644 --- a/languages/python/python/stackencrypt/_cipher.py +++ b/languages/python/python/stackencrypt/_cipher.py @@ -1,10 +1,10 @@ """``Cipher``: encrypts and decrypts the instances of one declared class. It checks the class and the credentials, and builds the model's plan once. -``encrypt`` is here. ``decrypt`` and ``query`` come in later tickets. +``encrypt`` and ``decrypt`` are here. ``query`` comes in a later ticket. """ -from typing import Dict, List, Sequence +from typing import Dict, Generic, List, Mapping, Sequence, Type, TypeVar from stackencrypt import _engine from stackencrypt._markers import Passthrough @@ -12,7 +12,10 @@ from stackencrypt._errors import AuthenticationError, DeclarationError -class Cipher: +T = TypeVar("T") + + +class Cipher(Generic[T]): """The cipher for one class declared with ``stackencrypt.model``:: users = stackencrypt.Cipher(User) @@ -23,10 +26,11 @@ class Cipher: that ``stash auth login`` writes. Building a cipher makes no request. Every cipher in the process shares one engine client. It is built on the - first ``encrypt`` of any cipher, which loads the default keyset once. + first ``encrypt`` or ``decrypt`` of any cipher, which loads the default + keyset once. """ - def __init__(self, cls: type) -> None: + def __init__(self, cls: Type[T]) -> None: # The type check comes first: declaration_of reads the class's own # __dict__, and a string or an instance with __slots__ has none. declaration = declaration_of(cls) if isinstance(cls, type) else None @@ -66,6 +70,25 @@ def encrypt(self, models: Sequence[object]) -> List[Dict[str, object]]: """ return _engine.encrypt(self._plan, list(models)) + def decrypt(self, rows: Sequence[Mapping[str, object]]) -> List[T]: + """Decrypt ``rows`` into instances of the declared class. + + Each row is a mapping from column name to value, such as a row that + psycopg's ``dict_row`` returns. A sealed column's value is the EQL + JSON as a ``str``, as UTF-8 ``bytes``, or as a parsed ``dict``. + Columns the class does not declare are ignored. Every declared + field's column must be in every row, passthrough columns included. + An ``Ignore`` field is not passed to the constructor. + + Every row is checked first. A row that is not a mapping, a missing + column, a value that is not EQL JSON, or a value stored under + another table or column raises ``DecryptError`` naming the field, + and no request is made. A passthrough value of another type than the + field's raises ``InvalidValueError``; it is never converted. The + call is all or nothing: no instance is built when any row fails. + """ + return _engine.decrypt(self._plan, self._cls, list(rows)) + @property def declaration(self) -> Declaration: """What the decorator read from the class.""" diff --git a/languages/python/python/stackencrypt/_engine.pyi b/languages/python/python/stackencrypt/_engine.pyi index eeff8de00..ec521e9e8 100644 --- a/languages/python/python/stackencrypt/_engine.pyi +++ b/languages/python/python/stackencrypt/_engine.pyi @@ -30,6 +30,11 @@ def encrypt(plan: Plan, models: Sequence[Any]) -> List[Dict[str, object]]: """One dict per model: EQL JSON as ``str`` for a sealed field, the value for a passthrough. Raises ``InvalidValueError`` before any request.""" +def decrypt(plan: Plan, cls: type, rows: Sequence[Any]) -> List[Any]: + """One instance of ``cls`` per row, built with the plan's fields as + keyword arguments. Raises ``DecryptError`` or ``InvalidValueError`` + before any request for what the rows themselves can answer.""" + def client_builds() -> int: """How many times the shared engine client has been built. For tests.""" diff --git a/languages/python/python/stackencrypt/_errors.py b/languages/python/python/stackencrypt/_errors.py index 1b1e33022..82af23bbf 100644 --- a/languages/python/python/stackencrypt/_errors.py +++ b/languages/python/python/stackencrypt/_errors.py @@ -54,3 +54,16 @@ class InvalidValueError(StashError, ValueError): field's kind, such as ``int32``, or ``passthrough``. The message never contains the value. """ + + +class DecryptError(StashError): + """A stored row that ``decrypt`` cannot turn into a model. + + Before any key request: ``decrypt_row_shape`` (a row that is not a + mapping), ``decrypt_missing_column`` (a declared field's column is not in + the row), ``decrypt_malformed`` (the column value is not an EQL value), + and ``decrypt_foreign_row`` (the value was stored under another table or + column). After the key requests: ``decrypt_kind_mismatch`` (the value was + stored as another type) and ``decrypt_not_representable`` (an opened value + Python cannot hold). ``field`` names the column where it is known. + """ diff --git a/languages/python/tests/test_decrypt.py b/languages/python/tests/test_decrypt.py new file mode 100644 index 000000000..e8e1819e3 --- /dev/null +++ b/languages/python/tests/test_decrypt.py @@ -0,0 +1,152 @@ +"""``Cipher.decrypt`` before any request: the row checks and the empty list. + +Every row is checked before the shared client is built: its shape, its +columns, each EQL value's JSON and the table and column it names, and each +passthrough value's type. So these tests need fake credentials only, and +make no request. The live round trip is in ``test_encrypt_live.py``. +""" + +import dataclasses +import json +from typing import Annotated + +import pytest + +import stackencrypt +from stackencrypt import ( + Cipher, + DecryptError, + InvalidValueError, + Passthrough, + StashError, + TextEq, +) +from stackencrypt import _engine + +# The `i` of a value stored under users.email, and the rest of an EQL value +# left out: the checks before any request read only `i`. +STORED = json.dumps({"i": {"t": "users", "c": "email"}, "v": 2}) + + +@pytest.fixture(autouse=True) +def credentials(monkeypatch, tmp_path): + """Credentials that parse, so ``Cipher`` builds. Nothing here sends them.""" + for name in ("CS_CLIENT_ID", "CS_CLIENT_KEY", "CS_CTS_HOST", "CS_ZEROKMS_HOST", "CS_VITUR_HOST"): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", "CSAKtestKeyId.testKeySecret") + monkeypatch.setenv("CS_WORKSPACE_CRN", "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY") + monkeypatch.setenv("CS_CONFIG_PATH", str(tmp_path / "no-profile")) + + +@pytest.fixture +def untouched(): + """Fails the test if the shared client was built or a request was sent.""" + builds, requests = _engine.client_builds(), _engine.request_count() + yield + assert _engine.client_builds() == builds + assert _engine.request_count() == requests + + +@stackencrypt.model("users") +@dataclasses.dataclass +class User: + email: Annotated[str, TextEq] + age: Annotated[int, Passthrough] + + +def refused(rows, error=DecryptError): + with pytest.raises(error) as raised: + Cipher(User).decrypt(rows) + return raised.value + + +def test_an_empty_list_is_an_empty_list_with_no_request(untouched): + assert Cipher(User).decrypt([]) == [] + + +def test_a_decrypt_error_is_a_stash_error(): + assert issubclass(DecryptError, StashError) + + +def test_a_tuple_row_is_the_wrong_shape_and_names_dict_row(untouched): + error = refused([(STORED, 3)]) + assert error.code == "decrypt_row_shape" + assert "dict_row" in error.message + + +def test_a_row_without_a_declared_column_names_it(untouched): + error = refused([{"email": STORED}]) + assert error.code == "decrypt_missing_column" + assert error.field == "age" + + +@pytest.mark.parametrize( + "stored", + [ + json.dumps({"i": {"t": "users", "c": "login"}}), + json.dumps({"i": {"t": "accounts", "c": "email"}}), + ], + ids=["another-column", "another-table"], +) +def test_a_value_stored_under_another_column_is_a_foreign_row(untouched, stored): + error = refused([{"email": stored, "age": 3}]) + assert error.code == "decrypt_foreign_row" + assert error.field == "email" + + +@pytest.mark.parametrize( + "value", + [STORED.encode(), json.loads(STORED)], + ids=["bytes", "dict"], +) +def test_bytes_and_dict_values_are_read_as_eql_json(untouched, value): + # Each form reaches the same checks as the str: the row is refused for + # its passthrough, after the EQL value was read and accepted. + error = refused([{"email": value, "age": "3"}], InvalidValueError) + assert error.field == "age" + + +@pytest.mark.parametrize( + "value", + [STORED[: len(STORED) // 2], b"\xff\xfe", 42, None, "[]"], + ids=["truncated", "not-utf-8", "int", "null", "not-an-object"], +) +def test_a_value_that_is_not_eql_json_is_malformed(untouched, value): + error = refused([{"email": value, "age": 3}]) + assert error.code == "decrypt_malformed" + assert error.field == "email" + + +def test_a_passthrough_int_that_arrives_as_str_is_the_wrong_type(untouched): + error = refused([{"email": STORED, "age": "3"}], InvalidValueError) + assert error.code == "value_wrong_type" + assert error.field == "age" + assert error.reason == "passthrough" + + +def test_a_null_passthrough_is_refused(untouched): + error = refused([{"email": STORED, "age": None}], InvalidValueError) + assert error.code == "value_none" + assert error.field == "age" + + +def test_one_bad_row_in_ten_fails_the_call_before_any_request(untouched): + rows = [{"email": STORED, "age": i} for i in range(9)] + [{"email": STORED}] + error = refused(rows) + assert error.code == "decrypt_missing_column" + assert "row 9" in error.message + + +def test_extra_columns_are_not_read(untouched): + # The extra column holds what every check would refuse. The row is + # refused for its passthrough instead, so the extra column was not read. + error = refused([{"email": STORED, "age": "3", "id": object()}], InvalidValueError) + assert error.field == "age" + + +def test_a_row_that_passes_every_check_reaches_the_engine(): + # Refused later by the engine for want of a client key, not by the row + # checks: they passed. + with pytest.raises(StashError) as raised: + Cipher(User).decrypt([{"email": STORED, "age": 3}]) + assert not isinstance(raised.value, (DecryptError, InvalidValueError)) diff --git a/languages/python/tests/test_encrypt_live.py b/languages/python/tests/test_encrypt_live.py index b6a54e547..e0c3d4f7a 100644 --- a/languages/python/tests/test_encrypt_live.py +++ b/languages/python/tests/test_encrypt_live.py @@ -1,4 +1,5 @@ -"""``Cipher.encrypt`` against ZeroKMS and a PostgreSQL with the EQL v4 bundle. +"""``Cipher.encrypt`` and ``Cipher.decrypt`` against ZeroKMS and a PostgreSQL +with the EQL v4 bundle. These tests need CipherStash credentials (``CS_CLIENT_ACCESS_KEY``, ``CS_WORKSPACE_CRN``, ``CS_CLIENT_ID``, ``CS_CLIENT_KEY``) and a database @@ -9,13 +10,18 @@ import dataclasses import decimal +import json import os -from typing import Annotated +from typing import Annotated, List, NamedTuple +import attrs +import pydantic import pytest +from sqlalchemy import ForeignKey +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship import stackencrypt -from stackencrypt import Cipher, Passthrough, TextEq +from stackencrypt import Cipher, DecryptError, Ignore, Passthrough, TextEq from stackencrypt import _engine CREDENTIALS = ("CS_CLIENT_ACCESS_KEY", "CS_WORKSPACE_CRN", "CS_CLIENT_ID", "CS_CLIENT_KEY") @@ -104,3 +110,137 @@ def test_a_decimal_passthrough_comes_back_as_the_same_decimal(): assert type(amount) is decimal.Decimal # The scale is kept: 19.50, not 19.5. assert str(amount) == "19.50" + + +# The five class kinds, each with one TextEq, one Passthrough and one Ignore. + + +@stackencrypt.model("users") +@dataclasses.dataclass +class DataclassUser: + email: Annotated[str, TextEq] + age: Annotated[int, Passthrough] + note: Annotated[str, Ignore] = "" + + +@stackencrypt.model("users") +@attrs.define +class AttrsUser: + email: Annotated[str, TextEq] + age: Annotated[int, Passthrough] + note: Annotated[str, Ignore] = "" + + +@stackencrypt.model("users") +class PydanticUser(pydantic.BaseModel): + email: Annotated[str, TextEq] + age: Annotated[int, Passthrough] + note: Annotated[str, Ignore] = "" + + +class TupleUser(NamedTuple): + email: Annotated[str, TextEq] + age: Annotated[int, Passthrough] + note: Annotated[str, Ignore] = "" + + +stackencrypt.model("users")(TupleUser) + + +class Base(DeclarativeBase): + pass + + +class Post(Base): + __tablename__ = "posts" + id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) + user_id: Mapped[Annotated[int, Passthrough]] = mapped_column(ForeignKey("users.id")) + + +@stackencrypt.model("users") +class SqlAlchemyUser(Base): + __tablename__ = "users" + id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) + email: Mapped[Annotated[str, TextEq]] + age: Mapped[Annotated[int, Passthrough]] + posts: Mapped[Annotated[List["Post"], Ignore]] = relationship() + + +def build(cls, at): + """The model for row ``at``: an id only where the class declares one.""" + fields = {"email": f"user{at}@example.com", "age": 30 + at} + if cls is SqlAlchemyUser: + fields["id"] = at + 1 + return cls(**fields) + + +def fields(model): + return {name: getattr(model, name) for name in ("email", "age")} + + +CLASS_KINDS = [DataclassUser, AttrsUser, PydanticUser, TupleUser, SqlAlchemyUser] + + +@needs_database +@pytest.mark.parametrize("cls", CLASS_KINDS, ids=lambda cls: cls.__name__) +def test_rows_read_back_with_dict_row_decrypt_into_the_models(cls): + psycopg = pytest.importorskip("psycopg") + from psycopg.rows import dict_row + + cipher = Cipher(cls) + models = [build(cls, at) for at in range(3)] + rows = cipher.encrypt(models) + with psycopg.connect(DATABASE_URL) as conn, conn.cursor(row_factory=dict_row) as cur: + cur.execute( + "CREATE TEMPORARY TABLE users " + "(id serial PRIMARY KEY, email public.eql_v4_text_eq, age int)" + ) + for row in rows: + cur.execute("INSERT INTO users (email, age) VALUES (%s, %s)", (row["email"], row["age"])) + # SELECT *: the id column is extra for every class but SQLAlchemy's, + # and is ignored. + cur.execute("SELECT * FROM users ORDER BY id") + decrypted = cipher.decrypt(cur.fetchall()) + + assert [type(model) for model in decrypted] == [cls] * 3 + assert [fields(model) for model in decrypted] == [fields(model) for model in models] + + +def test_str_bytes_and_dict_forms_decrypt_to_the_same_value(): + cipher = Cipher(User) + [row] = cipher.encrypt([User(email="forms@example.com", age=3)]) + forms = [row["email"], row["email"].encode(), json.loads(row["email"])] + + decrypted = cipher.decrypt([{"email": form, "age": 3} for form in forms]) + + assert [user.email for user in decrypted] == ["forms@example.com"] * 3 + + +def test_a_value_encrypted_for_another_column_is_a_foreign_row(): + [row] = Cipher(Contact).encrypt([Contact(email="a@example.com")]) + with pytest.raises(DecryptError) as raised: + Cipher(User).decrypt([{"email": row["email"], "age": 3}]) + assert raised.value.code == "decrypt_foreign_row" + assert raised.value.field == "email" + + +def test_a_sqlalchemy_class_is_built_without_its_ignored_relationship(): + cipher = Cipher(SqlAlchemyUser) + rows = cipher.encrypt([build(SqlAlchemyUser, 0)]) + + [user] = cipher.decrypt(rows) + + assert (user.id, user.email, user.age) == (1, "user0@example.com", 30) + assert "posts" not in user.__dict__ + + +def test_1001_rows_open_with_three_key_requests(): + cipher = Cipher(Contact) + rows = cipher.encrypt([Contact(email=f"user{i}@example.com") for i in range(1001)]) + before = _engine.request_count() + + decrypted = cipher.decrypt(rows) + + # ceil(1001 / 500): the keys for the whole list, 500 per request. + assert _engine.request_count() - before == 3 + assert decrypted[1000].email == "user1000@example.com" From b989f6b95f9a093a767b29755db82269267fb299 Mon Sep 17 00:00:00 2001 From: Yuji Yokoo Date: Mon, 2 Nov 2026 23:17:35 +0900 Subject: [PATCH 2/5] fix(python): address the review of decrypt (#1390) Open rows under the client scope, as Go's Client.Open does. A row sealed under another keyset of the same client now decrypts. Map stack_encrypt_foreign_keyset and stack_encrypt_context_mismatch to AuthenticationError. Name the row in a malformed-value error, and check for a duplicate typed null as the scalar path does. Borrow str and bytes column values, so the EQL text is copied once, into the engine's frame. Share the frame helpers between encrypt and decrypt, and move the credential fixtures into conftest.py. A kind mismatch names its field only when the model has one sealed field. The engine does not name the field yet. #1508 tracks that. The README says that a passthrough datetime needs a timestamptz column, because psycopg reads timestamp as a naive datetime. Co-Authored-By: Claude Opus 5.5 --- languages/python/README.md | 2 + languages/python/crates/core/src/decrypt.rs | 160 +++++++++--------- languages/python/crates/core/src/encrypt.rs | 6 +- languages/python/crates/core/src/values.rs | 64 ++++--- languages/python/crates/engine/src/decrypt.rs | 59 ++++--- languages/python/crates/engine/src/errors.rs | 70 +++++++- languages/python/tests/conftest.py | 39 +++++ languages/python/tests/test_cipher.py | 17 +- languages/python/tests/test_decrypt.py | 17 +- languages/python/tests/test_encrypt.py | 9 +- 10 files changed, 280 insertions(+), 163 deletions(-) create mode 100644 languages/python/tests/conftest.py diff --git a/languages/python/README.md b/languages/python/README.md index 161c30cf5..5b6a3dc24 100644 --- a/languages/python/README.md +++ b/languages/python/README.md @@ -80,6 +80,8 @@ with psycopg.connect(DATABASE_URL) as conn: `decrypt` builds whole `User` instances, so the `SELECT` must name every declared column, passthrough columns included. A row without one raises `stackencrypt.DecryptError` with the code `decrypt_missing_column`. Other columns in the row are ignored. +A passthrough `datetime` field needs a `timestamptz` column. psycopg reads a `timestamp` column as a naive `datetime`, with no `tzinfo`. `decrypt` refuses a naive `datetime` with `value_naive_datetime`, as `encrypt` does. + `decrypt` checks every row before it sends anything. A value stored under another table or column raises `DecryptError` with the code `decrypt_foreign_row`. A passthrough value of another type than the field declares raises `InvalidValueError`. When any row fails, `decrypt` returns no instances. ## Run the tests diff --git a/languages/python/crates/core/src/decrypt.rs b/languages/python/crates/core/src/decrypt.rs index f75a21401..25b832983 100644 --- a/languages/python/crates/core/src/decrypt.rs +++ b/languages/python/crates/core/src/decrypt.rs @@ -12,20 +12,19 @@ use std::future::Future; -use stack_encrypt::dynamic::plaintext::{ - Error as SinkError, FixedType, Logical, MapSink, Physical, SeqSink, Sink, SourceError, -}; +use stack_encrypt::dynamic::frame; +use stack_encrypt::dynamic::plaintext::{FixedType, Logical, MapSink, Physical, SeqSink, Sink}; use stack_encrypt::dynamic::record::{self, EQL_KEY}; use stack_encrypt::dynamic::{Output, Scope, ValueFrame}; use stack_encrypt::kms::{DataKeySource, IndexKeySource}; use stack_encrypt::{CipherText, Encrypt, PassthroughFrame, StackCipher, StackCipherText}; use crate::client; -use crate::encrypt::{Field, ModelPlan}; +use crate::encrypt::{engine_fault, Field, ModelPlan}; use crate::plan; use crate::refusal::Refusal; use crate::targets; -use crate::values::{self, Misfit, Scalar, Value}; +use crate::values::{self, Misfit, Never, ReadError, Scalar, Value}; /// The stored rows of one `decrypt` call, as the engine's record tree. #[derive(Default)] @@ -99,9 +98,7 @@ impl Row { ), )); } - let frame = ValueFrame::of(Carried(Value::Bytes(json))) - .map_err(|_| refusal("stack_encrypt_unknown", field, "bytes".to_owned()))?; - self.add(field, EQL_KEY, frame); + self.add(field, EQL_KEY, eql_frame(json)); Ok(()) } @@ -116,12 +113,12 @@ impl Row { values::scalar(None, value)?; // A value that fits always writes. let frame = ValueFrame::of(Carried(value)).map_err(|_| Misfit::OutOfRange)?; - self.add(field, Output::Passthrough.key(), frame); + self.add(field, Output::Passthrough.key(), frame.risky_ref().to_vec()); Ok(()) } - fn add(&mut self, field: &Field, key: &str, frame: ValueFrame) { - let node = CipherText::Passthrough(PassthroughFrame(frame.risky_ref().to_vec())); + fn add(&mut self, field: &Field, key: &str, frame: Vec) { + let node = CipherText::Passthrough(PassthroughFrame(frame)); self.fields.push(( field.name.clone(), CipherText::Map(vec![(key.to_owned(), node)]), @@ -129,6 +126,20 @@ impl Row { } } +/// An EQL value's JSON text as the frame of one bytes scalar, the way the +/// engine stores it: written straight into the frame's buffer, so the text +/// is copied once. Bytes are ciphertext here, not plaintext. +fn eql_frame(json: &[u8]) -> Vec { + struct Bytes<'a>(&'a [u8]); + impl serde::Serialize for Bytes<'_> { + fn serialize(&self, serializer: S) -> Result { + serializer.serialize_bytes(self.0) + } + } + // One bytes scalar always encodes. + frame::serde::to_vec(&Bytes(json)).expect("a bytes scalar encodes as a frame") +} + /// The `i.t` and `i.c` of an EQL value's JSON text, or `None` when it is not /// JSON text of an object with both. fn identifier(json: &[u8]) -> Option<(String, String)> { @@ -170,8 +181,10 @@ pub fn decrypt(plan: &ModelPlan, rows: Rows) -> Result>, Refusal client.block_on(decrypt_with(client.cipher(), plan, rows)) } -/// [`decrypt`] on `cipher`, under its default keyset: a value sealed under -/// another keyset is refused before any key is requested. +/// [`decrypt`] on `cipher`. Each value opens under the keyset that sealed +/// it, with the keys requested per keyset, as the Go SDK's `Client.Open` +/// does: rows another SDK wrote under another keyset of the same client +/// decrypt here too. /// /// # Errors /// @@ -186,7 +199,7 @@ where { let count = rows.len(); async move { - let scope = Scope::Keyset(cipher.default_keyset()); + let scope = Scope::Client(cipher); let tree = CipherText::Sequence(rows.rows); let pending = record::decrypt_with(scope, tree, plan.plan(), None, &targets::resolver()) .map_err(|error| opening(plan, &plan::refusal(&error)))?; @@ -204,7 +217,7 @@ where /// an `i` but is not the field's EQL type. `stack_encrypt_opened_type` comes /// after: the value opened, but was sealed as another type. The engine does /// not say which field that was. So the field is named only when the plan -/// has one sealed field. +/// has one sealed field. #1508 tracks the engine change that names it. fn opening(plan: &ModelPlan, refused: &Refusal) -> Refusal { let code = match refused.code.as_str() { "stack_encrypt_target_stored" => "decrypt_malformed", @@ -248,18 +261,6 @@ fn read_rows( Ok(rows) } -fn engine_fault(what: &str) -> Refusal { - Refusal { - code: "stack_encrypt_unknown".to_owned(), - message: format!( - "the engine's opened record does not have the shape the SDK reads: {what}. \ - Report this message as a stackencrypt bug." - ), - field: None, - reason: None, - } -} - /// A value the engine reads as it is: what [`Row`] stores. #[derive(Clone, Copy)] struct Carried<'a>(Value<'a>); @@ -270,15 +271,6 @@ impl Encrypt for Carried<'_> { } } -#[derive(Debug)] -struct ReadError(&'static str); - -impl SinkError for ReadError { - fn source(_: SourceError) -> Self { - ReadError("the frame could not be walked") - } -} - fn not_expected(what: &'static str) -> Result { Err(ReadError(what)) } @@ -293,7 +285,7 @@ impl<'a> Sink for Opened<'a> { type Ok = (); type Error = ReadError; type Seq = Self; - type Map = Nothing; + type Map = Never<(), OpenedValue<'static>>; fn scalar(self, _: Option, _: Physical<'_>) -> Result<(), ReadError> { not_expected("a scalar in place of the rows") @@ -307,7 +299,7 @@ impl<'a> Sink for Opened<'a> { Ok(self) } - fn map(self, _: Option) -> Result { + fn map(self, _: Option) -> Result>, ReadError> { not_expected("one record in place of the rows") } @@ -345,7 +337,7 @@ struct OpenedRow<'a> { impl<'a> Sink for OpenedRow<'a> { type Ok = (); type Error = ReadError; - type Seq = Nothing; + type Seq = Never<(), OpenedValue<'static>>; type Map = OpenedFields<'a>; fn scalar(self, _: Option, _: Physical<'_>) -> Result<(), ReadError> { @@ -356,7 +348,7 @@ impl<'a> Sink for OpenedRow<'a> { not_expected("a null in place of a row") } - fn seq(self, _: Option) -> Result { + fn seq(self, _: Option) -> Result>, ReadError> { not_expected("a sequence in place of a row") } @@ -414,30 +406,34 @@ struct OpenedValue<'a> { slot: &'a mut Option, } +impl OpenedValue<'_> { + fn put(self, scalar: Scalar) -> Result<(), ReadError> { + if self.slot.replace(scalar).is_some() { + return not_expected("a field twice in one row"); + } + Ok(()) + } +} + impl Sink for OpenedValue<'_> { type Ok = (); type Error = ReadError; - type Seq = Nothing; - type Map = Nothing; + type Seq = Never<(), OpenedValue<'static>>; + type Map = Never<(), OpenedValue<'static>>; fn scalar(self, ty: Option, value: Physical<'_>) -> Result<(), ReadError> { - let scalar = values::scalar_of(ty, value).map_err(ReadError)?; - if self.slot.replace(scalar).is_some() { - return not_expected("a field twice in one row"); - } - Ok(()) + self.put(values::scalar_of(ty, value).map_err(ReadError)?) } fn typed_null(self, _: FixedType) -> Result<(), ReadError> { - *self.slot = Some(Scalar::Null); - Ok(()) + self.put(Scalar::Null) } - fn seq(self, _: Option) -> Result { + fn seq(self, _: Option) -> Result>, ReadError> { not_expected("a sequence in place of a value") } - fn map(self, _: Option) -> Result { + fn map(self, _: Option) -> Result>, ReadError> { not_expected("a map in place of a value") } @@ -446,37 +442,6 @@ impl Sink for OpenedValue<'_> { } } -/// A container sink that is never built. -enum Nothing {} - -impl SeqSink for Nothing { - type Ok = (); - type Error = ReadError; - type Element<'s> = OpenedValue<'s>; - - fn element(&mut self) -> Result, ReadError> { - match *self {} - } - - fn end(self) -> Result<(), ReadError> { - match self {} - } -} - -impl MapSink for Nothing { - type Ok = (); - type Error = ReadError; - type Entry<'s> = OpenedValue<'s>; - - fn entry(&mut self, _: &str) -> Result, ReadError> { - match *self {} - } - - fn end(self) -> Result<(), ReadError> { - match self {} - } -} - #[cfg(test)] mod tests { use super::*; @@ -587,6 +552,13 @@ mod tests { } } + #[test] + fn an_eql_frame_is_the_frame_of_one_bytes_scalar() { + let json = br#"{"i":{"t":"users","c":"email"}}"#; + let scalar = ValueFrame::of(Carried(Value::Bytes(json))).unwrap(); + assert_eq!(eql_frame(json), scalar.risky_ref()); + } + #[test] fn a_batch_round_trips_through_encrypt_and_decrypt() { let cipher = cipher(); @@ -604,6 +576,30 @@ mod tests { ); } + #[test] + fn a_row_sealed_under_another_keyset_of_the_client_decrypts() { + let cipher = cipher(); + let plan = users(); + let named = stack_kms::IdentifiedBy::Name("acme".to_owned().into()); + let acme = run(cipher.keyset(named)).unwrap(); + let source = ValueFrame::of(Source(vec![vec![ + ("email", Some(Kind::String), Value::Text("a@example.com")), + ("age", None, Value::Int(3)), + ]])) + .unwrap(); + let stored = run(encrypt_with(&acme, &plan, source, 1)).unwrap(); + + let opened = run(decrypt_with(&cipher, &plan, rows_of(&plan, &stored))).unwrap(); + + assert_eq!( + opened, + vec![vec![ + Scalar::Text("a@example.com".to_owned()), + Scalar::Int(3) + ]] + ); + } + #[test] fn a_value_from_another_table_is_a_foreign_row() { let cipher = cipher(); diff --git a/languages/python/crates/core/src/encrypt.rs b/languages/python/crates/core/src/encrypt.rs index 06edac6d0..d8bbae87b 100644 --- a/languages/python/crates/core/src/encrypt.rs +++ b/languages/python/crates/core/src/encrypt.rs @@ -185,11 +185,13 @@ fn read_row(plan: &ModelPlan, row: StackCipherText) -> Result, Refus Ok(columns) } -fn engine_fault(what: &str) -> Refusal { +/// A refusal for an engine result of another shape than the SDK reads: +/// a bug, never the caller's input. +pub(crate) fn engine_fault(what: &str) -> Refusal { Refusal { code: "stack_encrypt_unknown".to_owned(), message: format!( - "the engine's output does not have the shape the SDK reads: {what}. \ + "the engine's result does not have the shape the SDK reads: {what}. \ Report this message as a stackencrypt bug." ), field: None, diff --git a/languages/python/crates/core/src/values.rs b/languages/python/crates/core/src/values.rs index 0841a9c2c..d19edecd7 100644 --- a/languages/python/crates/core/src/values.rs +++ b/languages/python/crates/core/src/values.rs @@ -18,6 +18,9 @@ use stack_encrypt::dynamic::plaintext::{ SourceError, TimeUnit, }; +use std::convert::Infallible; +use std::marker::PhantomData; + use stack_encrypt::Encrypt; use crate::refusal::Refusal; @@ -286,8 +289,9 @@ fn malformed(what: &str) -> Refusal { } } +/// Why a frame the engine returned could not be read: what was found. #[derive(Debug)] -struct ReadError(&'static str); +pub(crate) struct ReadError(pub(crate) &'static str); impl SinkError for ReadError { fn source(_: SourceError) -> Self { @@ -300,8 +304,8 @@ struct Reader; impl Sink for Reader { type Ok = Scalar; type Error = ReadError; - type Seq = Never; - type Map = Never; + type Seq = Never; + type Map = Never; fn scalar(self, ty: Option, value: Physical<'_>) -> Result { scalar_of(ty, value).map_err(ReadError) @@ -311,11 +315,11 @@ impl Sink for Reader { Ok(Scalar::Null) } - fn seq(self, _: Option) -> Result { + fn seq(self, _: Option) -> Result, ReadError> { Err(ReadError("a sequence")) } - fn map(self, _: Option) -> Result { + fn map(self, _: Option) -> Result, ReadError> { Err(ReadError("a map")) } @@ -324,34 +328,52 @@ impl Sink for Reader { } } -/// A container sink that is never built: a passthrough is one scalar. -enum Never {} +/// A container sink that is never built, for a sink that refuses +/// containers: `O` is its `Ok`, `E` the sink of its elements and entries. +pub(crate) enum Never { + #[allow(dead_code)] + Void(Infallible, PhantomData (O, E)>), +} -impl SeqSink for Never { - type Ok = Scalar; +impl> SeqSink for Never { + type Ok = O; type Error = ReadError; - type Element<'s> = Reader; + type Element<'s> + = E + where + Self: 's; - fn element(&mut self) -> Result { - match *self {} + fn element(&mut self) -> Result { + match self { + Self::Void(never, _) => match *never {}, + } } - fn end(self) -> Result { - match self {} + fn end(self) -> Result { + match self { + Self::Void(never, _) => match never {}, + } } } -impl MapSink for Never { - type Ok = Scalar; +impl> MapSink for Never { + type Ok = O; type Error = ReadError; - type Entry<'s> = Reader; + type Entry<'s> + = E + where + Self: 's; - fn entry(&mut self, _: &str) -> Result { - match *self {} + fn entry(&mut self, _: &str) -> Result { + match self { + Self::Void(never, _) => match *never {}, + } } - fn end(self) -> Result { - match self {} + fn end(self) -> Result { + match self { + Self::Void(never, _) => match never {}, + } } } diff --git a/languages/python/crates/engine/src/decrypt.rs b/languages/python/crates/engine/src/decrypt.rs index 863ee7d5c..778fc65c1 100644 --- a/languages/python/crates/engine/src/decrypt.rs +++ b/languages/python/crates/engine/src/decrypt.rs @@ -9,6 +9,8 @@ //! only after every value has been turned into a Python object, so a call //! that fails returns no instance. +use std::borrow::Cow; + use pyo3::prelude::*; use pyo3::sync::PyOnceLock; use pyo3::types::{PyBytes, PyDict, PyList, PyString}; @@ -106,9 +108,10 @@ fn stored_row(plan: &Plan, row: &Bound<'_, PyAny>, at: usize) -> PyResult { ) }; match &field.eql { - Some(_) => stored - .eql(&plan.plan, field, &eql_text(field, &value)?) - .map_err(in_row)?, + Some(_) => { + let text = eql_text(field, &value)?.map_err(in_row)?; + stored.eql(&plan.plan, field, &text).map_err(in_row)?; + } None => passthrough(&mut stored, field, *kind, &value).map_err(in_row)?, } } @@ -116,45 +119,53 @@ fn stored_row(plan: &Plan, row: &Bound<'_, PyAny>, at: usize) -> PyResult { } /// An EQL column's JSON text, as UTF-8 bytes: from a `str`, from `bytes` -/// that are UTF-8, or from a `dict` the driver already parsed. -fn eql_text(field: &Field, value: &Bound<'_, PyAny>) -> PyResult> { +/// that are UTF-8, or from a `dict` the driver already parsed. A `str` or +/// `bytes` value is borrowed, not copied: the text is copied once, into the +/// engine's frame. +/// +/// The outer error is Python's own; the inner one is the refusal, for the +/// caller to name the row in. +fn eql_text<'a>( + field: &Field, + value: &'a Bound<'_, PyAny>, +) -> PyResult, Refusal>> { let py = value.py(); let malformed = |what: String| { - raise( - py, - Refusal { - code: "decrypt_malformed".to_owned(), - message: format!("field {:?}: the column value {what}.", field.name), - field: Some(field.name.clone()), - reason: None, - }, - ) + Ok(Err(Refusal { + code: "decrypt_malformed".to_owned(), + message: format!("field {:?}: the column value {what}.", field.name), + field: Some(field.name.clone()), + reason: None, + })) }; if let Ok(text) = value.cast::() { - return Ok(text.to_cow()?.as_bytes().to_vec()); + return Ok(Ok(match text.to_cow()? { + Cow::Borrowed(text) => Cow::Borrowed(text.as_bytes()), + Cow::Owned(text) => Cow::Owned(text.into_bytes()), + })); } if let Ok(bytes) = value.cast::() { let bytes = bytes.as_bytes(); return match std::str::from_utf8(bytes) { - Ok(_) => Ok(bytes.to_vec()), - Err(_) => Err(malformed("is bytes that are not UTF-8".to_owned())), + Ok(_) => Ok(Ok(Cow::Borrowed(bytes))), + Err(_) => malformed("is bytes that are not UTF-8".to_owned()), }; } if value.cast::().is_ok() { return match dumps(py)?.call1((value,)) { - Ok(text) => Ok(text.cast::()?.to_cow()?.as_bytes().to_vec()), - Err(_) => Err(malformed("is a dict that is not JSON".to_owned())), + Ok(text) => Ok(Ok(Cow::Owned( + text.cast::()?.to_cow()?.into_owned().into_bytes(), + ))), + Err(_) => malformed("is a dict that is not JSON".to_owned()), }; } if value.is_none() { - return Err(malformed( - "is None, a SQL NULL, not an EQL value".to_owned(), - )); + return malformed("is None, a SQL NULL, not an EQL value".to_owned()); } - Err(malformed(format!( + malformed(format!( "is a {}, not the str, bytes or dict of an EQL value", type_name(value) - ))) + )) } /// A passthrough column's value, checked by exact type as on `encrypt`. diff --git a/languages/python/crates/engine/src/errors.rs b/languages/python/crates/engine/src/errors.rs index 7c5ecdd6d..63bb42dca 100644 --- a/languages/python/crates/engine/src/errors.rs +++ b/languages/python/crates/engine/src/errors.rs @@ -4,13 +4,21 @@ //! the pure-Python checks and the extension raise the same ones. The class //! is chosen by the code's prefix, as the spec's "Errors" section assigns //! them: `value_*` is `InvalidValueError`, `decrypt_*` is `DecryptError`, -//! `stack_auth_*` is `AuthenticationError`, and anything else is the base -//! `StashError`. +//! `stack_auth_*` and the engine's context and keyset refusals are +//! `AuthenticationError`, and anything else is the base `StashError`. use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; use stackencrypt_core::refusal::Refusal; +/// The engine codes the spec's "Errors" puts under `AuthenticationError` +/// besides `stack_auth_*`: a context the value was not sealed under, and a +/// value from a keyset this call does not open. +const AUTHENTICATION: [&str; 2] = [ + "stack_encrypt_context_mismatch", + "stack_encrypt_foreign_keyset", +]; + /// The exception for `refused`, ready to raise. pub fn raise(py: Python<'_>, refused: Refusal) -> PyErr { match exception(py, &refused) { @@ -29,7 +37,9 @@ fn exception(py: Python<'_>, refused: &Refusal) -> PyResult { "InvalidValueError" } else if refused.code.starts_with("decrypt_") { "DecryptError" - } else if refused.code.starts_with("stack_auth_") { + } else if refused.code.starts_with("stack_auth_") + || AUTHENTICATION.contains(&refused.code.as_str()) + { "AuthenticationError" } else { "StashError" @@ -43,3 +53,57 @@ fn exception(py: Python<'_>, refused: &Refusal) -> PyResult { let instance = class.call(args, Some(&kwargs))?; Ok(PyErr::from_value(instance)) } + +#[cfg(test)] +mod tests { + use super::*; + + /// The class `raise` picks for `code`, by name. + fn class_of(code: &str) -> String { + Python::initialize(); + Python::attach(|py| { + // `stackencrypt._errors` from the source tree. The package is + // registered empty, so its `__init__`, which imports this + // extension, does not run. + let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../python/stackencrypt"); + let package = pyo3::types::PyModule::new(py, "stackencrypt").unwrap(); + package.setattr("__path__", vec![dir]).unwrap(); + py.import("sys") + .unwrap() + .getattr("modules") + .unwrap() + .set_item("stackencrypt", package) + .unwrap(); + let refused = Refusal { + code: code.to_owned(), + message: "a message".to_owned(), + field: None, + reason: None, + }; + raise(py, refused).get_type(py).name().unwrap().to_string() + }) + } + + #[test] + fn a_context_or_keyset_refusal_is_an_authentication_error() { + assert_eq!( + class_of("stack_encrypt_foreign_keyset"), + "AuthenticationError" + ); + assert_eq!( + class_of("stack_encrypt_context_mismatch"), + "AuthenticationError" + ); + assert_eq!( + class_of("stack_auth_not_authenticated"), + "AuthenticationError" + ); + } + + #[test] + fn each_prefix_has_its_class() { + assert_eq!(class_of("value_none"), "InvalidValueError"); + assert_eq!(class_of("decrypt_malformed"), "DecryptError"); + assert_eq!(class_of("stack_encrypt_aead"), "StashError"); + } +} diff --git a/languages/python/tests/conftest.py b/languages/python/tests/conftest.py new file mode 100644 index 000000000..672c3ed38 --- /dev/null +++ b/languages/python/tests/conftest.py @@ -0,0 +1,39 @@ +"""The credential fixtures every offline test module uses. + +One copy, so that a variable added here is cleared for every module. A +variable left set in one module could let its tests reach the network. +""" + +import pytest + +ACCESS_KEY = "CSAKtestKeyId.testKeySecret" +WORKSPACE_CRN = "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY" + +# Every variable the engine client reads: the credentials, the client key, +# and the service hosts. +CS_VARIABLES = ( + "CS_CLIENT_ACCESS_KEY", + "CS_WORKSPACE_CRN", + "CS_CLIENT_ID", + "CS_CLIENT_KEY", + "CS_CTS_HOST", + "CS_ZEROKMS_HOST", + "CS_VITUR_HOST", +) + + +@pytest.fixture +def no_credentials(monkeypatch, tmp_path): + """No CipherStash variable in the environment, and an empty profile store.""" + for name in CS_VARIABLES: + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("CS_CONFIG_PATH", str(tmp_path / "no-profile")) + + +@pytest.fixture +def credentials(monkeypatch, no_credentials): + """Credentials that parse, so ``Cipher`` builds, and no client key. A + call that gets past every check fails building the engine client, with + no request.""" + monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", ACCESS_KEY) + monkeypatch.setenv("CS_WORKSPACE_CRN", WORKSPACE_CRN) diff --git a/languages/python/tests/test_cipher.py b/languages/python/tests/test_cipher.py index 656fca0bc..7876481a5 100644 --- a/languages/python/tests/test_cipher.py +++ b/languages/python/tests/test_cipher.py @@ -16,22 +16,9 @@ TextEq, ) -ACCESS_KEY = "CSAKtestKeyId.testKeySecret" -WORKSPACE_CRN = "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY" +from conftest import ACCESS_KEY, WORKSPACE_CRN - -@pytest.fixture(autouse=True) -def no_credentials(monkeypatch, tmp_path): - """No credentials in the environment, and an empty profile store.""" - for name in ("CS_CLIENT_ACCESS_KEY", "CS_WORKSPACE_CRN", "CS_CLIENT_ID", "CS_CLIENT_KEY"): - monkeypatch.delenv(name, raising=False) - monkeypatch.setenv("CS_CONFIG_PATH", str(tmp_path / "no-profile")) - - -@pytest.fixture -def credentials(monkeypatch): - monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", ACCESS_KEY) - monkeypatch.setenv("CS_WORKSPACE_CRN", WORKSPACE_CRN) +pytestmark = pytest.mark.usefixtures("no_credentials") def declared(): diff --git a/languages/python/tests/test_decrypt.py b/languages/python/tests/test_decrypt.py index e8e1819e3..2dfba355b 100644 --- a/languages/python/tests/test_decrypt.py +++ b/languages/python/tests/test_decrypt.py @@ -28,14 +28,7 @@ STORED = json.dumps({"i": {"t": "users", "c": "email"}, "v": 2}) -@pytest.fixture(autouse=True) -def credentials(monkeypatch, tmp_path): - """Credentials that parse, so ``Cipher`` builds. Nothing here sends them.""" - for name in ("CS_CLIENT_ID", "CS_CLIENT_KEY", "CS_CTS_HOST", "CS_ZEROKMS_HOST", "CS_VITUR_HOST"): - monkeypatch.delenv(name, raising=False) - monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", "CSAKtestKeyId.testKeySecret") - monkeypatch.setenv("CS_WORKSPACE_CRN", "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY") - monkeypatch.setenv("CS_CONFIG_PATH", str(tmp_path / "no-profile")) +pytestmark = pytest.mark.usefixtures("credentials") @pytest.fixture @@ -117,6 +110,14 @@ def test_a_value_that_is_not_eql_json_is_malformed(untouched, value): assert error.field == "email" +@pytest.mark.parametrize("value", [None, ""], ids=["null", "empty"]) +def test_a_malformed_value_names_its_row(untouched, value): + rows = [{"email": STORED, "age": i} for i in range(2)] + [{"email": value, "age": 2}] + error = refused(rows) + assert error.code == "decrypt_malformed" + assert "row 2" in error.message + + def test_a_passthrough_int_that_arrives_as_str_is_the_wrong_type(untouched): error = refused([{"email": STORED, "age": "3"}], InvalidValueError) assert error.code == "value_wrong_type" diff --git a/languages/python/tests/test_encrypt.py b/languages/python/tests/test_encrypt.py index 3364979a3..3efcadaa7 100644 --- a/languages/python/tests/test_encrypt.py +++ b/languages/python/tests/test_encrypt.py @@ -34,14 +34,7 @@ SECRET = "secret@example.com" -@pytest.fixture(autouse=True) -def credentials(monkeypatch, tmp_path): - """Credentials that parse, so ``Cipher`` builds. Nothing here sends them.""" - for name in ("CS_CLIENT_ID", "CS_CLIENT_KEY", "CS_CTS_HOST", "CS_ZEROKMS_HOST", "CS_VITUR_HOST"): - monkeypatch.delenv(name, raising=False) - monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", "CSAKtestKeyId.testKeySecret") - monkeypatch.setenv("CS_WORKSPACE_CRN", "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY") - monkeypatch.setenv("CS_CONFIG_PATH", str(tmp_path / "no-profile")) +pytestmark = pytest.mark.usefixtures("credentials") @pytest.fixture From 635767bc114a5958bdba8c64a4ed61975e4d5358 Mon Sep 17 00:00:00 2001 From: Yuji Yokoo Date: Mon, 2 Nov 2026 23:32:09 +0900 Subject: [PATCH 3/5] fix(python): encode a str column value once, and share untouched (#1390) Encode a str column value with encode_utf8, and use the bytes it returns. Under abi3-py39, PyString::to_cow encodes the text and then copies it again, on every Python version. Borrowing the UTF-8 of a str is in the limited API only from Python 3.10. Add a test that the engine names the field of a value with only an identifier, in a model with two sealed fields. So the field of decrypt_malformed does not depend on #1508. Move the untouched fixture into conftest.py, so the decrypt and encrypt tests share one copy. Co-Authored-By: Claude Opus 5.5 --- languages/python/crates/core/src/decrypt.rs | 30 ++++++++++++++++ languages/python/crates/engine/src/decrypt.rs | 36 +++++++++---------- languages/python/tests/conftest.py | 13 ++++++- languages/python/tests/test_decrypt.py | 10 ------ languages/python/tests/test_encrypt.py | 10 ------ 5 files changed, 59 insertions(+), 40 deletions(-) diff --git a/languages/python/crates/core/src/decrypt.rs b/languages/python/crates/core/src/decrypt.rs index 25b832983..a413266fa 100644 --- a/languages/python/crates/core/src/decrypt.rs +++ b/languages/python/crates/core/src/decrypt.rs @@ -668,6 +668,36 @@ mod tests { assert_eq!(refused.field.as_deref(), Some("email")); } + #[test] + fn an_eql_value_with_only_an_identifier_names_its_field_among_several() { + let cipher = cipher(); + let stored = stored(&cipher, &users(), &["a@example.com"]); + let plan = ModelPlan::new( + "users", + vec![ + sealed("email", "TextEq", Kind::String), + sealed("login", "TextEq", Kind::String), + ], + ) + .unwrap(); + let mut row = Row::new(); + row.eql(&plan, &plan.fields()[0], eql(&stored[0]).as_bytes()) + .unwrap(); + row.eql( + &plan, + &plan.fields()[1], + br#"{"i": {"t": "users", "c": "login"}}"#, + ) + .unwrap(); + let mut rows = Rows::default(); + rows.push(row); + + let refused = run(decrypt_with(&cipher, &plan, rows)).unwrap_err(); + + assert_eq!(refused.code, "decrypt_malformed", "{refused:?}"); + assert_eq!(refused.field.as_deref(), Some("login"), "{refused:?}"); + } + #[test] fn a_value_stored_as_another_kind_is_a_kind_mismatch() { let cipher = cipher(); diff --git a/languages/python/crates/engine/src/decrypt.rs b/languages/python/crates/engine/src/decrypt.rs index 778fc65c1..063d2ad40 100644 --- a/languages/python/crates/engine/src/decrypt.rs +++ b/languages/python/crates/engine/src/decrypt.rs @@ -9,8 +9,6 @@ //! only after every value has been turned into a Python object, so a call //! that fails returns no instance. -use std::borrow::Cow; - use pyo3::prelude::*; use pyo3::sync::PyOnceLock; use pyo3::types::{PyBytes, PyDict, PyList, PyString}; @@ -110,7 +108,9 @@ fn stored_row(plan: &Plan, row: &Bound<'_, PyAny>, at: usize) -> PyResult { match &field.eql { Some(_) => { let text = eql_text(field, &value)?.map_err(in_row)?; - stored.eql(&plan.plan, field, &text).map_err(in_row)?; + stored + .eql(&plan.plan, field, text.as_bytes()) + .map_err(in_row)?; } None => passthrough(&mut stored, field, *kind, &value).map_err(in_row)?, } @@ -119,16 +119,20 @@ fn stored_row(plan: &Plan, row: &Bound<'_, PyAny>, at: usize) -> PyResult { } /// An EQL column's JSON text, as UTF-8 bytes: from a `str`, from `bytes` -/// that are UTF-8, or from a `dict` the driver already parsed. A `str` or -/// `bytes` value is borrowed, not copied: the text is copied once, into the -/// engine's frame. +/// that are UTF-8, or from a `dict` the driver already parsed. +/// +/// A `bytes` value is used as it is. A `str` is encoded once, by Python, +/// and the encoded `bytes` are used as they are. `PyString::to_cow` is not +/// used: under `abi3-py39` it encodes and then copies again, on every +/// Python version, because borrowing a `str`'s UTF-8 is limited API only +/// from 3.10. The text is then copied once more, into the engine's frame. /// /// The outer error is Python's own; the inner one is the refusal, for the /// caller to name the row in. -fn eql_text<'a>( +fn eql_text<'py>( field: &Field, - value: &'a Bound<'_, PyAny>, -) -> PyResult, Refusal>> { + value: &Bound<'py, PyAny>, +) -> PyResult, Refusal>> { let py = value.py(); let malformed = |what: String| { Ok(Err(Refusal { @@ -139,23 +143,17 @@ fn eql_text<'a>( })) }; if let Ok(text) = value.cast::() { - return Ok(Ok(match text.to_cow()? { - Cow::Borrowed(text) => Cow::Borrowed(text.as_bytes()), - Cow::Owned(text) => Cow::Owned(text.into_bytes()), - })); + return Ok(Ok(text.encode_utf8()?)); } if let Ok(bytes) = value.cast::() { - let bytes = bytes.as_bytes(); - return match std::str::from_utf8(bytes) { - Ok(_) => Ok(Ok(Cow::Borrowed(bytes))), + return match std::str::from_utf8(bytes.as_bytes()) { + Ok(_) => Ok(Ok(bytes.clone())), Err(_) => malformed("is bytes that are not UTF-8".to_owned()), }; } if value.cast::().is_ok() { return match dumps(py)?.call1((value,)) { - Ok(text) => Ok(Ok(Cow::Owned( - text.cast::()?.to_cow()?.into_owned().into_bytes(), - ))), + Ok(text) => Ok(Ok(text.cast::()?.encode_utf8()?)), Err(_) => malformed("is a dict that is not JSON".to_owned()), }; } diff --git a/languages/python/tests/conftest.py b/languages/python/tests/conftest.py index 672c3ed38..21f0c53bc 100644 --- a/languages/python/tests/conftest.py +++ b/languages/python/tests/conftest.py @@ -1,4 +1,4 @@ -"""The credential fixtures every offline test module uses. +"""The fixtures every offline test module uses. One copy, so that a variable added here is cleared for every module. A variable left set in one module could let its tests reach the network. @@ -6,6 +6,8 @@ import pytest +from stackencrypt import _engine + ACCESS_KEY = "CSAKtestKeyId.testKeySecret" WORKSPACE_CRN = "crn:ap-southeast-2.aws:ZVATKW3VHMFG27DY" @@ -37,3 +39,12 @@ def credentials(monkeypatch, no_credentials): no request.""" monkeypatch.setenv("CS_CLIENT_ACCESS_KEY", ACCESS_KEY) monkeypatch.setenv("CS_WORKSPACE_CRN", WORKSPACE_CRN) + + +@pytest.fixture +def untouched(): + """Fails the test if the shared client was built or a request was sent.""" + builds, requests = _engine.client_builds(), _engine.request_count() + yield + assert _engine.client_builds() == builds + assert _engine.request_count() == requests diff --git a/languages/python/tests/test_decrypt.py b/languages/python/tests/test_decrypt.py index 2dfba355b..9a4c94768 100644 --- a/languages/python/tests/test_decrypt.py +++ b/languages/python/tests/test_decrypt.py @@ -21,7 +21,6 @@ StashError, TextEq, ) -from stackencrypt import _engine # The `i` of a value stored under users.email, and the rest of an EQL value # left out: the checks before any request read only `i`. @@ -31,15 +30,6 @@ pytestmark = pytest.mark.usefixtures("credentials") -@pytest.fixture -def untouched(): - """Fails the test if the shared client was built or a request was sent.""" - builds, requests = _engine.client_builds(), _engine.request_count() - yield - assert _engine.client_builds() == builds - assert _engine.request_count() == requests - - @stackencrypt.model("users") @dataclasses.dataclass class User: diff --git a/languages/python/tests/test_encrypt.py b/languages/python/tests/test_encrypt.py index 3efcadaa7..305a92445 100644 --- a/languages/python/tests/test_encrypt.py +++ b/languages/python/tests/test_encrypt.py @@ -29,7 +29,6 @@ TextEq, Timestamp, ) -from stackencrypt import _engine SECRET = "secret@example.com" @@ -37,15 +36,6 @@ pytestmark = pytest.mark.usefixtures("credentials") -@pytest.fixture -def untouched(): - """Fails the test if the shared client was built or a request was sent.""" - builds, requests = _engine.client_builds(), _engine.request_count() - yield - assert _engine.client_builds() == builds - assert _engine.request_count() == requests - - def cipher_for(**fields): """A Cipher for a dataclass with ``email: TextEq`` and ``fields``.""" annotations = {"email": Annotated[str, TextEq], **fields} From 77c5bde3300d457dd9d9f0c36ff7b3eca9027184 Mon Sep 17 00:00:00 2001 From: Yuji Yokoo Date: Wed, 4 Nov 2026 17:50:54 +0900 Subject: [PATCH 4/5] test(python): name the target of the ignored SQLAlchemy relationship (#1390) SQLAlchemy finds a relationship's target in the type directly inside Mapped[...]. It does not look inside Annotated. So a bare relationship() on Mapped[Annotated[List["Post"], Ignore]] took List["Post"] as the target, and the first instance failed to configure the mapper. The live tests on SQLAlchemyUser failed on that. Naming the target, relationship("Post"), lets the mapper configure. posts is still not a list under this form. Spec #1341 records that, and leaves the fix to the SQLAlchemy layer. Co-Authored-By: Claude Opus 5.5 --- languages/python/tests/_declaration_cases.py | 4 ++-- languages/python/tests/test_encrypt_live.py | 6 +++++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/languages/python/tests/_declaration_cases.py b/languages/python/tests/_declaration_cases.py index 35c6a1705..a5912b1aa 100644 --- a/languages/python/tests/_declaration_cases.py +++ b/languages/python/tests/_declaration_cases.py @@ -120,7 +120,7 @@ class User(Base): id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) email: Mapped[Annotated[str, TextEq]] age: Mapped[Annotated[int, Passthrough]] - posts: Mapped[Annotated[List["Post"], Ignore]] = relationship() + posts: Mapped[Annotated[List["Post"], Ignore]] = relationship("Post") def test_a_sqlalchemy_class_declares_through_mapped(): @@ -315,7 +315,7 @@ class Book(LaterBase): __tablename__ = "books" id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) title: Mapped[Annotated[str, TextEq]] - authors: Mapped[Annotated[List["Author"], Ignore]] = relationship() # noqa: F821 + authors: Mapped[Annotated[List["Author"], Ignore]] = relationship("Author") # noqa: F821 error = refusal(build) assert error.code == "declaration_hints_unresolved" diff --git a/languages/python/tests/test_encrypt_live.py b/languages/python/tests/test_encrypt_live.py index e0c3d4f7a..e309c038e 100644 --- a/languages/python/tests/test_encrypt_live.py +++ b/languages/python/tests/test_encrypt_live.py @@ -163,7 +163,11 @@ class SqlAlchemyUser(Base): id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) email: Mapped[Annotated[str, TextEq]] age: Mapped[Annotated[int, Passthrough]] - posts: Mapped[Annotated[List["Post"], Ignore]] = relationship() + # SQLAlchemy does not look inside Annotated for the target, so the + # relationship names it. Without "Post", the first instance fails to + # configure the mapper. posts is still not a list: see spec #1341, the + # note on the Ignore relationship form. + posts: Mapped[Annotated[List["Post"], Ignore]] = relationship("Post") def build(cls, at): From 9d5188b24c4a4505a6029b583218e11ecb0cf173 Mon Sep 17 00:00:00 2001 From: Yuji Yokoo Date: Thu, 5 Nov 2026 00:05:04 +0900 Subject: [PATCH 5/5] docs(python): record that Ignore does not work on a SQLAlchemy collection (#1390) SQLAlchemy reads a relationship's target and collection type from the type directly inside Mapped[...]. It does not look inside Annotated. So Ignore works on a relationship to one object, and a collection marked Ignore maps as a single object. ADR-0001 records this in an amendment. The README shows a collection left without an annotation, one object marked Ignore, and the Optional form, which must name its target. Co-Authored-By: Claude Opus 5.5 --- languages/python/README.md | 45 +++++++++++++++++++ ...on-is-checked-when-the-class-is-defined.md | 29 ++++++++++++ 2 files changed, 74 insertions(+) diff --git a/languages/python/README.md b/languages/python/README.md index 5b6a3dc24..d89948324 100644 --- a/languages/python/README.md +++ b/languages/python/README.md @@ -84,6 +84,51 @@ A passthrough `datetime` field needs a `timestamptz` column. psycopg reads a `ti `decrypt` checks every row before it sends anything. A value stored under another table or column raises `DecryptError` with the code `decrypt_foreign_row`. A passthrough value of another type than the field declares raises `InvalidValueError`. When any row fails, `decrypt` returns no instances. +## Declare a SQLAlchemy relationship + +A SQLAlchemy relationship is not a column. Every annotated field needs a marker, so write a relationship in one of two ways: + +- **A collection,** such as an author's books: leave the relationship without an annotation. The decorator never reads it. +- **One object,** such as a book's author: mark it `Ignore`. The SDK then leaves it out. + +```python +from sqlalchemy import ForeignKey +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship +from stackencrypt import Ignore + + +class Base(DeclarativeBase): + pass + + +@stackencrypt.model("authors") +class Author(Base): + __tablename__ = "authors" + id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) + name: Mapped[Annotated[str, TextEq]] + books = relationship("Book", back_populates="author") # a list of Book + + +@stackencrypt.model("books") +class Book(Base): + __tablename__ = "books" + id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) + author_id: Mapped[Annotated[int, Passthrough]] = mapped_column(ForeignKey("authors.id")) + author: Mapped[Annotated["Author", Ignore]] = relationship(back_populates="books") +``` + +Do not mark a collection `Ignore`. SQLAlchemy reads the type directly inside `Mapped[...]`, and it does not look inside `Annotated`. So `Mapped[Annotated[list["Book"], Ignore]]` does not give a list. The cost of the unannotated form is that a type checker does not know the type of `books`. + +If a book can have no author, write `Optional["Author"]`. Then the relationship must name its target, as `relationship("Author")` does: + +```text +author: Mapped[Annotated[Optional["Author"], Ignore]] = relationship("Author", back_populates="books") +``` + +With this form, a new `Book` has `author` set to `None` until you set it. A bare `relationship()` fails here, because SQLAlchemy takes `Optional["Author"]` as the target. + +`Author` must be defined before `Book`, because the decorator evaluates `"Author"` in the hint when `Book`'s class statement runs. + ## Run the tests From `languages/python`: diff --git a/languages/python/docs/adr/0001-a-declaration-is-checked-when-the-class-is-defined.md b/languages/python/docs/adr/0001-a-declaration-is-checked-when-the-class-is-defined.md index 90dc0100a..4ee1b91a0 100644 --- a/languages/python/docs/adr/0001-a-declaration-is-checked-when-the-class-is-defined.md +++ b/languages/python/docs/adr/0001-a-declaration-is-checked-when-the-class-is-defined.md @@ -240,6 +240,35 @@ Option 2. checked by its own decorator, or not at all, and `Cipher(Post)` refuses an undecorated `Post` as decision 4 says. + *Amended 2026-11-04:* the `Ignore` form works for a relationship to + one object. It does not work for a collection. SQLAlchemy reads the + type directly inside `Mapped[...]` to choose between the two. + SQLAlchemy does not look inside `Annotated`, so it maps a single + object in both cases. + + - One object: `author: Mapped[Annotated["User", stash.Ignore]] = + relationship()` maps, and `author` is the related row. With + `Optional["User"]`, the relationship must name its target, as in + `relationship("User")`. + - A collection: with a bare `relationship()`, the first instance fails + to configure the mapper, because SQLAlchemy takes `List["Post"]` as + the target. With `relationship("Post")` the mapper configures, but + `posts` is one object, not a list. `uselist=True` does not change + that: SQLAlchemy sets `uselist` from the annotation. + + So a collection relationship takes no annotation: + `posts = relationship("Post")`. The reader never sees it, and `posts` + is a list. The cost is that a type checker does not know its type. + + A typed collection needs the SQLAlchemy layer, because that layer can + read the mapper. See spec #1341, the note on the `Ignore` relationship + form. + + These results are from SQLAlchemy 2.1.6 on Python 3.14.10. The + bare collection also failed on SQLAlchemy 2.0.57. The probe of + 2026-10-16 under "What was run" found the markers. It never made an + instance, so the mapper never configured. + The first two are checked at decoration. The constructor check reads `inspect.signature(cls)` and accepts a `**kwargs` parameter. It also checks that every `Ignore` field is