diff --git a/AGENTS.md b/AGENTS.md index 197730ef2..4ddc42071 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,7 +93,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates". - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do. - `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. The stack-encrypt guest has a second build with the EQL types (`mise run wasm:guest:build:eql`, cargo feature `eql`, linking `eql-bindings` by path; `wasm:guest:build:eql:deterministic` is its test build), embedded by `encrypt/eql` and registered on import, so a program whose generated code names an EQL type runs it (ADR-0007, amended). `encrypt/eql`'s types and `Types` table are generated from the EQL catalog by `eql-codegen` (`eql_gen.go`, written by `mise run types:generate` in `packages/eql` and drift-gated by `types:check` and `cargo test -p eql-codegen`). Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. Releases are cut by hand with `.github/workflows/release-golang.yml` (`gh workflow run release-golang.yml -f version=X.Y.Z[-pre] --ref `): because the guests are gitignored on main and Go fetches a version as the files committed at its tag, the workflow builds them, commits them on a release-only commit no branch points at, tags it `languages/golang/vX.Y.Z` and pushes the tag alone, then attaches the EQL v4 SQL bundle to the GitHub release. The ref must carry the EQL v4 bundle, or the build stops at "Build the EQL SQL bundles". `scripts/go-release-version.mjs` refuses, before anything is built, every version Go would refuse (non-canonical semver, `+build` metadata, a v2+ major on a module path with no `/vN` suffix, a version shaped like a pseudo-version), a final version dispatched from anywhere but `main`, and a final version while `languages/golang/go.mod` pins a `github.com/cipherstash/vitaminc/…` module at a pseudo-version (vitaminc's Go bindings have no tags, and a TEMPORARY pin on an unreleased commit must be gone before the module is tagged); a pre-release may come from a branch and may carry such a pin. The build job also refuses a version the Go proxy already lists in `@v/list`, which catches a tag deleted after the proxy fetched it; it never asks the proxy for the version itself before the tag exists, because the proxy would cache "not found" for up to 30 minutes and `verify` would fail after the push. The build checks the `Cargo.lock` of the EQL workspace and both guest workspaces with `--locked` first, since none of those builds passes it. It runs in a job with no write token and hands the guests to a `publish` job that holds `contents: write` and runs only git and gh. `publish` fetches the commit with plain unauthenticated git rather than `actions/checkout`, whose `token` input defaults to the job's writable token, and refuses a release commit whose tree is not the one the build tested. `publish` takes the paths it commits from the workflow's fixed `GUESTS` list, never from the artifact, and refuses a `guests.sha256` that is not exactly one SHA-256 line for each of those three guests: the build job runs third-party code after writing that manifest, so a path read from it could put any file on the tag (`scripts/__tests__/go-release-artifact.test.mjs`). A version is immutable once the Go proxy has fetched it, so a tag is never moved or deleted: if `publish` or `verify` fails after the tag is pushed, use "Re-run failed jobs" on the same run, which keeps a tag that run pushed and creates or completes the release; a new dispatch of the same version is refused. Never tag a main commit by hand: `go get` would fetch a module with no guests. -- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. `crates/core` compiles the Go guest's EQL resolver, `languages/golang/encrypt/guest/src/targets.rs`, in place through `#[path]`. So both SDKs run one resolver, and a change to that file is a change to both. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. A `live` job runs `mise run test:live`: `encrypt` and `decrypt` against ZeroKMS with the repository's `CS_*` secrets, and the rows inserted into and read back from a `postgres:17` service with the EQL v4 bundle built from the tree. It skips on a fork's pull request, and the task fails on any skipped test. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, attrs, Pydantic, SQLAlchemy, and psycopg. The middle three are for the declaration tests, and psycopg is for the live tests. None has an upper bound, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: Python ADR-0001 under `languages/python/docs/adr/`, the spec in #1341. +- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. `crates/core` compiles the Go guest's EQL resolver, `languages/golang/encrypt/guest/src/targets.rs`, in place through `#[path]`. So both SDKs run one resolver, and a change to that file is a change to both. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. A `live` job runs `mise run test:live`: `encrypt`, `decrypt` and `query` against ZeroKMS with the repository's `CS_*` secrets, and the rows inserted into, searched in and read back from a `postgres:17` service with the EQL v4 bundle built from the tree. It skips on a fork's pull request, and the task fails on any skipped test. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, attrs, Pydantic, SQLAlchemy, and psycopg. The middle three are for the declaration tests, and psycopg is for the live tests. None has an upper bound, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: the Python ADRs under `languages/python/docs/adr/` (ADR-0001, the declaration; ADR-0002, the call shape of `query`), the spec in #1341. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). diff --git a/languages/python/README.md b/languages/python/README.md index d89948324..3316ce2e5 100644 --- a/languages/python/README.md +++ b/languages/python/README.md @@ -2,7 +2,7 @@ `stackencrypt` is the Python SDK for CipherStash Stack. It encrypts the fields of a Python model into values you store in PostgreSQL with EQL v4. EQL is CipherStash's set of PostgreSQL types for encrypted values. -This is a stub. The package is not released. Today it can encrypt a model and decrypt it again. Querying comes in a later release. +This is a stub. The package is not released. Today it can encrypt a model, decrypt it again, and search an encrypted column by equality. ## Before you start @@ -84,6 +84,27 @@ A passthrough `datetime` field needs a `timestamptz` column. psycopg reads a `ti `decrypt` checks every row before it sends anything. A value stored under another table or column raises `DecryptError` with the code `decrypt_foreign_row`. A passthrough value of another type than the field declares raises `InvalidValueError`. When any row fails, `decrypt` returns no instances. +## Search by equality + +`cipher.query` takes a field's name and a value, and returns a query term as a `str`. Cast the term to the field's query type in the SQL: + +```python +term = cipher.query("email", "a@example.com") +with psycopg.connect(DATABASE_URL) as conn: + with conn.cursor(row_factory=dict_row) as cur: + cur.execute( + "SELECT email, age FROM users WHERE email = %s::eql_v4.query_text_eq", + (term,), + ) + users = cipher.decrypt(cur.fetchall()) +``` + +The query type is the marker's `query_sql_domain`. For `TextEq` it is `eql_v4.query_text_eq`. The `users_email_eq` index from "Create the table" lets PostgreSQL find the row without reading every row. + +`query` sends no request to CipherStash after the first operation of the process. The term is made from a key that the first operation loads. + +`query` makes equality terms only. A field whose marker has no `eq` index raises `stackencrypt.QueryError` with the code `query_unsupported`. Examples are `Text`, `Passthrough`, `IntegerOrd`, and `TextMatch`. A name that is not a declared field raises `QueryError` with the code `query_unknown_field`. The value is checked as `encrypt` checks it. + ## Declare a SQLAlchemy relationship A SQLAlchemy relationship is not a column. Every annotated field needs a marker, so write a relationship in one of two ways: diff --git a/languages/python/crates/core/src/decrypt.rs b/languages/python/crates/core/src/decrypt.rs index a413266fa..667206201 100644 --- a/languages/python/crates/core/src/decrypt.rs +++ b/languages/python/crates/core/src/decrypt.rs @@ -443,20 +443,20 @@ impl Sink for OpenedValue<'_> { } #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; use crate::encrypt::{encrypt_with, Column}; use crate::values::Kind; use stack_kms::FakeDataKeySource; - fn run(future: F) -> F::Output { + pub(crate) fn run(future: F) -> F::Output { tokio::runtime::Builder::new_current_thread() .build() .unwrap() .block_on(future) } - fn cipher() -> StackCipher { + pub(crate) fn cipher() -> StackCipher { run(StackCipher::builder().kms(FakeDataKeySource::new()).init()).unwrap() } @@ -474,7 +474,7 @@ mod tests { } } - fn users() -> ModelPlan { + pub(crate) fn users() -> ModelPlan { ModelPlan::new( "users", vec![sealed("email", "TextEq", Kind::String), passthrough("age")], @@ -501,7 +501,7 @@ mod tests { } /// Encrypt one `users` row per email, with `age` 30 + its index. - fn stored( + pub(crate) fn stored( cipher: &StackCipher, plan: &ModelPlan, emails: &[&str], diff --git a/languages/python/crates/core/src/lib.rs b/languages/python/crates/core/src/lib.rs index 8be71d7ce..fd2faf5fe 100644 --- a/languages/python/crates/core/src/lib.rs +++ b/languages/python/crates/core/src/lib.rs @@ -4,8 +4,8 @@ //! It holds the engine commit the build recorded (`build.rs`), the engine's //! check of a model's plan ([`plan`]), the credential lookup //! ([`credentials`]), the one engine client a process shares ([`client`]), -//! the value rules ([`values`]), `encrypt` itself ([`encrypt`]) and -//! `decrypt` ([`decrypt`]). Each +//! the value rules ([`values`]), `encrypt` itself ([`encrypt`]), +//! `decrypt` ([`decrypt`]) and `query` ([`query`]). Each //! reports an engine error as a [`refusal::Refusal`]. The Python SDK spec //! (#1341, "Layout") says what else belongs here. @@ -14,6 +14,7 @@ pub mod credentials; pub mod decrypt; pub mod encrypt; pub mod plan; +pub mod query; pub mod refusal; pub mod values; diff --git a/languages/python/crates/core/src/query.rs b/languages/python/crates/core/src/query.rs new file mode 100644 index 000000000..b594b6d8c --- /dev/null +++ b/languages/python/crates/core/src/query.rs @@ -0,0 +1,214 @@ +//! `query`: the engine's equality query term for one field and one value. +//! +//! The term is the EQL query value the engine derives through the field's +//! EQL type, as JSON text. The SQL casts it to the type's query domain, such +//! as `eql_v4.query_text_eq`, and EQL's `=` compares it with the stored +//! values' `eq_term`. A term is derived from the index key in the loaded +//! keyset, so no ZeroKMS request is made once the shared client is built. + +use stack_encrypt::dynamic::plaintext::Sink; +use stack_encrypt::dynamic::{self, OpSpec, Operand}; +use stack_encrypt::kms::{DataKeySource, IndexKeySource}; +use stack_encrypt::{Encrypt, KeysetCipher}; + +use crate::client; +use crate::encrypt::{engine_fault, ModelPlan}; +use crate::plan; +use crate::refusal::Refusal; +use crate::targets; +use crate::values::{self, Kind, Value}; + +/// The equality query term for `value` on the sealed field `field` of +/// `plan`, as JSON text. +/// +/// Checks first, with no I/O, that the field answers equality. Then builds +/// the shared client on the first call in the process. The caller has +/// released the GIL, and has checked that `value` fits the field's kind. +/// +/// # Errors +/// +/// `query_unsupported` for a field that answers no equality query, before +/// any request. Then the client's refusal, or the engine's for the query. +pub fn query( + plan: &ModelPlan, + field: &str, + kind: Kind, + value: Value<'_>, +) -> Result { + answers_equality(plan, field)?; + let client = client::shared()?; + client.block_on(query_with(&client.keyset(), plan, field, kind, value)) +} + +/// [`query`] under `keyset`. +/// +/// # Errors +/// +/// The engine's refusal for the query. +pub async fn query_with( + keyset: &KeysetCipher<'_, K>, + plan: &ModelPlan, + field: &str, + kind: Kind, + value: Value<'_>, +) -> Result +where + K: DataKeySource + IndexKeySource + Sync + 'static, +{ + let term = dynamic::query_term( + keyset, + plan.plan(), + field, + OpSpec::Eq, + Typed(kind, value), + &targets::resolver(), + ) + .map_err(|error| plan::refusal(&error))? + .await + .map_err(|error| Refusal::of(&error))?; + let Operand::One(json) = term.operand() else { + return Err(engine_fault("the equality term is not one operand")); + }; + // A copy of the term's bytes, about a hundred: `QueryTerm` has no + // accessor that gives up its operand. + String::from_utf8(json.clone()).map_err(|_| engine_fault("the equality term is not UTF-8")) +} + +/// Whether `field` answers an equality query, from the plan alone: the +/// engine answers one through the field's `eq` index only, and a type with +/// only an order or match index answers none. `Cipher.query` checks the +/// marker's indexes the same way. This check is for a caller that did not. +/// +/// # Errors +/// +/// `query_unsupported`, or the engine's refusal of the field. +pub fn answers_equality(plan: &ModelPlan, field: &str) -> Result<(), Refusal> { + let operators = dynamic::field_operators(plan.plan(), field, &targets::resolver()) + .map_err(|error| plan::refusal(&error))?; + if operators + .iter() + .any(|(operator, _)| *operator == OpSpec::Eq) + { + return Ok(()); + } + Err(Refusal { + code: "query_unsupported".to_owned(), + message: format!("field {field:?} has no eq index, so it answers no equality query."), + field: Some(field.to_owned()), + reason: None, + }) +} + +/// A value written as the field's kind, as `encrypt` writes it. +struct Typed<'a>(Kind, Value<'a>); + +impl Encrypt for Typed<'_> { + fn visit(self, sink: S) -> Result { + values::write(sink, Some(self.0), self.1) + } +} + +#[cfg(test)] +mod tests { + use stack_encrypt::StackCipher; + use stack_kms::FakeDataKeySource; + + use super::*; + use crate::decrypt::tests::{cipher, run, stored, users}; + use crate::encrypt::{Column, Field}; + + fn term(cipher: &StackCipher, email: &str) -> serde_json::Value { + let json = run(query_with( + &cipher.default_keyset(), + &users(), + "email", + Kind::String, + Value::Text(email), + )) + .unwrap(); + serde_json::from_str(&json).unwrap() + } + + #[test] + fn the_term_is_the_stored_eq_term_with_no_ciphertext() { + let cipher = cipher(); + let term = term(&cipher, "a@example.com"); + let Column::Eql(json) = &stored(&cipher, &users(), &["a@example.com"])[0][0] else { + panic!("not EQL"); + }; + let stored: serde_json::Value = serde_json::from_str(json).unwrap(); + + // `hm` is the equality term EQL's `=` compares. The query carries + // no ciphertext: a term reveals nothing a stored row does not. + assert_eq!(term["hm"], stored["hm"], "{term}"); + assert_eq!(term["i"], stored["i"], "{term}"); + assert!(term.get("c").is_none(), "{term}"); + } + + #[test] + fn the_same_value_gives_the_same_term_and_another_value_another() { + let cipher = cipher(); + + assert_eq!( + term(&cipher, "a@example.com"), + term(&cipher, "a@example.com") + ); + assert_ne!( + term(&cipher, "a@example.com"), + term(&cipher, "b@example.com") + ); + } + + #[test] + fn a_field_with_only_an_order_or_match_index_answers_no_equality() { + for eql in ["IntegerOrd", "IntegerOrdOre", "TextMatch"] { + let kind = if eql == "TextMatch" { + Kind::String + } else { + Kind::Int32 + }; + let plan = ModelPlan::new( + "users", + vec![Field { + name: "rank".to_owned(), + eql: Some((eql.to_owned(), kind)), + }], + ) + .unwrap(); + + let refused = answers_equality(&plan, "rank").unwrap_err(); + + assert_eq!(refused.code, "query_unsupported", "{eql}"); + assert_eq!(refused.field.as_deref(), Some("rank")); + } + } + + #[test] + fn a_field_with_an_eq_index_answers_equality() { + answers_equality(&users(), "email").unwrap(); + } + + #[test] + fn a_passthrough_field_answers_no_equality() { + assert_eq!( + answers_equality(&users(), "age").unwrap_err().code, + "query_unsupported" + ); + } + + #[test] + fn a_passthrough_field_is_refused_by_the_engine() { + let cipher = cipher(); + + let refused = run(query_with( + &cipher.default_keyset(), + &users(), + "age", + Kind::Int64, + Value::Int(1), + )) + .unwrap_err(); + + assert_eq!(refused.field.as_deref(), Some("age"), "{refused:?}"); + } +} diff --git a/languages/python/crates/engine/src/encrypt.rs b/languages/python/crates/engine/src/encrypt.rs index 8084b431b..6a86bec11 100644 --- a/languages/python/crates/engine/src/encrypt.rs +++ b/languages/python/crates/engine/src/encrypt.rs @@ -18,7 +18,7 @@ use stackencrypt_core::refusal::Refusal; use stackencrypt_core::values::{self, Kind}; use crate::errors::raise; -use crate::values::{read, to_python, PyKind}; +use crate::values::{read, to_python, PyKind, Read}; /// One plan field as the decorator hands it over: `(name, eql_type, kind, /// value_type)`. @@ -161,6 +161,45 @@ pub fn request_count() -> u64 { stackencrypt_core::client::request_count() } +/// `value` read as `field`'s Python type and checked against the range of +/// its kind, as `encrypt` and `query` both take it. A refusal names the +/// field, with the kind as its reason. +/// +/// The outer error is Python's own, such as one a `__index__` raises. +pub(crate) fn field_value<'a>( + value: &'a Bound<'_, PyAny>, + field: &Field, + kind: PyKind, +) -> PyResult, Refusal>> { + let engine_kind = field.eql.as_ref().map(|(_, kind)| *kind); + let reason = engine_kind.map_or("passthrough", Kind::name); + let named = |refused: Refusal| Refusal { + message: format!("field {:?}: {}", field.name, refused.message), + field: Some(field.name.clone()), + reason: Some(reason.to_owned()), + ..refused + }; + let read = match read(value, kind, reason)? { + Ok(read) => read, + Err(refused) => return Ok(Err(named(refused))), + }; + // The range rules first, so a misfit is refused with its own code + // and the field's name, not as the sink's error. + if let Err(misfit) = values::scalar(engine_kind, read.value()) { + let what = match engine_kind { + Some(kind) => format!("the value does not fit {}.", kind.name()), + None => "the value is outside what the engine stores.".to_owned(), + }; + return Ok(Err(named(Refusal { + code: misfit.code().to_owned(), + message: what, + field: None, + reason: None, + }))); + } + Ok(Ok(read)) +} + /// The models as the engine's source: a sequence of maps, one entry per /// plan field, read from each model's attribute of that name. /// @@ -188,49 +227,21 @@ impl Models<'_, '_> { field: &Field, kind: PyKind, ) -> Result { - let py = model.py(); // The attribute, as the user's class gives it: a property that // raises propagates unchanged. let attribute = model .getattr(&field.name) .map_err(|error| self.stop(error))?; - let engine_kind = field.eql.as_ref().map(|(_, kind)| *kind); - let reason = engine_kind.map_or("passthrough", Kind::name); - let refuse = |refused: Refusal| { - self.stop::(raise( - py, - Refusal { - message: format!("field {:?}: {}", field.name, refused.message), - field: Some(field.name.clone()), - reason: Some(reason.to_owned()), - ..refused - }, - )) - }; - let read = match read(&attribute, kind, reason) { + let read = match field_value(&attribute, field, kind) { Ok(Ok(read)) => read, - Ok(Err(refused)) => return Err(refuse(refused)), + Ok(Err(refused)) => return Err(self.stop(raise(model.py(), refused))), Err(error) => return Err(self.stop(error)), }; - let value = read.value(); - // The range rules first, so a misfit is refused with its own code - // and the field's name, not as the sink's error. - if let Err(misfit) = values::scalar(engine_kind, value) { - let what = match engine_kind { - Some(kind) => format!("the value does not fit {}.", kind.name()), - None => "the value is outside what the engine stores.".to_owned(), - }; - return Err(refuse(Refusal { - code: misfit.code().to_owned(), - message: what, - field: None, - reason: None, - })); - } + let engine_kind = field.eql.as_ref().map(|(_, kind)| *kind); // A passthrough is written as a plain scalar: the plan's // `"passthrough"` output carries the field's frame as it is, so a // passthrough node here would come back nested in one. - values::write(sink, engine_kind, value) + values::write(sink, engine_kind, read.value()) } } diff --git a/languages/python/crates/engine/src/errors.rs b/languages/python/crates/engine/src/errors.rs index 63bb42dca..cc05ec88a 100644 --- a/languages/python/crates/engine/src/errors.rs +++ b/languages/python/crates/engine/src/errors.rs @@ -3,7 +3,7 @@ //! The exception classes live in `stackencrypt._errors`, in Python, so that //! the pure-Python checks and the extension raise the same ones. The class //! is chosen by the code's prefix, as the spec's "Errors" section assigns -//! them: `value_*` is `InvalidValueError`, `decrypt_*` is `DecryptError`, +//! them: `value_*` is `InvalidValueError`, `decrypt_*` is `DecryptError`, `query_*` is `QueryError`, //! `stack_auth_*` and the engine's context and keyset refusals are //! `AuthenticationError`, and anything else is the base `StashError`. @@ -37,6 +37,8 @@ fn exception(py: Python<'_>, refused: &Refusal) -> PyResult { "InvalidValueError" } else if refused.code.starts_with("decrypt_") { "DecryptError" + } else if refused.code.starts_with("query_") { + "QueryError" } else if refused.code.starts_with("stack_auth_") || AUTHENTICATION.contains(&refused.code.as_str()) { @@ -104,6 +106,7 @@ mod tests { fn each_prefix_has_its_class() { assert_eq!(class_of("value_none"), "InvalidValueError"); assert_eq!(class_of("decrypt_malformed"), "DecryptError"); + assert_eq!(class_of("query_unsupported"), "QueryError"); assert_eq!(class_of("stack_encrypt_aead"), "StashError"); } } diff --git a/languages/python/crates/engine/src/lib.rs b/languages/python/crates/engine/src/lib.rs index 64c525190..1447c254d 100644 --- a/languages/python/crates/engine/src/lib.rs +++ b/languages/python/crates/engine/src/lib.rs @@ -6,6 +6,7 @@ mod decrypt; mod encrypt; mod errors; +mod query; mod values; use pyo3::prelude::*; @@ -96,6 +97,7 @@ fn _engine(module: &Bound<'_, PyModule>) -> PyResult<()> { module.add_function(wrap_pyfunction!(encrypt::build_plan, module)?)?; module.add_function(wrap_pyfunction!(encrypt::encrypt, module)?)?; module.add_function(wrap_pyfunction!(decrypt::decrypt, module)?)?; + module.add_function(wrap_pyfunction!(query::query, module)?)?; module.add_function(wrap_pyfunction!(encrypt::client_builds, module)?)?; module.add_function(wrap_pyfunction!(encrypt::request_count, module)?)?; #[cfg(feature = "gate-probe")] diff --git a/languages/python/crates/engine/src/query.rs b/languages/python/crates/engine/src/query.rs new file mode 100644 index 000000000..70cebf698 --- /dev/null +++ b/languages/python/crates/engine/src/query.rs @@ -0,0 +1,57 @@ +//! `query` at the boundary: one value checked as `encrypt` checks it, and +//! the engine's equality term back as a `str`. +//! +//! `Cipher.query` has already refused a name that is not a declared field, +//! and a field with no `eq` index (the spec in #1341, "Errors"). Core +//! checks the second again from the plan, before the client is built. + +use pyo3::prelude::*; +use stackencrypt_core::query as core; +use stackencrypt_core::refusal::Refusal; + +use crate::encrypt::{field_value, Plan}; +use crate::errors::raise; + +/// The equality query term for `value` on the sealed field `field`. +/// +/// The value is checked first, by exact type and range, with no request. +/// Then the shared client is built on the first call in the process, and +/// the engine derives the term with the GIL released. A term needs no +/// request of its own. +#[pyfunction] +pub fn query( + py: Python<'_>, + plan: &Plan, + field: &str, + value: &Bound<'_, PyAny>, +) -> PyResult { + let found = plan + .plan + .fields() + .iter() + .zip(&plan.kinds) + .find(|(candidate, _)| candidate.name == field); + let Some((declared, &kind)) = found else { + return Err(raise(py, not_sealed(field))); + }; + let Some((_, engine_kind)) = declared.eql else { + return Err(raise(py, not_sealed(field))); + }; + let read = field_value(value, declared, kind)?.map_err(|refused| raise(py, refused))?; + let value = read.value(); + py.detach(|| core::query(&plan.plan, field, engine_kind, value)) + .map_err(|refused| raise(py, refused)) +} + +fn not_sealed(field: &str) -> Refusal { + Refusal { + code: "stack_encrypt_unknown".to_owned(), + message: format!( + "field {field:?} reached the engine's query, but it is not a sealed field of the \ + plan. Cipher.query should have refused it. Report this message as a stackencrypt \ + bug." + ), + field: Some(field.to_owned()), + reason: None, + } +} diff --git a/languages/python/docs/adr/0002-a-query-names-its-field-by-a-string.md b/languages/python/docs/adr/0002-a-query-names-its-field-by-a-string.md new file mode 100644 index 000000000..32b583207 --- /dev/null +++ b/languages/python/docs/adr/0002-a-query-names-its-field-by-a-string.md @@ -0,0 +1,60 @@ +--- +status: proposed +date: 2026-11-02 +ticket: "#1391, the first ticket that adds `query`, as the spec (#1341) says" +extends: ADR-0001 +--- + +# A query names its field by a string + +`Cipher.query` takes the field's name as a `str`, and one value: + +```python +term = users.query("email", "a@example.com") +``` + +The call returns the equality query term as a `str`. +There is no operator argument. The first version answers equality only. + +The spec in #1341 says the first ticket that adds `query` decides its call shape. This Architecture Decision Record (ADR) records that decision. + +## The options + +### A. The field's name as a string (chosen) + +```python +users.query("email", "a@example.com") +``` + +- A works for every class kind that core supports: dataclasses, attrs, Pydantic, `NamedTuple`, and SQLAlchemy. +- Core reads only the declaration. Core does not change the user's class. +- A misspelt name fails at the first call, before any request, with `query_unknown_field`. The message lists the fields that have a query. +- A type checker cannot find a misspelt name. An editor's "rename symbol" does not update the string. + +### B. The class attribute + +```python +users.query(User.email, "a@example.com") +``` + +- A type checker and an editor see the name. A misspelt name fails before the program runs. A rename updates every call. +- With a typed attribute, such as `Field[str]`, a type checker can also check the value's type. +- B matches the SQLAlchemy style, `User.email == x`. +- B does not work for a plain dataclass, attrs, or Pydantic class. A dataclass field with no default has no class attribute, so `User.email` raises `AttributeError`. A field with a default gives the default value, not the field. +- So core would have to put an attribute on the class for each field. Core does not change the user's class today. ADR-0001 stores the declaration on the class, and adds no attribute that a field would use. + +## Decision + +A, for now. + +B stays open for the later SQLAlchemy layer. A SQLAlchemy model already has a column attribute for each field, so B works there without core changing the class. The layer reads the attribute's name and calls core with the string. The spec in #1341 keeps that layer out of scope for now. + +A does not block B. A later version can accept a string or an attribute in the same position. A caller who passes a string sees no change. + +An operator can come later as a keyword argument, such as `op=`. A caller who passes none still gets equality. + +## Consequences + +- `query_unknown_field` and `query_unsupported` are raised in Python, by `Cipher.query`, before the extension is called. +- A typo is found when the call runs, not when a type checker runs. The error message lists the fields that have a query, so the caller can correct the name. +- The SQLAlchemy layer's spec decides whether it accepts `User.email`. diff --git a/languages/python/python/stackencrypt/__init__.py b/languages/python/python/stackencrypt/__init__.py index c85c6c767..d31ff4df8 100644 --- a/languages/python/python/stackencrypt/__init__.py +++ b/languages/python/python/stackencrypt/__init__.py @@ -3,7 +3,8 @@ Walking skeleton. The package loads its native extension, ``_engine``, exposes two facts about the build, exports the markers a model's fields are declared with, reads a model's declaration with ``model``, and builds a ``Cipher`` for -a declared class that encrypts its instances. +a declared class that encrypts its instances, decrypts them, and makes +query terms. """ from stackencrypt import _engine, _eql @@ -17,6 +18,7 @@ DeclarationError, DecryptError, InvalidValueError, + QueryError, StashError, ) from stackencrypt._markers import Ignore, Passthrough @@ -34,6 +36,7 @@ "Ignore", "InvalidValueError", "Passthrough", + "QueryError", "StashError", "__engine__", "__version__", diff --git a/languages/python/python/stackencrypt/_cipher.py b/languages/python/python/stackencrypt/_cipher.py index e9bd4f5de..4d538ec0d 100644 --- a/languages/python/python/stackencrypt/_cipher.py +++ b/languages/python/python/stackencrypt/_cipher.py @@ -1,15 +1,20 @@ """``Cipher``: encrypts and decrypts the instances of one declared class. It checks the class and the credentials, and builds the model's plan once. -``encrypt`` and ``decrypt`` are here. ``query`` comes in a later ticket. +``encrypt``, ``decrypt`` and ``query`` are here. """ from typing import Dict, Generic, List, Mapping, Sequence, Type, TypeVar from stackencrypt import _engine from stackencrypt._markers import Passthrough -from stackencrypt._declaration import Declaration, check_unchanged, declaration_of -from stackencrypt._errors import AuthenticationError, DeclarationError +from stackencrypt._declaration import ( + Declaration, + DeclaredField, + check_unchanged, + declaration_of, +) +from stackencrypt._errors import AuthenticationError, DeclarationError, QueryError T = TypeVar("T") @@ -89,6 +94,55 @@ def decrypt(self, rows: Sequence[Mapping[str, object]]) -> List[T]: """ return _engine.decrypt(self._plan, self._cls, list(rows)) + def query(self, field: str, value: object) -> str: + """The equality query term for ``value`` on the declared field + ``field``, named as a ``str``:: + + term = users.query("email", "a@example.com") + cur.execute( + "SELECT * FROM users WHERE email = %s::eql_v4.query_text_eq", + (term,), + ) + + The term is a ``str``. The SQL casts it to the field marker's + ``query_sql_domain``. A term is derived from the keyset already + loaded: after the shared client is built, ``query`` makes no request. + + A name that is not a declared field raises ``QueryError`` + (``query_unknown_field``). A field whose marker has no ``eq`` index + raises ``QueryError`` (``query_unsupported``). Examples are ``Text``, + ``Passthrough``, ``IntegerOrd`` and ``TextMatch``. The value is checked as ``encrypt`` checks + it, and a value that does not fit raises ``InvalidValueError``. All + three are raised before any request. + """ + declared = next( + (f for f in self._declaration.fields if f.name == field), None + ) + if declared is None: + raise QueryError( + "query_unknown_field", + f"{field!r} is not a declared field of " + f"{self._cls.__qualname__!r}. {self._queryable()}", + field=field if isinstance(field, str) else None, + ) + if not _has_equality(declared): + # Passthrough names no EQL type, so it has no `name` attribute. + marker = declared.marker.__name__ + raise QueryError( + "query_unsupported", + f"field {field!r} is declared {marker}, which has no eq " + f"index, so it answers no equality query. {self._queryable()}", + field=field, + reason=marker, + ) + return _engine.query(self._plan, field, value) + + def _queryable(self) -> str: + names = [repr(f.name) for f in self._declaration.fields if _has_equality(f)] + if not names: + return "The class has no field with an equality query." + return "Fields with an equality query: " + ", ".join(names) + "." + @property def declaration(self) -> Declaration: """What the decorator read from the class.""" @@ -98,6 +152,14 @@ def __repr__(self) -> str: return f"Cipher({self._cls.__qualname__})" +def _has_equality(field: DeclaredField) -> bool: + """Whether ``query`` answers ``field``: the engine answers an equality + query through the ``eq`` index alone (``OpSpec::answered_by`` in + ``stack-encrypt``). A marker with only an order or match index has a + query type, but not for equality.""" + return field.marker is not Passthrough and "eq" in field.marker.indexes + + def _not_decorated(cls: object) -> DeclarationError: if not isinstance(cls, type): return DeclarationError( diff --git a/languages/python/python/stackencrypt/_engine.pyi b/languages/python/python/stackencrypt/_engine.pyi index ec521e9e8..4ad91366c 100644 --- a/languages/python/python/stackencrypt/_engine.pyi +++ b/languages/python/python/stackencrypt/_engine.pyi @@ -35,6 +35,11 @@ def decrypt(plan: Plan, cls: type, rows: Sequence[Any]) -> List[Any]: keyword arguments. Raises ``DecryptError`` or ``InvalidValueError`` before any request for what the rows themselves can answer.""" +def query(plan: Plan, field: str, value: Any) -> str: + """The equality query term for ``value`` on the sealed field ``field``. + Raises ``InvalidValueError`` before any request for a value that does + not fit the field.""" + def client_builds() -> int: """How many times the shared engine client has been built. For tests.""" diff --git a/languages/python/python/stackencrypt/_errors.py b/languages/python/python/stackencrypt/_errors.py index 82af23bbf..4837ac930 100644 --- a/languages/python/python/stackencrypt/_errors.py +++ b/languages/python/python/stackencrypt/_errors.py @@ -67,3 +67,14 @@ class DecryptError(StashError): stored as another type) and ``decrypt_not_representable`` (an opened value Python cannot hold). ``field`` names the column where it is known. """ + + +class QueryError(StashError): + """A ``query`` call the declared class cannot answer, raised before any + request. + + ``query_unknown_field``: the name is not a declared field. + ``query_unsupported``: the field's marker has no ``eq`` index, such as + ``Text``, ``Passthrough`` or ``IntegerOrd``. ``reason`` is the marker's + name. + """ diff --git a/languages/python/tests/test_encrypt_live.py b/languages/python/tests/test_encrypt_live.py index e309c038e..88ce3cecc 100644 --- a/languages/python/tests/test_encrypt_live.py +++ b/languages/python/tests/test_encrypt_live.py @@ -1,5 +1,5 @@ -"""``Cipher.encrypt`` and ``Cipher.decrypt`` against ZeroKMS and a PostgreSQL -with the EQL v4 bundle. +"""``Cipher.encrypt``, ``Cipher.decrypt`` and ``Cipher.query`` against ZeroKMS +and a PostgreSQL with the EQL v4 bundle. These tests need CipherStash credentials (``CS_CLIENT_ACCESS_KEY``, ``CS_WORKSPACE_CRN``, ``CS_CLIENT_ID``, ``CS_CLIENT_KEY``) and a database @@ -248,3 +248,64 @@ def test_1001_rows_open_with_three_key_requests(): # ceil(1001 / 500): the keys for the whole list, 500 per request. assert _engine.request_count() - before == 3 assert decrypted[1000].email == "user1000@example.com" + + +@needs_database +def test_a_query_term_finds_the_one_matching_row(): + psycopg = pytest.importorskip("psycopg") + from psycopg.rows import dict_row + + cipher = Cipher(User) + rows = cipher.encrypt( + [User(email="a@example.com", age=1), User(email="b@example.com", age=2)] + ) + with psycopg.connect(DATABASE_URL) as conn, conn.cursor(row_factory=dict_row) as cur: + cur.execute("CREATE TEMPORARY TABLE users (email public.eql_v4_text_eq, age int)") + for row in rows: + cur.execute("INSERT INTO users (email, age) VALUES (%s, %s)", (row["email"], row["age"])) + # The README's query: the term cast to the marker's query domain. + cur.execute( + f"SELECT * FROM users WHERE email = %s::{TextEq.query_sql_domain}", + (cipher.query("email", "a@example.com"),), + ) + found = cipher.decrypt(cur.fetchall()) + + assert found == [User(email="a@example.com", age=1)] + + +@needs_database +def test_a_query_uses_the_eq_term_index(): + psycopg = pytest.importorskip("psycopg") + + cipher = Cipher(User) + rows = cipher.encrypt([User(email=f"user{i}@example.com", age=i) for i in range(20)]) + with psycopg.connect(DATABASE_URL) as conn, conn.cursor() as cur: + cur.execute("CREATE TEMPORARY TABLE users (email public.eql_v4_text_eq, age int)") + # The README's index. + cur.execute("CREATE INDEX users_email_eq ON users USING hash (eql_v4.eq_term(email))") + for row in rows: + cur.execute("INSERT INTO users (email, age) VALUES (%s, %s)", (row["email"], row["age"])) + cur.execute("ANALYZE users") + # A table this small is cheaper to scan, so the planner is told not + # to. The test is that the index can answer the query at all. + cur.execute("SET LOCAL enable_seqscan = off") + cur.execute( + "EXPLAIN SELECT * FROM users WHERE email = %s::eql_v4.query_text_eq", + (cipher.query("email", "user7@example.com"),), + ) + plan = "\n".join(line for (line,) in cur.fetchall()) + + assert "users_email_eq" in plan, plan + + +def test_100_queries_make_no_request(): + cipher = Cipher(User) + # The first call in the process may build the client and load the + # keyset. Warm it up first, so the count below is the queries alone. + cipher.query("email", "warm@example.com") + before = _engine.request_count() + + terms = {cipher.query("email", f"user{i}@example.com") for i in range(100)} + + assert _engine.request_count() == before + assert len(terms) == 100 diff --git a/languages/python/tests/test_query.py b/languages/python/tests/test_query.py new file mode 100644 index 000000000..db9271efc --- /dev/null +++ b/languages/python/tests/test_query.py @@ -0,0 +1,140 @@ +"""``Cipher.query`` before any request: the field checks and the value checks. + +A name that is not a declared field, a field that answers no query, and a +value that does not fit are all refused before the shared client is built. +So these tests need fake credentials only, and make no request. The live +search, the index, and the request count are in ``test_encrypt_live.py``. +""" + +import dataclasses +from typing import Annotated + +import pytest + +import stackencrypt +from stackencrypt import ( + Cipher, + IntegerEq, + IntegerOrd, + InvalidValueError, + Passthrough, + QueryError, + StashError, + Text, + TextEq, + TextMatch, +) + +pytestmark = pytest.mark.usefixtures("credentials") + + +@stackencrypt.model("users") +@dataclasses.dataclass +class User: + email: Annotated[str, TextEq] + bio: Annotated[str, Text] + age: Annotated[int, Passthrough] + rank: Annotated[int, IntegerEq] + level: Annotated[int, IntegerOrd] + notes: Annotated[str, TextMatch] + + +def refused(field, value, error=QueryError): + with pytest.raises(error) as raised: + Cipher(User).query(field, value) + return raised.value + + +def test_a_query_error_is_a_stash_error(): + assert issubclass(QueryError, StashError) + + +def test_an_unknown_name_lists_the_fields_with_a_query(untouched): + error = refused("emial", "a@example.com") + assert error.code == "query_unknown_field" + assert error.field == "emial" + assert "'emial'" in error.message + assert "Fields with an equality query: 'email', 'rank'." in error.message + + +def test_a_field_that_is_not_a_str_is_an_unknown_name(untouched): + error = refused(None, "a@example.com") + assert error.code == "query_unknown_field" + assert error.field is None + + +def test_a_text_field_answers_no_query(untouched): + error = refused("bio", "hello") + assert error.code == "query_unsupported" + assert error.field == "bio" + assert error.reason == "Text" + + +def test_a_passthrough_field_answers_no_query(untouched): + error = refused("age", 3) + assert error.code == "query_unsupported" + assert error.field == "age" + assert error.reason == "Passthrough" + + +@pytest.mark.parametrize( + ("field", "value", "marker"), + [("level", 3, "IntegerOrd"), ("notes", "hello", "TextMatch")], +) +def test_a_field_with_a_query_type_but_no_eq_index_answers_no_query( + untouched, field, value, marker +): + # The marker has a query type, for order or match. `query` answers + # equality only, which the engine answers through the eq index alone. + error = refused(field, value) + assert error.code == "query_unsupported" + assert error.field == field + assert error.reason == marker + + +def test_a_class_with_no_queryable_field_says_so(untouched): + @stackencrypt.model("notes") + @dataclasses.dataclass + class Note: + body: Annotated[str, Text] + + with pytest.raises(QueryError) as raised: + Cipher(Note).query("body", "x") + assert "The class has no field with an equality query." in raised.value.message + + +def test_a_value_of_another_type_is_the_wrong_type(untouched): + error = refused("email", 3, InvalidValueError) + assert error.code == "value_wrong_type" + assert error.field == "email" + assert error.reason == "string" + + +def test_a_bool_on_an_integer_field_is_the_wrong_type(untouched): + error = refused("rank", True, InvalidValueError) + assert error.code == "value_wrong_type" + assert error.field == "rank" + + +def test_none_is_refused(untouched): + error = refused("email", None, InvalidValueError) + assert error.code == "value_none" + + +def test_a_value_out_of_range_is_refused(untouched): + error = refused("rank", 2**31, InvalidValueError) + assert error.code == "value_out_of_range" + assert error.reason == "int32" + + +def test_the_message_never_contains_the_value(untouched): + error = refused("email", 123456789, InvalidValueError) + assert "123456789" not in str(error) + + +def test_a_query_that_passes_every_check_reaches_the_engine(): + # Refused later by the engine for want of a client key, not by the + # checks: they passed. + with pytest.raises(StashError) as raised: + Cipher(User).query("email", "a@example.com") + assert not isinstance(raised.value, (QueryError, InvalidValueError))