diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md new file mode 100644 index 000000000..fe6f7cec7 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -0,0 +1,1504 @@ +# Stack Encrypt API plan + +Date: 7 November 2026. Checked against `next` at commit `e4990e49b`. + +Stack Encrypt encrypts each value under its own data key from ZeroKMS, +CipherStash's key service. It also makes the search terms that let +PostgreSQL query the encrypted value. The Rust crate `stack-encrypt` is the +engine. The Go SDK runs the same engine through the dynamic interface. + +This plan is for the engineers who build Stack Encrypt, and for the lead +who approves its design. It has three parts: + +1. [Target state](#target-state): the API we are building, shown as code. +2. [Remaining work](#remaining-work): what is left to build to reach it. +3. [Open questions and decisions to be + made](#open-questions-and-decisions-to-be-made). + +The history of each decision is in the [review +log](2026-10-24-stack-encrypt-api-review.md). + +## Terms + +| Term | Meaning | +| --- | --- | +| AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | +| Binding | A part of the context that ties a ciphertext to its row, such as a row ID. It is in the stored context and not in the query context. Passed as `Bind::new(..)` | +| Ciphertext | One encrypted value, as the Rust `Ciphertext` type. It is stored as bytes, or as Base64 text | +| Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Rust and Go run the same cases. The files are in `conformance/stack-encrypt/v1/` | +| Context name | The last declared scope of a field: its name by default, pinned when the Rust field is renamed. Two fields of one record never share one | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | +| Dynamic interface | The engine's interface for languages other than Rust. It takes a declaration and values as data, not as Rust types | +| EQL (Encrypt Query Language) | The PostgreSQL types and functions in `packages/eql` that store encrypted values and answer queries on them | +| Guest | The Rust engine compiled to WebAssembly. The Go SDK runs it inside the Go process | +| Keyset | A set of keys in ZeroKMS. Every data key belongs to one keyset. A tenant can have its own keyset | +| Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | +| Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | +| Query context | All the scopes of a context, declared and caller-supplied. Search terms derive under it. See [Query context and stored context](#query-context-and-stored-context) | +| Scope | A part of the context that narrows the data and the queries over it. The declaration supplies the first scopes, such as `users` and `email`; the caller adds more, such as a tenant ID, with `Scope::new(..)` | +| Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | +| Stored context | Every part of a value's context. The data key and the AAD derive under it | +| Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | +| Target | The encrypted type that a value is stored as, such as `Ciphertext`, an EQL type like `TextEq`, or a record type | + +## Target state + +This section shows what a developer writes when every item in this plan is +built. A comment marks each piece of code that does not compile on `next` +today. [Remaining work](#remaining-work) says what each of those needs. + +### Rust with static types + +These examples use the `stack-encrypt` crate directly, with a Rust type for +each encrypted value. + +#### Connect and choose a keyset + +A client reads its credentials from the environment, or from the profile +that `npx stash auth login` writes. Every encryption uses one keyset. A +program can use the client's default keyset, or select another by name or +ID. + +```rust +use stack_encrypt::{kms::IdentifiedBy, StackCipher}; + +let cipher = StackCipher::new().await?; +let keyset = cipher.default_keyset(); +// Not on `next` yet: the `IdentifiedBy::name` and `IdentifiedBy::uuid` +// constructors. `name` checks the name, so it returns a `Result`. +let tenant = cipher.keyset(IdentifiedBy::name("acme")?).await?; +let by_id = cipher.keyset(IdentifiedBy::uuid(keyset_id)).await?; +``` + +#### Encrypt and decrypt one value + +`encrypt` returns one `Ciphertext`. A context ties the ciphertext to where +it belongs, so it cannot be decrypted anywhere else. + +`encrypt` accepts an owned value or a borrowed one. It wipes an owned +value, such as a `String`. It only reads a borrowed value, such as a +`&str`, so the caller is responsible for wiping the original. + +```rust +use stack_encrypt::Ciphertext; + +// Not on `next` yet: borrowed values such as `&str`. +let secret: Ciphertext = keyset.encrypt("secret").await?; +let plaintext: String = keyset.decrypt(secret).await?; + +let number: String = read_card_number()?; +let card: Ciphertext = keyset + .encrypt_with_context(number, "users/42/card") // wipes `number` + .await?; + +// Store the ciphertext as bytes, or as Base64 text through Serde. +let bytes: Vec = card.to_bytes(); +let card = Ciphertext::from_bytes(&bytes)?; +let number: String = keyset.decrypt_with_context(card, "users/42/card").await?; +``` + +`encrypt` accepts only single values: text, numbers, booleans, dates, +times, and decimals. A struct or a `Vec` does not compile. Use a record, a +collection, or an opaque blob instead. + +#### Encrypt a record + +A record type says how each field of a plaintext struct is stored. The +derive seals each encrypted field under its own context, such as +`("users", "name")`. All the fields in a record share one batch of key +requests. + +```rust +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::target::{DecryptSource, EncryptTarget}; +use stack_encrypt::{Ciphertext, DecryptFrom, EncryptInto}; +use zeroize::Zeroize; + +#[derive(Zeroize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +// A searchable email: its ciphertext and an equality term. +#[derive(EncryptTarget, DecryptSource)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, +} + +// Not on `next` yet: the record derive and `passthrough`. +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &cipher).await?; +``` + +The derives need no Serde. A program that stores an encrypted value as +JSON adds `Serialize` and `Deserialize` to the encrypted type. That is +the same as for any other type. + +To tie every field to one row, the caller supplies the row ID as a part of +the context, and supplies it again to decrypt. The row ID is a binding +part. It is in the stored context, so the key and the AAD depend on it. It +is not in the query context, so a query still finds the row across the +column. + +[Query context and stored context](#query-context-and-stored-context) gives the rule. +The API for declaring a binding is an [open +question](#open-questions-and-decisions-to-be-made). + +```rust +// Not on `next` yet: `Bind`. Today the row ID is passed bare. +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, Bind::new(user_id)) + .await?; +let user = User::decrypt_from_with_context(encrypted, &keyset, Bind::new(user_id)).await?; +``` + +#### Store a field as an EQL type + +A field can use an EQL type from the `eql-bindings` crate as its target. +The field is then stored as one EQL column, such as +`public.eql_v4_text_eq`. + +Go records already do this through the dynamic interface. A field there +names its EQL type, such as `"target": "TextEq"`, and the engine runs that +type for the field. The Rust record derive does not support it yet. + +```rust +use eql_bindings::v4::{integer::IntegerOrd, text::TextEq}; + +#[derive(Zeroize)] +struct Member { + id: u64, + email: String, + age: i32, +} + +// Not on `next` yet in Rust: EQL types as fields of a derived record. +// Go records do this today through the dynamic interface. +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Member, context = "members")] +struct EncryptedMember { + #[stash(passthrough)] + id: u64, + email: TextEq, + age: IntegerOrd, +} +``` + +#### Query a field + +A query encrypts the search value into the query type of the field. It uses +the same keyset and the same context as the stored value. Each EQL type has +a query type, such as `TextEqQuery` for `TextEq` and `IntegerOrdQuery` for +`IntegerOrd`. The query value binds to SQL as a parameter, and the SQL +operator chooses the comparison. + +```rust +use eql_bindings::v4::{integer::IntegerOrdQuery, text::TextEqQuery}; +use eql_bindings::Identifier; + +let email = Identifier::for_column("members", "email")?; +let age = Identifier::for_column("members", "age")?; + +// Not on `next` yet: the new traits for EQL types, and a borrowed value +// and context. Today this needs `target::legacy::EncryptInto`, an owned +// value, and an owned context. +let by_email: TextEqQuery = "alice@example.com" + .encrypt_into_with_context(&keyset, Scope::new(&email)) + .await?; +let older: IntegerOrdQuery = 30_i32 + .encrypt_into_with_context(&keyset, Scope::new(&age)) + .await?; +``` + +```sql +SELECT id, email, age FROM members WHERE email = $1::eql_v4.query_text_eq; +SELECT id, email, age FROM members WHERE age > $1::eql_v4.query_integer_ord; +``` + +The context `Identifier::for_column("members", "email")` is the same as the +context that the record derive gives the `email` field of +`EncryptedMember`. So the query matches the stored values. + +A match query works in the same way, on a field with a match index. Its +answer can include rows that do not match. So the caller decrypts the rows, +and checks each one. + +#### Encrypt a JSON document + +A JSON field is stored as one encrypted document in a +`public.eql_v4_json_search` column. + +The simplest form encrypts straight into `JsonTreeDocument`, with the +default options. The context is the column. The default options order no +paths. They also store the whole document in its root entry, so it +decrypts in one step. + +```rust +use eql_bindings::{v4::json_tree::JsonTreeDocument, Identifier}; +use serde_json::json; +use stack_encrypt::json::JsonPlaintext; + +let column = Identifier::for_column("customers", "profile")?; + +// Not on `next` yet: the new traits for EQL types, and the public +// `JsonPlaintext`. Today this needs `target::legacy::EncryptInto`, a +// `serde_json::Value`, and an owned context. +let profile: JsonPlaintext = json!({ "name": "Ada", "age": 35 }).into(); +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; + +// On decrypt the column is checked against the one stored in the document. +let opened = JsonPlaintext::decrypt_from_with_context(encrypted, &keyset, Scope::new(&column)).await?; +let name = &opened.value()["name"]; // `opened` wipes the document when it is dropped +``` + +`JsonPlaintext` contains a `serde_json::Value`. It wipes the value when it +is dropped. It converts from a `serde_json::Value`, or parses JSON text with +the strict checks on JSON input. A program can also encrypt a borrowed +`&serde_json::Value`. Then the program is responsible for wiping it, as for +any [borrowed value](#encrypt-borrowed-values). + +With the default options, no path has an order term. So a query cannot +compare the order of the values at a path. To order some paths, declare a +named type with typed Rust options: + +```rust +use stack_encrypt::json::JsonOptions; + +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].at"]) +} + +// Not on `next` yet: a JSON target with typed options. +#[derive(EncryptTarget, DecryptSource)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("customers", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); + +let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?; +let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; +let opened = JsonPlaintext::decrypt_from(encrypted, &keyset).await?; +``` + +#### Query a JSON document + +A JSON query works like a [scalar query](#query-a-field). An operand type +encrypts into its EQL query type with `encrypt_into`, the same verb as +every other value. A query needs two things from the field's declaration: + +- **The column context.** Each path's selector is derived under it, so the + query must use the same context as the stored documents. +- **The options.** They say which paths have order terms, and how far a + wildcard such as `[*]` expands. An order query on a path without order + terms would find no rows, so Stack Encrypt refuses it. + +With the default options, the operand needs only the column. The caller +passes it as the context, as a scalar query does: + +```rust +use eql_bindings::v4::json_tree::{JsonTreeLeafQuery, JsonTreeQuery}; +use stack_encrypt::json::{AtPath, Contains}; + +// Not on `next` yet: the public JSON operand types. +let ada: JsonTreeQuery = Contains(json!({ "name": "Ada" })) + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; +``` + +With custom options, the operand names the target type of the field. The +target's declaration supplies the column and the options. An order query +on a path that the target does not order is refused. + +```rust +use eql_bindings::v4::json_tree::JsonTreeElementsQuery; +use stack_encrypt::json::ElementMatch; + +// Not on `next` yet: the public JSON operand types. +let ada: JsonTreeQuery = Contains::(json!({ "name": "Ada" })) + .encrypt_into(&keyset) + .await?; +let older: JsonTreeLeafQuery = AtPath::::gt("$.age", 30) + .encrypt_into(&keyset) + .await?; +let adults: JsonTreeLeafQuery = AtPath::::between("$.age", 18, 65) + .encrypt_into(&keyset) + .await?; +let home: JsonTreeElementsQuery = ElementMatch::::at("$.visits[*]") + .eq("place", "home") + .gt("at", 15) + .encrypt_into(&keyset) + .await?; +``` + +```sql +SELECT id FROM customers WHERE profile @> $1::eql_v4.query_json; + +-- $1 is the selector of `$.age`. $2 and $3 are the bounds. +SELECT id FROM customers +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text); + +SELECT id FROM customers +WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements); +``` + +The kinds of JSON query, and what each one finds: + +| Query | What it finds | EQL query type | +| --- | --- | --- | +| Containment | Rows whose document contains the given document, or is contained by it | `JsonTreeQuery` | +| Comparison at a path | Rows where the value at an ordered path equals, orders against, or lies between the given values | `JsonTreeLeafQuery` | +| Element match | Rows where one element of an array meets several conditions together | `JsonTreeElementsQuery` | +| Jsonpath predicate | Rows where a SQL/JSON path predicate is true | `JsonTreePredicateQuery` | +| Children and child count | The entries directly below one node, or how many there are | A node token | + +The operand types are named `Contains`, `AtPath`, and `ElementMatch`, +with `Predicate` for a jsonpath predicate and `Children` for the entries +below a node. The user agreed these names on 7 November 2026. + +The operand types are not new. The dynamic interface already has one +private input type for each operator, in `stack_encrypt::dynamic`: + +- `DocumentInput` for containment; +- `AtInput` for a comparison at a path; +- `ElementsInput` for element match. + +The dynamic parser reads the JSON object a binding sends into one of +them. The engine then derives the operand from it. The static form makes +those types public, with constructors. The parser reads into the same +types, so both forms share one set of input rules. + +The public forms do not exist on `next`. Today a Rust program queries a +JSON document [through a plan](#query-through-a-plan). A plan is part +of the dynamic interface. The dynamic interface is still the right tool for a +query known only at run time, such as a jsonpath predicate that a user +types. + +#### Encrypt a whole value as one ciphertext + +`OpaqueBlob` encodes any Serde value and seals it as one ciphertext. Go +writes the same format for its opaque structs. So each language can read +the blobs that the other one writes. + +```rust +use stack_encrypt::blob::OpaqueBlob; + +// Not on `next` yet: `OpaqueBlob`. +let sealed = keyset + .encrypt_with_context(OpaqueBlob::new(settings), ("users", "settings")) + .await?; +let settings: OpaqueBlob = keyset + .decrypt_with_context(sealed, ("users", "settings")) + .await?; +``` + +#### Encrypt a collection + +A `Vec` of values encrypts into a `Vec` of targets, with one batch of key +requests for the whole collection. Each element has its own data key, so +each one decrypts on its own. The order and the length of the `Vec` are +not authenticated, because a database can return rows in any order. + +```rust +let column: Vec = vec![1_i32, 3, 5].encrypt_into(&keyset).await?; +let values = Vec::::decrypt_from(column, &keyset).await?; + +let rows: Vec = users.encrypt_into(&keyset).await?; +``` + +That is the default, because it is the common case. A caller who needs +the order and the length authenticated asks for it by type. This is the +same way that `OpaqueBlob` changes how a value is sealed. The wrapper seals each +element under its position and the length of the `Vec`, as part of the +context. An element that is swapped, dropped, or added then fails to open. + +```rust +use stack_encrypt::Positioned; + +// Not on `next` yet: `Positioned`. The name is a proposal. +let steps: Positioned> = Positioned(vec![1_i32, 3, 5]) + .encrypt_into(&keyset) + .await?; +let values = Positioned::>::decrypt_from(steps, &keyset).await?; +``` + +`Positioned` is a new stored form, so it does not change the data that +Go rc.3 writes. ADR-0010 records that stricter authentication of a +container is opt-in, by type. + +#### Encrypt a byte string + +A `Vec` is the exception to the collection rule. It encrypts as one +value, into one `Ciphertext`, because a byte string is almost always one +thing: a file, a key, or an image. A fixed-size array `[u8; N]` is a byte +string too. `u8` on its own is not a value that encrypts, so a `Vec` +cannot encrypt byte by byte. + +A Go `[]byte` field takes the same path. The Go SDK sends the bytes to +the engine as one bytes value. The engine seals them with the same +Vitamin C code that seals a Rust `Vec`. The dynamic interface and the +Rust API share one engine, so they write the same bytes for the same +declaration (ADR-0007). + +The byte string is not a special case of either language. It is a rule of +the engine. + +A `Vec` of byte strings follows the collection rule. It encrypts into a +`Vec`, one for each byte string, with no wrapper type. + +```rust +let file: Ciphertext = std::fs::read("photo.jpg")?.encrypt_into(&keyset).await?; +let photo = Vec::::decrypt_from(file, &keyset).await?; + +let key: Ciphertext = [7u8; 16].encrypt_into(&keyset).await?; + +let files: Vec = vec![vec![1u8, 2, 3], vec![], vec![9]] + .encrypt_into(&keyset) + .await?; +let opened = Vec::>::decrypt_from(files, &keyset).await?; +``` + +To encrypt each byte of a `Vec` as its own ciphertext, name a type +for one byte. A `Vec` of that type is an ordinary collection, so it +encrypts into one `Ciphertext` for each element. The named type is a +single value because its borrow implements Vitamin C's `Scalar`. That is +the same rule every built-in single value follows. + +```rust +use stack_encrypt::Scalar; +use zeroize::Zeroize; + +/// One byte, sealed on its own. +// Not on `next` yet: `#[derive(Scalar)]`. Today `Byte` needs `Encrypt` +// and `Scalar` on `&Byte`, and `FromScalar` on `Byte`, by hand. +#[derive(Zeroize, Scalar)] +struct Byte(u8); + +let bytes: Vec = vec![0u8, 1, 255].into_iter().map(Byte).collect(); +let each: Vec = bytes.encrypt_into(&keyset).await?; +let opened = Vec::::decrypt_from(each, &keyset).await?; +``` + +`#[derive(Scalar)]` works on a struct with one field, whose type is a +built-in single value. The derive encrypts the struct as that field, and +decrypts the field back into the struct. So `Byte` is sealed exactly as +the `u8` inside it would be, if `u8` were a single value on its own. + +The user agreed this behaviour on 7 November 2026. It is what pull +request #1501 built. + +### The dynamic interface + +The dynamic interface takes a declaration and values as data, not as Rust +types. A program uses it in two cases: + +- when it learns the declaration only at run time, such as a query that + chooses its operator and path from user input; +- when it is written in another language, such as Go. + +The engine turns a declaration into the same plan that the Rust derive +builds. So the dynamic interface writes the same bytes as the static API +for the same declaration. + +#### A declaration as data + +A declaration names each field and its context. It also says how the field +is stored: as a passthrough field, with the indexes it lists, or as an EQL +type. + +```json +{ + "id": { "context": ["users", "id"], "outputs": ["passthrough"], "type": "int64" }, + "email": { "context": ["users", "email"], "target": "TextEq" }, + "age": { "context": ["users", "age"], "outputs": ["c", "eq", "ore"], "type": "int32" }, + "profile": { "context": ["users", "profile"], + "target": { "name": "JsonSearch", + "options": { "sealed_root": true, "ordered": ["$.age"] } } } +} +``` + +Each row crosses the boundary as one transport frame. A query names a field +and an operator, such as `eq` or `contains`, and gives a value. The engine +returns the query value, with its EQL type. + +The output keys and the type names are part of the stored format, so they +cannot change. The output keys are `c`, `eq`, `match`, `ore`, `ope`, and +`passthrough`. The type names include `int64` and `string`. + +#### Query through a plan + +On `next`, a Rust program queries a JSON document through a plan. The plan +is the declaration as data, so this is the dynamic interface with typed +operators. `stack_encrypt::op` names the operator. `on` names the plan, and the plan +supplies the column and the options: + +```rust +use stack_encrypt::{json::JsonPath, op, NonEmpty}; + +// Containment: documents that contain this document. +let ada = keyset + .query(op::Contains, &json!({ "name": "Ada" })) + .on(&plan) + .await?; + +// A comparison at a path. `$.age` must be an ordered path. +let older = keyset + .query(op::AtPath(op::Gt), &("$.age", 30)) + .on(&plan) + .await?; +let adults = keyset + .query(op::AtPath(op::Between), &("$.age", (18, 65))) + .on(&plan) + .await?; + +// The selector of a path, for `->` in SQL. +let age = keyset + .json_selector(&JsonPath::parse("$.age")?, NonEmpty::new(column.clone())?) + .await?; +``` + +The query values convert to the EQL query types: `JsonTreeQuery` for +containment, and `JsonTreeLeafQuery` for a comparison at a path. + +A chain call that needs a scope or a binding takes it through the paired +entry point, as the typed API does: + +```rust +// Not on `next` yet: the paired chain entry points and `Scope`. +let by_tenant = keyset + .query_with_context(op::Eq, "alice@example.com", Scope::new(tenant)) + .on(&plan) + .await?; +let user: User = keyset + .decrypt_with_context(encrypted, Scope::new(tenant).bind(row_id)) + .using(&plan) + .await?; +``` + +Three kinds of JSON query have no typed operator on `next`. They exist +only as data, through `stack_encrypt::dynamic::query_term`, with the +operator named as text: + +- element match: `"element_match"`; +- jsonpath predicates: `"match"`; +- children and child count: `"children"` and `"child_count"`. + +Go has generated methods for the first two, such as +`Fields.Profile.Elements.Visits.Match`. + +#### Query through a field marker + +The derive emits one marker for each field of a record, such as +`EncryptedMember::EMAIL`. A marker names the field, its plaintext type, +and its accessor. It is the value the plan builder takes to declare the +field, and the value a query takes to select it. + +The derive also emits `encryption_plan()`. It builds the record's `Plan` +through the plan builder, once, and keeps it. A query through a marker +reaches that plan itself. Decided on 7 November 2026. + +```rust +use stack_encrypt::op; + +// Not on `next` yet: field markers, `encryption_plan()`, and `using` for +// a query. Today a query names a plan written by hand: +// `cipher.query(op::Eq, value).on(&plan)`. +let by_email = keyset + .using(EncryptedMember::EMAIL) + .query(op::Eq, "alice@example.com") + .await?; +let adults = keyset + .using(EncryptedMember::AGE) + .query(op::Between, &(18, 65)) + .await?; +let ada = keyset + .using(EncryptedProfile::DOCUMENT) + .query(op::Contains, &json!({ "name": "Ada" })) + .await?; + +// The same plan, for the chain and the dynamic interface. +let plan: &Plan = EncryptedMember::encryption_plan(); +``` + +```sql +SELECT id, profile FROM customers WHERE profile @> $1::eql_v4.query_json; +``` + +Whether `using` comes before `query`, as above, or after it, is decided +with the query design. A program can also build a `Plan` by hand, from a +declaration it reads at run time, and select it with `using`. + +Below the definition, `stack_encrypt::dynamic::record` has the functions +that take a declaration as a frame: `plan_with`, `encrypt_with`, `query`, +and `decrypt_with`. Each takes a `TargetResolver`. The resolver runs the EQL +type that a field names. + +On `next`, each program writes its own resolver over +`eql-bindings`. In the target state, `eql-bindings` provides it. See [A +ready-made EQL resolver](#a-ready-made-eql-resolver). + +#### Call it from Go + +A Go program declares its records with struct tags. The `stashgen` +generator reads the tags, and writes the encrypted type, its functions, and +the declaration it sends to the engine. The examples show how a customer +uses the Go SDK today, on Go `v0.1.0-rc.3`. + +The `context=` tag gives the record's context. The tag on each field says +how the field is stored: as a passthrough field, as an EQL type, or with +the indexes it names. + +```go +//go:generate go tool stashgen -type User +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt_into=TextEq"` + Name string `stash:"name,encrypt_into=TextMatch"` + Age int32 `stash:"age,encrypt_into=Integer"` +} + +//go:generate go tool stashgen -type Customer +type Customer struct { + _ struct{} `stash:"context=customers"` + ID int64 `stash:"id,passthrough"` + Profile Profile `stash:"profile,encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)"` +} + +//go:generate go tool stashgen -type Settings +type Settings struct { + _ struct{} `stash:"context=settings,opaque"` // sealed as one blob + Theme string `json:"theme"` +} +``` + +A `Cipher` is the client bound to one keyset. `Scope` adds a scope to the +context of every field, such as a tenant ID. `Bind` adds a binding, such +as a row ID. The generated `Encrypt` and +`Decrypt` functions take a slice. They send one ZeroKMS request for each +500 values. + +```go +client, err := encrypt.NewClient(ctx) +defer client.Close() +// Not on rc.3 yet: `Scope` and `Bind`. Today `Scope` is `Extend`. +cipher := client.DefaultKeyset().Scope("tenant-42") + +encrypted, err := Encrypt(ctx, cipher, people) // []EncryptedUser +opened, err := Decrypt(ctx, cipher, encrypted) // []User + +term, err := Fields.Email.Eq(ctx, cipher, "bob@example.com") +// SELECT id, email, name, age FROM users WHERE email = $1::eql_v4.query_text_eq + +needle, err := Fields.Profile.Contains(ctx, cipher, map[string]any{"name": "Ada"}) +older, err := Fields.Profile.At.Age.Gt(ctx, cipher, 30) +visit, err := Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.JSONReadOptions{}, + encrypt.AtPath(Paths.Profile.Visits.Place).Eq("home"), + encrypt.AtPath(Paths.Profile.Visits.At).Gt(15), +) +``` + +A field has a query method only for the operators it supports, so an +unsupported query does not compile. The generator asks the engine which +operators each field supports. It does not keep its own copy of the rules. + +Two parts of the target state are not in Go yet: + +- A Go program cannot set match options. +- A Go program cannot open one child of a JSON node, bound to its parent. + +Both are in [Remaining work](#remaining-work). + +### Rules in the target state + +#### The stored format stays compatible with Go rc.3 + +The Go customer already uses Go `v0.1.0-rc.3`. So the data that rc.3 +writes must stay readable and searchable. The stored format is everything +that Stack Encrypt writes to the database: + +- ciphertexts, with their envelopes and headers; +- search terms; +- the entries of JSON documents; +- the declaration names that a stored row depends on. + +The baseline is the tag `languages/golang/v0.1.0-rc.3`, at commit +`114e08448`. It includes the `eql-v4-install.sql` bundle attached to that +release. + +The code and the APIs can still change in ways that break callers. A change +to the stored format must keep rc.3 data readable and searchable, or add a +new format version beside it. See [Changes that would alter the stored +format](#changes-that-would-alter-the-stored-format). + +#### Context and AAD + +The context is the data that a ciphertext is tied to, such as a column +name or a row ID. The field's declaration supplies part of the context, +such as `users/age`. The caller can supply the rest, such as a row ID. + +Stack Encrypt uses the context in three ways: + +1. **To derive the data key.** Stack Encrypt sends the context to ZeroKMS + as a text descriptor, such as `users/age`. ZeroKMS needs the same + descriptor to derive the key again. +2. **To build the AAD of each ciphertext.** The AAD is data that + encryption checks without encrypting it. Stack Encrypt builds it from + the context and the place of the value in its record. The AAD also + includes the format version, the keyset ID, and the key tag. +3. **To derive the search terms** for the field. + +A different context gives a different key and a different AAD, so +decryption fails. + +A parameter that takes the caller's context is named `context`. The name +`aad` is used only for a value that is an AAD. Some public methods do not +follow this rule yet. See [Name context parameters +`context`](#name-context-parameters-context). + +Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` +describes the descriptor and the AAD. The module documentation of +`packages/stack-encrypt/src/target/context.rs` describes the search terms. + +#### Query context and stored context + +Decided on 7 November 2026. Every part of a context is a scope or a +binding. The declaration supplies the first scopes: the record name, then +the field's context name. The caller supplies the rest. + +- **The query context** is all the scopes: the declared ones, then the + caller's. Search terms derive under it. A query supplies the caller's + scopes, and matches every row stored under the same query context. +- **The stored context** is the query context followed by the bindings. + The data key and the AAD derive under it. + +A field's **context name** is the last declared scope. It is the field's +name by default. It can be pinned when the Rust field is renamed, so that +stored data keeps opening. + +The context name is always the last declared scope, so two fields of one +record never share a context. If they did, they would share a key and an +AAD, and a ciphertext copied between the two columns would open. ADR-0006 +records this. + +A part of the context is one of three kinds. The declaration says which, +and fixes the order: the scopes first, then the bindings. + +| Kind of part | Verb | Example | In the query context | In the descriptor | In the AAD | Verified by | +| --- | --- | --- | --- | --- | --- | --- | +| Scope | `scope` | A record name, a field's context name, or a tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | +| Binding | `bind` | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | +| Post-seal part | None. The database adds it | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | + +The words were decided on 7 November 2026. A caller scopes a value with a +scope. A scope narrows the data and the queries over it. A caller binds a +value with a binding. A binding ties the ciphertext to its row and leaves +the queries as they are. + +"Extend" and "extension" are no longer used. Go `Extend` becomes `Scope`, with +`Extend` kept as a deprecated alias until Go 0.1.0. `Bind` is added to +Go. + +A part is a scope unless the declaration marks it as a binding, because +that is what Go rc.3 writes. A binding can be stored in the row as a context field, +so the reader takes it from the row. ZeroKMS logs the descriptor at every +key retrieval, so a binding such as the writer's IP address is in the +access log. The descriptor has a length limit. It is also sent to ZeroKMS, so the +part leaves the customer's system. + +ZeroKMS version 2 is planned to remove the limit and to encrypt the +access log. + +A post-seal part is added after sealing, by a database trigger that +refines the stored tag. ZeroKMS verifies the refined tag and returns the +original. The AAD binds the original. The design allows for this kind. The work +depends on ZeroKMS, and is the [refined tags](#changes-that-would-alter-the-stored-format) +item. + +The use case is access intelligence. Each value is bound to its row by a +client-generated UUID. A query by column or by tenant still finds it. + +#### A developer can see the context + +Decided on 7 November 2026. A wrong context is silent: a query matches +nothing, or a row opens somewhere it should not. So every declaration +and every call can show the context it uses, before any key is fetched. + +```rust +// Not on `next` yet: `contexts()` and `context()`. + +// For a declaration: each field, its query context, and its stored +// context, with a placeholder for every part the caller supplies. +println!("{}", EncryptedUser::encryption_plan().contexts()); +// email +// query context: users / email / {tenant: u64} +// stored context: users / email / {tenant: u64} / {row: Uuid} +// search terms: under the query context + +// For a call: the exact context, with the caller's values filled in. +let call = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(42_u64)); +println!("{}", call.context()); +// query context: users / email / 42u64 +``` + +The output shows each part's type, because `7u64`, `7i64`, the text `7`, +and the bytes `37` are four different contexts. The descriptor that +ZeroKMS logs is coarser, so it is not enough on its own. A call's context +can include the caller's values, such as a tenant ID or an IP address. +The plan form shows placeholders instead, so it is the one to put in +logs. + +The same output serves a unit test. A testing feature of the crate gives assertions over it. A developer can +pin the context of a field or a call, and the build fails when it +changes. The Go module has the same +assertions, in a testing package. + +```rust +// Not on `next` yet: the `testing` feature and `assert_context!`. +#[cfg(test)] +mod tests { + use stack_encrypt::testing::assert_context; + + #[test] + fn email_is_scoped_to_the_tenant_and_bound_to_the_row() { + assert_context!( + EncryptedUser::EMAIL, + query = ["users", "email", scope::()], + stored = ["users", "email", scope::(), bind::()], + ); + } +} +``` + +#### Other rules + +- **Rust and Go behave the same.** Every built-in capability has the same + meaning and the same stored bytes in Rust and in Go. The conformance + profiles test this. A custom Rust query stage can be Rust-only. It is + then marked as not portable. +- **Checks run before keys are used.** Every check that needs no key runs + before Stack Encrypt asks ZeroKMS for a key, or loads an index key. +- **Encryption wipes an owned plaintext.** `encrypt` and `encrypt_into` wipe an + owned plaintext, including when they fail. They only read a borrowed + plaintext, so the caller is responsible for wiping it. Stack Encrypt + wipes every copy it makes. A decrypted value is an ordinary owned value. + The caller is responsible for wiping it. +- **Context is passed by reference.** A method that takes a context + borrows it while the request runs. So a program can use one context for + a write, a query, and a read without cloning it. An owned context also + works. +- **Context is explicit.** A declaration lists the parts the caller must + supply, each with its kind and type. A part that is not declared is + refused. A declaration can mark a part as optional. That is the rare + case, because a query that omits an optional part matches nothing and + gives no error. Stack Encrypt does not use a type's `Default` + implementation to decide. Decided on 7 November 2026. +- **The dynamic grammar can declare required parts too,** with their kind + and type, so the Go generator can require them when it generates the + code. Its default stays "any part or none", because Go rc.3 callers use + that, and the stored bytes are the same either way. +- **Each element of a collection stands alone.** Each element has its own + data key. Each element is authenticated on its own. +- **Order terms do not answer equality.** An order-revealing encryption + (ORE) term does not answer an equality query. A field that needs both + declares both indexes. + +## Remaining work + +The work is in the order that the customer needs it. The Go customer work +comes first. The Rust API work follows, and the crate moves come after the +customer release. + +### 1. The Go customer release + +#### Finish the specification around the conformance profiles + +- **What to build.** JSON Schema files for declarations, requests, and + responses. Protocol version identifiers. A release compatibility matrix. + Tests that exchange new JSON values between languages. +- **Why.** The specification lets another language implement the dynamic + interface and get the same results. +- **Done already.** Conformance profiles exist for these areas (#1460, + #1461, #1462, #1469, #1491, #1492, #1493): + - the declaration grammar; + - runtime values and value kinds; + - stored reads and context; + - queries; + - EQL and JSON; + - the lifecycle of values and keys. + + The profiles are experimental. They do not certify a whole SDK. +- **Finished when** these are all true: + - the profiles are no longer experimental; + - the schemas exist; + - the version identifiers exist; + - the compatibility matrix lists rc.3. + +#### Open one child of a JSON node, bound to its parent + +- **What to build.** An operation that opens one child value of a JSON + node, and checks that the child belongs to that parent. This is J17 in + the [JSON operations](2026-10-25-stack-encrypt-json-operations.md) + document. J14 is the matching SQL. +- **Why.** Today a program must read and open the parent's whole subtree to + see one child. +- **Done already.** Nothing. +- **Finished when** Go can open one child, and the `eql-json-v1` + conformance profile covers it. + +#### Configurable match options in Go + +- **What to build.** Match options that a Go program can set: the + tokeniser, the filters, and the n-gram length. Issue #1112 tracks the + part in the generator. +- **Why.** Today Go uses only the default match options. +- **Done already.** Words of two letters can be searched (#1397). +- **Finished when** a Go field can declare match options, and its queries + use the same options as its stored terms. + +#### Freeze the release candidate + +- **What to build.** Fixed test vectors for the JSON storage format and the + matching EQL bundle, recorded as the rc.3 baseline. +- **Why.** The vectors make sure that later work keeps rc.3 data readable + and searchable. +- **Done already.** The conformance jobs run in CI. +- **Finished when** CI fails on any change to the stored bytes of rc.3 + data. + +#### Replace the temporary Vitamin C pins in Go + +- **What to build.** Replace the Go module's pseudo-versions of Vitamin C + with released module tags. +- **Why.** A final Go release cannot depend on unreleased commits. +- **Done already.** Prereleases can keep the pins, so rc.3 uses them. +- **Finished when** the release workflow accepts a final version. + +### 2. The stored format + +#### Changes that would alter the stored format + +Each change below would alter the stored format if it were built as first +planned. Each one must keep rc.3 data readable and searchable, or add a new +format version beside it. + +| Planned change | What it would alter | +| --- | --- | +| Refined tags | The tag stored in the envelope, and what the AAD binds | +| Search terms under the query context | A new form of search term, beside the rc.3 form, where terms derive under every part | +| Text encodings as field targets (Base64 and Base85) | How a ciphertext is written as text in a column | +| The `SealedValue` envelope and the JSON header version | The offsets of the key tag, and header version 1, which rc.3 writes | +| Configurable match options | The terms of columns written with the rc.3 defaults | + +#### Opaque blobs + +- **What to build.** `OpaqueBlob` in Rust, using Vitamin C's storage + profile. The storage profile is Vitamin C's versioned byte format for an + opaque value that is stored. +- **Why.** Go rc.3 already writes opaque structs in this format, so Rust + must read them. +- **Done already.** Go writes and reads the format (#1369, #1370). #1503 + adds `OpaqueBlob`, and is in review. +- **Finished when** Rust and Go each open blobs that the other wrote, and + the conformance profiles cover it. + +#### Authenticated metadata in an EQL payload + +- **What to build.** An optional field, `m`, in every EQL payload type, + for authenticated metadata. It contains named bindings, stored in the clear inside the payload, such as the address a value was written from + and when. The + Rust declaration marks a binding as stored `in = metadata`, and the + derive writes it into the field's payload. The dynamic grammar has the + same form, so Go can declare it. The EQL bundle accepts the field in + every domain and ignores it in every operator. +- **Why.** Access intelligence needs each value's writing facts bound to + it. As row columns, a reader would have to select every one of them to + decrypt the value. Inside the payload, one column stores the value and its bindings, so + `SELECT body` is enough. Scenario 9 of the [context + scenarios](2026-10-24-stack-encrypt-context-examples.md) shows it. +- **Without EQL.** The same shape works in a target type of our own, with + the bindings as passthrough fields of the target. Scenario 10 shows it. + That needs bindings declared on a target type, from the [record + derives](#record-derives) item, and no change to EQL. +- **Stored format.** The field is optional, so a payload without it is + unchanged. A payload with it is a new form beside the rc.3 form. +- **Done already.** Nothing. +- **Finished when** these are all true: + - scenario 9 encrypts, and is read back from one column; + - it fails to open when the metadata is changed; + - the EQL conformance profile covers the field in Rust and in Go. + +#### Typed text encodings and key tag extraction + +- **What to build.** + - `to_base64` and `to_base85` methods on `Ciphertext`. They return the + existing `Base64Text` and `Base85Text` types, not `String`. + - Field targets that store a ciphertext as text. Their working names are + `Base64Ciphertext` and `Base85Ciphertext`. Each one checks both the text + and the envelope. + - A SQL and PL/pgSQL function that reads the ZeroKMS key tag from a stored + ciphertext. It must check the version and the lengths before it reads. +- **Why.** Some columns store ciphertext as text. SQL needs to read the key + tag for refined tags. +- **Limits.** PostgreSQL can decode Base64, but not Base85. So SQL can read + the key tag from Base85 text only if we supply a decoder. Reading the tag + does not authenticate it. +- **Done already.** `Ciphertext` is stored as bytes, and Serde writes it as + Base64. `Ciphertext::key_tag` reads the tag in Rust (#1404). +- **Finished when** the methods and targets exist, and Rust, Go, and SQL + test vectors show the same key tag for the same stored value. + +### 3. The Rust API + +#### Record derives + +- **What to build.** The `EncryptTarget` and `DecryptSource` derives for + records, with passthrough fields, renamed fields, nested records, and + stored context. +- **Why.** The derives give the [record example](#encrypt-a-record). +- **Done already.** One value (#1404) and collections (#1501). #1504 adds + the record derives, and is in review. +- **Finished when** the public examples for mixed and searchable records + run as tests. + +#### EQL types as fields + +- **What to build.** EQL types such as `TextEq` and `JsonTreeDocument` as + field types of the Rust record derive. In Rust today, an EQL type works + only on its own, through the older structural API, with a + `NonEmpty` context. The EQL types do derive the structural + `EncryptFrom` and `DecryptInto`. A record still cannot use them as + fields. + A record with a `TextEq` field, built against `next` on 7 November 2026, + fails with three errors. `NonEmpty` and + `ExpectedContext` do not implement `From`, and + the record derive gives each field a `CallerContext`. +- **The smallest fix.** Convert a `CallerContext` of two parts into an + `Identifier`, with the table first and the column second. `Identifier` + already gives the same context as that pair, so the stored bytes do not + change. +- **Why.** This gives the [EQL field example](#store-a-field-as-an-eql-type) + and the [JSON example](#encrypt-a-json-document). +- **Done already.** The dynamic interface supports EQL fields, so Go + records use them today. A field there names its EQL type, and the engine + looks the type up by name through `eql-bindings` (the `TargetResolver` + in `dynamic/record.rs`). `Identifier` already gives the same context as + the `(table, column)` pair that a record derive builds. +- **Finished when** these are all true: + - a Rust record with EQL fields encrypts, decrypts, and queries through + the new API; + - it writes the same bytes as a Go record with the same declaration. + +#### A ready-made EQL resolver + +- **What to build.** A `TargetResolver` in `eql-bindings`, behind its + `targets` feature. It runs the EQL type that a field of a dynamic + declaration names, for encryption, decryption, and queries. +- **Why.** On `next`, every Rust program that uses + `stack_encrypt::dynamic::record` with EQL types writes its own resolver. + The EQL tests and the Go guest each have a copy. +- **Done already.** The functions it calls exist in `eql-bindings`: + `encrypt_with`, `decrypt`, and `query` in `v4::targets` and + `encryption::targets`. +- **Finished when** the tests and the Go guest use the resolver from + `eql-bindings`, and their own copies are deleted. + +#### Static JSON queries + +- **What to build.** Make the dynamic interface's input types public. + There is one for each kind of JSON query: containment, comparison at a + path, element match, jsonpath predicate, and reads of children. Give + each one constructors, and an `EncryptTarget` implementation that + encrypts it into its EQL query type. The target type of the field + supplies the column and the options. The dynamic parser keeps reading + into the same types, so there is one definition of each input. +- **Names.** `Contains`, `AtPath`, `ElementMatch`, `Predicate`, and + `Children`, in `stack_encrypt::json`. Agreed on 7 November 2026. +- **Why.** On `next`, a Rust program needs a plan for any JSON query, and + needs the dynamic interface for element match, predicates, and children. + Go already has typed methods for these. +- **Done already.** The input types and their parsing rules, as private + types in `stack_encrypt::dynamic::json_query`. Typed containment and + comparison at a path, through a plan (`op::Contains`, `op::AtPath`). The + EQL query types exist in `eql-bindings`. +- **Finished when** the [static JSON query + example](#query-a-json-document) compiles, and an order + query on a path the target does not order is refused. + +#### Field markers and the encryption plan + +Decided on 7 November 2026, as difference six of the context artifact +review. + +- **What to build.** The derive emits one marker for each field, as an + associated constant such as `EncryptedMember::EMAIL`, with the field's + name, its plaintext type, and its accessor. The marker is the `FieldRef` + the plan builder takes, and the handle a query takes. The derive also + emits `encryption_plan()`. It returns the record's `Plan`, built once + through the plan builder with the markers as its arguments. `using` + selects a plan or a marker, for encryption and for a query; `on` goes. + Typed Rust option factories, and the same options read from data in the + dynamic interface. +- **Why.** A query must use the same declaration as the stored values. The + caller must not write a second plan by hand. One marker type serves the + builder and the query, so the derive lowers to the builder with no + second mechanism. +- **Done already.** `FieldRef` and `pick` exist in `plan/field_ref.rs`. + The derive already lowers a structural record to the builder. Queries + work through a plan written by hand: `cipher.query(op, value).on(&plan)`. +- **Cannot fail.** `build()` refuses a repeated field name or a shared context name. + The derive sees every field, so it refuses those at compile time, and + `encryption_plan()` returns `&'static Plan`, not a `Result`. The + hand-written builder keeps the runtime check. The dynamic grammar needs + it. +- **Later.** A compile-time duplicate-field check for the hand-written + builder, through a type-level list of markers, stays a follow-up, as the + artifact says. +- **Finished when** the [marker example](#query-through-a-field-marker) + compiles, and tests show that a query cannot run without its plan or its + required context. + +#### Context from fields and from the caller + +- **What to build.** One mechanism for where a field's context comes + from: the plan builder. The derive and the dynamic grammar lower to it. + A field's context comes from one of three sources: + - the declaration, through `Plan::context(record_name)`, the only way + to put the record name in a plan, with each field's context name + after it; + - a row field stored unsealed in the record, so the reader takes it + from the row. A reader may pass it on decrypt to check it; + - the caller, on every write, query, and read, as a scope or a + binding. + + Parts are separate and ordered: the declared scopes, the caller's + scopes, then the bindings. Grouping is available inside one part. Nothing + is read from the plaintext on one side and from the caller on the + other. Decided on 7 November 2026, with the context artifact. +- **Why.** A program must never need to open a ciphertext to learn the + context that opens it. One mechanism keeps Rust and Go writing the same + bytes. +- **Scenarios.** The [context + examples](2026-10-24-stack-encrypt-context-examples.md) are the + acceptance matrix: ten scenarios, from a static context alone to the + bindings that access intelligence stores. Two earlier scenarios were removed, because + each sealed a value that was also another field's context, so nothing + could open without the plaintext. +- **Done already.** The order in which parts are joined is fixed and + tested in Rust and Go (#1491). The context inputs `RequiredContext`, + `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds + `ContextExtension` and `ExpectedContext`, and is in review. + `ContextExtension` is renamed for the scope vocabulary; its new + name is decided with the field reference design. The plan builder's + `.context(..)` and `.context_field(..)` and the derive's + `#[stash(context_field)]` are the forms this replaces. +- **Required parts.** A Rust declaration lists the parts the caller must + supply, with kind and type, as the artifact's `refined_by::()` does, + extended to bindings. An undeclared part is refused. Optional is + the marked case. The dynamic grammar can declare required parts; its + default stays "any", and the Go generator requires a declared part at + compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` + (#1404) are the three states this keeps. +- **Finished when** these are both true: + - all ten scenarios run as tests in Rust and Go, through the + builder; + - a Rust call with an undeclared part fails to compile. + +#### Query context and stored context + +- **What to build.** The binding kind of part, in the declaration and in + the dynamic grammar. Search terms derive under the query context, and the + key and the AAD under the stored context. A binding can be a context + field stored in the row. The order is fixed: the scopes, then the + bindings. +- **Why.** Access intelligence needs each value bound to its row, and found + by a column or tenant query. [Two scopes of one + context](#query-context-and-stored-context) has the rule. +- **Vocabulary.** The Rust chain's `.extend(..)` is removed; the context + argument carries the scopes and bindings instead. Go `Extend` becomes + `Scope`, with `Extend` as a deprecated alias until 0.1.0. `Bind` is added + to Go. The context contract and the conformance README say "scope" + instead of "extension". The frozen vectors do not change, because no key + on the wire names the operation. +- **Done already.** Nothing. Today every part is in both scopes. +- **Finished when** these are all true, in Rust and in Go: + - a record with a binding row ID encrypts; + - a column query finds it; + - it fails to open under another row ID; + - the stored bytes of a record with no binding are unchanged. + +#### One context argument, and paired entry points + +Decided on 7 November 2026, as differences two and five of the context +artifact review. + +- **What to build.** The context argument is one of three values, built + before the call: `Scope::new(..)`, `Bind::new(..)`, or + `Scope::new(..).bind(..)`. Several of one kind chain: + `Scope::new(tenant).scope(region).bind(id)`. The order is fixed by type: + after the first `bind`, `scope` is not offered. `Scope::new((a, b))` is + one grouped part. That is a different context from two parts. + Both layers take it through paired entry points: + - the typed API: `encrypt` with `encrypt_with_context`; `encrypt_into` + with `encrypt_into_with_context`; `decrypt_from` with + `decrypt_from_with_context`; + - the chain: `encrypt` with `encrypt_with_context`; `query` with + `query_with_context`; `decrypt` with `decrypt_with_context`. The + chain's `open` is renamed `decrypt`. No chain state offers + `.context(..)`, `.extend(..)`, `.scope(..)`, `.bind(..)`, or + `.keyset(..)`. The keyset is chosen before the chain starts. + + A query entry point accepts a `Scope` and nothing else, so a binding in + a query does not compile. A part whose source is a row field is read + from the plaintext on encrypt. On decrypt, a stored part may be passed, + and is then checked against the row before any key is fetched. That + check is what `ExpectedContext` does today; it stops being a public + type. +- **Why.** The typed API already pairs its entry points (#1404). A + value for the context reads the same on both layers. A builder + state for the context, and a part added later in a chain, are the + mistakes the artifact describes. +- **Done already.** The scalar pairs (#1404). The record pairs, in + #1504. The chain's `encrypt`, `query`, and `open`, with `.extend(..)` + and `.context(..)` on several states. +- **Order of calls.** The keyset is chosen first. It is the receiver of + every entry point: `keyset.encrypt(..)`, `keyset.using(..)`. Whether + `using` comes before or after `query` is decided with the query design. + Difference seven of the context artifact review, 7 November 2026. +- **Rename.** `Scope<'a, K>` in `dynamic::record` and `CipherScope` in the + owned API both mean the cipher a call runs under. Both are renamed in + the naming review, so that `Scope` means a part of the context. +- **Finished when** these are all true: + - the examples in [Target state](#target-state) compile; + - the compile-fail cases in `tests/ui` cover a binding in a query, and + a scope after a bind; + - the chain has no `.context`, `.extend`, or `.keyset` state. + +#### Show the context + +Decided on 7 November 2026. + +- **What to build.** Two ways to see a context before any key is fetched: + - `contexts()` on a `Plan`, listing each field's query context and + stored context, with a typed placeholder for every caller-supplied + part. The derive's `encryption_plan()` gives it to every record, and + a hand-built plan has it too. + - `context()` on a prepared call. It gives the exact query context and + stored context the call will use, with the caller's values filled in. + Each part shows its type. + + Both render through the same code that derives the key descriptor and + the AAD, so what a developer sees is what the engine uses. The dynamic + interface has the same two forms. So `stashgen` can print a Go record's + contexts. A Go call can show its context. +- **Assertions for tests.** A `testing` feature of `stack-encrypt`, with + `assert_context!` over a field marker, a plan, or a prepared call. It + compares the query context and the stored context, with types, against + what the test states, and names the first part that differs. The Go + module has the same in a testing package, such as `encrypttest`, so a + Go test can pin a record's contexts the same way. Both read the same + output as `contexts()` and `context()`, so a test and a log never + disagree. +- **Why.** A wrong context loses data silently. A query under the wrong + context matches nothing and gives no error. A developer who cannot see + the context cannot find the mistake. A test that cannot state the + context cannot catch a change to it. +- **Done already.** `Descriptor` renders a context as text, and + `ContextMismatch` reports the stored descriptor. Neither shows the + types, and neither is reachable from a plan or a call. +- **Finished when** these are both true: + - a test pins the output of `contexts()` for each of the [context + scenarios](2026-10-24-stack-encrypt-context-examples.md); + - a conformance case shows that Rust and Go print the same context for + the same declaration; + - the [context scenarios](2026-10-24-stack-encrypt-context-examples.md) + each carry an `assert_context!` test, in Rust and in Go. + +#### Name context parameters `context` + +These public methods in `packages/stack-encrypt/src/cipher.rs` take a +context through a parameter named `aad`: + +- `KeysetCipher::encrypt_tree`; +- `KeysetCipher::decrypt_tree`; +- `StackCipher::decrypt`; +- `PendingStackCipherText::plan`, whose bound is Vitamin C's `IntoAad`; +- `PendingStackCipherText::seal`; +- `PendingStackCipherText::into_pending`. + +Rename each parameter to `context`. This breaks the Rust API only. It does +not change the stored format. Renaming Vitamin C's `IntoAad` trait is a +follow-up in Vitamin C. The Go SDK and the dynamic interface have no public +parameter named `aad`. + +#### Pass context by reference + +- **What to build.** Let every method that takes a context accept a + borrowed context, such as `&Identifier`. The request borrows the context + for as long as it runs. `eql-bindings` needs `IntoContext` for + `&Identifier`. +- **Why.** On `next`, every context is taken by value. A program that uses + one column context for a write and a query must clone it for each call. +- **Done already.** Stack Encrypt implements `IntoContext` for `&Label`. + The context inputs, such as `RequiredContext`, still need an owned value. +- **Finished when** the public examples pass a context by reference, with + no clone, and compile. + +#### Encrypt borrowed values + +- **What to build.** Let `encrypt` and `encrypt_with_context` accept a + borrowed single value, such as `&str`, `&[u8]`, or `&i64`. Keep the + current behaviour for an owned value. +- **Why.** A program that encrypts a string literal must write + `"secret".to_owned()` on `next`. The copy adds nothing, because a literal + cannot be wiped. +- **How.** The engine already reads an owned value through a borrow + (`ScalarRef` in `scalar.rs`). But `OpaqueEncode` requires `Zeroize` and + `'static`, so `&str` does not meet it. A borrowed value needs its own + bound, and the request borrows the value for as long as it runs. +- **Done already.** Owned single values (#1404). +- **Finished when** `keyset.encrypt("secret")` compiles, and a test shows + that Stack Encrypt wipes its own copies of a borrowed value. + +#### A public `JsonPlaintext` + +- **What to build.** A public `stack_encrypt::json::JsonPlaintext`, with: + - `From`; + - `parse(&str)`, with the strict checks on JSON input; + - `value()`, to read the document by reference. + + Encryption into a JSON target accepts a `JsonPlaintext`, or a borrowed + `&serde_json::Value`. Decryption returns a `JsonPlaintext`. +- **Why.** The new traits need a plaintext that can be wiped. + `serde_json::Value` does not implement `Zeroize`. Rust's orphan rule + stops this repository adding that, so a wrapper is needed. A program + that already has a `serde_json::Value` must not have to print it and + parse it again. +- **Done already.** `stack_encrypt::dynamic::json::JsonPlaintext` wraps a + `serde_json::Value`, and wipes it when it is dropped. It is part of the + dynamic interface. It has only `value()`. The strict parser exists as + `stack_encrypt::json::parse_text`. +- **Finished when** the JSON examples in this plan compile, and a test + shows that a dropped `JsonPlaintext` leaves no plaintext behind. + +#### Constructors for `IdentifiedBy` + +- **What to build.** Two constructors that select a keyset: + - `IdentifiedBy::name(&str)`. It checks the name, so it returns a + `Result`; + - `IdentifiedBy::uuid(Uuid)`. +- **Why.** On `next`, selecting a keyset by name needs + `IdentifiedBy::Name("acme".to_string().into())`. The `.into()` converts + the `String` into the `Name` type. That conversion does not check the + name. Only `Name::try_from(&str)` checks it: at most 64 characters, not + empty, and only `A-Z a-z 0-9 _ - /`. +- **Where.** `IdentifiedBy` and `Name` are in the `zerokms-protocol` crate. + The `cipherstash-suite` repository publishes that crate. Rust allows inherent methods only in + the crate that defines the type. So the constructors go in + `zerokms-protocol`. Stack Encrypt re-exports `IdentifiedBy` from + `stack_encrypt::kms`, so callers get them through that path. +- **Done already.** `StackCipher::keyset` takes `impl Into`, + and `IdentifiedBy` implements `From` and `From`. +- **Finished when** a released `zerokms-protocol` has both constructors, + and the examples in this plan compile. + +#### Authenticate the order and length of a collection + +- **What to build.** A wrapper type, `Positioned>`, that seals each + element under its position and the length of the collection. Opening + checks both, so a swapped, dropped, or added element fails. The dynamic + interface needs the same form, so that Go can declare it. +- **Why.** The default collection does not authenticate order or length, + because a database returns rows in any order. Some callers store a + sequence whose order matters, such as the steps of a workflow. ADR-0010 + says this is opt-in, by type. It does not exist yet. +- **Done already.** Nothing. The default collection (#1501). +- **Finished when** a test shows that each of swapping, dropping, and + adding an element fails to open, in Rust and in Go. + +#### A derive for a single-value type + +- **What to build.** `#[derive(Scalar)]`, for a struct with one field + whose type is a built-in single value, such as `struct Byte(u8)` or + `struct Token(String)`. It gives the type what `OpaqueEncode` and + `OpaqueDecode` need: `Encrypt` and `Scalar` on its borrow, and + `FromScalar` on the type. Each one forwards to the field. +- **Why.** A newtype is how a caller changes the rule for a value, such as + encrypting a `Vec` byte by byte. On `next`, the three + implementations are written by hand. `FromScalar` is not documented for + that use. +- **Done already.** `OpaqueEncode` and `OpaqueDecode` already accept any + type that meets those bounds. Vitamin C documents a hand-written + `Blob(Vec)` newtype. +- **Finished when** the [byte example](#encrypt-a-byte-string) compiles + with the derive and no hand-written implementation. + +#### Remove the structural API + +- **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and + the helpers in `target::legacy`. Move EQL, Go, and the guest to the new + API in the same change. Rename the plan builder's output method to + `target`. +- **Why.** Two APIs for the same job make the right one hard to find. +- **Done already.** The new API exists beside the old one. Issue #1488 + tracks the removal. +- **Finished when** the old API is gone, and every consumer uses the new + API. + +#### Wipe plaintext in the remaining cases + +- **What to build.** Wipe the parts of a Serde value that were decoded + before a later part failed. Add an adapter that wipes a decrypted value + when it is dropped. +- **Why.** Plaintext must not stay in freed memory. +- **Done already.** Collections wipe the values decoded before a failure. + Cancellation and drop are tested (#1493, #1501). +- **Finished when** the lifecycle conformance profile covers both cases. + +### 4. After the customer release + +- **Packaging and names.** Crate moves, the shared format crate, and the + final names. The [packaging + document](2026-10-25-stack-encrypt-packaging.md) has the proposal. +- **Key providers.** Choosing keysets from data, and ZeroKMS lock context, + as optional capabilities of a key provider. The [provider capabilities + document](2026-10-25-stack-encrypt-provider-capabilities.md) has the + proposal. + +## Open questions and decisions to be made + +The lead who approves this plan decides each question. + +| Question | Options | Recommendation | +| --- | --- | --- | +| What is the name of the wrapper that authenticates a collection's order and length? | `Positioned>`. Or a name chosen in the naming review | `Positioned`. Decide in the naming review | +| For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | +| Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | +| How does a declaration mark a binding in the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences three to seven of the context artifact | +| Should keysets be chosen from data, and should Stack Encrypt use ZeroKMS lock context? | Optional capabilities of a key provider. Or leave them out | Optional capabilities, after the customer release | +| What privacy claim can be made for match terms of short words? | Accept what they reveal, and document it. Or change how short words are stored | No recommendation yet. Merging #1397 did not show that they reveal nothing new | +| Which crate should own `Base64Text` and `Base85Text`? | A small Vitamin C encoding crate. Or keep them in `eql-bindings` | A small Vitamin C crate, re-exported by Stack Encrypt and EQL | +| What names and crate layout should the final API use? | The proposal in the packaging document | Review the proposal after the customer release | diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md new file mode 100644 index 000000000..f01f34f7a --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -0,0 +1,1751 @@ +# Stack Encrypt API review + +Started on 24 October 2026. This review covers the current Rust +`stack-encrypt` API. It started with one task: encrypt one `i32` into +opaque ciphertext and store the result. + +This document is a running log of the issues found. Each numbered section +records one issue: what the code does now, what changes were proposed, and +which decisions are made or still open. The proposed APIs are illustrations, +not implemented code, unless a status line says otherwise. Add a numbered +section for each new issue, and keep observations, proposed changes, and +open decisions apart. + +On 7 November 2026, this log was checked against `next` at commit +`e4990e49b`. Some issues are now partly done. The owned API for one value and for collections is merged. A conformance corpus for Rust and Go is merged too. No issue is completely done. + +The [summary of +status](#summary-of-status) gives the status of each issue and the pull +requests behind it. "Current behaviour" sections describe the code when +each issue was written. Where a code block in one of them is now wrong, a +note follows the block. + +Three companion documents go with this log: + +- [Complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md) + is a draft API design that covers encryption, storage, decryption, and + queries. +- [Directional context examples](2026-10-24-stack-encrypt-context-examples.md) + explores where context comes from and where it is stored, using attribute + syntax modelled on Serde. +- [The implementation plan](2026-10-24-stack-encrypt-api-plan.md) records the + requirements, the recommended decisions, the order of work, and when the + work is complete. + +## The stored format must stay compatible from 7 November 2026 + +On 7 November 2026, the user decided: "The stored data must not break as of +now. The Go customer has already started implementation with rc.3. We can +make breaking changes to the code/API but not the stored format." + +The stored format is what is written to the database. It includes +ciphertexts with their envelopes and headers. It includes index and search +terms. It includes JSON tree entries. + +The baseline is what the Go module at +tag `languages/golang/v0.1.0-rc.3` writes, with the EQL v4 SQL bundle +attached to that release. The tag is commit `114e08448`, built on 2 +November 2026 from `next` at `929451b49`. + +From now on, a change that would alter the stored format must keep data +written by rc.3 readable and searchable. Otherwise it must add a new, +versioned format beside the rc.3 format. The Rust and Go code and APIs can +still change in breaking ways. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#changes-that-would-alter-the-stored-format) +lists the planned work that this decision constrains. + +### History: no backward compatibility with the prototype + +On 25 October 2026 the user clarified that this API has never shipped beyond +an early prototype. Backward compatibility, and migration of prototype data, +were not requirements. That replaced any earlier suggestion in this log to +keep old encodings or public imports. The new Rust, Go, guest, and EQL +(Encrypt Query Language) implementations must agree with each other. + +The decision of 7 November 2026 replaces this one for stored data. The +earlier decision still applies to the code and the APIs. Issue 22 had said +the exemption would end when the customer release shipped. It now ends for +stored data before that release. + +## Terms used in this log + +These terms appear in many issues. Terms used in only one issue are defined +where they first appear. + +| Term | Meaning | +| --- | --- | +| Vitamin C | CipherStash's Rust cryptography library, published as the `vitaminc-*` crates. Stack Encrypt used version 0.8.0 when this log started. #1501 moved it to 0.9 | +| ZeroKMS | CipherStash's key management service, which issues and retrieves data keys | +| KMS | Key management service | +| EQL | Encrypt Query Language, the SQL and storage format for searchable encrypted data in PostgreSQL | +| AEAD | Authenticated encryption with associated data | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| Leaf | One sealed value, `SealedValue`, in Vitamin C's ciphertext tree | +| Node | One element of Vitamin C's ciphertext tree, `Node`: a single leaf, a sequence, a map, an empty container, or a passthrough value | +| Passthrough | A field that is copied into the output in plaintext, without encryption | +| Target | An output type that declares how a plaintext is encrypted into it, through `EncryptFrom`, or `EncryptTarget` in the owned API | +| Owned API | The Rust API added by #1404 that consumes its plaintext input: `keyset.encrypt`, `EncryptInto`, and `DecryptFrom`, over `EncryptTarget` and `DecryptSource` | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](#the-stored-format-must-stay-compatible-from-7-november-2026) | +| Plan | The lower-level builder, `Plan`, that composes encryption operations. The derives generate plans, and the dynamic interface uses them | +| Dynamic interface | The interface driven by data instead of Rust types, used through FFI (foreign function interface) and by the Go SDK | +| Guest | The WebAssembly build of the Rust engine that the Go SDK runs | +| ORE | Order-revealing encryption, the index used for range queries and sorting | + +## Summary of status + +| Issue | Status on 7 November 2026 | +| --- | --- | +| 1. Cipher trait mismatch | Partly done. #1404 adds `encrypt`, `encrypt_with_context`, `decrypt`, and `decrypt_with_context` on the keyset. How Stack's ciphers relate to Vitamin C's traits is still open | +| 2. Opaque ciphertext storage | Partly done. #1404 adds `Ciphertext`; text encodings and `OpaqueBlob` (#1503, in progress) are still open | +| 3. Plaintext-side encryption | Partly done. #1404 implements the keyset receiver, and an empty context for `Ciphertext` through both APIs. The structural API keeps its different rules | +| 4. Fixed context on the scalar derive | Still open. #1491 makes the composition rules in this issue true for repeated extensions | +| 5. `EncryptFrom` has no `encrypt_from` | Partly done. #1404 adds `EncryptTarget` and `DecryptSource`; the structural traits remain | +| 6. Plan builder `encrypt_into` | Still open | +| 7. Mixed encrypted and plaintext fields | Still open. In progress in #1504 | +| 8. Whole-value encryption and layouts | Still open. In progress in #1503 and #1504 | +| 9. Ownership across Stack and Vitamin C | Still open. #1501 moves the Rust crates to Vitamin C 0.9 | +| 10. Public `encrypt_as` | Still open. `encrypt_as` is still public | +| 11. Context from an input struct | Still open | +| 12. `Default` compared with optional context | Partly done. #1404 adds explicit context inputs for the owned API. The structural API still uses `Default` | +| 13. JSON layouts and output adaptation | Still open | +| 14. Decrypt context variants | Partly done. #1404 implements both pairs for one value, and #1501 for collections. Records are in progress in #1504 | +| 15. Go JSON work in pull request #1353 | Coordination item. Unchanged | +| 16. Prior query design | Adopted as the baseline. Unchanged | +| 17. One name for definition selection | Still open | +| 18. Consuming inputs with cleanup | Partly done (#1404, #1493, #1501). Blob cleanup, partial Serde decoding, and drop-guarded results are still open | +| 19. First-release completeness | Partly done (#1445, #1446, #1451, #1452, #1492). Opening a child bound to its parent is still open | +| 20. Keysets chosen from plaintext | Still open | +| 21. Packaging and `AsyncCipher` | Still open | +| 22. Customer-driven order of work | Partly done. The conformance corpus is merged. Merged work was reconciled on 7 November 2026 | + +## 1. Cipher trait mismatch and the required context argument + +**Status:** Partly done. #1404 adds `encrypt` and `encrypt_with_context`, and +the matching `decrypt` pair, on the keyset. The open questions about +Vitamin C's cipher traits remain. + +The user asked for two separate methods, so that the common case needs no +empty context argument. Questions remain about how Stack's ciphers relate +to Vitamin C's cipher traits. + +### Decision of 7 November 2026: name context parameters `context` + +A parameter that takes the caller's context is named `context`. The name `aad` is kept only for a value that is an AAD. + +Several public methods, such as `encrypt_tree`, still name the context `aad`. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#name-context-parameters-context) +lists them as work to do. The rename breaks the Rust API only. It does not +change the stored format. + +### Requested API direction + +Passing an empty tuple, `()`, for the common case of no application context is +needless ceremony. Use separate methods, following the convention used +elsewhere: + +```rust +keyset.encrypt(42_i32).await?; +keyset.encrypt_with_context(42_i32, "users/age").await?; +``` + +The method without context selects the same empty context that `()` +represents today. Make this split even if `KeysetCipher` keeps inherent +methods instead of implementing a Vitamin C trait. + +### Current behaviour + +`KeysetCipher::encrypt` is an inherent method, not a trait method: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let ciphertext = keyset.encrypt(42_i32, "users/age").await?; +``` + +Note added on 7 November 2026: this block is no longer current. #1404 +renamed the two-argument method to `encrypt_tree`. `keyset.encrypt(value)` +now takes one argument and returns a `Ciphertext`. The second argument of +`encrypt_tree` is named `aad` in the code, but it takes a context. + +Vitamin C (`vitaminc-aead` 0.8.0) provides the `Cipher` trait and the +`AsyncCipher` trait. Stack implements the synchronous `Cipher` trait on +internal adapters after it fetches keys. `MintedKeys` seals leaves, and +`BoundKeys` opens them. Neither public cipher implements `AsyncCipher`. +Callers therefore cannot pass `KeysetCipher` to generic Vitamin C code that +needs that trait. + +Each internal adapter implements only half of `Cipher`. `MintedKeys::open` +and `BoundKeys::seal` always return errors. + +### Why the mismatch exists + +Batching alone does not explain the separate API, because `AsyncCipher` +already batches through `seal_many` and `open_many`. The real reasons are +these: + +- `AsyncCipher`'s methods receive each leaf's derived AAD. ZeroKMS needs a + descriptor of the original structured context, and the derived leaf AAD + does not contain that context. +- Stack's layer for pending operations combines key requests across + documents, fields, and rows. The trait's batch interface works on one + document at a time. + +Any trait integration has to keep meeting these constraints. The current +separation is deliberate. It still leaves a gap for code that is generic +over Vitamin C's traits, and internal implementations with operations that +always fail. + +### Open questions + +- Can an adapter bound to a context implement `AsyncCipher`, or does + Vitamin C's interface need to carry the original context explicitly? +- How does that integration keep batching across several documents? +- Should sealing and opening be separate trait capabilities, so that no + method is present but always fails? +- Should decryption have the matching `decrypt` and `decrypt_with_context` + split? + +**Source:** +[cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) +(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's +discussion of batching). Vitamin C's trait definitions are in +`vitaminc-aead` 0.8.0, `src/seal.rs`. + +## 2. The Node and SealedValue types make opaque ciphertext storage harder + +**Status:** Partly done. Several decisions are recorded below. Pull request #1404 adds `Ciphertext`. The type has `to_bytes`, `from_bytes`, `key_tag`, and Serde support as Base64 or bytes. The text encodings and `OpaqueBlob` are still open. + +### Decisions so far + +- **Ordinary encryption produces one opaque ciphertext.** To encrypt a + value structurally, the caller chooses an explicit target. To encrypt a + composite plaintext as one ciphertext, the caller encodes it first with + `OpaqueBlob`. The + [implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends + adapting a native single sealed leaf through the existing transcoder. The + transcoder is the `target::transcode` protocol described in issue 13. This + avoids serialising a whole encrypted tree or encrypting again. +- **Collections, decided on 25 October 2026.** Direct `encrypt` takes a + single value. It does not lift itself over collections automatically. + Collections are still composed through `EncryptInto`, for example + `let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. + To encode and seal a whole collection as one ciphertext, use + `OpaqueBlob>`. The collection question had been reopened earlier. + This decision settles it. +- **Text encodings, decided on 25 October 2026.** Text conversions must use + the existing `Base64Text` and `Base85Text` wrappers from the EQL bindings. + The + [text encodings item](2026-10-24-stack-encrypt-api-plan.md#typed-text-encodings-and-key-tag-extraction) + in the API plan records the remaining work and the limit on decoding + Base85 in SQL. + +The follow-up of 25 October 2026 also set these directions: + +- Explore moving these general encoding types to a layer below Stack and + EQL, and supporting encoded field targets specific to ciphertext. +- Rust and PL/pgSQL must both be able to extract the stored ZeroKMS key tag + with simple code. +- A protocol for refined tags is being designed separately. The proposed + refinement replaces the ZeroKMS tag with a tag derived from it. The user + clarified that ZeroKMS checks policy and writes logs against the refined + tag, and returns the original tag. The decrypt path uses the original tag + to rebuild the leaf AAD. Record these two separate roles in the + implementation. + +### Open questions + +- Should byte buffers be spelled `Vec`, or use an explicit bytes + wrapper? +- What form should a simple round trip take that stores and loads opaque + ciphertext for scalar values, with direct conversion to bytes and back? + It could be a dedicated scalar result type, a wrapper, or conversion + helpers on the existing API. +- Should Base64 and Serde be supported conveniences? +- Does serialising a whole tree belong in Stack, or does it stay with the + caller? If it stays with the caller, say so explicitly, and keep it out + of the simple scalar path. + +The prototype's canonical leaf encoding is a useful starting point. On 25 +October 2026 it was not a backward-compatibility obligation. Since 7 +November 2026 it is, because Go rc.3 writes that envelope. #1404's +`Ciphertext` uses the same envelope unchanged. + +### Current behaviour + +`KeysetCipher::encrypt` returns `StackCipherText`, an alias for Vitamin C's +`Node`. Encrypting an `i32` produces `Single(SealedValue)`. +The return type also allows sequences, maps, empty containers, and +passthrough nodes. It does not keep the plaintext type as a type parameter. + +| Capability | `StackCipherText` tree | `SealedValue` leaf | +| --- | --- | --- | +| Debug printing (`{:?}`) | Yes | Yes | +| Serde serialisation and deserialisation | No | Yes | +| Canonical byte encoding | None | `to_bytes()` and `from_bytes()` | +| Direct Base64 conversion | None | None. Encode the canonical bytes separately | + +A leaf's canonical bytes contain its keyset ID, the metadata needed to +retrieve its key, and the local ciphertext. That versioned byte format is +the storage commitment. The leaf's Serde representation is not the same +compatibility commitment. Since 7 November 2026, data written by Go rc.3 +must stay readable. + +### Effect on callers + +Even storing a scalar makes the caller handle the container +representation: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let StackCipherText::Single(leaf) = ciphertext else { + unreachable!("encrypting an i32 produces a single leaf"); +}; +let bytes = leaf.to_bytes(); + +// After loading the bytes: +let leaf = SealedValue::from_bytes(&bytes)?; +let ciphertext = StackCipherText::Single(leaf); +let value: i32 = cipher.decrypt(ciphertext, ()).await?; +``` + +Note added on 7 November 2026: this block is no longer current. With #1404, +`keyset.encrypt(42_i32, ())` does not compile. The tree form is `keyset.encrypt_tree(42_i32, ())`. + +For one value, `keyset.encrypt(42_i32)` returns a `Ciphertext`. A `Ciphertext` has `to_bytes` and `from_bytes`. `keyset.decrypt` opens it. `StackCipher::decrypt` still takes a tree and a +context, as in the last line. + +The caller has to: + +1. handle a shape mismatch that cannot happen for this input; +2. extract the leaf; +3. rebuild the node before decrypting. + +There is no direct `ciphertext.to_bytes()` or +`serde_json::to_string(&ciphertext)` path. Base64 adds a separate encoding +step. For collections, the caller also has to choose a representation that +keeps the tree structure and the node variants. + +Stored bytes alone are not enough to decrypt. The caller also needs +authorised access to ZeroKMS and the same application context. The storage +API should make that clear without exposing tree internals in the simple +scalar case. + +**Sources:** +[ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) +(`StackCipherText`, `SealedValue`) and +[storage documentation and public re-exports](../../packages/stack-encrypt/src/lib.rs). +Vitamin C's container definition is in `vitaminc-aead` 0.8.0, +`src/container.rs`. + +## 3. Plaintext-side encryption has different syntax and context requirements + +**Status:** Partly done. The receiver for ordinary `encrypt` was decided on 24 October 2026. Pull request #1404 implements it. Pull request #1404 also gives `Ciphertext` +the same empty context through the keyset and through `EncryptInto`. The +structural API keeps its different requirements. + +### Decision: the keyset is the receiver for ordinary encryption + +**Decided on 24 October 2026.** Use only the keyset as the receiver of the +ordinary `encrypt` method: + +```rust +tenant_keyset.encrypt(value).await?; +``` + +Do not add a plaintext-side counterpart, `value.encrypt(&keyset)`. + +**`EncryptInto` stays.** This decision covers only the `encrypt` method. It +does not remove the API that encrypts into a chosen target. Implementation +is pending. The current inherent method still requires its context argument +(issue 1). + +Both receiver forms take the same inputs and perform the same operation. +Neither gives better type safety, ownership behaviour, performance, or +batching by itself. Those depend on the signatures and the implementation, +not on which argument is `self`. + +| Consideration | Plaintext receiver | Keyset receiver | +| --- | --- | --- | +| Mental model | Transform this value using a capability | Ask this encryption capability to process a value | +| Discoverability | The caller imports an extension trait. Methods appear on plaintext types | Encryption methods are found on the encryption handle | +| Fluent composition | Reads naturally after an expression that produces plaintext | Reads naturally when a keyset is chosen before an operation | +| Operational scope | The keyset and its authority appear as an argument | The keyset and its authority come first in the call | +| API surface | Adds methods to primitives and application types | Keeps execution methods on handles the library owns | +| Generic integration | Can work through a cipher trait | Can work through a cipher trait | + +The reasons for the decision: + +- Stack performs an asynchronous operation through a configured KMS + capability. Putting that capability first makes its scope visible. It + also gives callers one place to find operations. +- Plaintext traits can describe how values are represented without also + owning the execution API. +- Two equivalent spellings add documentation, completion entries, and + naming choices, but no capability. Familiarity with an earlier API matters + for migration, but does not by itself justify two permanent entry points. + +The `EncryptInto` API can choose outputs other than ordinary ciphertext. The +decision keeps it explicitly. Its context inconsistency is open, separately +from this decision. This decision does not cover the receivers for +decryption or batch methods. + +### Current behaviour + +Vitamin C's current `Encrypt` trait describes a plaintext through `visit`. +It has no `encrypt` method. Importing Stack's re-export of that trait does +not make `10_i32.encrypt(&keyset)` work. Vitamin C 0.8.0 documents this +trait as the replacement for an older trait. The older trait used a visitor +to drive a cipher directly. + +Vitamin C's `ContextTag` wrapper has encryption methods, but those require +Vitamin C's cipher traits. The methods do not make the syntax available on +bare integers. They also do not accept Stack's public ciphers directly. + +Stack supplies its own blanket extension trait, `EncryptInto`: + +```rust +use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; + +let ciphertext: StackCipherText = 10_i32 + .encrypt_into_with_context(&keyset, nonempty!("users/age")) + .await?; +``` + +Note added on 7 November 2026: this block no longer compiles as written. This was checked by reading the source, not by compiling. + +Since #1404, the crate-root `EncryptInto` is the owned trait. `StackCipherText` is not an owned target on `next`. The structural form needs +`stack_encrypt::target::legacy::EncryptInto`. The owned equivalent is +`let ciphertext: Ciphertext = 10_i32.encrypt_into_with_context(&keyset, "users/age").await?;`. + +The output type selects the `EncryptFrom` implementation. For +`StackCipherText`, that implementation requires a source that implements +`Encrypt + Clone`, and an `AeadContext`. An `AeadContext` must not be +empty. + +`EncryptInto` also has `encrypt_into(&keyset)`. That method requires the +target's context to implement `Default`. `AeadContext` does not, so this +form without context cannot produce `StackCipherText`. The form works for +other targets whose contexts have defaults, such as declarations that +include their own contexts. + +The scalar derive also chooses a context without saying so. That choice is +hard to discover. `#[stash(plaintext = i32)]` generates +`type Context = CallerContext`, even for a record whose only field is a +`StackCipherText`, whose own context is `AeadContext`. The generated +operation converts between the two types. + +The reviewer found that hidden choice surprising. It is undecided whether +this needs an API change, or clearer diagnostics and documentation. The +`Default` bound belongs to `EncryptInto::encrypt_into`, not to the +`EncryptFrom` trait itself. + +### Effect on callers + +Plaintext-side encryption exists, but it is not a mirror image of +`keyset.encrypt(value, context)`. Compared with the keyset method, it: + +- uses different method names; +- selects an output through a target trait; +- adds a `Clone` bound for this output; +- refuses the empty context that the inherent method accepts. + +Resolve the differing requirements for an empty context together with +issue 1. Renaming alone would not make the two paths equivalent. This +inconsistency is separate from the receiver decision above. + +**Sources:** +[plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto`, `EncryptFrom for StackCipherText`) and +[context types](../../packages/stack-encrypt/src/target/context.rs) +(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, +`src/plaintext/mod.rs`. + +## 4. The scalar derive cannot declare a fixed literal context + +**Status:** Direction agreed. Still open. #1491 makes repeated Rust +extensions append, so the composition table below now matches the code. + +### Agreed direction + +Allow a scalar `plaintext = T` derive to declare a fixed literal context: + +```rust +#[derive(EncryptFrom)] +#[stash(plaintext = i32, context = "example/integer")] +struct EncryptedInteger { + ciphertext: StackCipherText, +} + +let encrypted: EncryptedInteger = 10_i32.encrypt_into(&keyset).await?; +``` + +With a fixed context, the generated operation supplies that context. It +exposes a caller-context type that has a default, so `encrypt_into` works. +Without a fixed context, the ordinary scalar derive keeps using +`CallerContext` and requires `encrypt_into_with_context`. + +Keep the explicit `context_type` and `context_field` configurations. Their +interaction with a fixed context needs validation rules, not a silent +override. + +This closes the scalar gap in the derive. It does not need `CallerContext` +itself to implement `Default`. It also leaves the `EncryptFrom` trait +unchanged. A bare scalar has no input field for the derive to take context +from. The caller can still supply context at run time. + +### Combining caller context with the fixed context + +Extra context from the caller is accepted. Reuse the existing mechanism, +`DeclaredContext::under(own)`. Operation descriptions expose it through +`Encryption::under(own)`. + +`DeclaredContext` implements `Default`, and its +empty default leaves the declared context unchanged. A caller extension is +combined as `(own, caller)`. It never replaces the declared context. + +The recommended rules follow this existing mechanism: + +| Caller input | Effective structured context, with static context `C` | +| --- | --- | +| No extension | `C` | +| One extension `P` | `(C, P)` | +| Extensions `P`, then `Q`, through `DeclaredContext::with` | `((C, P), Q)` | +| One extension that contains the pair `(P, Q)` | `(C, (P, Q))` | + +This is structured composition, not string concatenation. Order and +grouping matter, and decryption must rebuild the same structure. +`NonEmpty::with` builds the matching pairs directly. The existing test +`under_gives_a_subtree_its_own_context_which_the_callers_extends` in +[target.rs](../../packages/stack-encrypt/tests/target.rs) checks one case. +An operation's declared context plus a caller extension must agree with a +context composed directly. + +The derive implementation still has to specify how a static literal is +represented: for example, as text or as a parsed path label. Encryption and +decryption then produce the same descriptor and AAD. This choice is +separate from keeping the `EncryptInto` trait. + +### Current behaviour and workaround + +The goal is a derived output for a bare `i32`, a fixed context declared on +that output, and `10_i32.encrypt_into(&keyset)`. The current macro rejects +`#[stash(plaintext = i32, context = "example")]`. The literal `context` +attribute works only with `struct = ..`. + +A workaround that uses only the derive wraps the integer in a plaintext +struct: + +```rust +struct IntegerInput { + value: i32, +} + +#[derive(stack_encrypt::EncryptFrom)] +#[stash(struct = IntegerInput, context = "example")] +struct EncryptedInteger { + value: stack_encrypt::StackCipherText, +} + +// With EncryptInto imported: +let encrypted: EncryptedInteger = IntegerInput { value: 10 } + .encrypt_into(&keyset) + .await?; +``` + +Note added on 7 November 2026: since #1404, "EncryptInto" here must be +`stack_encrypt::target::legacy::EncryptInto`. The crate-root `EncryptInto` is the owned trait. The `EncryptFrom` derive does not implement it. +This was checked by reading the source, not by compiling. + +This form declares the field's context as the pair `("example", "value")`. +ZeroKMS receives it as the descriptor `example/value`. For optional caller +extensions, the form uses `DeclaredContext`, a type with a default. It is +not a direct conversion from a bare integer. + +A manual `EncryptFrom` implementation can supply its own context. The +ordinary literal attribute cannot do that on a scalar derive today. + +**Sources:** +[macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`a_container_context_requires_a_struct`) and +[generated implementations](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`struct_derive`, `value_derive`). + +## 5. EncryptFrom does not define encrypt_from + +**Status:** Partly done. #1404 adds the declaration traits `EncryptTarget` +and `DecryptSource`, executed by the owned `EncryptInto` and `DecryptFrom`. +The structural `EncryptFrom` and `DecryptInto` remain. #1504 is in +progress. + +Review the trait names and method names, or where the convenience function +lives. Keep the distinction between declaring an operation and executing +it. + +`EncryptFrom` defines the associated type `Context`, the constructor +for an operation description, `encryption()`, and `indexes()`. It does not +define a function named `encrypt_from`. + +A function with that name exists on the blanket extension trait +`EncryptInto`. `Target::encrypt_from(&value, &keyset, context)` delegates to +`keyset.encrypt_as`. To call it, the caller needs `EncryptInto` in scope and +a target that implements `EncryptFrom`. + +This breaks the standard Rust naming pattern. `From` defines `from`, and +`Into` defines `into`. The names here suggest the same pattern, but two +things differ: + +- `EncryptFrom` declares an operation. It does not execute a conversion. +- The execution function on the target side belongs to the extension trait + on the source side. + +As a result, finding the method, and knowing which trait to import, is +surprising. + +**Source:** +[target operation traits](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptFrom`, `EncryptInto`, including `EncryptInto::encrypt_from`). + +## 6. The plan builder's encrypt_into reads like the execution method + +**Status:** Still open. Revisit during the plan API discussion. + +`Plan::value::().encrypt_into::()` is an inherent builder method. It +selects an output target and requires `T: EncryptFrom`. It does not +implement `EncryptInto`. It does not execute encryption either. + +Its name is the same as `value.encrypt_into(&keyset).await?`. The shared +name caused confusion during this review. + +A proposal for later discussion is to rename the builder method to +`target`: + +```rust +Plan::value::() + .target::() + .build() +``` + +The name `target` describes the choice being made, and separates it from +executing a conversion. The rename is not a decision. The API has not +changed. The proposal leaves `EncryptInto::encrypt_into` unchanged. + +**Source:** +[value plan builder](../../packages/stack-encrypt/src/plan/value.rs) +(`ValueStart::encrypt_into`). + +## 7. The output derive cannot mix encrypted and plaintext fields directly + +**Status:** The passthrough attribute is accepted in principle. Still open. Pull request #1504 adds it to the owned derives. That pull request is in progress. + +### Accepted proposal: a passthrough attribute + +The proposed `#[stash(passthrough)]` attribute on output fields was accepted +as sensible. It copies the matching source field unchanged, next to fields +whose types select encryption. + +Passthrough fields stay unencrypted and unauthenticated. Making a field +part of the encryption context is a separate choice. Copying a field does +not imply it. + +### Current behaviour + +Take a `User` with encrypted `name` and `email`, and plaintext `id` and +`created_at`. The fields-plan API supports this directly with +`.encrypt(pick(..))` and `.passthrough(pick(..))`. It batches the encrypted +fields and returns `FieldValues`. Callers can extract those values into a +typed storage struct. + +The current `EncryptFrom` derive has no attribute to copy a source field +through. `#[stash(default)]` initialises an output from a default or an +expression. It does not copy a field from the input. Keeping the original +field type does not select passthrough either, because derived fields need +target traits. + +### How Vitamin C's derive differs + +This gap applies only to Stack's output-side `EncryptFrom` derive. Vitamin +C already has a plaintext-side `#[derive(Encrypt, Decrypt)]`, with +`#[aead(passthrough)]` on source fields. Stack's +`keyset.encrypt(&user, aad)` can take that derived source. Since #1404, +that method is `keyset.encrypt_tree(&user, aad)`, and its `aad` parameter +takes a context. The result is a +generic `StackCipherText` map with sealed and passthrough nodes. + +That route has three limits: + +- It does not generate a named `EncryptedUser` type. The output is not + exposed as typed Rust fields. +- It derives AAD for map entries differently from the per-field contexts of + a Stack fields plan. Do not present the two routes as interchangeable + storage formats. +- No source-side derive exists that generates the typed encrypted + counterpart shown in this review. + +**Sources:** +[plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) +(`FieldsBuilder::passthrough`) and +[derive field attributes](../../packages/stack-encrypt-derive/src/attrs.rs) +(`FieldAttrs`). + +## 8. Clarify whole-value encryption, layouts, and searchable outputs + +**Status:** Open architectural discussion. The recommendations below are +not decisions. #1503 is in progress for workflow 1, and #1504 for workflows +2 and 3. + +Three user workflows need clear support: + +1. Serialise a complete value into bytes, and encrypt those bytes as one + opaque ciphertext. +2. Produce a chosen layout, such as `EncryptedUser`, with fields encrypted + independently and explicit plaintext fields. +3. Produce a layout with extra encrypted forms or search terms, such as + EQL targets. + +### Recommended model + +Separate the encryption granularity from the output representation and the +operations it requests. The granularity is either one encoded value or +selected fields. Workflow 3 should extend workflow 2 through field targets +that request ciphertext, search terms, or both. It should not add a separate +execution system. The generic tree can stay as one layout option without +being the required storage representation for scalars. + +- **Whole-value path.** The follow-up review favours an explicit named + wrapper, provisionally `OpaqueBlob`. The wrapper owns serialisation + and deserialisation through one codec that the library chooses. It + produces one opaque ciphertext that can be stored. Decrypting into + `OpaqueBlob` selects the matching decoding path. Today callers can + serialise into `Vec`. Vitamin C treats a `Vec` as a single byte + leaf. Callers still meet the storage gap between node and leaf (issue 2). +- **Layout path.** Select a target with `EncryptFrom` and `EncryptInto`, or + a plan, with an explicit passthrough choice. A source-side derive that + generates a target would make authoring easier on the same system. It + would not add another set of encryption semantics. +- **Searchable layout path.** Compose the core encryption and index + operations through targets. EQL adapters own their storage + representation, and Stack owns execution. Write operations and query + operations must share the relevant context and normalisation rules. + +Inside the library, keep these four things separate: source +representation, requested operations, output encoding, and execution. Give +each workflow a short public path. An application should not need to learn +visitor machinery or plans to encrypt and store a scalar. Arbitrary +serialisation must not silently replace the typed normalisation that +indexes use. + +Before choosing more trait names, write examples for all three workflows +from start to end: encrypt, store, load, and decrypt, plus query for +searchable targets. Use those examples to judge the public API and its +error messages. + +### Why Vitamin C's derive is not whole-value encryption + +The existing Vitamin C plaintext derive produces a structural ciphertext +tree, not one encrypted serialisation. It is a generic output layout for +workflow 2, not workflow 1. The difference affects four properties: the +number of keys, independent access to fields, the visible structure, and +the authenticated boundaries. Encoding the completed ciphertext tree +into one byte string would not give it those properties of whole-value +encryption. + +### Follow-up: a whole-value wrapper chosen by the library + +The first public example now uses +`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` instead of +encoding JSON by hand. The wrapper name is provisional. The direction is to +make blob serialisation an explicit behaviour that the library owns. + +The stored result stays opaque ciphertext. Decryption returns an +`OpaqueBlob` that the caller can unwrap. The codec and its +authenticated versioning still need a specification. The wrapper does not +silently change how unwrapped structs are encrypted. + +### Constant-time serialisation is not yet established + +Constant-time serialisation and deserialisation is a property to +investigate. Choosing a binary codec, or using constant-time AEAD, does not +establish it. The first revised example uses Serde traits provisionally. +Arbitrary user implementations of those traits cannot give that guarantee. + +Before promising execution that does not depend on secrets, define which +lengths and shapes may be public. Also decide whether a stricter set of +supported types is needed. The examples document this open question +separately from how easy the wrapper is to use. + +Investigate `serdect` as part of that work. Stack Encrypt already depends +on it. Check what its encoding and decoding guarantees are and what they +cover. Then judge whether it applies to serialising ciphertext and to the +proposed blob codec. Its current use does not imply a constant-time +guarantee for whole objects, or a codec choice. + +## 9. Ownership of the value-encryption API across Stack and Vitamin C + +**Status:** Open architectural discussion. Moving the traits is proposed. +It is not authorised implementation work. #1501 moves the Rust crates to +Vitamin C 0.9. `OpaqueEncode` now follows Vitamin C's `Scalar` marker for +which types are single values. + +### Recommendation + +Stack should own the public model for encrypting values. That includes: + +- the plaintext traits and derives; +- target declarations; +- contexts; +- storing ciphertext; +- execution semantics. + +Vitamin C can keep reusable primitives for cryptography and for protecting +memory. Moving files alone does not fix three problems: + +- ambiguous names: for example, `Encrypt` currently describes a source; +- duplicate entry points; +- the missing path for whole values and storage. + +One core layer owned by Stack could serve both Rust and language guests, +without pulling in KMS transport or HTTP. Applications should still have one +consistent entry point, `stack_encrypt`. An internal split of dependencies +should not become another public API that users have to understand. + +### Where the proposal came from + +The review kept crossing from Stack's target declarations into Vitamin C's +plaintext visitor traits, cipher traits, contexts, derives, and ciphertext +tree. The user proposes moving the encryption traits into Stack Encrypt, +and keeping general primitives, such as protected values and randomness, in +Vitamin C. + +### Migration constraints + +- Today `stack-encrypt` depends on both `vitaminc-aead` and + `vitaminc-encrypt`, and `vitaminc-encrypt` also depends on + `vitaminc-aead`. Moving the traits of `vitaminc-aead` into the top-level + Stack crate, and making the existing backend depend on Stack, would create + a dependency cycle. Either move the backend adapters for values too, or + draw a separate boundary for a lower-level core of primitives. +- These must agree across the new consumers: context composition, type + tags, ciphertext framing, index normalisation, and the transport frames of + the bindings. On 25 October 2026 they could change without keeping + prototype data and without a migration. Since 7 November 2026, stored + data written by Go rc.3 must stay readable and searchable. +- Keep one canonical identity for each trait and type. Update consumers + directly. Re-exports for compatibility are not needed. Independent copies + of trait definitions would need duplicate implementations and adapters. +- Account for guest and FFI consumers and EQL target implementations, not + only Rust application imports. The path without HTTP, for WebAssembly, must + stay usable. + +**Sources:** [Stack dependencies](../../packages/stack-encrypt/Cargo.toml), +[guest application binary interface (ABI) dependencies](../../packages/stack-guest-abi/Cargo.toml), +and +[EQL encryption adapters](../../packages/eql/crates/eql-bindings/src/encryption.rs). +The installed `vitaminc-encrypt` 0.8.0 manifest confirms that it depends on +`vitaminc-aead`. + +## 10. The public encrypt_as duplicates encrypt_into + +**Status:** Decision agreed. Still open: `encrypt_as` is still public on +`next`. + +### Decision + +Remove `encrypt_as` from the public API. Keep `encrypt_into` and +`encrypt_into_with_context` as the public entry points for encrypting into +a target: + +```rust +let encrypted: EncryptedInteger = value.encrypt_into(&keyset).await?; +let encrypted: EncryptedInteger = value + .encrypt_into_with_context(&keyset, context) + .await?; +``` + +The machinery that executes operations can stay internal. Removing the +public method does not remove target declarations, batching, or execution. +Public examples of the proposed API should use the entry points that stay. +Earlier mentions of `encrypt_as` in this log describe the current code. + +This is separate from issue 3's decision about ordinary encryption. +`keyset.encrypt(value)` stays the only spelling of ordinary `encrypt`. +`value.encrypt_into(&keyset)` selects a particular output type. The +question in issue 5 about the name and location of `encrypt_from` is still +open. + +### Current behaviour + +`KeysetCipher::encrypt_as` is currently an inherent method. It requires the +output to implement `EncryptFrom`, and executes that target's operation +declaration. `EncryptInto::encrypt_into` and +`EncryptInto::encrypt_into_with_context` delegate to it. They supply either +the default context or the caller's context. + +**Source:** +[target operations](../../packages/stack-encrypt/src/target/operations.rs) +(`KeysetCipher::encrypt_as`, `EncryptInto`). + +## 11. Context fields do not yet take context from an input struct + +**Status:** Still open. The syntax is still open. The shorthand +`context(field = "id")` is recommended. No merged pull request adds it. + +### Current behaviour + +`#[stash(context_field)]` exists on derived output types: + +- On encryption, it stores the supplied context in that output field. +- On decryption, it uses the stored context. It can also check the stored + context against an expected value that the caller supplies separately. + +It does not automatically select a field from a plaintext input struct. + +The macro treats this field as the complete context, and rejects it next to +a literal `context`. Because `struct = User` requires a literal context, +that form cannot mark its `id` with `context_field`. The public examples +first described passing the ID explicitly, without explaining this +difference. + +### Clarified model: where context comes from and where it is stored + +Two similar scenarios need to be told apart: + +1. The caller supplies context as a function argument, and the encrypted + output stores it for later use. This is what scalar-output + `context_field` does today. +2. The encryption declaration takes context from one or more fields of the + plaintext. Whether those values are also stored is a separate choice. + +The second scenario can include storage too. It reads `User.id`, uses it to +bind the encrypted fields, and copies it into `EncryptedUser.id`. The caller +does not pass the same value twice. Several fields, such as a tenant ID and +a user ID, can supply a structured composite context. + +| Context source | Stored in the encrypted output | What decryption needs | +| --- | --- | --- | +| Caller argument | Yes | Read the stored context. Optionally check it against an expected value supplied separately | +| Caller argument | No | Supply the same context again | +| One or more plaintext fields | Yes | Rebuild the context from the stored fields. Optionally check expected values | +| One or more plaintext fields | No | Supply equivalent context separately before decrypting | + +Store a context taken from the source separately from the ciphertext that +needs it. Decryption then does not have to open that ciphertext first to +find its own context. Binding a ciphertext to a stored context +authenticates that relationship. It does not by itself prove that the whole +record is where the application expected it to be. + +### Recommended syntax + +The derive should express two separate concepts: selecting the source, and +storing in the output. A shorthand can cover the common case of reading and +storing. Do not assume that one `context_field` attribute should silently +do both. + +The syntax is still open. Define each of these: + +- the ordering and grouping of fields for a composite context; +- literal prefixes; +- optional caller extensions; +- `from` mappings; +- checks of expected context. + +These capabilities are proposed, not implemented. + +The recommended shorthand for the common case is `context(field = "id")`. +It expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. It +selects the same Rust field name on the two different inputs. It does not +copy or store that field implicitly. + +- Keep the directional form for different names or different sources. +- At first, reject mixing the shorthand with directional selectors, so + that precedence is never ambiguous. + +The context examples now use this shorthand for the case where a plaintext +ID becomes a stored ID. + +**Sources:** +[macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`Record::parse`, `context_type`) and +[context-field output assembly](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`assignments`, `value_derive`). + +## 12. Default is not the same as optional caller context + +**Status:** Partly done. The directional-context examples showed this +problem. Pull request #1404 implements the first option below for the owned API. It adds `RequiredContext`, with the capabilities `AllowsNoContext` and `AcceptsContext`. #1501 adds compile-fail tests for a `u64` context. The +structural API still uses `Default`. + +`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` offer the form +without context whenever the associated `Context` type implements +`Default`. That works for types that are meant to have defaults, such as +`DeclaredContext` and `ExpectedContext`. It currently excludes +`CallerContext` and `AeadContext`. + +A required caller value can still have a Rust default. If a declaration +exposes `Context = u64`, the form without context supplies zero. The +directional proposal `encrypt(caller = u64)` must not accidentally mean +"optional, with zero as the default". + +Separate the application's type for the context value from the trait's +type or capability for caller input. Two options: + +- a `RequiredContext` wrapper with no default, accepting convenient + conversions from `T`; +- a dedicated marker that controls the entry points without context. + +Required context matters on decryption too. Context taken from the source +may need no argument for encryption. Decryption may need an external value, +or accept an optional expectation of the stored context. Do not assume that +the two associated input types must match. Do not assume that either one +always equals the complete logical context that encryption uses. + +See the +[context scenario examples](2026-10-24-stack-encrypt-context-examples.md) +for the proposed selectors and what each one requires as input. + +**Source:** +[convenience methods](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). + +## 13. JSON layouts and adapting encrypted output + +**Status:** Open. This is a first assessment of the JSON tree +implementation on the `next` branch at `b1049824b`. No replacement API is +chosen. + +**Agreed follow-up:** investigate replacing the serialisation round trips in +the EQL JSON adapter with direct typed conversions. This does not yet choose +an extended visitor protocol, or authorise a particular replacement API. + +### How JSON configuration and targets work today + +`Layout` describes a target that is selected by a value carrying run-time +options. It links a `Target: EncryptFrom` with an `Index`. It supplies an +`encryption(&self)` description, and the configured index for queries. +`EncryptFrom::encryption()`, by contrast, selects a description by +output type. + +For JSON, both routes can produce `JsonDocument`: + +- Selecting that type uses the default options. +- `Json::new(options)` can set ordered paths, and whether the root is + sealed. Under a sealed root, the root entry's ciphertext contains the + whole document, so it can be opened as a complete document. + +A plan keeps those options for its queries. + +This goes beyond choosing the field layout of a Rust record. JSON options +affect the cryptographic structure, what queries can do, and what +information is exposed to the server. The same output type can represent +different configurations, so the type alone does not identify the +configuration. The JSON tree belongs to the searchable-layout workflow +(issue 8). It is not an opaque serialised blob. + +### The existing transcode protocol does a different job + +The existing `target::transcode` protocol works at a later stage. A +`Reader` hands completed ciphertext, structural markers, metadata, and +search terms to a destination `Visitor`. `Encryption::transcode()` uses it +to adapt encrypted output without an intermediate serialisation tree. The +protocol does not choose JSON indexing options. It does not generate +missing search terms either. + +The new EQL adapter takes another route: + +- `JsonTree` implements `Layout`. It + delegates encryption to the engine's `Json` layout, and wraps the + completed `JsonDocument` with EQL version and column metadata. +- Its envelope conversion goes through a `serde_json::Value` intermediate. +- A private helper that is also named `transcode` converts query types and + extraction types. It calls `to_value`, then `from_value`. + +Neither conversion is the existing `target::transcode` protocol. That +protocol has no reader or visitor event for a JSON document today. The +conversion processes encrypted output and query metadata. It is not a way +to skip plaintext serialisation. + +On decryption, the EQL target checks its stored column against any expected +value supplied. It rebuilds the engine document through the wire form, and +delegates opening to the engine. Its layout object is not needed for that +conversion. + +The sources for this section are: + +- + [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) + (`Layout`, `json_document`, `EncryptFrom for JsonDocument`); +- + [encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs); +- [JSON options and document](../../packages/stack-encrypt/src/json/mod.rs); +- + [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) + (`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). + +### Open questions about JSON configuration + +- How should the API keep encryption configuration separate from adapting + encrypted output? +- Is `Layout` the clearest public name for a configured target, given that + this log earlier used "layout" for output records? +- How do run-time target options fit the `EncryptInto` API that stays? + Today type-directed conversion selects the target's defaults, and plans + accept a configured layout through `encrypt_with_layout`. +- Should JSON use a direct typed adapter for JSON documents, or an extended + visitor for encrypted output? Do not force the JSON protocol into generic + map and sequence events. Those events lose its header, shared document + key, projection, and extraction semantics. A shared architectural boundary + does not have to mean one universal visitor. +- Should a general abstraction for configured targets require an index? + Today's `Layout` requires one, but targets that only encrypt can also need + options. +- How does the design keep the link between write configuration and query + configuration? A plan's reuse of the configured index is useful whatever + the names are. + +### Static Rust API compared with run-time configuration + +The intended public Rust experience is traits and derives. Plans are the +lower-level composition interface. They support those APIs and the dynamic +FFI interface. Rust callers should not have to write a plan for JSON. + +The derive already emits plans. Its implementation can keep that shared +execution path without making the plan builder the main user example. + +JSON already has a type-directed path. Given an initialised `cipher` and its +`keyset`, this code uses existing methods and the default JSON options: + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use eql_bindings::Identifier; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, EncryptInto}; + +let profile = json!({"name": "Alice", "age": 34}); +let column = Identifier::for_column("users", "profile")?; + +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored: Value = encrypted.decrypt_into(&keyset, column).await?; +assert_eq!(restored, profile); +``` + +This produces searchable JSON with a sealed root and no ordered paths. The +explicit decryption context checks the expected column against the stored +identifier. The snippet was checked against the source, not compiled. + +The earlier `SearchableEmail` example fixes its recipe in the target type: +ciphertext plus an equality term. `EncryptFrom::encryption()` takes no +configuration value. JSON's default target also fixes a recipe in its type. + +What `Layout::encryption(&self)` adds is access to configuration on an +instance. One `JsonTreeDocument` output type can be produced with `$.age` +ordered, with `$.salary` ordered, or with different root settings, chosen +at run time. Its configured index also supplies the matching query +behaviour. + +Search metadata itself does not require `Layout`: + +- A custom target could fix ordered JSON paths in a manual `EncryptFrom` + implementation, using the existing JSON operation. +- A derive could later generate such a fixed policy. +- Targets configured at run time need an extra configuration input. Today + `encrypt_into_with_context` accepts context, but no layout or options. + Custom JSON configurations use the lower-level plan, or the `keyset.run` + path. + +Keep those options separate from cryptographic context. + +The cryptographic difference is separate from the API difference. JSON uses +a specialised document operation, with a shared document data key and its +own node and extraction semantics. Adding a generic equality-term field to +the ordinary ciphertext tree cannot implement it. That explains why the +JSON operation exists. It does not by itself require a public `Layout` +trait. + +### Names hide the roles of the JSON types + +| Current type | Role | +| --- | --- | +| `serde_json::Value` | Plaintext input, and recovered plaintext | +| `stack_encrypt::json::JsonOptions` | Run-time configuration | +| `stack_encrypt::target::Json` | Configured index. Also implements `Layout` | +| `stack_encrypt::json::JsonDocument` | Encrypted engine document, including search metadata | +| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target or layout | +| `eql_bindings::v4::json_tree::JsonTreeDocument` | Encrypted EQL storage envelope | + +Review names that state these roles: + +- `Encrypted` for documents that contain ciphertext; +- `Index` for index descriptions; +- `Target` or `Config` for configuration; +- module paths that separate EQL types from engine types. + +No replacement names are chosen. Consider `Json`'s double role as index and +layout before renaming it. + +### Candidate: a fixed-policy wrapper around the EQL output + +This is a proposal. It is not implemented. Its syntax is not agreed. + +The fixed policy is attached to an output newtype. The inner library type +defines the database format. The alias below only makes the existing +type's role clearer. It does not rename anything in the library. + +The later [implementation plan](2026-10-24-stack-encrypt-api-plan.md) +recommends a general factory for typed options, `target(options = factory)`, +instead of the JSON-specific attribute below. Dynamic callers deserialise +options into the same concrete type. Rust and Go support ship together. This +log keeps the sketch to explain the earlier alternative. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use eql_bindings::Identifier; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, DecryptInto, EncryptFrom, EncryptInto}; + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = Value, + transparent, + json_tree(ordering_paths = ["$.salary"]) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); + +// Given a keyset: +let payroll = json!({"name": "Alice", "salary": 120_000}); +let column = Identifier::for_column("employees", "payroll")?; + +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +// The wire representation is the inner EQL envelope, with no extra field. +let wire = serde_json::to_vec(&encrypted)?; +let loaded: EncryptedPayroll = serde_json::from_slice(&wire)?; +let restored: Value = loaded.decrypt_into(&keyset, column).await?; +assert_eq!(restored, payroll); +``` + +The proposed `stash(transparent, json_tree(...))` selects the configured +JSON operation for the inner target, and wraps its completed output. It +must not: + +- run the ordinary scalar or record derivation on the wrapper; +- run the inner target's default encryption and then try to add ordering + terms. + +The wrapper inherits the EQL target's caller context and decryption +behaviour. Serde's existing `transparent` attribute keeps the EQL envelope. +It does not provide trait implementations for database drivers. Those stay +a separate integration concern. Under this candidate's defaults, the root +stays sealed. + +The generated declaration must also expose the configured index metadata +for future typed queries. Queries can then reuse the policy instead of +repeating the paths. The syntax for queries is not designed yet. The core +derive must delegate the construction of configured output to a capability +of the inner target, instead of depending directly on the EQL crate. It is +still open whether `Layout` or another trait expresses that capability. + +### Alternative: configuration on a plaintext wrapper + +The user proposed `JsonPlaintext::new(value).with_ordering_path(path)`. It +can still produce the exact EQL wire format. Where the configuration comes +from is independent of the output representation. + +An explicit +`EqlEncryptedJsonTree` target can accept that input and execute the same +configured JSON operation. The wrapper must be treated as an encryption +request, and not serialised as part of the plaintext. The current API does +not provide this wrapper. + +This wrapper is attractive when paths come from run-time configuration. +However, it repeats a fixed policy at each call site, and needs a way to +reuse that policy for queries. For the fixed-policy example, prefer the +output newtype. + +If the run-time wrapper is added, do not implement conversion from it into +fixed-policy targets at first. This avoids silently overriding or merging +conflicting policies. Neither syntax lets callers redefine EQL's required +fields or its cryptographic wire protocol. + +A fixed Rust type describes the policy for new writes. Transparent Serde +deserialisation alone does not prove that a stored document used that +policy. The document header does not record all ordered paths. Policy +changes, and existing rows written with different policies, need explicit +compatibility and migration rules. Only then can the API claim stronger +guarantees when it reads. + +## 14. Decrypt convenience methods need explicit context variants + +**Status:** Partly done. #1404 implements `T::decrypt_from` and +`decrypt_from_with_context`, and the keyset pair, for one value. #1501 +adds them for collections. The structural `decrypt_into` remains in +`target::legacy`. + +**Required direction:** provide a method without context and a separate +`_with_context` method, mirroring encryption. The user prefers the name +`decrypt_from`. + +**Recommendation:** use the pair directed at the destination in the +preferred public examples. `User::decrypt_from(encrypted, &keyset)` follows +Rust's `From` convention. `encrypted.decrypt_from(...)` would reverse it. + +These are not settled user decisions. Do not change the declaration trait +only because its convenience spelling changes. Handle trait discoverability +with issue 5, and the capability for optional context with issue 12. + +### Current behaviour + +These methods exist today on the blanket extension trait `DecryptFrom`: + +- `ciphertext.decrypt_into(cipher, context)`; +- `Plaintext::decrypt_from(ciphertext, cipher)`; +- `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)`. + +Since #1404, these three methods are in `target::legacy::DecryptFrom`. The +crate-root `DecryptFrom` is the owned trait, with `decrypt_from` and +`decrypt_from_with_context` and no `decrypt_into`. + +`DecryptInto

` separately declares the opening operation. These method +and trait names are easy to confuse. + +## 15. Coordinate the Go JSON work with the shared API redesign + +The user identified +[pull request #1353](https://github.com/cipherstash/stack/pull/1353) as +work by a parallel agent. The pull request is currently a design document +for Go JSON, with implementation tasks #1348 to #1352, tracked by #1347. + +These parts of it must be reconciled with the contracts for configured +targets, traits, and context developed here: + +- its dynamic path for JSON and options; +- its public names; +- its context behaviour; +- its staged support for extraction. + +Keep the compatible work on the catalog, the transport, the engine, and +testing. Do not design it twice, and do not implement it twice. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records the +specific points to reconcile, written after reading the pull request. This +is a local coordination item. No message or review was posted to that agent +or pull request. + +## 16. Integrate the prior query design instead of reopening it + +The user supplied **Stack Encrypt query design**, dated 21 October 2026. It +was read in full on 25 October 2026 from +`/Users/dan/Downloads/Stack Encrypt query design.md`. It is explicitly a +draft for discussion, with open questions. It proposes: + +- index selection owned by the field; +- one index that answers each operator; +- typed field handles; +- one declaration of capabilities; +- shared rules for deriving and refusing queries. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses +that draft as the baseline. The plan records specific points to reconcile: + +- this log's output-side derives and factories for configured options; +- JSON operators and extraction; +- context inputs; +- portability between the static and dynamic interfaces; +- when validation happens. + +The user's clarification about the prototype replaced older compatibility +requirements. The decision of 7 November 2026 adds one back: the stored +format of Go rc.3 must not break. The source document was not changed. + +## 17. Use the same name to select a definition for encryption and queries + +**Status:** Still open. `.using(...)` is recommended. The final spelling is +not yet accepted. + +The user finds it confusing that encryption uses `.using(...)` and queries +use `.on(...)`, and asked to consider one name for both. The recommendation +is `.using(...)`, because the selected definition or field supplies the +operation's policy. + +Keep the bounds specific to each operation. This is a change of name, not +a merger of the two capabilities: + +- Encryption selects the associated definition of the output target. +- Queries require a field, or a single-value target, that implements + `QueryTarget`. + +The user also prefers `cipher.using(&field).query(op, input)`. They +explicitly require strict typestate: the request's type records which steps +are done, so the compiler refuses an incomplete request. They accept +supporting both orders. The plan recommends putting the definition first as +the canonical order. The order that starts with the operator then produces +the same typed request. + +On the static path: + +- a missing definition, or missing required context, must prevent awaiting + the request; +- selecting a definition again must not silently replace the request's + definition. + +This log does not claim that the current `.query(...).on(...)` +implementation already provides this API. + +## 18. A single ciphertext needs inputs that are consumed and cleaned up + +**Status:** Decided on 25 October 2026, with the follow-up decisions below. +Partly done: #1404 and #1501 guard and wipe owned sources, and #1493 adds +lifecycle conformance. No completed audit is claimed. + +### Decision on encoding traits and consumed input + +**Decided on 25 October 2026.** Use `OpaqueEncode` and `OpaqueDecode` to +encode and decode single values. These names separate one opaque ciphertext +from target layouts. + +Both direct `encrypt` and target-directed `encrypt_into` consume their +input, including the variants that take context. Both are responsible for +cleaning up the owned plaintext data they create. Explicit passthrough +fields move to the output. + +The plan records: + +- what cleanup each source must support; +- guarding immediately, before returning a request that has not been + polled; +- cleaning up encoded buffers and scratch space on every ordinary exit + path; +- the limits for copies held by the caller, custom serialisation, + passthrough output, and ownership between the Go host and the guest. + +`Serialize` alone cannot ensure that an `OpaqueBlob` source is cleaned +up. Consuming a value and zeroising it are different things. + +### Follow-up decision: use the existing Zeroize trait + +Plaintext types must implement the existing `Zeroize` trait. Do not add a +cleanup trait of our own for now. For the proposed Serde blob, this means +`Serialize + Zeroize` on input, and `DeserializeOwned + Zeroize` on owned +decoding. Library guards must actually call the cleanup. Serde values that +are only partly built need separate handling. + +### Decryption decision + +Keep `keyset.decrypt` and `decrypt_with_context` for ordinary opaque +ciphertext. Keep `T::decrypt_from` and `decrypt_from_with_context` for +declared targets. On success, return an ordinary owned `T: Zeroize`. This +hands the cleanup responsibility to the caller. + +Stack guards intermediate plaintext values while Stack is their owner. Support +results with an explicit drop guard, but do not make them the default. +`Zeroize` alone does not mean the value is cleaned up when dropped. This +does not resolve the separate requirement to clean up after a partial +deserialisation. + +### Codec recommendation + +Reuse Vitamin C's frame machinery with a mapping for blobs. The 0.8.0 Serde +implementation for transport already exists and guards serialisation +buffers. Its frame writer wipes allocations when they grow. + +Do not reuse that mapping unchanged, for three reasons: + +- It can lose whether an option is present: `Some(())` and `None` both + encode as null. +- Maps require text keys. +- A successful `to_vec` returns a vector with no guard. + +The +[opaque blobs](2026-10-24-stack-encrypt-api-plan.md#opaque-blobs) item in +the API plan records the codec now chosen: Vitamin C's storage profile. +Go rc.3 already writes that format. Neither the codec choice nor the `Zeroize` +bound establishes constant-time behaviour, or cleanup after partial +decoding. + +### Priority update + +First specify stable, versioned storage bytes. The existing frame codec was +designed for short-lived transport. Later improvements to zeroisation and +timing can follow without changing the stored format. The action needed +upstream is +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). + +## 19. First-release completeness and remaining review decisions + +**Status:** The decisions below were accepted on 25 October 2026. The +detailed JSON operation contract is still under user review. Partly done: +#1492 enrols the JSON operations in conformance. Opening a child bound to +its parent is still open. + +### Encoded ciphertext and equality + +- Encoded ciphertext wrappers are first-class field targets. Their final + names and package placement are decided last. +- ORE does not answer equality at first, even for integers. +- Allow for a future explicit capability for encodings that preserve + equality. Never infer exact equality from ordering support for text or + other lossy encodings. +- Keep the validation that checks for overlapping operator providers. + Avoid silently changing existing definitions. + +### The JSON first version must be comprehensive + +The first JSON version must be comprehensive across Rust and the dynamic +interface for Go. It includes extraction, opening entries, listing and +counting children, and walks over subtrees. Claude's earlier Go plan +deferred walks. The new scope replaces that deferral. + +The +[operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) +lists: + +- the operations required; +- the current evidence; +- the proposed Rust and dynamic APIs; +- how SQL uses them; +- the rules for missing paths and null; +- the acceptance checks for implementation. + +Names and packaging are explicitly the last design activity. + +### Further accepted decisions + +- Custom Rust stages can stay explicitly Rust-only, and use `NotPortable` + where dynamic export is not supported. Built-in stages keep parity between + Rust and Go. +- Every check that does not need a key runs before keys are resolved or + loaded. + +## 20. Keysets chosen from plaintext, and optional provider policies + +**Status:** Still open. The recommendations below are not yet accepted. + +### Request + +The user asked for three things: + +- route a plaintext record to a keyset, by ID or name, taken from one of its + fields; +- an assessment of Vitamin C's unused `AsyncCipher`; +- ZeroKMS lock context, including selectors for claims in the principal's + JWT (JSON Web Token). Lock context is a ZeroKMS feature that binds a data + key to claims about the authenticated identity, the principal, that makes + the request. + +Keyset and lock-context capabilities must stay optional, for future AWS KMS +and Vault backends. The +[provider capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) +gives the current evidence from the source, proposed static and dynamic +declarations, and the work needed for acceptance. + +### Current facts + +- Stack resolves keysets by ID or name, but each generation batch uses one + keyset chosen beforehand. +- Stack deliberately coordinates provider requests outside Vitamin C's walk + over leaves, and already implements the synchronous `Cipher`. +- Its layer for provider payloads supports lock context, but Stack Encrypt + sends an empty lock context. +- The existing abstractions for data keys and index keys still embed + ZeroKMS protocol types. Future providers need a neutral base plus optional + capabilities, not empty implementations of required ZeroKMS features. + +### Recommendations + +- Derive key routing as an explicit target selector, and resolve it in the + client executor. +- Split batches by the key, provider, and principal requirements they + resolve to. +- Carry lock context as a separate typed provider policy, on both key + generation and key retrieval. +- Reject unsupported required features before any I/O. +- Keep these four things separate: key routing, ciphertext context, search + scope, and principal policy. Search scope is the set of values that index + terms are derived under, such as a column and a tenant. A query can then + match every row in that scope. +- Revisit cleanup of the cipher adapters during packaging. Do not lose + batching across records, or native and WebAssembly support. + +## 21. Packaging, naming, and retiring AsyncCipher + +**Status:** Still open. The names and package moves are proposals. Pull request #1404 uses some agreed names. No package has moved. + +The user favours removing Vitamin C's `AsyncCipher`, and moving encryption +behaviour for applications into Stack Encrypt. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes: + +- Stack owns value traversal, declarations, target adapters, index + adapters, and execution; +- a small shared `stack-encrypt-format` crate contains codecs and typed + encodings; +- a client crate specific to ZeroKMS; +- reusable primitives stay in Vitamin C. + +It includes a dependency map with no cycles, and a glossary of public names. + +Stack does not use `AsyncCipher`. Vitamin C's async helpers and tests do +use it. Retire that execution path together with its consumers, and move the +relevant test coverage. Deleting the trait alone is not complete. + +The adapters for normalisation, PRF (pseudorandom function), and ORE also +depend on the existing value model. They need to be separated from their +primitives. + +Moving the frame implementation refines the earlier recommendation about +who owns the codec. Coordinate the work on stable storage under +Vitamin C #519. + +## 22. Customer-driven order of work: Go JSON and match search first + +**Status:** Order of work set by the customer's needs. Merged work +reconciled on 7 November 2026. The conformance corpus is merged. + +### The customer's needs come first + +The customer needs Go JSON support. They also need dependable Bloom-filter +handling with tokenizers and token filters, including input of two +characters. The quality of a standalone Rust release is secondary. + +- Claude's stack of JSON-tree pull requests, headed by #1363, has now + merged. Revise the older Go plan in #1353 against it, and against the + agreed API direction. +- Vitamin C #520 and #526 implement the storage profile: Vitamin C's versioned byte format for an opaque value that is stored. Review by a + colleague, and the release, are pending. Stack still needs to adopt that + release. + +### What to freeze before the customer release + +Freeze the public Go contract, and the stored binary and search semantics, +before the customer release. Internal plans can change behind that +interface. + +Adopt and stabilise the Vitamin C codec in its current repository now. +Defer these: + +- moving crates physically; +- retiring `AsyncCipher`; +- the final Rust derives. + +The earlier exemption for prototype compatibility does not allow breaking +customer data. Since 7 November 2026, that applies from now, not only +after the final release, because the customer builds on Go rc.3. + +### Short words in match search + +The supplied `Short-ngrams.md` explores Bloom filters for short whole words, +and queries over variants. These must be kept separate from search for any +two-character substring. + +The current tokenizer allows a length of 2. To support named filters and +Boolean alternatives, these need to grow: + +- the current output of one match; +- the current shapes of a single query term and a range query term. + +Resolve the scheme's leakage, its SQL representation, and its performance +before the release. Do not silently defer the customer's requirement. + +### Reconciliation with merged work, 27 October 2026 + +This review compared `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` +with the previous baseline, `ebdb5de0a`. + +#### What the merged Go work provides + +Pull requests #1372, #1373, #1378, and #1402 provide: + +- the Go JSON catalog; +- configured targets; +- operators; +- reads that Rust verifies against the original request. A prepared read + keeps its request, and Rust derives and checks the selectors and read keys + again from that request when it opens rows; +- adapters for typed documents; +- documentation that runs as code. + +Items #1347 to #1352 are closed. Pull requests #1369 and #1370 integrate +opaque storage in Go. Final Go releases still need the temporary Vitamin C +pseudo-version pins replaced. Rust dependencies stay on 0.8.0. + +#### Cascaded tags and random order + +Pull request #1379 adds cascaded tags: each container's tag covers the tags +of its children. It also randomises the order of stored entries and of +projections. A wildcard `->` returns an unspecified match, not the first +element of the document. + +The format to freeze is the data written after #1379. The header version is +still 1, but that does not mean the data is compatible with the old +prototype AAD. Verification on read still does not authenticate that a +result is complete, that something is absent, or which database row it +came from. + +#### Short-word terms + +Pull request #1397 implements short-word terms that are deterministic and +domain-separated, inside the existing Bloom filter. EQL's pinned match configuration turns them on. This avoids a separate index for short words. + +It is not the random-variant mechanism explored earlier. It does not by +itself resolve the user's concern about frequency and leakage. The native Go +index options still refuse configuration (#1112). The uncommitted research +on ngrams and the Go delivery document are unchanged. + +#### Remaining work + +The refreshed API plan and the JSON operation inventory mark landed work as +implemented. The remaining items include: + +- the formal interoperability specification and conformance corpus; +- opening a child bound to its parent; +- configurable match pipelines in Go; +- direct typed EQL conversion; +- later, separating the roles of context. + +The customer release does not wait for the final Rust traits or crate +moves. + +### Reconciliation with merged work, 7 November 2026 + +This review compared `next` at `e4990e49b` with the previous baseline, +`ffeed7adb`. It read each pull request description, and the code where the +description was not enough. It ran no test suite. + +#### What merged + +- #1404 adds the owned API for one value. #1501 extends it to `Vec`, and + moves the Rust crates to Vitamin C 0.9. +- #1491 makes repeated Rust context extensions append in order, as Go + already did. +- #1460, #1461, #1462, #1469, #1491, #1492, and #1493 add a conformance + corpus that native Rust and the Go guest both run. +- #1444, #1445, #1446, #1451, and #1452 add element-scoped and jsonpath + predicate queries over JSON documents, in the engine, EQL, and Go. +- #1500 documents Go `v0.1.0-rc.3`. That version was tagged on 2 November 2026. +- #1403 fixes findings on the release candidate, and bumps the crate + versions that `stack-encrypt` 0.3.0 needs. + +#### Remaining work on 7 November 2026 + +- the specification around the conformance corpus: versions, schemas, + and the Python and TypeScript runners; +- opening a child bound to its parent; +- configurable match pipelines in Go; +- direct typed EQL conversion; +- freezing the release candidate's stored format; +- later, separating the roles of context. + +#### Differences from the plan + +#1501 encrypts `Vec` as one value. The plan had recommended encrypting +it byte by byte. On 7 November 2026, the user agreed with #1501: a byte +string is one value, and a `Vec` of byte strings is a collection. + +#### What the reconciliation of 27 October 2026 checked + +This review inspected the source and the existing tests. It did not run the +run-time or database test suites again, and it changed no production code. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md new file mode 100644 index 000000000..e7461bda5 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -0,0 +1,345 @@ +# Context scenarios for the Stack Encrypt API + +Date: 7 November 2026. + +This document lists ten ways a record can get its context, each as a +code example with its use case. Together they are the acceptance matrix +for the work item [Context from fields and from the +caller](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) +in the API plan. The plan's section [Query context and stored +context](2026-10-24-stack-encrypt-api-plan.md#query-context-and-stored-context) +gives the rules, and its terms table defines every term used here. + +In short: every part of a context is a scope or a binding. The query +context is all the scopes. The stored context is the scopes followed by +the bindings. Search terms derive under the query context, and the data +key and the AAD derive under the stored context. + +The code below is proposed. None of it compiles on `next` today. The +rules are decided; the attribute spellings, such as `scope(tenant: u64)`, +are illustrative. + +## How to read the examples + +- `scope(tenant: u64)` declares a scope the caller supplies on every call. + `bind(row: Uuid)` declares a binding the caller supplies. +- `scope(field = "tenant")` declares a scope stored in the row, in the + passthrough field `tenant`. `bind(field = "id")` declares a stored + binding in the same way. +- A caller passes its parts as `Scope::new(..)`, `Bind::new(..)`, or + `Scope::new(..).bind(..)`. A query accepts a `Scope` only. +- A stored part is read from the plaintext on encrypt. On decrypt, the + reader takes it from the row. If the reader passes it as well, Stack + Encrypt checks it against the row before it fetches any key. + +## 1. A static context only + +The record's own name and each field's name make up the whole context. +Nothing comes from the caller or the row. This is the default for a +record that has no tenant and no row binding, such as a settings table. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Settings, context = "settings")] +struct EncryptedSettings { + theme: Ciphertext, +} + +let encrypted: EncryptedSettings = settings.encrypt_into(&keyset).await?; +let settings = Settings::decrypt_from(encrypted, &keyset).await?; +``` + +## 2. A scope the caller supplies and does not store + +The caller knows the tenant from the request, and passes it on every +write, query, and read. Nothing about the tenant is stored with the row. +A query for one tenant matches only that tenant's rows. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(tenant: u64))] +struct EncryptedUser { + email: SearchableEmail, +} + +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, Scope::new(tenant)) + .await?; +let user = User::decrypt_from_with_context(encrypted, &keyset, Scope::new(tenant)).await?; + +let by_email = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(tenant)) + .await?; +``` + +## 3. A scope the caller supplies and the record stores + +The tenant is a field of the plaintext, and the record stores it in the +clear. The declaration marks that field as the stored copy, so encrypt +reads it from the plaintext. A reader takes the tenant from the row, so a +batch job can decrypt rows from many tenants without knowing each one +first. A reader that serves one +tenant passes it as well, to check that the row belongs to that tenant. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"))] +struct EncryptedUser { + #[stash(passthrough)] + tenant: u64, + email: SearchableEmail, +} + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; // tenant comes from `user.tenant` + +let user = User::decrypt_from(encrypted, &keyset).await?; // tenant comes from the row +let user = User::decrypt_from_with_context(encrypted, &keyset, Scope::new(session.tenant)).await?; +``` + +The second decrypt is refused with `ContextMismatch` if the row's tenant +is not `session.tenant`, before any key is fetched. + +## 4. A binding stored in the row + +The row's own ID is part of the context, so each ciphertext opens only +in its row. A ciphertext copied into another row fails to open. The ID is +stored in the clear, as any primary key is. A query is not affected, +because a binding is not in the query context. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", bind(field = "id"))] +struct EncryptedUser { + #[stash(passthrough)] + id: Uuid, + email: SearchableEmail, +} + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &keyset).await?; + +// A query covers the whole column. It does not take the row ID. +let by_email = keyset.using(EncryptedUser::EMAIL).query(op::Eq, "alice@example.com").await?; +``` + +## 5. A stored scope and a stored binding + +The tenant and the row ID are both stored in the row. A query for one +tenant matches that tenant's rows, and each ciphertext opens only in its +own row. This is the shape for access intelligence, where each decrypted +value must be tied to the row it came from. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"), bind(field = "id"))] +struct EncryptedUser { + #[stash(passthrough)] + tenant: u64, + #[stash(passthrough)] + id: Uuid, + email: SearchableEmail, +} + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &keyset).await?; + +// Check both stored parts on decrypt. +let user = User::decrypt_from_with_context( + encrypted, + &keyset, + Scope::new(session.tenant).bind(requested_id), +) +.await?; + +let by_email = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(session.tenant)) + .await?; +``` + +## 6. A stored part and a caller-supplied part together + +The tenant is stored in the row. The row ID comes from the caller on +every call, and is never stored. The row ID is known from elsewhere, such +as the request path. A reader passes the row ID, and may pass the tenant +as well to check it. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"), bind(row: Uuid))] +struct EncryptedUser { + #[stash(passthrough)] + tenant: u64, + email: SearchableEmail, +} + +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, Bind::new(row_id)) + .await?; + +// The tenant comes from the row. The row ID must be passed. +let user = User::decrypt_from_with_context(encrypted, &keyset, Bind::new(row_id)).await?; + +// Passing the tenant as well checks it against the row. +let user = User::decrypt_from_with_context( + encrypted, + &keyset, + Scope::new(session.tenant).bind(row_id), +) +.await?; +``` + +## 7. A scope the caller supplies on every call and never stores + +The scope is a value that exists only where the data may be used, such as +a host or application ID. The record is fully stored, but part of its +context is not. So the data opens only on a host that has the value. A +row copied to another host fails to open, and a query from another host +matches nothing. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Secret, context = "secrets", scope(host: String))] +struct EncryptedSecret { + value: Ciphertext, +} + +let host = host_id()?; +let encrypted: EncryptedSecret = secret + .encrypt_into_with_context(&keyset, Scope::new(&host)) + .await?; +let secret = Secret::decrypt_from_with_context(encrypted, &keyset, Scope::new(&host)).await?; +``` + +## 8. An EQL type, with its column as the context + +An EQL type such as `TextEq` is one column's value, stored in a +PostgreSQL column of type `public.eql_v4_text_eq`. Its context is the +table and column, as an `Identifier`. The type stores the identifier in +its `i` field, so a reader takes it from the stored value. A reader that +knows which column it is reading passes the identifier as well. If the +stored identifier differs, the value is refused before any key is +fetched. + +Inside a record, the derive supplies the identifier from the record's +name and the field's context name, so none of this is written by hand. + +```rust +use eql_bindings::{v4::text::{TextEq, TextEqQuery}, Identifier}; + +let column = Identifier::for_column("users", "email")?; + +// Encrypt one value for the column. The identifier is the context, and +// `TextEq` stores it in `i`. +let stored: TextEq = "alice@example.com" + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; + +// Decrypt, taking the column from the stored value. +let email = String::decrypt_from(stored, &keyset).await?; + +// Decrypt, checking that the stored column is the one expected. A value +// from `users.name` is refused with `ContextMismatch` before any key +// is fetched. +let email = String::decrypt_from_with_context(stored, &keyset, Scope::new(&column)).await?; + +// A query for the column uses the same context, so its term matches. +let query: TextEqQuery = "alice@example.com" + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; +``` + +## 9. Bindings stored inside the EQL payload, for access intelligence + +Access intelligence needs to know which value was read, and the facts +about its writing: the address it was written from, and when. Each fact +is a binding. If each one were its own column, a reader would have to +select every one of them to decrypt the value. So the bindings travel +inside the EQL payload instead, in an authenticated metadata field. One column then stores the value and its bindings, so `SELECT body` is +enough to decrypt it. + +The metadata is stored in the clear. It is part of the stored context, so +the AAD and the descriptor cover it. A changed address fails to open. +Each binding is in the descriptor, so ZeroKMS logs it at every key +retrieval. That log is where the access record comes from. + +EQL has no such field today; the plan's [stored format +work](2026-10-24-stack-encrypt-api-plan.md#authenticated-metadata-in-an-eql-payload) +adds it. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Note, context = "notes", bind(field = "id"))] +struct EncryptedNote { + #[stash(passthrough)] + id: Uuid, // the row's primary key column + #[stash(bind(field = "written_from", in = metadata))] + #[stash(bind(field = "written_at", in = metadata))] + body: TextSearch, // one EQL column: ciphertext, terms, and the metadata +} + +let note = Note { id: Uuid::new_v4(), written_from: request.ip(), written_at: now(), body }; +let encrypted: EncryptedNote = note.encrypt_into(&keyset).await?; + +// The payload stored in the `body` column, with the metadata in `m`: +// {"v": 1, "i": {"t": "notes", "c": "body"}, "c": "...", "bf": [...], +// "m": {"written_from": "203.0.113.5", "written_at": "2026-11-07T10:15:00Z"}} + +// `SELECT id, body FROM notes WHERE id = $1` is enough. The reader passes +// nothing: the row ID comes from its column, and the address and the +// time come from the payload. ZeroKMS logs all three with the key. +let note = Note::decrypt_from(encrypted, &keyset).await?; +``` + +ZeroKMS today limits the length of a descriptor, and logs it in the +clear. ZeroKMS version 2 is planned to remove the limit and to encrypt +the access log. + +## 10. The same bindings, in a type of our own instead of EQL + +The shape of scenario 9 does not depend on EQL. A target type of our +own can store the ciphertext and its bindings as one value. It needs only the derives Stack Encrypt already has or plans. + +The bindings are passthrough fields of the target, stored in the clear inside +it. They are part of the stored context. One column stores +the whole value, through Serde, and the reader passes nothing. + +What EQL adds is the PostgreSQL side: a domain for the column, and the +operators that run a search on it. A type of our own has neither. Add a +search term to it only if the application writes the SQL that compares +the term. + +```rust +#[derive(Zeroize)] +struct Written { + text: String, + from: IpAddr, + at: Timestamp, +} + +/// One stored value: the ciphertext and two bindings. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[stash(plaintext = Written, bind(field = "from"), bind(field = "at"))] +struct EncryptedNoteBody { + ciphertext: Ciphertext, + #[stash(passthrough)] + from: IpAddr, + #[stash(passthrough)] + at: Timestamp, +} + +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Note, context = "notes", bind(field = "id"))] +struct EncryptedNote { + #[stash(passthrough)] + id: Uuid, + body: EncryptedNoteBody, // one column, stored as JSON through Serde +} + +let note = Note { id: Uuid::new_v4(), body: Written { text, from: request.ip(), at: now() } }; +let encrypted: EncryptedNote = note.encrypt_into(&keyset).await?; + +// `SELECT id, body FROM notes WHERE id = $1` is enough, as in scenario 9. +let note = Note::decrypt_from(encrypted, &keyset).await?; +``` diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md new file mode 100644 index 000000000..15d708f87 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -0,0 +1,869 @@ +# Stack Encrypt: complete public API examples for review + +Date: 24 October 2026. Implementation status checked on 7 November 2026, +against `next` at commit `e4990e49b`. + +This document tests the proposed public Rust API of Stack Encrypt against +three complete workflows. The code in it is a proposal. The code does not +compile against today's crate. This document is not documentation for a +released version. +Reviewers need to decide whether the API these examples show is the one to +build. + +The questions to settle are listed in +"What to review across the three workflows", near the end. + +The three workflows are: + +1. Encrypt a whole user record as one encrypted value, store it, load it, + and decrypt it. +2. Encrypt only some fields of the record, store it, load it, and decrypt + it. +3. Do the same as workflow 2, but also store a search term for the email. + Then build a query and use it to find the record. + +A fourth program shows how the same search works today with the Encrypt +Query Language (EQL) types, using APIs that already exist. + +## Related documents + +This document is a companion to the +[API review log](2026-10-24-stack-encrypt-api-review.md). Two other +documents cover related ground: + +- The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) + work through nine scenarios. They cover where a context comes from, whether + the context is stored, and what a decrypt call takes as input. The + attributes in that document are proposals, not implemented APIs. +- The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) + covers all the JSON workflows planned for the first release, with Rust, + dynamic, and SQL examples, including extraction and walks. + +The user clarified on 25 October 2026 that this API is an early prototype. +The new API does not need to stay backward compatible with the old APIs. + +On 7 November 2026, the user decided that the stored format must not break +from now on, because the Go customer builds on Go `v0.1.0-rc.3`. The stored +format is what is written to the database, such as ciphertext envelopes and +search terms. The APIs can still change. This replaces the decision of 25 +October 2026 for stored data. See the [API +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). + +## Terms used in this document + +The Stack Encrypt crate is `packages/stack-encrypt`. This document uses +these terms from it: + +- **Keyset**: a set of keys in ZeroKMS, the CipherStash key service. + Every encrypted value is sealed under one keyset, and the sealed value + records which keyset that was. +- **Data key**: the key that encrypts one value. Stack Encrypt requests data + keys from ZeroKMS in batches. +- **Context**: data that a ciphertext is tied to, such as + `("users", "email")` or a record id. Decryption needs the same context. + Search terms are also derived from it. See [Context and + AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad). +- **AAD**: additional authenticated data. Encryption checks it for one ciphertext, but does not encrypt it. Stack Encrypt builds it from the + context. +- **Leaf**: one encrypted value. "Sealing" a leaf means encrypting and + authenticating it. +- **Envelope**: the complete stored form of a ciphertext. It includes the + sealed bytes and the metadata needed to fetch the key again, so the caller + does not store anything separately. +- **Target**: a Rust type that a value is encrypted into or decrypted from. + The target decides which ciphertext and which search terms are produced. +- **Passthrough field**: a field copied unencrypted from the plaintext type + to the encrypted type, and back again on decrypt. +- **Plan**: the internal description of which fields a conversion encrypts, + indexes, or passes through. Today a caller can build one with + `Plan::fields()`. +- **Equality term**: a value stored next to a ciphertext that lets a query + find an exact match without decrypting. +- **Zeroize**: overwrite memory that held plaintext, so the plaintext does + not stay in memory after use. `Zeroize` is the Rust trait that does this. + +## Decisions already made, and what these examples propose + +The first parts of this section record decisions made on 25 October 2026. +The table after them lists every API surface in the examples and its status. + +### Ownership decision (25 October 2026) + +`encrypt` and `encrypt_into` take ownership of (consume) their inputs. This +includes the variants that take a context. Their implementations must guard +the owned plaintext and temporary buffers, so they are cleaned up on +success, on error, and when the request is dropped. Passthrough fields are +deliberately moved into the output. + +Other parts of this decision: + +- The capabilities for a single ciphertext are named `OpaqueEncode` and + `OpaqueDecode`. +- Plaintext types must implement `Zeroize`. No new cleanup trait is planned. +- `Serialize` alone does not establish the cleanup contract for + `OpaqueBlob`. + +The candidate programs below were written before the `Zeroize` requirement. +When making them executable, update their source derives, and remove any +reuse of input that has been consumed. + +### Decrypt decision (25 October 2026) + +Stack Encrypt keeps two ways to decrypt: + +- the direct `keyset.decrypt` and `decrypt_with_context`, for an opaque + ciphertext; +- the destination-directed `T::decrypt_from` and + `decrypt_from_with_context`, for a declared target. + +A successful call returns an ordinary owned `T: Zeroize`. The caller is then +responsible for cleaning it up. Stack Encrypt guards intermediate plaintext. + +The API must support results that are wiped when they are dropped, if the +caller asks for them, without forcing a wrapper on every caller. `Zeroize` +alone does not wipe a returned value when it is dropped. These are accepted +API contracts. This document does not claim that the prototype already +implements them. + +### Collection decision (25 October 2026) + +The direct `keyset.encrypt(value)` takes one single-value input and produces +one `Ciphertext`. It does not encrypt a collection element by element. + +To encrypt each element of a collection in one batch, use `EncryptInto`, +for example: +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. + +`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` produces one +ciphertext for the encoded collection instead. + +A byte string is the one exception. `Vec` and `[u8; N]` encrypt as one +value, as pull request #1501 built them. The user agreed this on 7 November +2026. The [API plan](2026-10-24-stack-encrypt-api-plan.md#encrypt-a-byte-string) +shows it. These calls need the proposed opaque ciphertext and support for +targets that take no context. + +### Status of each API surface + +| Surface | Status in this review | Implementation on 7 November 2026 | +| --- | --- | --- | +| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction | Done for one value (#1404) | +| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | Done for one value and for `Vec` (#1404, #1501) | +| No public `encrypt_as` in these examples | Agreed direction | Still open. `encrypt_as` is still public | +| `#[stash(passthrough)]` on output fields | Accepted in principle | Still open. In progress in #1504 | +| A fixed context on a scalar output derive | Agreed direction; not required by these examples | Still open | +| `OpaqueBlob` | Agreed wrapper direction; provisional name | Still open. In progress in #1503 | +| `Ciphertext` | Proposed name for an opaque single-value ciphertext | Done (#1404) | +| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the canonical sealed-leaf encoding | Done (#1404). The envelope is unchanged | +| `Ciphertext: Serialize + Deserialize` | Proposed: Base64 for human-readable formats, bytes for binary formats | Done (#1404) | +| Encoded ciphertext field targets | Accepted; final names remain for the naming pass | Still open | +| `Ciphertext` as an encryption or decryption target | Proposed single-value target | Done (#1404). `Vec` is one value (#1501), agreed on 7 November 2026 | +| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path | Still open. In progress in #1503 | +| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair | Done for one value (#1404) | +| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair | Done for one value and `Vec` (#1404, #1501). Records are in #1504 | +| `SearchableEmail` in example 3 | Application-defined target | Still open. #1504 turns example 3 into a test | + +The first three Rust programs use the proposed `OpaqueBlob`, `Ciphertext`, +and passthrough APIs, or some of them. Each is a complete candidate program. +None is claimed to compile against today's crate. The names and contracts in +the table can still change where they are marked proposed or provisional. + +### Notes on the proposed types + +The proposed opaque ciphertext deliberately carries no plaintext type. The +type that the decrypt call expects selects how the plaintext is decoded. A +typed wrapper is still an open alternative. + +The ciphertext's byte decoder +only parses a stored envelope. Authentication happens when it is decrypted. +The new Rust implementation and the dynamic implementation must agree on +the context and leaf encodings. On 24 October 2026 they did not need to keep the prototype's encoding. Since 7 November 2026, they must keep reading the encodings that Go rc.3 writes. + +The trait name `DecryptFrom` may surprise readers. The trait supplies the +convenience functions, while `#[derive(DecryptInto)]` describes how an +encrypted output recovers its plaintext. These examples keep that existing +split visible, rather than redesigning it without saying so. + +### What the examples leave out, and why + +The examples deliberately keep plans and operation descriptions out of the +application code. The application code shows only the public calls. + +Each main example is a separate program, with all imports and types shown. +All of them use the same user fields and made-up data. `created_at` is an +`i64` Unix timestamp in seconds, so that configuring a date library does not +distract from the encryption API. + +Files stand in for a database. The programs do encode, write, read, and +decode their stored forms. The files are written to a temporary directory. +The directory is removed when the program exits. + +## Setup shared by the programs + +The proposed programs use these crates: + +- `stack-encrypt`, with its default `http` feature; +- `serde`, with `derive`; +- `serde_json`; +- `tempfile`; +- `tokio`, with `macros` and `rt-multi-thread`. + +No published version of `stack-encrypt` supplies all the proposed APIs yet. +This draft therefore gives no Cargo manifest, because one would suggest the +programs can be installed and run. + +`StackCipher::new()` uses the configured ZeroKMS credentials. Those come +from a CLI login, or from these environment variables: `CS_CLIENT_ID`, +`CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. + +The second client in each program stands for a later reader that uses the +same authorised configuration. The examples select the default keyset to +keep them short. A tenant keyset that the application has already selected +can be used in its place. + +Decryption uses the client, because the sealed value records its keyset id. +The reader still needs access to the matching keys and the correct context. +The examples do not rely on in-memory fake keys surviving a reload. + +## 1. Encrypt the whole user through an explicit blob wrapper + +**What this example shows:** how to store one encrypted document. All four +fields, including the id and creation time, are inside one encrypted +serialization. The stored ciphertext contains no fields that can be read +separately, and no search terms. + +`OpaqueBlob::new(user)` explicitly asks for that treatment. Stack Encrypt +owns the serialization and the matching deserialization, so each +application does not have to choose and call a codec. The proposed wrapper +takes ownership of the input. `new` only wraps the value. The +encryption call returns any serialization error. + +The context names the application's document schema +(`"user-document-v1"`). The context also binds this document to a record id +that the application knows from outside the ciphertext. + +In this draft, the wrapper accepts `Serialize + Zeroize` on write and +`DeserializeOwned + Zeroize` on read. This does not claim that every Serde +implementation can meet a constant-time contract. + +The first version of the +wrapper has one versioned binary codec that the library chooses. The caller +cannot choose a codec. Choosing and specifying that codec is still design +and implementation work. This document does not claim that any binary format +is implemented, or that any is constant-time. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{nonempty, Ciphertext, OpaqueBlob, StackCipher}; +use std::error::Error; + +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let expected_id = 42_u64; // Supplied by the application's record lookup. + let user = User { + id: expected_id, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + let expected = user.clone(); // Retained only for the round-trip assertion. + + // The wrapper requests one blob and lets Stack handle serialization. + let sealed: Ciphertext = keyset + .encrypt_with_context( + OpaqueBlob::new(user), + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + + // Persist the complete ciphertext envelope as binary bytes. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.ciphertext"); + std::fs::write(&path, sealed.to_bytes())?; + + // Later: reconstruct the ciphertext from storage using a new client. + let reader = StackCipher::new().await?; + let stored_bytes = std::fs::read(&path)?; + let loaded = Ciphertext::from_bytes(&stored_bytes)?; + + // Stack authenticates/decrypts first, then decodes the blob into User. + let opened: OpaqueBlob = reader + .decrypt_with_context( + loaded, + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + let restored = opened.into_inner(); + assert_eq!(restored, expected); + Ok(()) +} +``` + +### What the API must guarantee for example 1 + +- The serialized bytes are one encrypted leaf under one data key. +- The envelope contains the metadata needed to fetch the key again. The + caller never needs to save an initialisation vector (IV) or a key tag + separately. +- A different context must fail to open the ciphertext. +- Stack Encrypt owns the matching encode and decode pair, and the contract + for keeping them compatible. +- The blob format's identifier and version must be authenticated, for + example by putting them inside the encrypted payload. Decoding then does + not depend on guessing a codec, or on format metadata that is not + authenticated. +- The blob can reuse the canonical outer ciphertext envelope. The blob + payload is a new, explicit format. It must not quietly reinterpret raw + bytes that were encrypted before. +- Changes to the application's schema still need explicit handling. A codec + version does not migrate changes to `User` automatically. + +### Encrypting without an application context + +Without an application context, use the form +`keyset.encrypt(OpaqueBlob::new(user))`. A matching no-context +`reader.decrypt(loaded)` is accepted, so that encrypt and decrypt match. +Since #1404, `keyset.decrypt(ciphertext)` takes one argument for an opaque +value. `OpaqueBlob` itself is not merged. +This document uses an explicit context in the full example so that the +saved format is unambiguous. + +### What the wrapper means + +`OpaqueBlob` is the request on the plaintext side, and the decoded +value. It is not the stored ciphertext type. `Ciphertext` is still the +stored envelope. `into_inner()` returns an ordinary `User`. + +The wrapper does not by itself promise to zeroize arbitrary input or +returned types. The codec should keep temporary plaintext buffers in memory +that can be wiped. + +`#[derive(Encrypt)]` on `User` currently means a structural tree of +ciphertexts, one for each field. That derive cannot replace wrapping the +value as one blob. + +### Timing contract to investigate + +The goal is serialization whose timing does not depend on the secret +contents, in both directions. An explicit policy sets what may vary in +lengths and shape. + +Constant time does not mean that messages of different sizes take the same +elapsed time. The policy must declare which properties may vary, and treat +them as public. In general, neither control flow nor memory access may +depend on secrets. That applies outside the cryptographic primitive too. +See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). + +A binary codec that the library owns gives us control over the +implementation, but does not give the property by itself. Each of these can +add work that depends on the values: + +- strings; +- collection lengths; +- enum variants; +- validation errors; +- memory allocations; +- Serde code that users write. + +Supporting arbitrary `Serialize` and `Deserialize` implementations cannot +guarantee constant-time behaviour for the whole operation. + +Before claiming such a guarantee, define these things: + +- the supported types; +- which shape and length information may be public; +- the restrictions on custom implementations. + +A stricter set of supported types may need bounded shapes, fixed-width encodings, padding, +and controlled encoders and decoders, with verification at the +implementation level. Padding the ciphertext to a fixed size does not by +itself make the serialization before it constant-time. Authentication must +finish before the plaintext blob is decoded. + +This section states a design requirement to investigate. These example +programs make no such guarantee. This section is not the result of a +constant-time audit. + +**Candidate to investigate: `serdect`.** `serdect` is already a dependency +of `stack-encrypt`. The investigation needs to: + +- review its supported encodings and its exact constant-time guarantees, + including its timing when decoding and on errors, and its assumptions + about lengths; +- assess where it can help serialize ciphertexts and bytes; +- assess whether any of its techniques apply to the `OpaqueBlob` codec. + +Do not assume that its guarantees for encoding bytes extend to serializing +arbitrary objects through Serde. This is a follow-up investigation. It is +not a finished assessment. It does not select a codec. + +## 2. Encrypt selected fields into a chosen output layout + +**What this example shows:** how to store a record in which `id` and +`created_at` stay as plaintext. `name` and `email` are each encrypted +separately. The output type declares the layout. The two +encrypted fields need no search indexes. + +The proposed Serde implementation for `Ciphertext` lets persistence be +built from ordinary parts. Ordinary Serde derives on the output type encode +the plaintext fields as usual, and encode the ciphertext fields as Base64 +envelopes. The application does not walk any `Node` values by hand. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let user = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + // The output type selects the conversion and declares its own contexts. + let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + assert_eq!(encrypted.id, user.id); + assert_eq!(encrypted.created_at, user.created_at); + + // Persist the typed output, including both plaintext and ciphertext fields. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: deserialize the same output layout and recover the plaintext type. + let reader = StackCipher::new().await?; + let loaded: EncryptedUser = serde_json::from_slice(&std::fs::read(&path)?)?; + let restored: User = User::decrypt_from(loaded, &reader).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +### Context and execution in example 2 + +`name` uses the structured context `("users", "name")`, and `email` uses +`("users", "email")`. The two fields share the keyset, but each encrypted +field gets its own data key. One conversion sends the key requests for both +fields in one batch. The decrypt derive rebuilds `User`: it opens the +ciphertext fields and copies the passthrough fields back. + +### What passthrough fields do and do not protect + +Copying `id` does not bind the sealed fields to it. Both plaintext fields +stay unencrypted and unauthenticated. That is the passthrough behaviour that +was asked for. It does not promise integrity for the whole record. The +example's `name` and `email` contexts tell columns apart, not rows. + +If the application wants each ciphertext bound to its row, it can use the +existing entry point `user.encrypt_into_with_context(&keyset, expected_id)` +and the matching +`User::decrypt_from_with_context(loaded, &reader, expected_id)`. The +contexts that take effect are then `(("users", "name"), expected_id)` and +`(("users", "email"), expected_id)`. The reader supplies the expected id +from its record lookup. This is an explicit variation. The passthrough +attribute does not cause it. + +### Existing `context_field` support + +The derive already has `#[stash(context_field)]`. Its current behaviour is +to store the context the caller supplied in an output field, and to use it +again when decrypting. It does not read an id from the plaintext `User`. + +`context_field` also supplies the complete context, so the macro refuses to +combine it with a literal `context = "users"`. The current `struct = User` +form requires that literal. As a result, the attribute cannot be used today +in example 2's field-by-field declaration. + +### Where context comes from, compared with where it is stored + +Taking context from plaintext fields and carrying context into the encrypted +output are separate choices. An extended derive could read `User.id`, +combine it with the field contexts declared on the type, and also copy it +into `EncryptedUser.id`. That would bind each ciphertext to its row without +passing the id twice. The same model should handle several source fields, +such as `(tenant_id, id)`, with explicit ordering and grouping. + +There are other combinations. Context can be: + +- passed as an argument and stored; +- passed as an argument and not stored; +- taken from plaintext fields and not stored. + +In either case where the context is not stored, the reader needs the same +context from somewhere else before decrypting. The API should let callers +express the source and the storage separately. Whether one annotation can +ask for both is still a syntax choice. Today's `context_field` only covers +the case where the caller supplies the context and it is stored. + +Suppose the context is taken from plaintext fields and also stored. A +reader could then use the stored id. To reject a whole record that was moved +to another place, the reader could also supply an expected id from +elsewhere. Changing the stored id alone must make decryption fail. The values needed to rebuild the +context must be available without first opening the ciphertext that +requires them. + +### Review points for example 2 + +The declaration is short, but it requires an explicit `EncryptedUser` type. +A future derive on the source type could generate that type. If it does, it +should generate this same layout and the same context behaviour. + +Reviewers also need to decide whether `User::decrypt_from(...)` is the +public spelling for decryption that we want. That decision comes before +adding a competing no-context convenience, `loaded.decrypt_into(...)`. + +## 3. Use the same layout model with a searchable email target + +**What this example shows:** the same plaintext fields as example 2, with +the name encrypted as before. The email is stored as its ciphertext plus an +equality term. Search terms are made from the original typed plaintext, +through the operation the target declares. They are not made from the JSON +encoding used to store the output. + +`SearchableEmail` is a scalar target that the application defines. It uses +the context that the outer record supplies: `("users", "email")`. It does +not declare a fixed context of its own. It does not add another `email` part +to the context. + +The member names `ciphertext` and `equality` only describe the output +layout. They do not add parts to the cryptographic +context. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::{ + nonempty, Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, + StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct SearchableUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let alice = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + let bob = User { + id: 43, + name: "Bob".to_owned(), + email: "bob@example.com".to_owned(), + created_at: 1_700_000_100, + }; + + // One collection conversion batches key generation across both users. + let users = vec![alice, bob]; + let encrypted: Vec = users.encrypt_into(&keyset).await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("users.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: a query uses the same keyset, field context, and equality operation. + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let wanted_email = "alice@example.com".to_owned(); + let query: EqualityTerm = wanted_email + .encrypt_into_with_context( + &query_keyset, + nonempty!("users").with("email"), + ) + .await?; + + // A file-backed comparison demonstrates matching without a SQL adapter. + let loaded: Vec = + serde_json::from_slice(&std::fs::read(&path)?)?; + let matches: Vec = loaded + .into_iter() + .filter(|row| row.email.equality == query) + .collect(); + assert_eq!(matches.len(), 1); + + // The query term cannot decrypt. Open the matched ciphertext rows in a batch. + let restored: Vec = Vec::::decrypt_from(matches, &reader).await?; + assert_eq!(restored.as_slice(), &users[..1]); + Ok(()) +} +``` + +### What the query in example 3 shows + +The filter scans the stored encrypted records in memory. It does not claim +that a database index is used. The filter shows the whole query contract: + +1. Make the query term. +2. Compare it with the stored term. +3. Open the matching ciphertext. + +An equality match does not authenticate the stored record. The decrypt +operation authenticates the record's encrypted fields. Passthrough fields +work as they do in example 2. + +### How collection conversion batches key requests + +Stack Encrypt already supports collection conversion, through its blanket +implementation `EncryptFrom> for Vec` when `T: EncryptFrom` and +the target's context can be cloned. Since #1501, the owned API has the same +batching for `Vec` into `Vec`. The batching counts below apply to +both. + +The conversion collects every row's +pending operations before sending any of them. The two users need four data +keys, one each for name and email in each row. All four are issued through +one batched `generate_keys` call. Equality terms use the current local +backend for index keys. + +This count is about the encryption work only. It +does not include setting up or authenticating the client. + +One logical `generate_keys` call is not always a single HTTP request. The +KMS (key management service) client splits a batch into chunks, according +to its configured limit on the number of keys. This example's four keys fit +in one normal request. A larger collection may need several requests, with +no change to the public call. + +The matching collection decrypt implementation also batches key retrieval +across rows. It makes one `retrieve_keys` call for each keyset found in the +ciphertexts. An empty collection needs no data-key call. The conversion +keeps the collection's order. It does not add any authentication of the +collection as a whole. + +### Three different ways to encrypt a vector + +This target-directed vector is a vector of separate output records. It is +different from both of these: + +- encrypting a plaintext vector into a single `StackCipherText` sequence + tree; +- wrapping the vector in `OpaqueBlob`, to encode and encrypt the whole + collection as one value. + +Batching is a property of how the work is run. It does not change the +layout that was chosen. + +### Limits of the search context + +This example uses the same field context for every row, so that one query +term can search all of them. Adding a row id to that context would also +limit the term to that row. How to bind each ciphertext to its row while +keeping a search context that covers the whole column is still an open +design question. These examples do not promise it. + +In the same way, an application that queries across keysets needs a +separate query term for each keyset. The examples do not imply any +comparison across keysets. + +### EQL targets: a concrete adapter boundary + +The example above is a generic searchable layout. It is **not an EQL +payload**. For EQL, use the adapter's types, metadata, serialization, and +query target. Here, the adapter is the `eql-bindings` crate. The crate +converts between Stack Encrypt values and the stored EQL format. + +For example, today's `eql_bindings::v4::text::TextEq` already contains an email +ciphertext plus an equality term. `TextEqQuery` is the matching query type. + +The program below is a separate example for one field. It uses existing APIs +and avoids the proposed `Ciphertext` and passthrough attributes. It needs: + +- `eql-bindings`, with its `stack-encrypt` feature; +- `stack-encrypt`; +- `serde_json`; +- `tempfile`; +- `tokio`. + +The program is marked `no_run` because it needs live credentials. The program +has been checked against the source signatures. It has not been compiled or +run. + +```rust +use eql_bindings::{ + v4::text::{TextEq, TextEqQuery}, + Identifier, +}; +use stack_encrypt::{DecryptFrom, EncryptInto, StackCipher}; +use std::error::Error; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let column = Identifier::for_column("users", "email")?; + let email = "alice@example.com".to_owned(); + + let encrypted: TextEq = email + .encrypt_into_with_context(&keyset, &column) + .await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("email.eql.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let query: TextEqQuery = email + .encrypt_into_with_context(&query_keyset, &column) + .await?; + let loaded: TextEq = serde_json::from_slice(&std::fs::read(&path)?)?; + assert_eq!(loaded.hm, query.hm); + + // Supply the expected column, so stored identifier metadata is checked too. + let restored: String = loaded.decrypt_into(&reader, &column).await?; + assert_eq!(restored, email); + Ok(()) +} +``` + +Note added on 7 November 2026: this program no longer uses existing APIs +as written. Since #1404, the crate-root `EncryptInto` and `DecryptFrom` are +the owned traits. `TextEq` implements only the structural declarations. So +the program needs `stack_encrypt::target::legacy::{DecryptFrom, +EncryptInto}`, or `keyset.encrypt_as`, as `eql-bindings` uses. This was +checked by reading the source, not by compiling. + +The program passes the column by reference, as `&column`, so it does not +clone it for each call. That is the target form. On `next`, every context +is taken by value, so the program would need `column.clone()`. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#pass-context-by-reference) +lists this as remaining work. + +`TextEq` includes its EQL version, its identifier, the ciphertext, and the +equality term. Its context type is `NonEmpty`. The application +supplies the typed identifier for the expected table and column, on both +writes and reads. Decryption checks it against the stored identifier before +opening the ciphertext. + +The `v4` namespace is deliberate: this Rust producer targets EQL v4. +The TypeScript integrations use EQL v3. + +### Gap before `TextEq` can replace `SearchableEmail` + +There is a gap before `TextEq` can replace `SearchableEmail` in the +outer derive. That derive currently passes a `CallerContext`. The EQL target +needs a `NonEmpty`, and stores its identifier in the output. + +The conversion between the two contexts must be designed explicitly. The +design includes whether extensions that the caller adds to the context can +be represented. The EQL example shows what that adapter must keep. It does +not invent an implicit conversion, or an SQL client API that does not exist. + +PostgreSQL DDL (the statements that create tables and indexes), parameter +binding, and whether queries use the index belong in a separate adapter +example. That example is to be added once the boundary is settled. + +## What to review across the three workflows + +Each row of this table is one design question. The second column gives the +answer these examples chose. + +| Question | Concrete choice made by these examples | +| --- | --- | +| What is one encrypted unit? | One `OpaqueBlob` encoding in example 1; one selected field in examples 2 and 3 | +| How is the layout selected? | Ordinary encrypt returns an opaque value; `EncryptInto` selects a declared output | +| How is persistence exposed? | Canonical bytes for one ciphertext; Serde composed over a typed layout | +| How is plaintext recovered? | Decrypt and decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | +| How do indexes fit? | Additional field targets under the same layout and execution model | +| Who chooses context? | The application for a blob; the layout plus optional caller extensions for fields; an explicit identifier for EQL | +| Do examples need plans or visitors? | No; those remain implementation and advanced composition mechanisms | +| Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | + +### Main open questions + +The examples leave these questions open: + +- What is the opaque ciphertext called, and what is its storage contract? +- What is the blob codec, and what is its timing contract? +- How is an EQL context adapted? +- Does a derive on the source type generate the output type? + +None of them requires bringing back a public `encrypt_as`. + +## Source checks and validation limits + +The draft was checked against the existing code: + +- [target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs); +- [context composition](../../packages/stack-encrypt/src/target/context.rs); +- [derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs); +- [decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs); +- [canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs); +- [EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). + +`rustfmt` parsed all four programs without errors, using Rust edition 2024. +All local links resolve. The code fences and whitespace were checked. These +checks confirm the syntax and the document's integrity, not that the types +are correct. + +This document does not implement the candidate APIs. The three proposed +programs cannot yet be type-checked against the repository. No live +encryption, database runs, or cryptographic compatibility tests are claimed. +Once the API surface is agreed and implemented, turn these programs into +executable examples. Then verify the storage round trips, the matching +queries, and that a wrong context fails. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md new file mode 100644 index 000000000..38c2ef186 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -0,0 +1,428 @@ +# JSON tree: operation inventory and release contract + +Updated 7 November 2026 against merged `next` at commit `e4990e49b`. The +previous update was on 27 October 2026, at `ffeed7adb`. This document accompanies the +[API plan](2026-10-24-stack-encrypt-api-plan.md). + +## Summary + +The Go JSON feature is implemented, through #1372, #1373, #1378, and #1402. +This document lists every JSON operation the release needs, with what is +supported now and what work remains. Each operation has an ID from J01 to +J20, used in the table of operations below. Two operations still have +gaps: J14 and J17. The SQL types and functions come from Encrypt Query +Language (EQL). + +Since 27 October 2026, two operations have been added as J21 and J22. +J21 tests several conditions on one array element. J22 sends a jsonpath +predicate as one query parameter. Both are in the engine, EQL, and Go. The +`eql-json-v1` conformance profile, from #1492, enrols the JSON operations +in Rust, Go, and PostgreSQL tests. + +From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not +break. The stored format is what is written to the database, such as JSON +tree entries and their search terms. See the [API +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). + +The Rust example in this document uses the proposed public derive and +facade. That example is not executable code for the current engine. The +[refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and its findings on the engine and Go give the exact source evidence. + +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| EQL | The SQL in `packages/eql` that stores and queries encrypted values | +| Entry | The encrypted record for one node of a document. A document is stored as one entry for each object, array, and scalar in it | +| Selector | A keyed pseudorandom value derived from a node's path, with every array position replaced by the wildcard. Every element of an array shares one selector | +| Token | A value derived from a node's concrete path. The database uses it to find that node and the entries beneath it | +| Needle | The JSON value that a containment query looks for | +| EqAt | Equality of a scalar at a path: `EqAt` in Go, `eq_at` in the dynamic engine | +| Containment projection | The set of selector and value pairs, one for each node, stored beside the entries. PostgreSQL's `jsonb @>` compares two of these sets to test containment | +| Subtree read | A read that returns a node and every entry beneath it, through `eql_v4.json_tree_subtree`, and opens them as one value | +| Cascaded tags | Each container's authentication tag covers the tags of its children, so a change anywhere below a node stops that node opening. Added by #1379 | +| Sealed root | The default. The root entry's ciphertext is the whole document, and the root token finds no other entry | +| A JSON document whose root is not sealed | The root is stored as an ordinary container entry (`sealed_root` false). Called "open root" in the code | +| Walk budget | The most entries a subtree read returns, counted across all its tokens, before the database falls back to the root | +| Root fallback | Past the walk budget, the database returns the sealed root alone. The client opens the whole document and picks out the requested values | +| Wildcard bound | How many elements each `[*]` in a read path expands to. One extra probe at the bound detects a longer array | +| Reads that Rust verifies against the original request | Prepared reads: Rust derives the request again when the rows are opened, and checks every row against it | +| Post-cascade format | The JSON storage and search format after #1379 added cascaded tags. Go rc.3 writes it, so it must stay compatible | +| Conformance profile | A named, versioned set of test cases for one area, such as `eql-json-v1`. Each language runs the same cases. The files are in `conformance/stack-encrypt/v1/` | +| Facade | The proposed public Rust API of `stack-encrypt` | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | + +## Current names + +| Role | Current public name in Go, the dynamic engine, and SQL | +| --- | --- | +| Configured target | `JsonSearch` | +| Encrypted document | `eql.JSONSearch`, `public.eql_v4_json_search` | +| Operand for containment and EqAt | `eql.JSONSearchQuery`, `eql_v4.query_json` | +| Operand for ordering | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | +| Encrypted entry | `eql.JSONEntry`, `public.eql_v4_json_entry` | +| Prepared selector and prepared subtree read | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | +| Result of opening one value | `JSONValue{Found, JSON}` | +| Operand for several conditions on one array element | `eql.JSONSearchElementsQuery`, `eql_v4.query_json_elements` | +| Operand for a compiled jsonpath predicate | `eql.JSONSearchPredicateQuery`, `eql_v4.query_json_predicate` | + +The handwritten Rust EQL adapters keep their `JsonTree*` names. The +internal engine type `JsonDocument` is not the Go ciphertext type. +`eql.JSON` is the type for storage only. Do not use the old domains +`query_json_tree` and `eql_v4_json_tree` in current recipes. + +## Context and scope policy + +The SDK resolves the target options configured for a field into native +`JsonOptions`. The SDK uses the same options for writes, queries, and +reads. The released configuration takes a document's context from its column, and its keys from the selected keyset. + +Separate row binding and search scope are accepted future API work. Row +binding and search scope are not supported as EQL context inputs today. +So: + +- reject a declaration that asks for them; +- do not add row IDs to the existing shared context. + +A query does not isolate rows only because the ciphertexts it returns +authenticate. + +## Required operations and current support + +| ID | Operation | Current support | Limits and remaining work | +| --- | --- | --- | --- | +| J01 | Encrypt an object, array, scalar, or null | Implemented. Raw JSON supports every kind of root. Typed adapters support declared structs, maps, and slices | None listed | +| J02 | Decrypt a complete document | Implemented, to raw JSON text or a declared Go type | None listed | +| J03 | Write and read in batches | Implemented, with a shared executor and chunking by the provider | None listed | +| J04 | Contains | Implemented: `Contains`, `contains`, and `@>` with `query_json` | Generated code checks the needle and refuses some shapes | +| J05 | Contained by | Implemented: `ContainedBy`, `contained_by`, and `<@` | None listed | +| J06 | Scalar equality at a path | Implemented: `EqAt`, `eq_at` | Compares value selectors, not ordering terms. J21 keeps several tests on one array element | +| J07 | The four ordering predicates | Implemented in runtime paths and in `At.` accessors | Ordered number and string paths only | +| J08 | Inclusive range | Implemented: `Between`, `between_at`, with named lower and upper operands | None listed | +| J09 | Sort | A selector and explicit SQL | Sorting at a wildcard path needs a defined reduction. Do not assume the first array element | +| J10 | Path exists | A selector and `json_tree_path_exists` | None listed | +| J11 | Extract one match | A selector and `->`, or the first result of a path query | At a wildcard path, which element is returned is unspecified | +| J12 | Extract as encrypted text | `->>` | The result is still encrypted entry JSON, not plaintext | +| J13 | Enumerate all matches | An SQL path query | The supported way to get grouped plaintext is a subtree read with a wildcard bound | +| J14 | Open extracted scalars | `JSONSelection.Open` opens one optional entry for each document group | Groups with more than one entry for a selector are refused. Do not label them supported | +| J15 | Prepare a read at a position or wildcard | Implemented: `Subtree`, `subtree`. Returns a token list, and keeps the trusted request | None listed | +| J16 | Fetch and open a subtree | `json_tree_subtree(document, tokens, budget)`, then `JSONRead.Open` or `OpenOne` | None listed | +| J17 | Immediate children | `Children` prepares a concrete token. SQL returns the child entries | Opening child scalars bound to their parent is not implemented | +| J18 | Count children | A `ChildCount` token. SQL returns a count that can be NULL | The count is not an authenticated proof of completeness | +| J19 | Walk budget and root fallback | Implemented. After root fallback, the client applies the selection originally requested | None listed | +| J20 | Comparisons between stored values | Existing document containment and entry ordering | Only within a compatible column, keyset, and storage format | +| J21 | Several conditions on one array element | Implemented in the engine and EQL (#1445), and in Go as `Elements..Match` (#1451) | Needs ordered paths. Uses no index | +| J22 | A jsonpath predicate as one parameter | Implemented in the engine and EQL as `match` and `@@` (#1446), and in Go as `Match` (#1452) | Supports a subset of jsonpath. Uses no index | + +Status on 7 November 2026: no merged pull request changes J14 or J17. The +`eql-json-v1` profile enrols tests for J01 to J08, J10, J11, J15, J16, and +J19, with J21 and J22. Its requirement IDs are not the J numbers. I made this mapping from that conformance profile's contract. I did not check it test by test. + +In J21, equality compares folded ordering terms, not exact values. In J22, the EQL reference documents five cases where the answer differs from `jsonb_path_match`. + +### Open requirements for J13 and J17 + +The release still needs every operation in the table. Closed tracking +issues do not mean an opener bound to the parent exists. + +For J17, build an opener for child scalars that is bound to their parent. +Do not pass child rows to an unrelated selector or subtree request. Until +the opener exists, fetch and open the parent's subtree to inspect its +children. Document that this transfers different data and leaks +differently. + +For J13, enumerating all matches in SQL is not the same as the supported +interface. The supported interface returns plaintext from a bounded +subtree read. + +## Semantics that changed or are now implemented + +- **Random order.** #1379 randomises the stored order of entries, the + ordinals of object members, and the order of the containment + projection. `->` at a wildcard selects an unspecified element. For + ordering or sorting, use all matches with an explicit `EXISTS`, `MIN`, + or `MAX` rule. +- **Cascaded tags.** Each container's tag covers its children's + ciphertext tags. A container entry with children, opened on its own, + fails with `JsonNotALeaf`. The same error is returned for an altered + entry. An empty container entry authenticates, but fails with + `OpenedType`. Every ordinary container needs a subtree read. A sealed + root that the caller asks for can still open as a complete document. +- **No positional paths in queries.** The dynamic engine now rejects + positional paths for selectors, ordering, and EqAt. Concrete positions + such as `[n]` are for subtree reads. A wildcard selector addresses every + path that differs only in its array positions. Two separate EqAt + predicates can match different elements of one array. One containment + needle keeps them matched to the same element. So does J21. +- **Native `jsonb` functions give wrong answers.** The encrypted column is + a PostgreSQL domain over `jsonb`. So PostgreSQL's own `jsonb` functions + accept it, and run over the stored envelope, not the document. For + example, `jsonb_path_match(col, ...)` returns false for every row, with + no error. #1444 documents this and does not block the functions. +- **Strict checks on JSON input.** The Rust parser now rejects duplicate + keys and out-of-range integer literals before key I/O, that is, before + any request to the key provider. For typed Go values, Go encodes first, + with `encoding/json`. That package replaces invalid UTF-8. The parser + checks raw JSON directly. Neither path promises to keep the JSON text + byte for byte. +- **NULL, missing, null, and empty are different results.** SQL NULL, a + missing selection, JSON null, and an empty container are different. `JSONValue.Found` records + whether the selection was present. `OpenOne` rejects wildcards. Subtree + reads that return several values return the requested values in + document order. +- **Reads that Rust verifies against the original request.** A prepared + read keeps the cipher, the field definition, the path, and the wildcard + bound. `Open(ctx, rows)` asks Rust to derive the request again and + verify it. No serialised key handle local to the guest is needed. Rows + cannot supply their own expected path. Keep one group for each database + document. +- **What cascaded tags do not prove.** Cascaded tags make verification + stronger beneath the node at the top of a returned read. Cascaded tags + do not authenticate the whole result set, the absence of a value, freshness, + or that the row is the intended database row. The database can still + withhold whole results or probes. +- **Defaults for reads.** The wildcard bound defaults to 64, with probes + at the bound and a cap of 8,192 tokens. The walk budget is separate, and + also defaults to 64. A walk budget of 0 turns off root fallback. A JSON + document whose root is not sealed cannot fall back to the sealed root. + The walk budget does not limit the size of every response. +- **Header version.** The header version is still 1, although the cascade + changed what the AAD binds. Freeze the post-cascade format explicitly. Go rc.3 writes this format. Since 7 November 2026, a later change must + keep it readable and searchable, or add a new header version beside it. + The unchanged header does not show compatibility with older prototype + ciphertexts. The query projection order is now random, so parity tests + need to compare projections in a canonical form, by meaning. + +## Current SQL recipes + +The examples assume a column declared as +`profile public.eql_v4_json_search`. Each parameter is a query, selector, +or token value that the SDK generates, never a plaintext path. Add the +application's usual authorisation and tenant predicates. + +```sql +-- Contains / contained by / EqAt operands. +SELECT id, profile FROM users WHERE profile @> $1::eql_v4.query_json; +SELECT id, profile FROM users WHERE profile <@ $1::eql_v4.query_json; + +-- A single-valued ordered path: selector, lower bound, upper bound. +SELECT id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Wildcard ordering: explicitly existential, not an unspecified -> match. +SELECT u.id FROM users AS u +WHERE EXISTS ( + SELECT 1 + FROM eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS x(entry) + WHERE x.entry > $2::eql_v4.query_json_leaf +); + +SELECT id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, profile -> $1::text AS entry FROM users; +SELECT id, profile ->> $1::text AS encrypted_entry_text FROM users; + +-- Token list, then walk budget. Group entries by id before opening. +SELECT u.id, w.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text[], $2::integer +) AS w(entry); + +-- Children/count take one concrete node token, with no gates parameter. +SELECT u.id, c.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children(u.profile, $1::text) AS c(entry); +SELECT id, eql_v4.json_tree_child_count(profile, $1::text) FROM users; +``` + +The child count is 0 for a scalar or an empty container. The count is +NULL for a missing node or for the sealed root. Do not convert NULL to 0. + +The earlier recipes with tokens and gates, and with +`json_tree_subtree_at`, are removed. +[The JSON example](../../languages/golang/encrypt/example/json/main.go) +contains complete current Go programs and examples of binding parameters. + +## Current dynamic grammar and results + +The example below shows the implemented syntax of a data plan. It leaves +out a versioned outer envelope: + +```json +{ + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "JsonSearch", + "options": {"sealed_root": true, "ordered": ["$.age"]} + } + } +} +``` + +Operations select the field through the application binary interface +(ABI) between the host and the guest. The named inputs of operations +include: + +- `{"json"}` +- `{"path","json"}` +- `{"path","low","high"}` +- `{"path"}` +- `{"path","wildcard_bound"}` + +Since #1445 and #1446, two more operations take these inputs: + +- `element_match` takes `{"path", "wildcard_bound", "predicates"}`. The + `wildcard_bound` is optional. +- `match` takes `{"predicate", "vars"}`, as `jsonb_path_match` does. + +JSON values are UTF-8 JSON text. Query results state the `encoding` +(`eql`, `text`, `text_array`, or scalar `bytes`), the SQL domain, and the +operands. Capabilities state the shapes of input. + +To open, call `json_open` or `se_json_open` with the original `selector` +or `subtree` request and the grouped encrypted rows. + +A general versioned package of definitions and schemas is still required +work. That package is not the current runtime grammar. + +## Proposed Rust write and read example + +The code below is proposed public code, not executable code for the +current engine. +`target(column = ...)` illustrates syntax that declares an explicit EQL +column identity. The coming specification and naming pass decides the +exact attribute spelling. The options factory is typed Rust, not JSON +hidden inside an attribute. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use serde::{Deserialize, Serialize}; +use stack_encrypt::{DecryptFrom, DecryptSource, EncryptInto, EncryptTarget, StackCipher}; +use stack_encrypt::json::{JsonOptions, JsonPlaintext}; +use stack_encrypt::sem::TermError; +use std::error::Error; +use zeroize::Zeroize; + +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].date"]) +} + +#[derive(Serialize, Deserialize, EncryptTarget, DecryptSource)] +#[serde(transparent)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("users", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); + +#[tokio::main] +async fn main() -> Result<(), Box> { + let client = StackCipher::new().await?; + let keyset = client.default_keyset(); + let _plan = EncryptedProfile::encryption_plan(); + + let profile = JsonPlaintext::parse( + r#"{"age":42,"name":"Alice","visits":[{"date":"2026-10-01"}]}"#, + )?; + let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; + + // These bytes can instead be bound to the EQL PostgreSQL column. + let stored = serde_json::to_vec(&encrypted)?; + let loaded: EncryptedProfile = serde_json::from_slice(&stored)?; + let mut opened = JsonPlaintext::decrypt_from(loaded, &keyset).await?; + // The caller now owns the returned plaintext and its cleanup. + opened.zeroize(); + + let profiles = vec![ + JsonPlaintext::parse(r#"{"age":42}"#)?, + JsonPlaintext::parse(r#"{"age":27}"#)?, + ]; + let encrypted: Vec = profiles.encrypt_into(&keyset).await?; + let mut opened = Vec::::decrypt_from(encrypted, &keyset).await?; + assert_eq!(opened.len(), 2); + opened.zeroize(); + Ok(()) +} +``` + +### Scope and row binding in the example + +The example's fixed standalone definition explicitly declares scope across +the whole column, and no row binding. The normative schema must state that +choice. The schema must not silently replace a required tenant scope that is +missing. + +Take an `EncryptedUser` whose definition is derived on the encrypted +output type, with `id: PrimaryKey` and `tenant_id: ScopeKey`. +For that type: + +1. Select those fields through the context and scope attributes described + in the API plan. +2. Get its JSON field handle from + `EncryptedUser::PROFILE`, and require `TenantId` on + query requests. +3. Store and return both ID fields for decryption. +4. If the caller supplies input, use the agreed `_with_context` variants, + with a typed input that keeps binding and scope separate. + +A row ID is never an index or query input implicitly. + +## Remaining work and release gates + +1. Write down the implemented wire grammar, errors, and presence states in the formal specification and the promised conformance tests. Write down the context rules of the released configuration there too. General Rust definitions, + caching, and more context roles can come later. +2. Close the stated scope gaps in J14 and J17. Subtree reads are the + supported way to get plaintext for all matches. An opener for children + bound to their parent is separate engine work. Do not add generic + decryption of raw entries as a shortcut. +3. Replace the internal EQL bridge with typed conversion. The bridge + serialises to `Value` and deserialises again. Keep the stored format. + The change is an optimisation, not a missing execution path for Go + JSON. +4. Run the existing Rust, guest, Go, and PostgreSQL suites on the release + candidate, and map the tests to J01 to J20. Include: + - typed and raw inputs; + - numeric and null cases; + - both root policies (sealed and not sealed); + - wildcard bounds; + - root fallback; + - tampering that the cascaded tags detect; + - the known limits on detecting omitted results. + + Inspect the real SQL parameter casts and index use. +5. Freeze the post-cascade storage and search vectors and the matching EQL + bundle. Compare projections by their canonical meaning, and test + cross-opening (one implementation opening another's ciphertexts). + Ordinary random bytes do not need to match. Replace the temporary Go + pins of Vitamin C before a final release. Opaque storage is already + adopted. Opaque storage is a separate format from JSON tree storage. + +### Status of this work on 7 November 2026 + +| Item | Status | +| --- | --- | +| 1. Specification and conformance | Partly done. #1492 adds the `eql-json-v1` conformance profile. That profile is experimental: it does not certify a whole SDK | +| 2. J14 and J17 | Still open | +| 3. Typed EQL conversion | Still open | +| 4. Run the suites on the release candidate | Partly done. #1492 runs them in CI, including a job against PostgreSQL | +| 5. Freeze the stored vectors and the EQL bundle | Still open. Go rc.3 already writes this format, so the frozen format must keep it readable | + +## What this refresh did not check + +This refresh is a review of source and documentation, not a new runtime +test run or a complete cryptographic audit. + +Later activities in the API plan include: + +- compiling the final Rust example; +- extracting packages; +- stronger zeroisation and constant-time work. + +These activities must keep the released Go and data contract. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md new file mode 100644 index 000000000..f62d3f4f3 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -0,0 +1,442 @@ +# Stack Encrypt packaging and public names + +Dated 25 October 2026, with implementation updates on 27 October 2026 and +7 November 2026. The last check was against `next` at commit `e4990e49b`. +This recommendation for review accompanies the +[implementation plan](2026-10-24-stack-encrypt-api-plan.md). + +## Status + +This document proposes boundaries and names. The proposals are not +implemented APIs or accepted renames. The existing companion examples keep their +working names until this proposal is accepted. The detailed JSON operation +contract is under separate user review. + +## Summary of recommendations + +- Stack Encrypt owns the application value model and its execution. + Vitamin C keeps the reusable cryptographic primitives. +- Add one new crate, `stack-encrypt-format`, and rename `stack-kms` to + `stack-zerokms`. Use modules, not crates, for the other boundaries. +- Retire Vitamin C's `AsyncCipher` trait. +- Adopt the public glossary for Rust. Keep the merged Go and SQL names for + JSON. + +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3) | + +## Update on 7 November 2026 + +Pull request #1404 implements some names from the glossary below. They are +`Ciphertext`, `EncryptInto`, `DecryptFrom`, `OpaqueEncode`, `OpaqueDecode`, +and the traits `target::EncryptTarget` and `target::DecryptSource`. Pull request #1504 adds the derives with those names. Pull request #1503 adds `OpaqueBlob`. Both are in progress. + +These parts of the proposal are not implemented: + +- the `Cipher` rename, `TargetDefinition`, `PrimaryKey`, and `ScopeKey`; +- the `encoding` module and the encoded ciphertext wrappers; +- `.target::()` on the plan builder, and the `provider` module; +- the `stack-encrypt-format` crate, and the rename of `stack-kms`; +- the retirement of `AsyncCipher`. + +`encrypt_as` is still public. The structural `decrypt_into` is still +public, in `target::legacy`. `StackCipherText` remains for the tree API. The tree methods are `encrypt_tree` and `decrypt_tree`. + +Other changes that affect packaging: + +- Pull request #1501 moves the Rust crates to Vitamin C 0.9. It also + re-exports `Buffer`, `ProtectedRef`, and `AsProtectedRef` from the crate + root. +- Pull request #1403 bumps `stack-kms` to 0.2.0 on crates.io. So the + proposed rename to `stack-zerokms` would now rename a published crate. + +From 7 November 2026, the stored format must not break. The user's +exemption for prototypes below now applies to the APIs only. Moving codecs +and adapters between crates must keep the bytes that Go rc.3 writes. + +## Update on 27 October 2026 + +Go JSON and opaque storage are merged into `next`, through #1369, #1370, +#1372, #1373, #1378, and #1402. #1397 adds match terms for short whole +words. + +What remains before customers can use them is configuration and +conformance work, and release readiness. That includes replacing the +temporary Go module pins of Vitamin C. Rust stayed on Vitamin C 0.8.0 +from crates.io until #1501 moved it to 0.9. A customer release does not wait for crate extraction or for the +final Rust facade (the proposed public Rust API). + +## Sources for this assessment + +This assessment is based on the current documentation branch and on the +local Vitamin C 0.8.0 sources that were inspected: + +- Stack's + [cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) + explain the boundary of its existing batching. +- The [EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) + shows that `eql-bindings`, the Rust crate for Encrypt Query Language + (EQL), sits downstream of the engine, with an optional dependency on + it. +- The + [encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) + are the concrete candidates for extracting shared types. + +This is a design assessment at the source level. It is not an audit of +every external consumer of Vitamin C. + +## Ownership rule + +Stack Encrypt owns the application value model and its execution. That +means turning owned Rust values and dynamic values into ciphertext, +structured targets, and search operands, and reconstructing plaintext. +Vitamin C owns reusable primitives for cryptography and protected memory. +An ordinary Stack Encrypt user should not have to import Vitamin C +encryption traits to implement an application type. + +Moving code should remove duplicate abstractions. Moving code should not +keep both public surfaces under new names. The user's compatibility +exemption for prototypes applies to this API, but not to the stored format +since 7 November 2026. Before deleting +dependencies upstream, review the other consumers of Vitamin C that use +them. + +## Proposed packages + +| Package | Responsibility | +| --- | --- | +| `stack-encrypt` | The public Rust API; traversing and reconstructing values; target declarations and definitions; context and scope roles; normalisation and adapters for searchable targets; plans; batching; JSON tree execution; and validating and executing dynamic definitions | +| `stack-encrypt-derive` | Derive macros that generate the same declarations and definitions a user can write by hand. `stack-encrypt` re-exports the derives | +| `stack-encrypt-format` (new) | Pure data representations and codecs shared by the engine and the bindings: frame primitives and the required scalar representations, the stable format of stored blobs, parsing and encoding of the ciphertext envelope, and validated Base64 and Base85 text. No key requests and no execution runtime | +| `stack-zerokms` (the current `stack-kms`, renamed) | The ZeroKMS protocol client, authentication integration, native keysets, request types for lock context, and responses. Not the neutral abstraction over several providers | +| `eql-bindings` | EQL storage and query types by version, JSON serialisation compatible with SQL, the catalog, and Stack target adapters. Stays downstream of the engine | +| Vitamin C crates | Protected memory; randomness; AEAD and key operations on bytes; HMAC, PRF, and ORE algorithm primitives; and reusable canonical encoding of context | + +The abbreviations in the last row are: + +- AEAD: authenticated encryption with associated data; +- HMAC: hash-based message authentication code; +- PRF: pseudorandom function; +- ORE: order-revealing encryption. + +### Crates that stay as they are + +Keep `stack-auth` and `stack-profile` separate, because they have +consumers outside encryption. Keep `stack-guest-abi` for the calling +convention between the host and the guest. Neither the application binary +interface (ABI) nor the Go host should own a second model of target +policy. + +Put the formal dynamic contract, its schemas, and its conformance fixtures +beside Stack Encrypt. The implementation of the dynamic contract uses the +same definition types and validation as static Rust. + +### Modules, not more crates + +At first, use modules for the remaining boundaries. In particular, do not +create separate crates for execution, JSON policy, queries, context, or +provider traits only so that each concept has its own crate. + +### Why the format crate is needed + +The format crate has a concrete consumer. EQL's wire types now work +without EQL's optional `stack-encrypt` integration. Those wire types need +shared encoding types without depending on an encryption client or +runtime. + +- Put Serde, schema, and TypeScript support behind features, where + practical. +- Define the trait implementations beside the types the crate owns. + Otherwise Rust's orphan rule forces downstream crates to duplicate + wrappers. The orphan rule says an implementation must be in the crate + of the trait or of the type. +- Re-export the ordinary format types through Stack Encrypt, so users + normally need only that one dependency. + +The format crate is one implementation with separate formats. Each of +these has its own contract and versioning requirements: blob storage, the +transport to and from the guest, and ciphertext envelopes. Sharing code must not +silently turn the Serde mapping for short-lived transport into the stored +blob format. + +The format crate does not add a way to store arbitrary internal `Node` +trees. Structured outputs still use their explicit target layout and +ordinary Serde composition. + +## Dependency direction + +In the diagram, an arrow points from a crate to a crate it depends on. + +```mermaid +flowchart TD + App[Application / Rust or guest adapter] --> EQL[eql-bindings] + App --> Engine[stack-encrypt] + EQL -->|optional target integration| Engine + EQL --> Format[stack-encrypt-format] + Engine --> Format + Engine --> Derive[stack-encrypt-derive] + Engine -->|optional ZeroKMS adapter| KMS[stack-zerokms] + Engine --> Vitamin[Vitamin C primitives] +``` + +- The derive crate emits Stack paths. The derive crate does not depend on + the engine as a normal runtime dependency. +- No Vitamin C primitive depends back on Stack Encrypt. +- `stack-zerokms` does not depend on the engine. +- The provider traits belong in `stack_encrypt::provider`. An adapter + that Stack owns implements them with the ZeroKMS client. This avoids a + dependency cycle between the engine and the client. + +### Providers as optional features + +Make ZeroKMS an optional provider feature, separate from the HTTP feature. +Then a different provider does not bring its protocol into the core +contract. Keep support for injected providers, and for guest and native +builds. + +Data key support is the base capability. Index key support, resolving +named keysets, and lock context stay explicit optional capabilities. A +required capability that is not supported fails before key I/O, that is, +before any request to the key provider. Future provider adapters can +become separate downstream crates when needed. Do not design invented AWS +or Vault keysets now. + +Do not re-export EQL from Stack Encrypt, because EQL's target integration +already depends on Stack Encrypt. Applications import their EQL types +explicitly. + +## What moves out of Vitamin C + +Move these parts into Stack's value, target, and plan layers: + +- the visitors for plaintext and ciphertext that applications use; +- structural nodes; +- the value conversion traits and their derives; +- tree walking; +- collection composition; +- the application encryption helpers. + +Keep advanced extension points public only where people who implement +targets need them. These extension points are not part of the normal API +at the crate root. Internal nodes do not become a second public type for +opaque ciphertext. + +### Frames and codecs + +Move the generic frame code and the codecs that Stack needs into the +format crate. Stack owns the stored blob format and its compatibility +promise. + +This refines the earlier recommendation to reuse Vitamin C's frame codec. +Reuse its implementation, with a lossless mapping to storage, but change +who owns it in the long term. Coordinate this handover under +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). Keep +that issue's work on the format specification and the fixtures. This +proposal has not changed any upstream issue or repository. + +### Normalisation and index adapters + +Also move the normalisation, PRF, and ORE adapters that understand Stack's +logical values, traversal, targets, or query semantics. Keep the +independent cryptographic algorithms below them in the dependency graph. + +The current manifests of `vitaminc-normalize`, `vitaminc-prf`, and +`vitaminc-ore` depend on `vitaminc-aead`. Normalisation even uses the +physical scalar types of `vitaminc-aead`. So moving only the encryption +traits would leave a dependency back to the old value model. + +As part of the move, separate primitive inputs from application adapters. +Do not make Vitamin C depend on Stack to get the code to compile. + +### Context encoding + +Keep the construction of canonical context bytes in `vitaminc-context`, +where independent algorithms use it. Stack owns the semantic roles and +the selection rules: `PrimaryKey`, `ScopeKey`, ciphertext context, query +scope, and caller input capabilities. Stack reuses that canonical encoding +instead of inventing a second encoding that is not compatible. Provider +policy stays a separate role. + +## Retire AsyncCipher + +Recommendation: remove Vitamin C's `AsyncCipher` trait and its helpers for +executing application encryption, as the Stack executor takes over that +work. `AsyncCipher` is the Vitamin C trait that seals and opens leaves +asynchronously. Do not add a replacement public async cipher trait in +Stack that covers every case. + +Stack already plans and groups the remote key operations, and then runs +the cryptography locally and synchronously. Stack's current `MintedKeys` +and `BoundKeys` implementations use synchronous operations of the leaf +cipher. `AsyncCipher` models a different place to put the asynchronous +work. The asynchronous boundary that matters is the provider and the +public pending operation, with Stack controlling the batching. + +Stack does not use `AsyncCipher`. Vitamin C does use it: `aead/src/seal.rs`, +`context.rs`, `typed.rs`, `walk.rs`, and `normalize/src/terms.rs` expose or +use async helpers, and their tests implement the trait. To retire it: + +1. Move the tests that cover batching, ordering, failures, and + reconstruction with the code they test. +2. Delete the retired trait and its helpers. +3. Audit external consumers before the upstream release. + +Where useful, separate the internal capabilities for sealing leaves and +opening leaves. Do not keep implementations that reject half of a +combined trait. + +## Recommended public glossary + +| Name or path | Meaning | +| --- | --- | +| `stack_encrypt::Cipher` | Execution handle that uses a provider for keys. Replaces `StackCipher` | +| `KeysetCipher` | Execution handle bound to a resolved keyset. Available for providers that support keysets | +| `Ciphertext` | One opaque encrypted value with a specified storage envelope. Replaces the public use of `StackCipherText = Node` | +| `OpaqueBlob` | Explicit adapter that serialises a whole owned value | +| `EncryptInto`, `DecryptFrom` | Convenience traits for encryption directed by the target and decryption directed by the destination, with variants that take context | +| `OpaqueEncode`, `OpaqueDecode` | Traits that represent a value as a single ciphertext | +| `target::{EncryptTarget, DecryptSource}` | Declaration traits. The matching derives are re-exported at the crate root | +| `target::TargetDefinition` | Validated, immutable definition shared by the static and dynamic paths | +| `context::{PrimaryKey, ScopeKey}` | Wrappers that mark a field as eligible. Neither selects a context or a keyset automatically | +| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes. Owned in the format crate and re-exported here | +| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated, encoded representations of opaque ciphertext, and supported field targets. Valid encoding alone is not enough | +| `plan::Plan` | Explicit low-level interface for composition. The output is selected with `.target::()` | +| `provider` | Neutral provider contracts and capability bounds | +| `provider::zerokms` | The ZeroKMS adapter and the surface for its policy extension, including `LockContext` | +| `dynamic` | Loading versioned definitions, discovering capabilities, and executing through the common engine | + +### Spellings already agreed + +Keep these agreed spellings: + +- `.using(...)` +- `encryption_plan()` +- `encrypt` +- `encrypt_with_context` +- `encrypt_into` +- `encrypt_into_with_context` +- `decrypt_from` +- `decrypt_from_with_context` + +Keep the direct `cipher.decrypt` and `decrypt_with_context`. As already +agreed, remove `.encrypt` on plaintext, the redundant `encrypt_as`, and +the public `decrypt_into`. + +### Definition, layout, seal, and open + +Use `TargetDefinition` for the configured public contract. Use `Layout` +only for the advanced internal concept of composition, where that +distinction is needed. A layout does not name a plaintext document or a +stored ciphertext. + +Name low-level operations on bytes `seal` and `open`. Keep the words +"encrypt" and "decrypt" for the public operations on application values. + +### Ciphertext types in the format crate + +Keep the ciphertext representations in the format crate independent of +traits. Stack implements its own target traits for those types. Plain +encoded text is not itself an encryption target. Constructing a +ciphertext wrapper must validate both layers: the encoding and the +ciphertext. Avoid a public string constructor without checks. + +## JSON names by role + +Keep the existing EQL package and its version modules. The merged public +Go surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery`, +and `JSONEntry`, configured through `JsonSearch`. In SQL, the names are +`eql_v4_json_search`, `query_json`, `query_json_leaf`, and +`eql_v4_json_entry`. The prepared Go objects are `encrypt.JSONSelection` +and `JSONRead`. + +The table below is still a proposed glossary for the Rust facade. Its +`JsonTree*` names describe the handwritten Rust adapters, not the +registered Go target. A customer release does not need the working Go +and SQL surface renamed to match this glossary. Do not rename it for that +reason. Later Rust renames must keep the frozen dynamic IDs. + +| Working or current name | Recommended name | Role | Home | +| --- | --- | --- | --- | +| `JsonPlaintext` | `JsonPlaintext` | Adapter for owned plaintext | `stack_encrypt::json` | +| `JsonOptions` | `JsonTreeOptions` | Validated policy for the tree target | `stack_encrypt::json` | +| `JsonSelection` | `JsonSelection` | Proposed result of a plaintext selection | `stack_encrypt::json` | +| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document | `eql_bindings::v4` | +| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry | `eql_bindings::v4` | +| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result | `eql_bindings::v4` | +| `JsonTreeQuery` | `JsonContainmentQuery` | Query operand for containment and for a value at a path | `eql_bindings::v4` | +| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Query operand for ordering | `eql_bindings::v4` | +| `JsonTreeSelector` | `JsonSelector` | Encoded path selector | `eql_bindings::v4` | +| `JsonSubtreeKey` | `JsonWalkRequest` | Walk token held by the client, and its binding to the request | Engine type, with a typed EQL adapter | + +### Notes on the JSON names + +`JsonContainmentQuery` can still be the physical operand of an equality +at a path, where the agreed SQL recipe uses containment. An operation and +its wire operand do not need the same name. + +`JsonWalkRequest` describes the real role better than "key" does. The +type contains sensitive capability and request state, not the data +encryption key. The separate review of the JSON contract decides the +exact behaviour of each operation. + +Rename the internal target index `Json` to `JsonTreeIndex`. Keep the +internal engine type `JsonDocument` out of the ordinary facade. If adapter +authors need public access to it, give it a name that states its role, +such as `EncryptedJsonTree`. + +`EncryptedJson` always means the stored form +that is compatible with EQL. `EncryptedJson` never means some other JSON +serialisation of encrypted nodes. An application's `EncryptedUser` stays +its own declared record layout. + +The dynamic identifiers of targets and operations are explicitly +versioned contract identifiers. They are not Rust type names serialised +into JSON. Choose their exact spellings in the interoperability +specification, and use the same glossary in Go. A Rust rename on its own +must not redefine stored SQL formats or dynamic discriminants by accident. + +## Implementation order and acceptance + +1. Specify and validate the Go JSON and opaque storage contracts. Both + are now implemented. Freeze the post-cascade JSON and search semantics + and the exact fixtures. "Post-cascade" means as they are after #1379. + That change made each container's tag cover the tags of its children. + Replace the temporary Go module pins of Vitamin C for a final release. +2. Complete the outstanding Go match configuration, and close the gaps in + JSON reads and conformance. Short whole-word matching is implemented in + EQL. Its leakage and its compatibility with the stored format still need explicit + acceptance for the release. +3. After the customer release, set up the shared format crate. Move the + value model, traversal, and the normalisation and index adapters that + depend on it into Stack. Keep the frozen encodings and an acyclic + graph. Move tests with their code. +4. Complete the neutral provider extraction and the renaming of the + protocol package. Then retire Vitamin C's surface for asynchronous + application execution. The policy and routing contracts that Go needs + must already have been handled before the release. +5. Complete the public Rust trait and derive names, imports, and + executable examples. These changes must not rename the released Go API + or reinterpret stored data. The changes must not require customers to + re-encrypt or reindex. + +Acceptance includes: + +- an acyclic Cargo graph; +- an EQL build of the wire types only, without the engine; +- engine builds without ZeroKMS and without HTTP; +- native and guest builds; +- compiled public examples; +- the behaviour tests that moved; +- shared fixtures for formats and operations across Rust, Go, and SQL. + +Full improvements to timing and memory clean-up remain follow-up work, +under the accepted priority that puts the format first. Record only the +guarantees that are true. + +This document does not claim any implementation changes, tests of the +proposed API, code moves, or upstream removals. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md new file mode 100644 index 000000000..381d3d7d2 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -0,0 +1,543 @@ +# Key selection, lock context, and provider capabilities + +Dated 25 October 2026, with updates on 27 October 2026 and 7 November +2026. The last check was against `next` at commit `e4990e49b`. This assessment +accompanies the [API implementation plan](2026-10-24-stack-encrypt-api-plan.md). + +## Status and requests + +This document contains recommendations for review. The recommendations +are not accepted public signatures. Nothing in them is implemented. On 7 +November 2026, no merged pull request implemented keyset routing from +plaintext, lock context, or the neutral provider split. + +From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not +break. The stored format is what is written to the database, such as +ciphertext envelopes and search terms. The refined-tag work in item 6 +below changes the stored tag, so it must keep rc.3 data readable. See the +[API +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). + +The user who requested this assessment asked for three things: + +- ZeroKMS keyset selection derived from plaintext; +- a review of the Vitamin C `AsyncCipher` trait; +- optional lock context support, without requiring future backends, such + as AWS Key Management Service (KMS) or Vault, to emulate it. + +The JSON operation contract is under separate user review. + +## Summary of recommendations + +1. Add a declaration that selects the keyset from a plaintext field. Never + fall back to the default keyset. +2. Do not implement `AsyncCipher` on `StackCipher`. The packaging + recommendation now proposes retiring the trait. +3. Add lock context as an optional, typed ZeroKMS policy input. +4. Split the provider boundary into a base capability for data keys and + optional capabilities for everything else. + +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| Keyset | A ZeroKMS key domain. Data keys are generated in a keyset, and each keyset has a deterministic index key | +| Lock context | A ZeroKMS policy that ZeroKMS applies when it generates or retrieves a data key, such as a claim of the authenticated JSON Web Token (JWT) | +| `DataKeySource` | The trait in `stack-kms` that generates fresh data keys and derives them again for stored ciphertexts | +| `IndexKeySource` | The trait in `stack-kms` that loads the deterministic index key of a keyset, used to derive search terms | +| `Pending` | The request object that `StackCipher` returns. Settling it (awaiting it) sends the batched ZeroKMS calls | +| Key I/O | Any request to the key provider to generate or retrieve a key | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| `MaybeSend` | A bound that means `Send` on native targets and no bound on WebAssembly (WASM) | +| Vitamin C | CipherStash's library of cryptographic primitives | + +## Four separate roles + +| Role | What it selects or binds | Example | +| --- | --- | --- | +| Key selection | The provider key domain used to encrypt and to derive index terms | A ZeroKMS keyset ID or name selected from `tenant_id` | +| Ciphertext context | The application binding verified when a ciphertext is opened | The column, the tenant scope, and the row ID | +| Search scope | The domain in which index and query terms compare | The column and the tenant ID | +| Provider policy | Conditions the provider applies when it generates or retrieves a key | ZeroKMS lock context that selects the `sub` claim of the authenticated JWT | + +These four roles are separate logical inputs, even where the protocol binds +several of them into its tag or AAD: + +- A `ScopeKey` does not select a ZeroKMS keyset by itself. +- A keyset selector does not add its source field to the AAD. +- Lock context is not another name for ciphertext context or for search + scope. + +For two roles to share a source value, the definition must declare it +explicitly. The engine must not infer it from a name. + +## Current implementation + +### Update on 7 November 2026 + +- Pull request #1501 batches an owned `Vec` collection. Encrypting makes + one `generate_keys` call. Decrypting makes one `retrieve_keys` call for + each keyset. A ciphertext from another keyset is refused before any key + is retrieved. +- Pull request #1493 adds lifecycle conformance, including bounded + concurrency when the KMS client splits a batch into chunks. +- Pull request #1501 moves the Rust crates to Vitamin C 0.9. The section + on `AsyncCipher` below describes 0.8.0, and was not checked against 0.9. +- Pull request #1403 bumps `stack-kms` to 0.2.0, and `stack-auth` to + 0.44.0. Neither bump adds lock context or routing. + +### Update on 27 October 2026 + +#1379 makes verification of a JSON document whose root is not sealed load +the index keyset as well as the data key. Settling a `Pending`, and the +generic execution entry points, now require +`DataKeySource + IndexKeySource`. + +The neutral provider split proposed below must keep that verification +requirement for JSON. The split must still allow capabilities with data +keys only, for operations that need no index key. + +#1379 does not implement keyset routing derived from plaintext, or the code +that passes ZeroKMS lock context through. + +### What the code does now + +- [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts + an `IdentifiedBy` universally unique identifier (UUID) or name. The method resolves the keyset and its + index key, and returns a `KeysetCipher` bound to that keyset. The client + has a keyset cache of limited size, and a policy for how fresh a name + resolution must be. `.default_keyset()` is already bound when the client + is initialised. Initialisation now requires both `DataKeySource` and + `IndexKeySource`. +- Each sealed leaf contains the UUID of the resolved keyset. Decryption + through the client groups key retrieval by the UUIDs in the leaves. A + bound keyset can restrict which keyset is allowed. Names are a + convenience for lookup, not persistent key identities. See the + architecture decision record (ADR) + [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). +- [Pending dispatch](../../packages/stack-encrypt/src/target/pending.rs) + now uses one selected keyset for every key it generates. Dispatch groups + key retrievals by their stored keyset IDs. Dispatch builds the payloads + for generating and retrieving keys without lock context, and passes no + unverified request context. +- The [key source traits](../../packages/stack-kms/src/key_source.rs) and + the [payloads](../../packages/stack-kms/src/payload.rs) already model + lists of ZeroKMS lock context, optional decryption policies, and + unverified context. These traits are abstractions specific to ZeroKMS, + used to inject a provider and for testing. They are not yet a neutral + contract for any KMS backend. The returned key types also follow the + existing integration with ZeroKMS and recipher. +- The [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) + accepts `.withLockContext({ identityClaim: ["sub"] })`. Its + documentation says the server resolves the claim values from the + principal that authenticated. Encrypt and decrypt operations pass that + selector to protect-ffi. Audit metadata is passed separately, as + `unverifiedContext`. +- The `Context` enum in the pinned `zerokms-protocol` 0.12.31 includes + `Tag`, `Value`, and `IdentityClaim`. Its `RetrievedKey` now exposes key + material only. `stack-kms::DataKey` contains the initialisation vector + (IV) and the key. `DataKey` does not contain the original + authentication tag that the proposed refinement flow returns. To support + that flow, the protocol and the client need to return that tag in their + responses. This matches the user's earlier explanation of the intended + behaviour. + +## 1. Keyset selection derived from plaintext + +Recommendation: add a key selection declaration to the existing target +definition, beside the source selectors for context and scope. A useful +minimal portable form is an explicit selector, by ID or by name, over a +plaintext field: + +```rust +// Proposed attribute fragment. tenant_id contains the actual keyset name. +#[stash(keyset(name(field = "tenant_id")))] +``` + +```rust +// Alternative: the selected field contains a keyset UUID, not an arbitrary +// application UUID assumed to be a keyset. +#[stash(keyset(id(field = "keyset_id")))] +``` + +Neither selector creates a keyset. A missing, unknown, unauthorised, +disabled, or incorrectly typed keyset is an error. Never fall back to the +default keyset. + +A business tenant ID does not have to be a keyset name or UUID. Where the +two differ, configure an explicit mapping or resolver from tenant to key. + +Start with direct selectors and one shared contract for registered +resolvers. Do not encode arbitrary Rust callbacks in dynamic definitions. +A pure mapping can return a selector. A network lookup belongs in the +execution or provider layer, not in the derive. + +### Selecting the keyset by hand and by declaration + +The existing form, where the caller selects the keyset by hand, already +fits the agreed API. This example uses the `IdentifiedBy::name` +constructor that the [API +plan](2026-10-24-stack-encrypt-api-plan.md#constructors-for-identifiedby) +proposes: + +```rust +let tenant = cipher.keyset(IdentifiedBy::name(&user.tenant_id)?).await?; +let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; +``` + +The proposed route, derived from the declaration, removes that +repetition: + +```rust +// EncryptedUser's definition declares how to select the keyset. +let encrypted: EncryptedUser = user.encrypt_into(&cipher).await?; +let encrypted: Vec = users.encrypt_into(&cipher).await?; +``` + +`&cipher` is the executor for the client and provider. That executor can +resolve keysets. `&cipher` is not a request without a scope that silently +encrypts under the default keyset. + +Extend the capability of the conversion executor so that it accepts +either a resolved keyset, or a client with a declared selection recipe. +The target's representation, and the description of its encryption +operation, do not change. A direct `tenant_keyset.encrypt(value)` keeps +its accepted meaning. + +### Execution order + +1. When the request is built, take guarded ownership of the source, as + already required. Before any provider I/O, validate: + - the target and its options; + - the input shape; + - the selectors; + - the role and capability requirements; + - every check on values that does not need a key. +2. Evaluate the key selection recipe on the guarded plaintext. Resolve the + selectors through the configured provider. Keep each resolved identity + fixed for the whole request, including index derivation. +3. Split a batch into groups by resolved key identity, by provider and + authentication session, and by compatible options at the request level. + Return results in the original order. A batch with many tenants does + not mean one remote call for the whole batch. +4. Use the index material of the selected keyset for its searchable + fields, and generate data keys in the selected keyset. Store the + resolved identity in the ciphertext's provider metadata, not a name + that can change. + +### A keyset selected both beforehand and by the target + +Define the behaviour for this conflict. A target's route must not silently +override a keyset selected beforehand. If both are given, either check +that the resolved route matches the bound keyset, or reject the +combination explicitly. Neither takes precedence implicitly, and there is +no fallback. + +Errors that need name resolution can only happen after that resolution. +The accepted preflight rule covers checks that need no key. The rule does +not cover facts about the provider that are not available locally. + +### Queries + +A query has no plaintext user record to inspect. So the same target +policy must either expose a tenant routing input that the caller supplies, +or use a keyset selected explicitly. One typed input can deliberately +supply both the key route and the search scope. The declaration decides +both uses. + +Do not derive a keyset from the email address being searched for. To +search across several keysets, fan out explicitly, with one query term for +each keyset. No single query term compares across all keysets. + +### Decryption + +Decryption uses the stable key identity stored with the ciphertext, as it +does today. Do not resolve a historical keyset name again and use what it +points to now. An expected tenant or keyset can restrict key retrieval, +with any historical key mapping supplied explicitly. + +Trusting a tenant field returned beside a ciphertext does not authenticate +the application's tenancy. Row binding, or a trusted expected context, +makes that separate check. In the same way, choosing a keyset from +plaintext that the caller controls is routing. That choice does not give +permission to use every tenant keyset a service account can access. + +## 2. Vitamin C AsyncCipher + +Recommendation: do not implement `AsyncCipher` on `StackCipher`. +`AsyncCipher` is the Vitamin C trait that seals and opens the leaves of +one document asynchronously. + +### Follow-up from packaging + +The user favours removing the trait. + +The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md#retire-asynccipher) +now recommends retiring this trait and its application helpers as +execution moves into Stack. That replaces the optional adapter considered +below. Vitamin C still has helpers and tests that use the trait. Those +helpers and tests need to be migrated. Nothing has been removed yet. + +### Why Stack leaves out AsyncCipher + +Stack leaves out `AsyncCipher` on purpose. The current design does not +omit it by accident. +[cipher.rs](../../packages/stack-encrypt/src/cipher.rs) documents the +reason. That file implements the synchronous Vitamin C `Cipher` trait +through two types: `MintedKeys` seals, and `BoundKeys` opens. + +In Vitamin C 0.8.0, `AsyncCipher` takes the plaintext of a leaf and its +derived AAD. Its `seal_many` and `open_many` work on the leaves of one +document. The trait does not match Stack in three ways: + +- Stack's executor keeps the original provider descriptor. The executor + combines requests across fields, documents, and records before it calls + the leaf cipher. The AAD of a leaf alone cannot be turned back into + those execution inputs. +- Future routed keys and lock policies add more execution metadata. They + do not remove the mismatch. +- The trait requires `Sync` and `Send` futures. To support both native + and WASM builds, Stack's provider traits use `MaybeSend`. + +### Original recommendation on AsyncCipher + +Keep Stack's orchestration of batches and providers, and Vitamin C's leaf +cryptography. Do not implement `AsyncCipher` on `StackCipher` only so that +the traits match. Do not encode provider policy secretly into AAD. Do not +lose batching across records. + +An adapter configured explicitly could implement the trait for a +restricted use, with the provider descriptor, key route, and policy +already supplied. The public redesign does not need that adapter. If +built, the adapter must state its limits. A general refactor would need a +richer request contract. It would also need evidence that batching, +metadata, and WASM behaviour still work. + +### A smaller clean-up + +Each synchronous adapter now implements the half of `Cipher` that it does +not use, by returning an error. During the final pass on dependencies and +packaging, consider separate capabilities for sealing and opening, or an +improvement to the internal adapter. Keep this clean-up separate from the +question of whether the public Stack client implements the async trait. + +## 3. Lock context as an optional provider policy + +Recommendation: pass lock context as its own typed ZeroKMS policy input. +Do not pass it as the ordinary `encrypt_with_context` argument, or as a +`SearchScope`. Proposed spelling: + +```rust +use stack_encrypt::providers::zerokms::{LockContext, LockContextExt}; + +let lock = LockContext::identity_claim("sub"); +let encrypted = user.encrypt_into(&tenant_keyset) + .with_lock_context(&lock) + .await?; +let user = User::decrypt_from(encrypted, &tenant_keyset) + .with_lock_context(&lock) + .await?; +``` + +The module and method names illustrate the idea until the naming pass. +The operation builder keeps its owned plaintext guards while the policy is +attached. The extension is present for a provider with the required +capability. A provider whose type is erased at runtime checks the same +capability before key I/O. + +Ordinary encryption without this optional policy works on providers that +do not support it. On those providers, a request or definition that +requires lock context must fail. The request must never discard the +policy, or substitute literal AAD. + +### Claims come from the provider + +The input selects the **name** of a claim. ZeroKMS verifies the principal +that authenticated, and resolves the value of its claim. Do not decode an +unverified JWT in the application and turn its `sub` into an assertion by +the caller. Authentication belongs to the configured provider or session. + +When different identities are active at the same time, bind each request +or session to its identity explicitly and immutably. Do not change the +identity of a global client between operations. + +### Rules for lock context + +- Support explicit literal tag and value items through the same provider + policy. Keep these assertions by the caller distinct from claims derived + from the principal. +- A missing claim, or a failure of the provider policy, is an error. Do + not retry without the lock. +- Send the policy on both key generation and key retrieval. +- A required policy can be part of the target definition, so callers + cannot leave it out by mistake. An operation can supply an optional + policy that the definition does not declare. An operation cannot replace + or weaken a policy that the definition requires. +- Define rules for combining policies and for conflicts between them. Do + not let the last call win. +- Reads may need the policy selectors. Define how the selectors are + stored, or which input the caller must supply. Do not serialise JWTs, + credentials, or claim values that the provider resolved. +- Keep audit metadata (`UnverifiedContext`) separate. Audit metadata is + not a trusted identity, ciphertext context, or a policy assertion. +- Keep lock context distinct from the protocol's separate decryption + policy feature. Do not silently treat one as the other. + +### Lock context does not protect search + +Deriving index and query terms can use a deterministic index key that is +already loaded, without generating or retrieving a data key. So lock +context on the data key endpoints does not by itself restrict who can +derive search terms or see query results. + +If search needs authorisation, define that capability separately. The +TypeScript query method accepts `withLockContext`. That does not prove +that the new index path is protected. Do not imitate a lock by +making search scope specific to a principal implicitly. + +### Batching and caching with lock context + +Batching must keep the lock inputs of each item. Batching must also keep +incompatible principal sessions and provider sessions apart. If a provider +supports a policy on each payload, keep the policy in each payload. If an +option applies to the whole request, split the batch by that option. + +Any future cache of data keys must respect the semantics of checking +policy and principal. A cache hit must not silently replace a provider +authorisation or audit event that is required. This is a design +requirement. It does not claim that an existing cache has a +vulnerability. Do not widen the current cache of keysets and index keys +into a data key cache that authorises without those checks. + +## A neutral provider base with optional capabilities + +The public model of values and targets needs extending, not replacing. The +larger refactor is at the provider boundary. That boundary now imports +ZeroKMS payloads, UUID keyset IDs, policies, errors, and returned key +types directly. + +The recommended split of capabilities, with working names: + +| Capability | Required for | +| --- | --- | +| Data key generation and recovery, and protected key material | Ordinary opaque and structural encryption and decryption | +| Index key support | Searchable targets and query derivation | +| Keyset resolution by ID or name | ZeroKMS-style keyset selectors. Not every provider offers it | +| Lock context support, including claim selectors | The ZeroKMS policy extension. Not every provider offers it | + +### Key references and metadata + +Define a reference to a resolved provider key, and versioned opaque +metadata for recovering the key. Do not force every provider to supply a +ZeroKMS UUID, an IV with its tag, or a recipher response. + +A caller selects and configures a provider. Ciphertext metadata identifies +a registered provider and key format. The metadata never names arbitrary +network endpoints or credentials to execute. + +Keep ZeroKMS's key tag simple for Rust and SQL consumers to extract, +within its documented envelope format. Other providers' metadata does not +need to invent such a tag. The stable encoding of blob payloads does not +depend on the envelope format of each provider. + +### Future providers + +Future integrations with AWS KMS or Vault implement the minimal +capabilities they offer. To encrypt an opaque value, they do not need lock +context, named keysets, or management of index keys for search. The +application can configure its own mapping from tenant to provider key, +without representing it as a native ZeroKMS keyset. + +A searchable target requires an index key strategy that is explicitly +supported. The engine must not assume that every data key backend supplies +deterministic search keys automatically. + +### Refined tags + +Support for refined tags also needs two distinct inputs: the tag stored +and sent in the request, and the original authentication tag returned. +When designing neutral responses, keep the ZeroKMS flow that the user +intended. Do not require every provider to implement refinable tags, or a +generic KMS to return that concept. + +## Dynamic interoperability + +Discovery and definition validation cover provider capabilities as well as +target capabilities. This example shows a proposed logical schema for an +extension specific to one provider: + +```json +{ + "provider": "zerokms", + "key_selection": { + "keyset": { "by": "name", "encrypt": { "field": "tenant_id" } } + }, + "provider_policy": { + "zerokms": { + "lock_context": [{ "identity_claim": "sub" }] + } + } +} +``` + +Rules for the dynamic definition: + +- The field's declared type must allow the chosen selector. A missing field + is a definition or input error, before the provider resolves anything. +- Query routing has no source record, so it declares an equivalent input + that the caller supplies. +- Providers and extensions are registered, versioned capabilities. +- Unsupported required key selection or policy is an error when the + definition is loaded or bound to an execution provider, before key I/O. +- Do not accept arbitrary extension maps, and do not ignore unfamiliar + entries. +- Authentication credentials stay in the provider setup. They are not in + this declaration or in stored policy metadata. + +Go code generation uses named selectors and policies, and the same +validated dynamic definition. Methods and capabilities specific to a +provider are explicit. The interoperability specification must cover: + +- batches with several keysets; +- policy inputs; +- refusals because a capability is missing; +- how key identities are stored; +- the difference between authenticated claims and audit metadata that the + caller supplies. + +## Implementation and acceptance work + +1. Specify neutral key references, recovery metadata, and optional + capability traits. Do not change the accepted conversion roles for + encryption and decryption. +2. Add selectors to definitions, and routed execution through the client. + Cover selection by ID compared with by name, explicit mappings, + conflict errors, and no fallback to the default. +3. Extend request planning and grouping to include the resolved route and + the provider policy. Across asynchronous resolution and cancellation, + keep the order of records. Keep the lifetimes of guards too. +4. Pass ZeroKMS lock context through both key generation and key + retrieval, in both the literal form and the principal claim form. Test + that a required policy is enforced. +5. Test these cases: + - batches with several tenants; + - matching keys for writes and queries; + - decryption by historical ID; + - wrong expected keysets; + - missing claims; + - failures for unsupported providers, before I/O; + - isolation between principals active at the same time. + + A minimal fake provider without keysets, locks, or index keys must + still encrypt opaque values. +6. Settle support for refined tags in responses with the ZeroKMS client + and protocol. Add integration fixtures without assuming the pinned API + already returns the original tag. +7. Leave the decision on the `AsyncCipher` adapter and the trait clean-up + to the packaging pass. Until then, keep the documented constraints on + batching and WASM. + +This assessment does not claim any implementation changes, any protocol +guarantees for future providers, or a completed security audit.