From a5a886a41af16ae688d118af1b0ceb82d3603216 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 20:57:54 +1100 Subject: [PATCH 01/42] docs: capture Stack Encrypt API review --- .../2026-10-24-stack-encrypt-api-review.md | 684 +++++++++++++++++ ...26-10-24-stack-encrypt-context-examples.md | 723 ++++++++++++++++++ ...026-10-24-stack-encrypt-public-examples.md | 572 ++++++++++++++ 3 files changed, 1979 insertions(+) create mode 100644 docs/plans/2026-10-24-stack-encrypt-api-review.md create mode 100644 docs/plans/2026-10-24-stack-encrypt-context-examples.md create mode 100644 docs/plans/2026-10-24-stack-encrypt-public-examples.md diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md new file mode 100644 index 000000000..ad6afc78e --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -0,0 +1,684 @@ +# Stack Encrypt API review + +Started: 2026-10-24. Scope: the current Rust `stack-encrypt` API, starting +with encrypting one `i32` into opaque ciphertext and storing the result. + +Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md), +an API design draft covering encryption, persistence, decryption, and queries. +The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) +explore context sources and storage using Serde-inspired attribute syntax. + +This is a running issue log. Add a numbered section as each issue is found; +keep observations, proposed changes, and unresolved decisions distinct. +All issues below remain unimplemented. Individual decisions are recorded +within each issue; proposed APIs are illustrative, not implemented. + +## 1. Cipher trait mismatch and explicit context ergonomics + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` is an inherent method, not a trait method: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let ciphertext = keyset.encrypt(42_i32, "users/age").await?; +``` + +Vitamin C (`vitaminc-aead` 0.8.0) provides both `Cipher` and `AsyncCipher`. +Stack implements the synchronous `Cipher` trait on internal adapters after +fetching keys: `MintedKeys` seals leaves and `BoundKeys` opens them. Neither +public cipher implements `AsyncCipher`, so callers cannot use +`KeysetCipher` directly with generic Vitamin C code requiring that trait. + +Each internal adapter implements only half of `Cipher`: `MintedKeys::open` +and `BoundKeys::seal` always return errors. + +### Why the mismatch exists + +`AsyncCipher` already supports batching through `seal_many` and `open_many`; +batching alone does not explain the separate API. Its methods receive each +leaf's derived AAD, while ZeroKMS needs the original structured context's +descriptor. That original context is not carried by the derived leaf AAD. +Stack's pending-operation layer also combines key requests across documents, +fields, and rows, beyond the trait's per-document batch interface. + +These constraints need to survive any trait integration. The current +separation is deliberate, but leaves an interoperability gap and internal +implementations with unsupported operations. + +### Requested API direction + +Requiring an empty tuple for the common case of no application AAD is +unnecessary ceremony. Use distinct methods, following the convention used +elsewhere: + +```rust +keyset.encrypt(42_i32).await?; +keyset.encrypt_with_context(42_i32, "users/age").await?; +``` + +The no-context method should select the same empty context currently +represented by `()`. This distinction should exist even if `KeysetCipher` +continues to use inherent methods rather than implementing a Vitamin C trait. + +### Open design questions + +- Can a context-bound adapter implement `AsyncCipher`, or should Vitamin C's + interface preserve the original context explicitly? +- How does that integration preserve batching across multiple documents? +- Should sealing and opening be separate trait capabilities, avoiding methods + that are present but always fail? +- Should decryption expose the corresponding `decrypt` / + `decrypt_with_context` distinction? + +**Source:** [cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) +(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's batching +discussion). Vitamin C trait definitions are in `vitaminc-aead` 0.8.0, +`src/seal.rs`. + +## 2. The Node / SealedValue boundary complicates opaque ciphertext storage + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` returns `StackCipherText`, an alias for Vitamin C's +`Node`. Encrypting an `i32` produces `Single(SealedValue)`, but +the return type also admits sequences, maps, empty containers, and passthrough +nodes. It does not retain the plaintext type as a type parameter. + +| Capability | `StackCipherText` tree | `SealedValue` leaf | +| --- | --- | --- | +| Debug printing (`{:?}`) | Yes | Yes | +| Serde serialization/deserialization | No | Yes | +| Canonical byte encoding | None | `to_bytes()` / `from_bytes()` | +| Direct Base64 conversion | None | None; encode the canonical bytes separately | + +The leaf's canonical bytes include its keyset id, key retrieval metadata, +and local ciphertext. That versioned byte format is the storage commitment; +the leaf's Serde representation is not the same compatibility commitment. + +### Caller impact + +Even the scalar storage path requires exposing the container representation: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let StackCipherText::Single(leaf) = ciphertext else { + unreachable!("encrypting an i32 produces a single leaf"); +}; +let bytes = leaf.to_bytes(); + +// After loading the bytes: +let leaf = SealedValue::from_bytes(&bytes)?; +let ciphertext = StackCipherText::Single(leaf); +let value: i32 = cipher.decrypt(ciphertext, ()).await?; +``` + +The caller must handle a shape mismatch that cannot occur for this input, +extract the leaf, and reconstruct the node to decrypt. There is no direct +`ciphertext.to_bytes()` or `serde_json::to_string(&ciphertext)` path. Base64 +adds a separate encoding step. Collections additionally require the caller +to choose a representation preserving the tree structure and node variants. + +Stored bytes are not sufficient by themselves to decrypt: the caller still +needs authorized ZeroKMS access and the same application context. The storage +API should make that requirement clear without exposing tree internals for +the simple scalar case. + +### Proposed direction and open decisions + +Provide a straightforward opaque ciphertext storage round trip for scalar +values, with direct byte conversion and reconstruction. Decide whether this +means a dedicated scalar result type, a wrapper, or conversion helpers on +the existing API. + +Also decide whether Base64 and Serde should be supported conveniences, and +whether whole-tree serialization belongs in Stack or remains caller-owned. +If it remains caller-owned, make that distinction explicit and keep it out +of the simple scalar path. Preserve the existing canonical leaf encoding; +an ergonomic improvement should not require changing stored ciphertext. + +**Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) +(`StackCipherText`, `SealedValue`) and +[storage documentation and public re-exports](../../packages/stack-encrypt/src/lib.rs). +Vitamin C's container definition is in `vitaminc-aead` 0.8.0, +`src/container.rs`. + +## 3. Plaintext-side encryption has different syntax and context requirements + +**Status:** Open. + +### Current behavior + +The current Vitamin C `Encrypt` trait describes a plaintext through `visit`; +it does not provide an `encrypt` method. Importing Stack's re-export of that +trait does not enable `10_i32.encrypt(&keyset)`. Vitamin C 0.8.0 documents +this trait as replacing the older visitor-based trait that drove a cipher +directly. Its `ContextTag` wrapper has encryption methods, but those require +Vitamin C cipher traits and do not make the syntax available on bare integers +or accept Stack's public ciphers directly. + +Stack supplies its own blanket `EncryptInto` extension trait: + +```rust +use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; + +let ciphertext: StackCipherText = 10_i32 + .encrypt_into_with_context(&keyset, nonempty!("users/age")) + .await?; +``` + +The output type selects the `EncryptFrom` implementation. For +`StackCipherText`, that implementation requires an `Encrypt + Clone` source +and an `AeadContext`, which must be nonempty. + +Although `EncryptInto` also has `encrypt_into(&keyset)`, that method requires +the target's context to implement `Default`. `AeadContext` does not, so this +no-context form cannot produce `StackCipherText`. It is available for other +targets with default contexts, such as declarations carrying their own +contexts. + +The scalar derive's implicit context choice is also a discoverability concern: +`#[stash(plaintext = i32)]` generates `type Context = CallerContext`, even +for a record containing only a `StackCipherText` field (whose own context is +`AeadContext`). The generated operation converts between those types. The +reviewer found that hidden choice surprising; whether it needs an API change +or clearer diagnostics/documentation remains undecided. The `Default` bound +belongs to `EncryptInto::encrypt_into`, not to the `EncryptFrom` trait itself. + +### Caller impact and open decisions + +Plaintext-side encryption exists, but is not a symmetric spelling of +`keyset.encrypt(value, context)`: it uses different method names, selects an +output through a target trait, adds a `Clone` bound for this output, and +refuses the empty context accepted by the inherent method. + +Resolve the differing empty-context requirements alongside issue 1; +renaming alone would not make the two paths equivalent. This inconsistency +is separate from the receiver choice below. + +### Receiver choice: plaintext or keyset? + +**Decision (2026-10-24):** use only the keyset receiver for the ordinary +`encrypt` method: + +```rust +tenant_keyset.encrypt(value).await?; +``` + +Do not add a plaintext-side `value.encrypt(&keyset)` counterpart. +**`EncryptInto` remains.** This decision concerns only the `encrypt` method, +not removal of the target-directed encryption API. Implementation is pending; +the current inherent method still requires its context argument (issue 1). + +The two receiver forms express the same inputs and operation. Neither inherently provides +better type safety, ownership behavior, performance, or batching; those +depend on the signatures and implementation, not which argument is `self`. + +| Consideration | Plaintext receiver | Keyset receiver | +| --- | --- | --- | +| Mental model | Transform this value using a capability | Ask this encryption capability to process a value | +| Discoverability | Requires importing an extension trait; methods appear on plaintext types | Encryption methods are found on the encryption handle | +| Fluent composition | Reads naturally after an expression producing plaintext | Reads naturally when selecting a keyset before an operation | +| Operational scope | Keyset and its authority appear as an argument | Keyset and its authority lead the call | +| API surface | Adds methods to primitives and application types | Keeps execution methods on library-owned handles | +| Generic integration | Can work through a cipher trait | Can work through a cipher trait | + +**Rationale:** Stack performs an asynchronous operation through a configured KMS +capability; putting that capability first makes its scope visible and gives +callers one place to discover operations. Plaintext traits can describe how +values are represented without also owning the execution API. + +Keeping both equivalent spellings adds documentation, completion, and naming +choices without adding capability. Familiarity with an earlier API is a +migration consideration, but does not by itself justify two permanent entry +points. + +The target-directed `EncryptInto` surface can choose outputs other than +ordinary ciphertext and is retained explicitly. Its context inconsistency +remains open independently of this receiver decision. No decision about +decryption or batch method receivers is made here. + +**Sources:** [plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto`, `EncryptFrom for StackCipherText`) and +[context types](../../packages/stack-encrypt/src/target/context.rs) +(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, +`src/plaintext/mod.rs`. + +## 4. The scalar derive cannot declare a fixed literal context + +**Status:** Direction agreed; implementation pending. + +The desired combination is a derived output for a bare `i32`, a fixed +context declared on that output, and `10_i32.encrypt_into(&keyset)`. +The current macro rejects `#[stash(plaintext = i32, context = "example")]`: +the literal `context` attribute is supported only with `struct = ..`. + +A supported derive-only workaround wraps the integer in a plaintext struct: + +```rust +struct IntegerInput { + value: i32, +} + +#[derive(stack_encrypt::EncryptFrom)] +#[stash(struct = IntegerInput, context = "example")] +struct EncryptedInteger { + value: stack_encrypt::StackCipherText, +} + +// With EncryptInto imported: +let encrypted: EncryptedInteger = IntegerInput { value: 10 } + .encrypt_into(&keyset) + .await?; +``` + +This form declares the field's context as the pair `("example", "value")`, +rendered as the ZeroKMS descriptor `example/value`, and uses defaultable +`DeclaredContext` for optional caller extensions. It is not a direct +conversion from a bare integer. A manual `EncryptFrom` implementation +can supply its own context; the ordinary literal attribute cannot express +that on a scalar derive today. + +### Agreed direction + +Allow a scalar `plaintext = T` derive to declare a fixed literal context: + +```rust +#[derive(EncryptFrom)] +#[stash(plaintext = i32, context = "example/integer")] +struct EncryptedInteger { + ciphertext: StackCipherText, +} + +let encrypted: EncryptedInteger = 10_i32.encrypt_into(&keyset).await?; +``` + +With a fixed context, the generated operation supplies that context and +exposes a defaultable caller-context type, enabling `encrypt_into`. Without +a fixed context, the ordinary scalar derive continues to use `CallerContext` +and requires `encrypt_into_with_context`. Preserve explicit `context_type` +and `context_field` configurations; their interaction with a fixed context +needs validation rules rather than a silent override. + +This addresses the scalar gap at the derive layer without requiring +`CallerContext` itself to implement `Default` or changing the `EncryptFrom` +trait. A bare scalar has no input field from which the derive can extract +context; runtime context can still be supplied by the caller. + +Additional caller context is accepted. The existing mechanism to reuse is +`DeclaredContext::under(own)`, exposed on operation descriptions through +`Encryption::under(own)`. `DeclaredContext` implements `Default`; its empty +default preserves the declared context unchanged. A caller extension is +combined as `(own, caller)`, never substituted for the declared context. + +Recommended combination semantics follow this existing mechanism: + +| Caller input | Effective structured context, with static context `C` | +| --- | --- | +| No extension | `C` | +| One extension `P` | `(C, P)` | +| Successive extensions `P`, then `Q` via `DeclaredContext::with` | `((C, P), Q)` | +| One extension containing the pair `(P, Q)` | `(C, (P, Q))` | + +This is structured composition, not string concatenation. Order and grouping +matter, and decrypt must reproduce the same structure. `NonEmpty::with` +builds the corresponding pairs directly. The existing test +`under_gives_a_subtree_its_own_context_which_the_callers_extends` in +[target.rs](../../packages/stack-encrypt/tests/target.rs) verifies that an +operation's declared context plus a caller extension agrees with a directly +composed context. + +The exact representation of a static literal (notably text versus a parsed +path label) still needs to be specified when implementing the derive, so +the descriptor and AEAD context remain consistent across encryption and +decryption. This decision is separate from retaining the `EncryptInto` trait. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`a_container_context_requires_a_struct`) and +[generated implementations](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`struct_derive`, `value_derive`). + +## 5. EncryptFrom does not define encrypt_from + +**Status:** Open. + +`EncryptFrom` defines the associated type `Context`, the operation-description +constructor `encryption()`, and `indexes()`. It does not define a function +named `encrypt_from`. + +A function with that name does exist, but on the blanket extension trait +`EncryptInto`: `Target::encrypt_from(&value, &keyset, context)` delegates to +`keyset.encrypt_as`. Calling that function requires `EncryptInto` in scope +and a target implementing `EncryptFrom`. + +This differs from the standard Rust naming relationship: `From` defines +`from`, while `Into` defines `into`. The names suggest that same +relationship here, but `EncryptFrom` declares an operation rather than +executing a conversion, and the target-side execution function belongs to +the source-side extension trait. That makes method discovery and the +required trait import surprising. + +Review the trait and method names, or where the convenience function lives, +while preserving the distinction between declaring an operation and +executing it. No replacement design has been selected. + +**Source:** [target operation traits](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptFrom`, `EncryptInto`, including `EncryptInto::encrypt_from`). + +## 6. Plan builder encrypt_into reads like the execution method + +**Status:** Open; revisit during the plan API discussion. + +`Plan::value::().encrypt_into::()` is an inherent builder method that +selects an output target requiring `T: EncryptFrom`. It does not implement +`EncryptInto` or execute encryption. Sharing the name with +`value.encrypt_into(&keyset).await?` caused confusion during this review. + +Consider renaming the builder method to `target`: + +```rust +Plan::value::() + .target::() + .build() +``` + +This names the selection being made and distinguishes it from executing a +conversion. The rename is a proposal for later discussion, not a decision +or an implemented API change. `EncryptInto::encrypt_into` remains unchanged +by this proposal. + +**Source:** [value plan builder](../../packages/stack-encrypt/src/plan/value.rs) +(`ValueStart::encrypt_into`). + +## 7. Mixed encrypted and plaintext fields lack a direct output-derive spelling + +**Status:** Passthrough attribute proposal accepted in principle; implementation pending. + +For a `User` with encrypted `name` and `email` and plaintext `id` and +`created_at`, the fields-plan API directly supports `.encrypt(pick(..))` +and `.passthrough(pick(..))`. It batches the encrypted fields and returns +`FieldValues`; callers can extract those values into a typed storage struct. + +The current `EncryptFrom` derive has no passthrough/copy attribute for source +fields. `#[stash(default)]` initializes an output from a default or expression; +it does not copy a field from the input. Merely keeping the original field +type does not select passthrough either: derived fields require target traits. + +The proposed `#[stash(passthrough)]` attribute on output fields was accepted +as sensible: it copies the matching source field unchanged, alongside fields +whose types select encryption. Plaintext passthrough fields remain unencrypted and +unauthenticated; making a field part of the encryption context is a separate +choice, not implied by copying it. + +This gap is specific to Stack's output-side `EncryptFrom` derive. Vitamin C +already has plaintext-side `#[derive(Encrypt, Decrypt)]` with +`#[aead(passthrough)]` on source fields. Stack's `keyset.encrypt(&user, aad)` +can consume that derived source, yielding a generic `StackCipherText` map +with sealed and passthrough nodes. It does not generate a named +`EncryptedUser` type or expose the output as typed Rust fields. Its map-entry +AAD derivation also differs from the per-field contexts of a Stack fields +plan; the two routes should not be presented as interchangeable storage +formats. There is currently no source-side derive that generates the typed +encrypted counterpart illustrated in this review. + +**Sources:** [plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) +(`FieldsBuilder::passthrough`) and +[derive field attributes](../../packages/stack-encrypt-derive/src/attrs.rs) +(`FieldAttrs`). + +## 8. Clarify whole-value encryption, layouts, and searchable outputs + +**Status:** Open architectural discussion; recommendations below are not decisions. + +Three user workflows need clear support: + +1. Serialize a complete value into bytes and encrypt those bytes as one + opaque ciphertext. +2. Produce a chosen layout, such as `EncryptedUser`, with independently + encrypted fields and explicit plaintext fields. +3. Produce a layout with additional encrypted forms or search terms, such + as EQL targets. + +The existing Vitamin C plaintext derive produces a structural ciphertext +tree, not a single encrypted serialization. It is a generic output layout +for workflow 2, not workflow 1. The distinction affects key count, +independent field access, visible structure, and authenticated boundaries. +Encoding the completed ciphertext tree into one byte string would not +change those properties into whole-value encryption. + +### Recommended model + +Distinguish the encryption granularity (one encoded value or selected +fields) from the output representation and its requested operations. +Workflow 3 should extend workflow 2 through field targets that request +ciphertext, terms, or both, rather than introducing a separate execution +system. The generic tree can remain a layout option without being the +mandatory scalar storage representation. + +- **Whole-value path:** the follow-up review favors an explicit named + wrapper, provisionally `OpaqueBlob`, which owns serialization and + deserialization through one library-selected codec. It produces one opaque + persistable ciphertext; decrypting into `OpaqueBlob` selects the matching + decode path. Today callers can serialize into `Vec`, which Vitamin C + treats as a single byte leaf, but still encounter the node/leaf storage gap. +- **Layout path:** select a target with `EncryptFrom`/`EncryptInto`, or a + plan, with an explicit passthrough choice. A source-side derive that + generates a target would be authoring convenience over the same system, + not another set of encryption semantics. +- **Searchable layout path:** compose core encryption/index operations + through targets. EQL adapters own their storage representation; Stack + owns execution. Write and query operations must share the relevant + context and normalization rules. + +Keep source representation, requested operations, output encoding, and +execution distinct internally, while giving each workflow a short public +path. An application should not need to learn visitor machinery or plans +to encrypt and persist a scalar. Arbitrary serialization must not become an +implicit replacement for the typed normalization used by indexes. + +Before selecting more trait names, write end-to-end examples for all three +workflows: encrypt, persist, load, decrypt, plus query for searchable targets. +Use those examples to evaluate the public surface and error messages. + +### Follow-up: opinionated whole-value wrapper + +The first public example now uses +`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` rather than +manual JSON encoding. The wrapper name is provisional; the direction is to +make blob serialization an explicit library-owned behavior. The stored result +remains opaque ciphertext, and decryption returns an `OpaqueBlob` that +can be unwrapped. The codec and its authenticated versioning must be specified; +the wrapper does not silently change how unwrapped structs are encrypted. + +Constant-time serialization/deserialization is a desired property to +investigate, not a property established by choosing a binary codec or using +constant-time AEAD. The first revised example provisionally uses Serde traits, +whose arbitrary user implementations cannot carry that guarantee. Define +which lengths and shapes may be public and whether a stricter supported-type +profile is needed before promising secret-independent execution. The examples +document this unresolved boundary separately from the wrapper ergonomics. + +Investigate `serdect` as part of that work. It is already a Stack Encrypt +dependency; check its encoding/decoding guarantees and their scope, and +evaluate applicability to ciphertext serialization and the proposed blob +codec. No whole-object constant-time guarantee or codec choice is inferred +from its use. + +## 9. Ownership of the value-encryption API across Stack and Vitamin C + +**Status:** Open architectural discussion; moving the traits is proposed, +not authorized implementation work. + +The review has repeatedly crossed from Stack's target declarations into +Vitamin C's plaintext visitor traits, cipher traits, contexts, derives, and +ciphertext tree. The user proposes moving the encryption traits into Stack +Encrypt and retaining general primitives such as protected values and +randomness in Vitamin C. + +**Recommendation:** Stack should own the public value-encryption model, +including the plaintext traits/derives, target declarations, contexts, +ciphertext persistence, and execution semantics. Vitamin C can retain +reusable cryptographic and memory-protection primitives. Moving files alone +does not resolve ambiguous names (`Encrypt` currently describes a source), +duplicate entry points, or the missing whole-value/storage path. + +One Stack-owned core layer may serve Rust and language guests without +pulling in KMS transport or HTTP. Applications should still have one +coherent `stack_encrypt` entry point; an internal dependency split need not +become another public API users must understand. + +### Migration constraints + +- Today `stack-encrypt` depends on both `vitaminc-aead` and + `vitaminc-encrypt`, while `vitaminc-encrypt` also depends on + `vitaminc-aead`. Moving the latter's traits into the top-level Stack crate + and making the existing backend depend on Stack would introduce a cycle. + Move the value-level backend adapters too, or separate a lower-level + core/primitive boundary. +- Context composition, type tags, ciphertext framing, index normalization, + and binding transport frames have compatibility consequences. Preserve + their existing encodings when changing ownership; any deliberate semantic + change needs separate versioning and migration treatment. +- Retain one canonical trait/type identity. Compatibility re-exports can + preserve an old path where dependencies permit; copied independent trait + definitions would require duplicate implementations and adapters. +- Account for guest/FFI consumers and EQL target implementations, not just + Rust application imports. The no-HTTP/WASM path must remain usable. + +**Sources:** [Stack dependencies](../../packages/stack-encrypt/Cargo.toml), +[guest ABI dependencies](../../packages/stack-guest-abi/Cargo.toml), and +[EQL encryption adapters](../../packages/eql/crates/eql-bindings/src/encryption.rs). +The installed `vitaminc-encrypt` 0.8.0 manifest confirms its dependency on +`vitaminc-aead`. + +## 10. Public encrypt_as duplicates encrypt_into + +**Status:** Decision agreed; implementation pending. + +`KeysetCipher::encrypt_as` is currently an inherent method requiring the +output to implement `EncryptFrom`. It executes that target's operation +declaration. `EncryptInto::encrypt_into` and +`EncryptInto::encrypt_into_with_context` delegate to it, supplying either +the default context or the caller's context. + +**Decision:** remove `encrypt_as` from the public API. Keep `encrypt_into` +and `encrypt_into_with_context` as the public target-directed entry points: + +```rust +let encrypted: EncryptedInteger = value.encrypt_into(&keyset).await?; +let encrypted: EncryptedInteger = value + .encrypt_into_with_context(&keyset, context) + .await?; +``` + +The operation-execution machinery may remain internal; removing the public +method does not remove target declarations, batching, or execution. Public +examples should use the retained entry points when illustrating the proposed +API. Earlier descriptions of `encrypt_as` in this log describe current code. + +This is distinct from issue 3's decision about ordinary encryption: +`keyset.encrypt(value)` remains the sole ordinary `encrypt` spelling, while +`value.encrypt_into(&keyset)` selects a particular output type. The +`encrypt_from` naming/location question in issue 5 remains open. + +**Source:** [target operations](../../packages/stack-encrypt/src/target/operations.rs) +(`KeysetCipher::encrypt_as`, `EncryptInto`). + +## 11. Context-field storage does not yet extract context from an input struct + +**Status:** Open. + +`#[stash(context_field)]` exists on derived output types. On encryption it +stores the supplied context in that output field; on decryption it uses the +stored context, optionally checking an independently supplied expected value. +It does not automatically select a field from a plaintext input struct. + +The macro currently treats this field as the complete context and rejects +it alongside a literal `context`. Since `struct = User` requires a literal +context, that form cannot currently mark its `id` with `context_field`. +The public examples originally described passing the id explicitly without +explaining this distinction. + +### Clarified model: context source and context storage are independent + +Two similar scenarios must be distinguished: + +1. The caller supplies context as a function argument; the encrypted output + stores it for later use. This is the existing scalar-output + `context_field` behavior. +2. The encryption declaration obtains context from one or more fields of + the plaintext. Whether those values are also stored is a separate choice. + +The second scenario can include storage too: read `User.id`, use it to bind +the encrypted fields, and carry it into `EncryptedUser.id`. The caller need +not pass the same value twice. Several fields, such as a tenant id and user +id, can supply a structured composite context. + +| Context source | Stored in the encrypted output? | Recovery requirement | +| --- | --- | --- | +| Caller argument | Yes | Read the stored context; optionally check an independently supplied expected value | +| Caller argument | No | Supply the same context again | +| Plaintext field(s) | Yes | Reconstruct context from the stored field(s); optionally check expected values | +| Plaintext field(s) | No | Supply equivalent context independently before decrypting | + +Store a source-derived context independently of the ciphertext that needs +it, so decrypt does not depend on first opening that ciphertext to discover +its own context. Binding ciphertext to a stored context authenticates that +relationship; it does not by itself establish that the entire record is at +the destination the application expected. + +The derive should express source selection and output storage as separate +concepts, even if a shorthand covers the common read-and-store case. Avoid +assuming that one `context_field` attribute should silently perform both. +Syntax remains open. Define field ordering/grouping for composite context, +literal prefixes, optional caller extensions, `from` mappings, and expected +context checks. These capabilities are proposed, not implemented. + +The recommended common-case shorthand is `context(field = "id")`, expanding +exactly to `encrypt(field = "id"), decrypt(field = "id")`. It selects the +same Rust field name on the two different inputs; it does not implicitly +copy or store that field. Keep the directional form for different names or +sources. Initially reject mixing shorthand and directional selectors to +avoid ambiguous precedence. The context examples now use this shorthand +for the plaintext-id-to-stored-id case. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`Record::parse`, `context_type`) and +[context-field output assembly](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`assignments`, `value_derive`). + +## 12. Default is not the same as optional caller context + +**Status:** Open; exposed by the directional-context examples. + +`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` make the +no-context form available whenever the relevant associated `Context` type +implements `Default`. That works for deliberately defaultable types such as +`DeclaredContext` and `ExpectedContext`, and currently excludes +`CallerContext` and `AeadContext`. + +However, a required caller value can have a Rust default: if a declaration +exposes `Context = u64`, the no-context form supplies zero. The directional +proposal `encrypt(caller = u64)` must not accidentally mean "optional, with +zero as the default". + +Separate the application's context value type from the trait's caller-input +type or capability. Options include a non-defaultable `RequiredContext` +wrapper that accepts ergonomic conversions from `T`, or a dedicated marker +controlling context-free entry points. No mechanism is decided yet. + +This matters equally to required context on decrypt. Source-derived context +can require no encrypt argument, while decrypt may require an external +value or accept an optional expectation of stored context. Do not assume the +two associated input types must match, or that either always equals the +complete logical context used by encryption. + +See the [context scenario examples](2026-10-24-stack-encrypt-context-examples.md) +for the proposed selectors and input requirements. + +**Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md new file mode 100644 index 000000000..c927b7d67 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -0,0 +1,723 @@ +# Context sources and storage: directional API examples + +Date: 2026-10-24. Design exploration accompanying the +[API review log](2026-10-24-stack-encrypt-api-review.md) and +[complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md). + +**Everything introduced here is proposed syntax, not an implemented API.** +The purpose is to compare context behavior across the meaningful source and +storage combinations before selecting attributes or changing traits. The +`Ciphertext` and `passthrough` proposals are inherited from the workflow draft. + +## Recommendation: directional sources, separate storage + +Borrow Serde's nested directional attribute style: it supports separate +serialization and deserialization names within one `rename` attribute. +See [Serde's field attributes](https://serde.rs/field-attrs.html). + +For encryption, use direction names to say **where to obtain context**: + +```rust +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(field = "id") +))] +``` + +The two `field` selectors refer to different objects: the plaintext input +when encrypting, and the encrypted input when decrypting. They must yield the +same typed context value. Unlike a Serde rename, these are not independent +cryptographic names: changing the context value or its shape between the two +directions prevents decryption. + +For the common case where both directions use the same Rust field name, +the recommended shorthand is: + +```rust +#[stash(context(prefix = "users", field = "id"))] +``` + +This expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. +It still reads two different objects: the plaintext input on encrypt and the +encrypted input on decrypt. It does not copy or store the field implicitly; +the output layout must make that value available, normally with passthrough. +Use the explicit directional form when the names or sources differ. + +Use a separate field annotation to **store the computed context** when it is +not already carried by ordinary passthrough fields: + +```rust +#[stash(from_context)] +scope: u64, +``` + +`from_context` fills an output metadata field from the computed logical +context. It does not choose the context source. A passthrough source field +can already carry the context value, in which case no `from_context` field +is needed. No new source-side encryption methods or public `encrypt_as` +entry point are introduced. + +This is preferable to bare `context(in)` / `context(out)`: either term could +mean an operation direction, an input/output object, or storing a value. +`encrypt`, `decrypt`, and `from_context` give those roles distinct names. + +## Proposed vocabulary and semantics + +| Syntax inside `context(...)` | Meaning | +| --- | --- | +| `prefix = "users"` | Static base for this layout's field contexts | +| `field = "id"` | Shorthand for both `encrypt(field = "id")` and `decrypt(field = "id")` | +| `encrypt(caller = u64)` | Obtain the logical context from a required `u64` argument | +| `encrypt(field = "id")` | Obtain it from the plaintext input's `id` | +| `decrypt(caller = u64)` | Obtain it from a required `u64` argument on decrypt | +| `decrypt(field = "scope")` | Obtain it from the encrypted input's cleartext `scope` | +| `encrypt(tuple(field = "tenant_id", field = "id"))` | Build one ordered, typed pair from plaintext fields | +| `decrypt(tuple(field = "id", caller = u64))` | Build one pair from a stored field and the decrypt argument | + +`tuple(...)` is a declarative selector, not a Rust expression or a function +that gets access to keys. It can nest. `field` names resolve at compile time; +the first design supports direct fields, not arbitrary paths or callbacks. +There is at most one caller argument selector per direction; use a tuple or +typed context object as that argument when it carries several values. + +Use either the shared `field` shorthand or both explicit directional +selectors. For the initial API, reject mixing the shorthand with directional +selectors instead of inventing precedence rules. That keeps the expansion +predictable; a record with asymmetric names uses the explicit form. +`context = "users"` +remains the existing simple spelling for a static prefix with optional caller +extensions; it is used in scenario 0. + +### Composition rule + +Call the logical row context `R`. For a field named `name`, its static base is +`B = ("users", "name")` and its effective context is `(B, R)`. With only a +static prefix and no extension it is just `B`, not `(B, ())`. + +The selector's result is **one structured value**. A pair `(tenant_id, id)` +produces `(("users", "name"), (tenant_id, id))`, not +`((("users", "name"), tenant_id), id)`. `from_context` stores `R`, not the +field-dependent effective context. Static prefixes and field identities are +reconstructed from the declaration. + +Names in this proposal do not change existing byte encodings. Mapping new +selectors onto the existing context machinery must preserve types, order, +grouping, and descriptor rendering. A new explicit selector is a new +declaration; it must not silently reinterpret existing stored data. + +### Storage and expected context + +When all of `R` can be reconstructed from stored fields, the default decrypt +call uses those fields. The caller may additionally supply an expected `R` +through the existing `ExpectedContext` pattern: check equality first, then +use the stored value. That expected value is **not another extension**. + +When a decrypt selector needs a caller argument, that argument is required +to construct `R`; no no-context convenience is available. It is not an +expected-value check on context already fully available from the record. + +If a future caller needs both a required input and a separate expectation +on the assembled `R`, use a distinct checked-context input or a named API. +Do not overload one unlabelled argument to mean both. That combined check +API remains outside this proposal; scenario 7 makes the need visible. + +## Scenario matrix + +| Scenario | Encrypt obtains context from | Context available in stored output | Decrypt obtains context from | +| --- | --- | --- | --- | +| 0 | Static prefix; no row context | Declaration | Declaration | +| 1 | Caller argument | Not stored | Caller argument | +| 2 | Caller argument | Dedicated metadata field | Metadata field | +| 3 | Plaintext `id` | No cleartext copy; `id` itself is encrypted | Caller argument | +| 4 | Plaintext `id` | Passthrough `id` | Stored `id` | +| 5 | Plaintext `(tenant_id, id)` | Two passthrough fields | Same pair of stored fields | +| 6 | Plaintext `id` plus caller scope | Dedicated pair-valued metadata field | Stored pair | +| 7 | Plaintext `id` plus caller scope | Only `id` is stored in cleartext | Stored `id` plus caller scope | +| 8 | Plaintext `(tenant_id, id)` | Neither component is stored in cleartext | Caller supplies the pair | + +An encrypted copy of a context field does not make that context available +before decrypt. Scenarios 3 and 8 retain encrypted copies to recover the +complete plaintext object; callers still need the context independently. + +## Shared code for the examples + +Each scenario defines an output type and an async function. Combine this +prelude, the scenario blocks, and the final `main` into one candidate program. +They use real file serialization through the same helper, so no example +relies on a context value being secretly carried inside an in-memory handle. +All use proposed APIs and therefore cannot yet be typechecked against Stack. + +Dependencies match the workflow draft: Stack Encrypt, Serde with derives, +serde_json, tempfile, and Tokio. ZeroKMS credentials are required once these +APIs exist. `scope = 7` in caller examples is external application context, +not a keyset selector or an authorization credential. + +```rust +use serde::{de::DeserializeOwned, Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, NonEmpty, + StackCipher, +}; +use std::error::Error; + +type Result = std::result::Result>; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_user() -> User { + User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +// The candidate Ciphertext Serde contract encodes its canonical envelope. +fn store_and_reload(value: &T) -> Result { + let directory = tempfile::tempdir()?; + let path = directory.path().join("record.json"); + std::fs::write(&path, serde_json::to_vec(value)?)?; + Ok(serde_json::from_slice(&std::fs::read(path)?)?) +} +``` + +## 0. Static prefix only + +This is the baseline. There is no dynamic row context to store or supply. +It retains the existing static-context form and proposed passthrough support. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct StaticUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_0() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StaticUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +For this shorthand, caller extensions remain available as previously agreed: +encrypt and decrypt must receive the same extension. The remaining examples +use explicit directional selectors, so their argument requirements follow +those selectors; there is no additional implicit extension layered on top. + +## 1. Caller context, not stored + +`scope` is absent from the output. The caller must provide it again. A +plaintext `id` in this layout is unrelated to the chosen context source. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(caller = u64) +))] +struct ExternalScopeUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_1() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: ExternalScopeUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context: `7_u64`. Effective name context: `(("users", "name"), 7_u64)`. + +## 2. Caller context, stored in an output metadata field + +The source is still the function argument. `from_context` stores its value; +the decrypt selector explicitly chooses that stored field. `scope` is output +metadata, not a field to add to the recovered `User`. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(field = "scope") +))] +struct StoredScopeUser { + #[stash(from_context)] + scope: u64, + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_2() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredScopeUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.scope, 7); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context remains `7_u64`. It is stored once, not appended a second time +on decrypt. This generalizes the existing caller-supplied-and-stored case to +the field-by-field layout. + +## 3. Plaintext field as context, no cleartext context stored + +The encrypt selector reads `User.id`; no argument is needed. The output's +`id` is encrypted, so it cannot supply the context required to open itself. +The reader obtains the id from its external record lookup. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(caller = u64) +))] +struct HiddenIdUser { + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_3() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let lookup_id = user.id; + + let stored: HiddenIdUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, lookup_id).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `R = 42_u64`. Supplying a different id fails to +open the encrypted fields. The id is not dropped or defaulted when rebuilding +`User`; its encrypted copy is opened using the externally supplied context. + +## 4. Plaintext field as context, stored as the corresponding output field + +This is the natural ID-bound form of workflow example 2. The context selector +and the passthrough annotation have separate jobs: select `id`, and copy it. +It uses the common-field shorthand, equivalent to the explicit pair of +directional selectors introduced above. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context(prefix = "users", field = "id"))] +struct IdBoundUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_4() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: IdBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + + // Alternatively, require the stored context to match the lookup's id. + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context( + loaded, + &cipher, + NonEmpty::from(user.id), + ) + .await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The second decrypt call uses the `ExpectedContext` conversion. It checks +the stored id instead of constructing `(stored_id, expected_id)`. Changing +`id` alone breaks decryption; moving an entire consistent record is detected +only when the reader supplies the expected id (or another placement check). +`created_at` remains ordinary unauthenticated passthrough metadata. + +## 5. Several plaintext fields as context, stored in separate fields + +Add a tenant id to the input. The context is the ordered pair `(tenant_id, id)`. +The output renames `id` to `user_id` to demonstrate why direction-specific +field selectors are useful. No duplicate context metadata field is needed. + +```rust +#[derive(Debug, PartialEq)] +struct TenantUser { + tenant_id: u64, + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_tenant_user() -> TenantUser { + TenantUser { + tenant_id: 7, + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(tuple(field = "tenant_id", field = "user_id")) +))] +struct TenantBoundUser { + #[stash(passthrough)] + tenant_id: u64, + #[stash(passthrough, from = "id")] + user_id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_5() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + + let stored: TenantBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = TenantUser::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `(7_u64, 42_u64)`. Rust field order does not choose +the context order: the selector does. A Serde wire rename also does not change +the selector; it names Rust fields, not serialized property names. + +## 6. Plaintext field plus caller argument, store the combined context + +Here `R = (User.id, caller_scope)`. `binding` stores that complete pair. The +original id is also retained as ciphertext so `User` can be reconstructed +without adding a new rule that maps pieces of metadata into plaintext fields. +That deliberately redundant representation keeps this example focused on +context composition rather than projection conveniences. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(field = "binding") +))] +struct StoredCombinedUser { + #[stash(from_context)] + binding: (u64, u64), + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_6() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredCombinedUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.binding, (42, 7)); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The encryption function accepts only the caller-supplied component (`7`), not +the pair the derive can construct itself. A checked decrypt would expect the +whole reconstructed pair `(42, 7)`, not just that caller component. + +## 7. Plaintext field plus caller argument, store only the field component + +Store `id` as a normal field, but keep the external scope out of the output. +Decryption combines the stored id with the scope supplied by the reader. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(tuple(field = "id", caller = u64)) +))] +struct PartiallyStoredUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_7() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: PartiallyStoredUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This also constructs `(42_u64, 7_u64)` in both directions. It shows why +"stored context" cannot always be represented as a single yes/no flag: +some components may be stored and others supplied externally. A separate +expected-id check can be performed against `loaded.id` before opening when +the application also needs to assert the record's destination. + +## 8. Several plaintext fields as context, keep neither component in cleartext + +The encrypt side obtains `(tenant_id, id)` from the source. The reader supplies +the complete pair from outside the encrypted record; both source fields are +recovered from ciphertext after that context has been established. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(caller = (u64, u64)) +))] +struct HiddenTenantUser { + tenant_id: Ciphertext, + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_8() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + let lookup_context = (user.tenant_id, user.id); + + let stored: HiddenTenantUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = + TenantUser::decrypt_from_with_context(loaded, &cipher, lookup_context).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This is a new typed selector contract. It does not mean flattening the pair +into text, or treating separately chained extensions as interchangeable with +one tuple-valued argument. + +## Candidate program entry point + +```rust +#[tokio::main] +async fn main() -> Result<()> { + example_0().await?; + example_1().await?; + example_2().await?; + example_3().await?; + example_4().await?; + example_5().await?; + example_6().await?; + example_7().await?; + example_8().await?; + Ok(()) +} +``` + +## Consequences for the traits and derive + +The logical context `R` is not always the value a caller passes. For example: + +| Scenario | Logical `R` | Caller input on encrypt | Caller input on decrypt | +| --- | --- | --- | --- | +| 1 | `u64` | Required `u64` | Required `u64` | +| 2 | `u64` | Required `u64` | Optional expectation of `u64` | +| 3 | `u64` | None | Required `u64` | +| 4 | `u64` | None | Optional expectation of `u64` | +| 6 | `(u64, u64)` | Required `u64` component | Optional expectation of the whole pair | +| 7 | `(u64, u64)` | Required `u64` component | Required `u64` component | +| 8 | `(u64, u64)` | None | Required pair | + +Today `EncryptFrom::Context` and `DecryptInto

::Context` already belong +to separate traits and can differ. They describe the input the caller needs +to supply, not necessarily the final context used by every operation. That +distinction should be explicit in documentation; `ContextInput` might be a +clearer name, but a rename is not decided here. + +No-caller-input selectors can expose a defaultable input, such as `()`, to +support `encrypt_into`. Stored-context decrypt uses the existing +`ExpectedContext` idea. Required caller selectors must remain required +even if their Rust type implements `Default`: a required `u64` must not +silently become zero merely because `u64: Default`. + +That exposes a real constraint in today's convenience methods, whose only +test is `Context: Default`. An implementation of this proposal would need +a non-defaultable `RequiredContext` input (with an ergonomic conversion +from `T`), or a separate trait capability controlling context-free calls. +The examples show bare caller values at call sites, but do not assume that +the associated input type can be bare `u64`. Choose that mechanism before +claiming the directional API enforces required inputs. + +The current public `Encryption` constructors must also gain or expose the +core-owned source-selection machinery needed by the derive. Existing plan +context fields demonstrate source-dependent context, but are label-based, +carry the field, and use different composition rules. They are not already +an implementation of this proposal's general typed, optionally stored `R`. + +## Rules the implementation should enforce + +- Selectors for both directions must produce the same context type and + ordered structure. Different Rust field names are fine; different values + or grouping are not a supported way to encrypt and decrypt one record. +- Resolve field names and types at compile time. A decrypt selector cannot + read an encrypted field to obtain the context needed to decrypt it. +- A `from_context` field is cleartext output metadata with the logical + context's type. It cannot also be encrypted, defaulted, or independently + copied from an unrelated source. Ordinary passthrough fields do not + become context merely because their names are `id` or `tenant_id`. +- Extra metadata is not invented as a plaintext field during recovery. + Every original plaintext field must have a defined recovery path; no + silent omission or defaulting to compensate for a context-only source. +- Check supplied expectations before key retrieval where possible. A + context value read from storage remains untrusted until encrypted fields + authenticate under it. An expectation check alone is not authentication. +- If a selected source value is duplicated in stored context metadata and + in a separately recovered plaintext field, validate their agreement + before returning the reconstructed object. Do not return a different id + from the one the encrypted fields were bound to. +- Stored context should be required in the serialized schema; a missing + field must not silently fall back to a default context. Empty-context + policy must be explicit and retain the current nonempty target rule. +- For indexed targets, the selected context also scopes their terms unless + a future API explicitly separates index context from ciphertext context. + A row-specific id therefore changes search semantics; these examples + concern ciphertext layouts and do not silently promise cross-row queries. + +Required tests include storage round trips for every scenario, missing +required arguments, wrong expected values, altered stored context, tuple +reordering/regrouping, renamed source/output fields, and attempted circular +decrypt dependencies. Same-typed field swaps may evade a type check, so +round-trip and mismatch tests remain necessary even with typed selectors. + +## Alternatives worth comparing + +1. **Field-level `context(encrypt, decrypt)` flags.** Concise for one id, + but references the output declaration while the encrypt source lives + elsewhere, leaves multi-field order implicit, and cannot readily describe + caller-plus-field composition. Prefer explicit container selectors first. +2. **One `context = expression` callback.** Flexible, but hides dependencies, + makes reverse reconstruction and cycle checks harder, and encourages + arbitrary application code in the declaration. Keep simple declarative + selectors as the initial surface. +3. **A typed context object with a dedicated derive.** Could reduce repetition + for large composite contexts and carry a stronger encoding contract. + Consider it after the scalar/tuple cases establish the desired semantics. + +The recommendation is the directional selector plus independent storage +model, not a commitment to every spelling shown here. In particular, +`from_context`, `prefix`, the tuple notation, and how required inputs are +represented remain review choices. + +## Evidence and validation + +Current behavior was checked in +[context types](../../packages/stack-encrypt/src/target/context.rs), +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[derive validation](../../packages/stack-encrypt-derive/src/shape.rs), and +[plan context fields](../../packages/stack-encrypt/src/plan/build.rs). +The directional nesting is inspired by Serde, not provided by Serde. + +These are candidate API examples. Syntax/link checks do not establish macro +support, type correctness, or encryption behavior. No live encryption or +database tests are claimed, and no library implementation is changed. + +Validation completed: `rustfmt` parsed the combined nine-scenario candidate +program and the introductory fragments (wrapped as Rust items), using edition +2024. Local links, Markdown fences, and trailing whitespace were checked in +this document and both companion documents. No proposed macro was expanded +or typechecked. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md new file mode 100644 index 000000000..da361fc04 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -0,0 +1,572 @@ +# Stack Encrypt: complete public API examples for review + +Date: 2026-10-24. Companion to the +[API review log](2026-10-24-stack-encrypt-api-review.md). +The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) +explore directional source selection, storing context, and decrypt inputs +across nine scenarios. Their attributes are proposals, not implemented APIs. + +This document tests the proposed public API against three complete workflows: +encrypt, persist, load, and decrypt a user; the searchable workflow also +constructs and applies a query. It is an API design draft, not documentation +for a released version. The examples deliberately leave plans and operation +descriptions out of the application code. + +Each main example is a separate program, with all imports and types shown. +They use the same user fields and synthetic data. `created_at` is an `i64` +Unix timestamp in seconds so that date-library configuration does not distract +from the encryption API. The storage files live in a temporary directory and +are removed when the program exits. Files stand in for a database boundary; +the examples really encode, write, read, and decode their stored forms. + +## What is agreed, and what these examples propose + +| Surface | Status in this review | +| --- | --- | +| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | +| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | +| No public `encrypt_as` in these examples | Agreed direction; current implementation still exposes it | +| `#[stash(passthrough)]` on output fields | Accepted in principle; needs implementation on both encrypt and decrypt derives | +| A fixed context on a scalar output derive | Agreed direction; not required by these particular examples | +| `OpaqueBlob` | Agreed wrapper direction; provisional name. Requests whole-value serialization and one encrypted leaf, using a library-owned codec | +| `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | +| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | +| `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | +| `Ciphertext` as an encryption/decryption target | Proposed scalar target, including support for `String` and `Vec`; does not silently flatten a struct into a blob | +| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | +| `cipher.decrypt_with_context` | Proposed companion to `encrypt_with_context`; ordinary decrypt naming remains a review decision | +| `User::decrypt_from` / `decrypt_from_with_context` | Existing `DecryptFrom` extension conveniences, retained here pending the decrypt naming discussion | +| `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | + +The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or +passthrough APIs. They are complete candidate programs, not programs claimed +to compile against today's crate. The names and contracts above remain +subject to review where marked proposed or provisional. + +The proposed opaque ciphertext is intentionally untyped: the decrypt call's +expected type selects decoding. A typed wrapper remains an open alternative. +Its byte decoder parses a stored envelope; authentication happens on decrypt. +For a derived field it must preserve the existing context and leaf encoding, +not introduce a new cryptographic format just to improve ergonomics. + +The trait name `DecryptFrom` is somewhat surprising: it supplies the +convenience functions, while `#[derive(DecryptInto)]` describes how an +encrypted output recovers plaintext. These examples keep that existing +distinction visible rather than silently redesigning it. + +## Setup shared by the programs + +The proposed programs use `stack-encrypt` with its default `http` feature, +`serde` with `derive`, `serde_json`, `tempfile`, and `tokio` with `macros` and +`rt-multi-thread`. No published version currently supplies all proposed APIs, +so this draft does not give a misleading installable Cargo manifest. + +`StackCipher::new()` uses the configured ZeroKMS credentials: a CLI login or +`CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. +The second client in each program represents a later reader using the same +authorized configuration. The default keyset is selected for brevity; an +already-selected tenant keyset can be used in its place. + +Decryption uses the client because the sealed value carries its keyset id. +It still requires access to the corresponding keys and the correct context. +The examples do not depend on in-memory fake keys surviving a reload. + +## 1. Encrypt the whole user through an explicit blob wrapper + +**Intent:** store one encrypted document. All four fields, including the ID +and creation time, are inside one encrypted serialization. There are no +independently readable fields or search terms in the stored ciphertext. + +`OpaqueBlob::new(user)` explicitly requests that treatment. Stack owns the +serialization and matching deserialization, rather than requiring every +application to select and invoke a codec. The proposed wrapper owns the input; +`new` only wraps it, and serialization errors surface from the encryption +operation. The context identifies the application's document schema and binds +this document to an externally known record id. + +For this draft the wrapper accepts Serde `Serialize` on write and +`DeserializeOwned` on read. That input-trait choice is provisional: it does +not claim that arbitrary Serde implementations can satisfy a constant-time +contract. The initial wrapper has one library-selected, versioned binary +codec, not a caller-selected codec parameter. Selecting and specifying that +codec remains implementation/design work; no particular binary format is +claimed to be implemented or constant-time here. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{nonempty, Ciphertext, OpaqueBlob, StackCipher}; +use std::error::Error; + +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let expected_id = 42_u64; // Supplied by the application's record lookup. + let user = User { + id: expected_id, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + let expected = user.clone(); // Retained only for the round-trip assertion. + + // The wrapper requests one blob and lets Stack handle serialization. + let sealed: Ciphertext = keyset + .encrypt_with_context( + OpaqueBlob::new(user), + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + + // Persist the complete ciphertext envelope as binary bytes. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.ciphertext"); + std::fs::write(&path, sealed.to_bytes())?; + + // Later: reconstruct the ciphertext from storage using a new client. + let reader = StackCipher::new().await?; + let stored_bytes = std::fs::read(&path)?; + let loaded = Ciphertext::from_bytes(&stored_bytes)?; + + // Stack authenticates/decrypts first, then decodes the blob into User. + let opened: OpaqueBlob = reader + .decrypt_with_context( + loaded, + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + let restored = opened.into_inner(); + assert_eq!(restored, expected); + Ok(()) +} +``` + +**What the API must guarantee:** the serialized bytes are one encrypted leaf +under one data key. The envelope contains the retrieval metadata, so the +caller never needs to save an IV or key tag separately. A different context +must fail to open it. Stack owns the matching encode/decode pair and its +compatibility contract. A blob-format discriminator and version must be +authenticated, for example inside the encrypted payload, so decoding does +not rely on guessing a codec or on unauthenticated format metadata. This +can reuse the canonical outer ciphertext envelope; the blob payload is a +new, explicit format rather than silently reinterpreting previously encrypted +raw bytes. Application schema evolution remains explicit: a codec version +does not automatically migrate changes to `User`. + +**Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` +form. A matching no-context `reader.decrypt(loaded)` is proposed for symmetry; +it is not the signature of today's two-argument decrypt method. This document +uses explicit context in the full example so the saved format is unambiguous. + +**Meaning of the wrapper:** `OpaqueBlob` is the plaintext-side request +and the decoded value, not the stored ciphertext type. `Ciphertext` remains +the persisted envelope. `into_inner()` returns an ordinary `User`. The +wrapper does not by itself promise to zeroize arbitrary input/returned types; +the codec should keep temporary plaintext buffers in wipeable storage. +`#[derive(Encrypt)]` on `User` currently means a structural ciphertext tree +and is not a substitute for wrapping the value as one blob. + +### Timing contract to investigate + +The desired property is serialization/deserialization whose timing does not +depend on secret contents, under an explicit policy for lengths and shape. +Constant time does not mean identical elapsed time for differently sized +messages; what may vary must be declared public. The general requirement is +secret-independent control flow and memory access, including outside the +cryptographic primitive. See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). + +A library-owned binary codec gives us control over that implementation, but +does not establish the property by itself. Strings, collection lengths, +enum variants, validation errors, allocations, and user-defined Serde code +can all introduce value-dependent work. Supporting arbitrary `Serialize` / +`Deserialize` implementations cannot guarantee constant-time behavior for +the whole operation. + +Before claiming such a guarantee, define supported types, permitted public +shape/length information, and restrictions on custom implementations. A +stricter profile may need bounded shapes, fixed-width encodings, padding, +and controlled encoders/decoders, with implementation-level verification. +Padding ciphertext to a fixed size alone does not make the preceding +serialization constant-time. Authentication must complete before decoding +the plaintext blob. This is a design requirement to investigate, not a +guarantee made by these example programs or a constant-time audit result. + +**Investigation candidate: `serdect`.** It is already a dependency of +`stack-encrypt`. Review its supported encodings and precise constant-time +guarantees, including decoding/error behavior and length assumptions. Assess +where it can help with ciphertext/byte serialization and whether any of its +techniques apply to the `OpaqueBlob` codec. Do not assume guarantees for byte +encoding extend to serializing arbitrary objects through Serde. This is a +follow-up investigation, not a completed assessment or codec selection. + +## 2. Encrypt selected fields into a chosen output layout + +**Intent:** store an ordinary record with plaintext `id` and `created_at`, +and independently encrypted `name` and `email`. The output type states the +layout. The two encrypted fields need no search indexes. + +The proposed ciphertext Serde implementation makes persistence compositional: +ordinary Serde derives on the output encode plaintext fields normally and +ciphertext fields as Base64 envelopes. There is no manual `Node` traversal. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let user = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + // The output type selects the conversion and declares its own contexts. + let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + assert_eq!(encrypted.id, user.id); + assert_eq!(encrypted.created_at, user.created_at); + + // Persist the typed output, including both plaintext and ciphertext fields. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: deserialize the same output layout and recover the plaintext type. + let reader = StackCipher::new().await?; + let loaded: EncryptedUser = serde_json::from_slice(&std::fs::read(&path)?)?; + let restored: User = User::decrypt_from(loaded, &reader).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +**Context and execution:** `name` uses the structured context +`("users", "name")`, and `email` uses `("users", "email")`. The keyset is +shared, but each encrypted field receives its own data key. One conversion +batches those key requests. The decrypt derive reconstructs `User`, opening +the ciphertext fields and copying the passthrough fields back. + +**Plaintext semantics:** copying `id` does not automatically bind the sealed +fields to it. Both plaintext fields remain unencrypted and unauthenticated. +That is the requested passthrough behavior, not a promise of whole-record +integrity. The example's name/email contexts distinguish columns, not rows. + +If the application wants row binding, it can use the existing +`user.encrypt_into_with_context(&keyset, expected_id)` entry point and the +matching `User::decrypt_from_with_context(loaded, &reader, expected_id)`. +The effective contexts then become `(("users", "name"), expected_id)` and +`(("users", "email"), expected_id)`. The reader supplies the expected id +from its record lookup. That is an explicit variation, not an automatic +effect of the passthrough attribute. + +**Existing `context_field` support:** the derive does have +`#[stash(context_field)]`, but its current behavior is to store the +caller-supplied context in an output field and use it again when decrypting. +It does not read an id from the plaintext `User`. It also supplies the complete +context: the macro rejects combining it with a literal `context = "users"`. +Because the current `struct = User` form requires that literal, the attribute +cannot currently be used on example 2's field-by-field declaration. + +**Separate source from storage:** obtaining context from plaintext fields +and carrying context into the encrypted output are independent choices. +An extended derive could read `User.id`, combine it with the declared field +contexts, and also copy it into `EncryptedUser.id`. That would enable row +binding without passing the id twice. The same model should accommodate +several source fields, such as `(tenant_id, id)`, with explicit ordering and +grouping. + +Alternatively, context can be passed as an argument and stored, passed and +not stored, or derived from plaintext fields without being stored. In either +non-stored case, the reader needs equivalent context from elsewhere before +decrypting. Source selection and storage should be separately expressible; +whether one annotation can conveniently request both remains a syntax choice. +Today's `context_field` only covers the caller-supplied-and-stored case. + +For the source-derived-and-stored case, a reader could use the stored id, +with an optional independently supplied expected id to reject a whole record +moved to another destination. Changing the stored id alone must fail +decryption. The values needed to reconstruct context must be available +without first opening the ciphertext that requires them. + +**Review point:** the declaration is concise, but requires an explicit +`EncryptedUser` type. A future source-side derive could generate that type; +it should generate this same layout and context behavior. Also decide whether +`User::decrypt_from(...)` is the desired public decrypt spelling before +adding a competing no-context `loaded.decrypt_into(...)` convenience. + +## 3. Use the same layout model with a searchable email target + +**Intent:** keep the same plaintext fields, encrypt the name normally, and +store the email's ciphertext plus an equality term. Search terms come from +the original typed plaintext through their declared operation, not from the +JSON encoding used to persist the output. + +`SearchableEmail` is an application-defined scalar target. It uses the caller +context supplied by the outer record: `("users", "email")`. It does not +declare another fixed context or append another `email` component. The +`ciphertext` and `equality` member names are output layout, not additional +cryptographic context segments. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::{ + nonempty, Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, + StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct SearchableUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let alice = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + let bob = User { + id: 43, + name: "Bob".to_owned(), + email: "bob@example.com".to_owned(), + created_at: 1_700_000_100, + }; + + // One collection conversion batches key generation across both users. + let users = vec![alice, bob]; + let encrypted: Vec = users.encrypt_into(&keyset).await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("users.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: a query uses the same keyset, field context, and equality operation. + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let wanted_email = "alice@example.com".to_owned(); + let query: EqualityTerm = wanted_email + .encrypt_into_with_context( + &query_keyset, + nonempty!("users").with("email"), + ) + .await?; + + // A file-backed comparison demonstrates matching without a SQL adapter. + let loaded: Vec = + serde_json::from_slice(&std::fs::read(&path)?)?; + let matches: Vec = loaded + .into_iter() + .filter(|row| row.email.equality == query) + .collect(); + assert_eq!(matches.len(), 1); + + // The query term cannot decrypt. Open the matched ciphertext rows in a batch. + let restored: Vec = Vec::::decrypt_from(matches, &reader).await?; + assert_eq!(restored.as_slice(), &users[..1]); + Ok(()) +} +``` + +The filter is an in-memory scan over persisted encrypted records, not a +claim of database index use. It demonstrates the entire query contract: +derive the term, compare it with the stored term, and open the matched +ciphertext. An equality match is not authentication of the stored record; +the decrypt operation authenticates its encrypted fields. Passthrough +semantics are the same as in example 2. + +Collection conversion is already supported by Stack's blanket +`EncryptFrom> for Vec` implementation when `T: EncryptFrom` +and its context supports cloning. It collects every row's pending operations +before dispatch: these two users require four data keys (name and email for +each), issued through one batched `generate_keys` call. Equality terms use +the current local index-key backend. This count concerns encryption work, +not client initialization or authentication. + +One logical `generate_keys` call is not an unlimited single HTTP request: +the KMS client chunks batches according to its configured key-count limit. +This four-key example fits in one normal request; larger collections may +span several requests without changing the public call. + +The matching collection decrypt implementation batches retrieval across +rows, with one `retrieve_keys` call per keyset represented in the ciphertext. +An empty collection requires no data-key call. Collection order is preserved +by the conversion; no new whole-collection authentication is implied. + +This target-directed vector is a vector of independent output records. It +is distinct from encrypting a plaintext vector into a single +`StackCipherText` sequence tree, and from wrapping the vector in `OpaqueBlob` +to encode/encrypt the whole collection as one value. Batching is a property +of execution, not a change to the chosen layout. + +This example uses the common field context for all rows so one query term +can search them. Adding a row id to that context would also scope the term +to that row. Separating ciphertext row binding from column-wide search +context is a further design question, not something these examples silently +promise. Likewise, applications querying across keysets need the appropriate +term per keyset; no cross-keyset comparison is implied. + +### EQL targets: a concrete adapter boundary + +The example above is a generic searchable layout, **not an EQL payload**. +For EQL, use the adapter's types, metadata, serialization, and query target. +For example, today's `eql_bindings::v4::text::TextEq` already represents an +email ciphertext plus equality term; `TextEqQuery` represents the query. + +The following separate field-level program uses existing APIs and avoids +the proposed `Ciphertext` and passthrough attributes. It needs `eql-bindings` +with its `stack-encrypt` feature, `stack-encrypt`, `serde_json`, `tempfile`, +and `tokio`. It is marked `no_run` because it needs live credentials; it has +been checked against source signatures here, not compiled or executed. + +```rust +use eql_bindings::{ + v4::text::{TextEq, TextEqQuery}, + Identifier, +}; +use stack_encrypt::{DecryptFrom, EncryptInto, StackCipher}; +use std::error::Error; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let column = Identifier::for_column("users", "email")?; + let email = "alice@example.com".to_owned(); + + let encrypted: TextEq = email + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("email.eql.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let query: TextEqQuery = email + .encrypt_into_with_context(&query_keyset, column.clone()) + .await?; + let loaded: TextEq = serde_json::from_slice(&std::fs::read(&path)?)?; + assert_eq!(loaded.hm, query.hm); + + // Supply the expected column, so stored identifier metadata is checked too. + let restored: String = loaded.decrypt_into(&reader, column).await?; + assert_eq!(restored, email); + Ok(()) +} +``` + +`TextEq` includes its EQL version, identifier, ciphertext, and equality term. +Its context is `NonEmpty`. The typed identifier comes from the +application's expected table/column on both writes and reads; decryption +checks it against the stored identifier before opening the ciphertext. +The `v4` namespace is intentional: this Rust producer profile targets EQL +v4, not the TypeScript EQL v3 integrations. + +There is a gap before we can simply replace `SearchableEmail` with `TextEq` +in the outer derive. That derive currently passes `CallerContext`; EQL's +target needs `NonEmpty` and carries its identifier in the output. +The context adaptation must be designed explicitly, including whether caller +extensions are representable. The EQL example demonstrates what that adapter +must preserve without inventing an implicit conversion or a nonexistent SQL +client API. PostgreSQL DDL, parameter binding, and index engagement are a +separate adapter example to add once that boundary is settled. + +## What to review across the three workflows + +| Question | Concrete choice made by these examples | +| --- | --- | +| What is one encrypted unit? | One `OpaqueBlob` encoding in example 1; one selected field in examples 2 and 3 | +| How is the layout selected? | Ordinary encrypt returns an opaque value; `EncryptInto` selects a declared output | +| How is persistence exposed? | Canonical bytes for one ciphertext; Serde composed over a typed layout | +| How is plaintext recovered? | Decrypt/decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | +| How do indexes fit? | Additional field targets under the same layout/execution model | +| Who chooses context? | The application for a blob; the layout plus optional caller extensions for fields; an explicit identifier for EQL | +| Do examples need plans or visitors? | No; those remain implementation and advanced composition mechanisms | +| Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | + +The principal unresolved choices exposed by the examples are the opaque +ciphertext name and storage contract, the public decrypt spelling, the blob +codec and timing contract, EQL context adaptation, and optional source-side output +generation. None requires reintroducing public `encrypt_as`. + +## Source checks and validation limits + +The draft was checked against the existing +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[context composition](../../packages/stack-encrypt/src/target/context.rs), +[derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs), +[decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs), +[canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs), and +[EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). + +All four programs were parsed successfully by `rustfmt` using Rust edition +2024. All local links resolve; code fences and whitespace were checked. +These checks establish syntax and document integrity, not type correctness. + +The candidate APIs are not implemented by this document. The three proposed +programs cannot yet be typechecked against the repository. No live encryption, +database execution, or cryptographic compatibility tests are claimed. Once the surface is agreed +and implemented, promote these programs to executable examples and verify +storage round trips, matching queries, and context mismatches. From 356ba45eee48c8b2032a0360b002fcf89c8cfe36 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 15:35:34 +1100 Subject: [PATCH 02/42] docs: develop Stack Encrypt API design plan --- .../2026-10-24-stack-encrypt-api-plan.md | 1528 +++++++++++++++++ .../2026-10-24-stack-encrypt-api-review.md | 453 ++++- ...026-10-24-stack-encrypt-public-examples.md | 55 +- ...026-10-25-stack-encrypt-json-operations.md | 570 ++++++ .../2026-10-25-stack-encrypt-packaging.md | 253 +++ ...-25-stack-encrypt-provider-capabilities.md | 342 ++++ 6 files changed, 3185 insertions(+), 16 deletions(-) create mode 100644 docs/plans/2026-10-24-stack-encrypt-api-plan.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-json-operations.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-packaging.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md new file mode 100644 index 000000000..600b52091 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -0,0 +1,1528 @@ +# Stack Encrypt API implementation plan + +Date: 2026-10-24. Status: proposed implementation direction, developed from +the [review log](2026-10-24-stack-encrypt-api-review.md), +[public examples](2026-10-24-stack-encrypt-public-examples.md), and +[context examples](2026-10-24-stack-encrypt-context-examples.md). + +This document turns the review into implementation choices and completion +criteria. User requirements are distinguished from recommendations that +have not yet been accepted. Examples below are proposed APIs, not working +code for the current release. + +**Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets +are accepted. The comprehensive JSON first-release contract and Rust/dynamic +examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), +including extraction and walks. ORE does not provide equality initially; +retain a future explicit capability for equality-preserving encodings. +Prioritize stable blob storage bytes via +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); +zeroization/timing implementation improvements may follow without changing +stored format. The naming/package review is now in progress. These decisions supersede +earlier sequencing recommendations below where they conflict. + +**Follow-up decisions:** custom Rust query stages may be explicitly +Rust-only, with `NotPortable` on dynamic export until registered; built-in +capabilities retain Rust/Go parity. Every validation independent of keys +runs before key resolution/loading. The user is still reviewing the JSON +operation contract; its detailed API proposals are not collectively +approved. New keyset-routing, lock-context and provider-capability questions +are assessed in [provider capabilities](2026-10-25-stack-encrypt-provider-capabilities.md). + +**Packaging review:** the user favors retiring Vitamin C `AsyncCipher` and +moving application encryption behavior into Stack. The +[packaging and naming recommendation](2026-10-25-stack-encrypt-packaging.md) +defines crate ownership, an acyclic dependency graph, the proposed shared +format crate, role-based public names and coordinated migration work. It is +a proposal for review, not an accepted rename or an implementation. + +**Compatibility scope (user clarification, 2026-10-25):** this API has only +existed as an early prototype. Do not preserve its APIs, stored ciphertexts, +or encodings for backward compatibility. No legacy decoders, compatibility +aliases, deprecation period, or data migration is required. Choose the +coherent new design and update its Rust, Go, guest, and EQL consumers +together. Engine output must still agree with the SQL that consumes it; +that is current interoperability, not a legacy-format requirement. + +## Requirements established in the review + +- Ordinary encryption is keyset-directed, with `encrypt` and + `encrypt_with_context`. Keep the separate target-directed `EncryptInto` + conversion interface; remove redundant public `encrypt_as`. +- Single-value encryption produces opaque ciphertext; `OpaqueBlob` + explicitly encodes a composite value before sealing it as one value. + Direct `encrypt` does not automatically lift over collections. Use + `EncryptInto` for collection batching, including `Vec` into + `Vec` without caller context. This division is agreed as of + 2026-10-25. The current byte-vector recommendation follows that same + collection rule and uses `OpaqueBlob` for a whole vector; there is no + prototype-compatibility reason to add a raw-byte exception. +- Rust traits and derives are the primary static interface. Plans support + those declarations and dynamic/FFI lowering. +- Design and deliver static and dynamic support together. Go support is an + acceptance criterion, not a later follow-up. +- Provide decrypt entry points with and without explicit context. The + no-context variant must not invent a default required application value. +- `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation + capabilities. Both `encrypt` and `encrypt_into`, including their + context-taking variants, consume input and take responsibility for + cleaning up owned plaintext and plaintext artifacts. Ownership alone is + not a zeroization implementation; require `Zeroize` on plaintext types + and enforce cleanup through guarded storage and lifecycle checks. Deliberate passthrough + fields transfer to the output rather than being erased. +- Keep EQL's engine/SQL contract coherent and retain shared batching/execution + behavior; prototype representations need not be preserved. +- Investigate and replace the EQL JSON adapter's serialization round trips + with direct typed conversion, maintaining engine/SQL agreement. + +## Recommended decision: general configured targets + +Use a general target-plus-typed-options contract. Do not add a public +`json_tree(...)` derive construct. Rust constructs concrete options directly; +the dynamic interface deserializes its options into that same type. JSON +supplies the options type, validation, and implementation behind the general +mechanism. Sharing a contract does not require sharing input syntax or +making static Rust configuration pass through deserialization. + +Three approaches were considered: + +| Approach | Benefit | Limitation | Recommendation | +| --- | --- | --- | --- | +| Typed Rust options factory | Compiler checks the options type; normal Rust APIs and tooling | The function itself is not portable, but its concrete output has a data form | Preferred static surface | +| Named policy type holding options | Reusable and discoverable | Adds a provider trait and marker type where a function can suffice | Optional later convenience | +| Target plus data options | Expressible in Go declarations and dynamic data | Requires deserialization and semantic validation | Dynamic surface, producing the same typed options | + +This supersedes the initial recommendation to put option field literals in +Rust attributes. Rejecting a Rust factory because its code cannot cross FFI +was unnecessarily restrictive: the common boundary is its typed result, +not the code used to construct it. No arbitrary encryption callback is +introduced; the factory only supplies configuration. + +The target owns its source capabilities, canonical output, options/defaults, +validation, encryption description, index/query capabilities, and opening +behavior. Context remains a separate input. Configuration does not supply +arbitrary execution callbacks or let a caller redefine EQL's wire fields. + +Implement configuration as a target capability (working name: +`ConfigurableTarget`), lowering into the existing operation descriptions. +Its associated options type has a data representation. Rust factories +return that type directly; registered dynamic targets deserialize into it. +Validate and normalize once when a reusable declaration is built. Both +paths use the same target-owned semantic validator, before key I/O. Use +validated deserialization, for example a wire helper plus `TryFrom`, rather +than a derived deserializer that bypasses checked builders. Keep any raw +options fields private if the public type promises validated invariants. + +Compile-time guarantees cover factory resolution and signatures, the +target/options association, and typed builder arguments. They do not imply +that a string is a valid JSON path or that the factory evaluates at compile +time. JSON path construction is currently fallible. Begin with a fallible +factory contract, `fn() -> Result`, with errors converted +to the engine's configuration error path. Propagate errors without panics +or `unwrap`; a later literal-path macro can add stronger static validation. + +Resolve the factory once per reusable configured declaration and reuse its +normalized options for writes and queries. A factory can read mutable +external state, so its type alone cannot prove a fixed policy; document +that fixed-policy providers must be stable. Do not independently reevaluate +the provider when generating a query. Intentionally variable configuration +belongs in an explicitly constructed configured-target instance. + +Unlike today's `Layout`, the general capability must not require one index: +targets may have none or multiple. Adapt the existing JSON layout internally +while migrating; do not expose two competing configured-target concepts. +The target catalog resolves registered targets for Go. Custom Rust targets +remain possible without registration, but cannot be dynamically addressed +until their adapter and descriptor are included in the guest build. + +## Candidate static syntax + +`target(options = payroll_options)` names a Rust factory for the inner +target of a transparent newtype. The inner Rust type selects the output +representation and associated options type. The spelling is proposed; +typed factories for Rust and typed deserialization for dynamic callers are +the revised recommendation. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use stack_encrypt::json::JsonOptions; +use stack_encrypt::sem::TermError; + +fn payroll_options() -> Result { + JsonOptions::new().ordering(["$.salary"]) +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = serde_json::Value, + transparent, + target(options = payroll_options) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); +``` + +The factory body uses today's `JsonOptions` builder; the derive support is +proposed. The derive calls the factory and type-checks its result against +the inner target's associated options type. It knows neither JSON path +semantics nor JSON-specific option names, and does not serialize or +deserialize the factory's result. The configured target constructs its +output, which the derive wraps. Serde transparency preserves that output's +wire envelope; SQL driver traits are a separate integration requirement. +The same factory attribute should work on a field of a derived output +record, with existing context selection rules. + +```rust +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored = serde_json::Value::decrypt_from_with_context( + encrypted, &keyset, column, +).await?; +``` + +Fixed policy is a recipe for writes and queries. Transparent deserialization +does not attest that an existing row was created with that policy. In +particular, the full ordered-path policy is not stored in the JSON header. + +## Candidate dynamic representation and Go requirements + +Extend the existing string target with an object form, using the current +type name provisionally rather than deciding a registry rename here: + +```json +{ + "payroll": { + "context": ["employees", "payroll"], + "target": { + "name": "JsonTreeDocument", + "options": { "ordered": ["$.salary"] } + } + } +} +``` + +Keep `"target": "TextEq"` as shorthand for that target's default options. +The object form is new: current dynamic targets accept only a string. +Reject unknown options, incompatible options, unsupported target names, and +invalid paths during declaration validation. Do not silently fall back to +defaults. Specify one consistent JSON wire format and context encoding +across the Rust, dynamic, Go, and database paths. + +Go's generator and policy API must represent these same options, emit the +same declaration, and obtain the EQL value from the guest. Go must not +assemble the EQL envelope or reproduce JSON indexing. Discovery must expose +the options contract and capabilities; extend the existing target catalog +and resolver rather than installing a parallel JSON registry. +The dynamic registry invokes the concrete target options deserializer; it +does not look up or invoke the Rust application's factory by name. A Go +caller supplies equivalent options data. If a Rust policy must be exported, +serialize its resolved options, not its function identifier. This contract +does not require all Rust-only targets to depend on Serde; data conversion +is required for targets registered with the dynamic interface. + +Current gaps: JSON tree is absent from EQL's catalog/target table, the +dynamic JSON index is refused with `JsonIndexPending`, target fields carry +only a name, and Go's generated target declarations carry no target options. +Supporting an index's option syntax in Go does not establish executable +JSON-tree support. Include JSON input transport for objects, arrays, scalar +roots, and null; preserve the engine's numeric semantics and reject values +the cross-language contract cannot faithfully represent. + +Writes and queries must reuse the normalized configured target in Rust and +Go. An ordering query on an undeclared path must fail before key I/O. +Containment, path equality, ordering, and extraction need capability metadata +that can express their different operand/result shapes, not a single +assumed scalar query twin. Decryption must use stored cryptographic metadata +and context, not require the caller to remember the write's ordered paths. + +## Accepted decrypt surface and return ownership + +**Decision, 2026-10-25:** retain ordinary keyset-directed decryption for one +opaque ciphertext and destination-directed `DecryptFrom` for declared +targets/layouts. Both paths have context-free and context-taking variants: + +```rust +let value: i32 = keyset.decrypt(encrypted).await?; +let value: i32 = keyset.decrypt_with_context(encrypted, context).await?; + +let plaintext = User::decrypt_from(encrypted, &keyset).await?; +let plaintext = User::decrypt_from_with_context(encrypted, &keyset, context) + .await?; +``` + +These are independent call examples, not sequential reuse of a consumed +ciphertext. Direct decrypt mirrors direct encrypt; `DecryptFrom` mirrors +target-directed `EncryptInto`. The existing input capabilities still govern +whether omission of context is available. Do not add a competing public +ciphertext-receiver `decrypt_into` method. + +Successful decryption returns an ordinary owned `T: Zeroize`. Stack guards +recovered bytes and intermediate plaintext while it owns them; successful +return transfers responsibility for the value to the caller. `Zeroize` does +not automatically wipe that returned value on drop. Support explicitly +requested drop-guarded results without requiring a wrapper around every +decrypted application struct; the precise opt-in adapter is implementation +design within this decision. Guarding the completed value does not resolve +partial-deserialization cleanup, which remains required work. + +This reads as constructing `User` from ciphertext. Calling +`encrypted.decrypt_from(...)` would conflict with the ordinary From/Into +receiver convention; retain the agreed destination receiver. +The destination-directed pair already exists on the blanket `DecryptFrom` +extension trait. Today's ciphertext method is +`encrypted.decrypt_into(cipher, context)`, also on `DecryptFrom`, and has +no separate context-free overload. The current declaration trait +`DecryptInto

` is another layer; do not rename it accidentally while +settling method ergonomics. Consolidate declaration and convenience-trait +naming with review issue 5. The public examples use the destination-directed +pair; remove the competing ciphertext method when implementing this surface. + +## Recommended public traits and context input capabilities + +Status: direction accepted by the user on 2026-10-24: separate conversion +methods from `EncryptTarget`/`DecryptSource` declarations, and use explicit +caller-context capabilities instead of `Default`. Implementation, complete +signatures, and call-site updates remain pending. This section resolves the +direction of work areas A and B below; earlier examples retain historical +derive names until their migration. + +Keep two user-facing conversion traits, implemented by blanket library +implementations: + +| Trait | Public methods | Receiver | +| --- | --- | --- | +| `EncryptInto` | `encrypt_into`, `encrypt_into_with_context` | Plaintext value; output inferred or selected | +| `DecryptFrom` | `decrypt_from`, `decrypt_from_with_context` | Plaintext destination type; ciphertext supplied as an argument | + +These methods run declarations through the common engine and return the +existing batched awaitable. Target implementers do not override execution. +Ordinary `keyset.encrypt` and `encrypt_with_context` remain inherent methods +and do not gain a plaintext-side `encrypt` synonym. Their accepted decrypt +counterparts are `keyset.decrypt` and `decrypt_with_context`. + +Give the declaration traits role names, rather than From/Into names that +promise conversion methods they do not define: + +| Proposed declaration | Implemented on | Describes | +| --- | --- | --- | +| `target::EncryptTarget

` | Encrypted output or term target | How to produce this target from `P` | +| `target::DecryptSource

` | Recoverable encrypted output | How this stored value yields `P` | + +The corresponding derives should use the same names. This replaces today's +declaration traits `EncryptFrom

` and `DecryptInto

`; it does not remove +the retained `EncryptInto` extension trait. A term-only target implements +only `EncryptTarget`: no decrypt method that always fails. Plaintext visitor +traits remain a separate Stack-owned representation layer; moving/renaming +them must retain a single canonical trait implementation and a consistent +wire contract across the new consumers; prototype compatibility is not needed. +Configured-target options are another associated capability, not context. + +Declaration methods continue returning core-owned operation descriptions. +Encryption declarations do not receive the plaintext and cipher together; +decrypt declarations inspect stored output and select its opening, without +performing key I/O. Plans, derives, and dynamic resolution reuse those +descriptions. Keep the declaration module public for custom targets and +keep normal method imports/derives available from the Stack entry point. + +Use `ContextInput` as the associated-type name on each declaration trait. +It describes only what the caller must or may supply. It is not the complete +cryptographic context, which can also include a prefix and fields selected +from plaintext or stored output. Encrypt and decrypt declare this type +independently. + +Recommend library-owned input wrappers with explicit capabilities: + +| Proposed input | Omitted argument | Supplied argument | Meaning | +| --- | --- | --- | --- | +| `NoCallerContext` | Allowed | Not accepted | Declaration derives all context and exposes no caller input | +| `RequiredContext` | Unavailable | Required `C` | Supplies context or the missing component selected by the declaration | +| `OptionalContext` | Allowed | Accepted `C` | For ordinary ciphertext: omission means empty context; supplied context is used directly | +| `ContextExtension` | Allowed | Accepted `C` | Omission preserves the declared base; supply extends it | +| `ExpectedContext` | Allowed | Accepted `C` | Omission reconstructs stored context; supply additionally checks it | + +`OptionalContext` does not append a synthetic empty prefix, and omission +of an extension does not append `()`. `ExpectedContext` never extends AAD. +The table specifies behavior; names may be refined without changing it. +Nonempty requirements belong to the accepted argument type or validated +resolved context, not to whether an input is optional. Search targets still +require a nonempty effective context. Ordinary scalar ciphertext should +accept the same empty context through both keyset and target-directed APIs. + +No-context methods require an explicit, sealed `AllowsNoContext` capability; +`_with_context` methods require a corresponding `AcceptsContext` capability +with its accepted argument type. Neither is inferred from `C: Default`. +`RequiredContext` consequently remains required even though `u64` +defaults to zero. The helpers accept ordinary argument values (or ergonomic +checked conversions); users do not manually wrap `RequiredContext` or +`ExpectedContext` at every call. Conversion errors propagate before key I/O. + +The core-owned wrappers carry presence and intent; target declarations +perform source selection and context composition. For composite required +input, `C` may be a tuple or named struct; it does not have to be the full +logical context. Preserve existing grouping rules and context encodings. + +Examples of independent encrypt/decrypt requirements: + +| Declaration | Encrypt input | Decrypt input | +| --- | --- | --- | +| Caller row ID stored in output | `RequiredContext` | `ExpectedContext` | +| Source row ID also stored in output | `NoCallerContext` | `ExpectedContext` | +| Source row ID not recoverable before decrypt | `NoCallerContext` | `RequiredContext` | +| Static field base with optional unstored extension | `ContextExtension` | `ContextExtension` | + +In the last row, decrypting an extended ciphertext requires supplying the +same extension; availability of the omitted method is not a promise that +it opens every ciphertext of that Rust type. A derive can expose an optional +extension instead of `NoCallerContext` where its declaration explicitly +allows that behavior. Expected-context checks do not authenticate stored +data themselves: successful cryptographic opening remains necessary. + +The dynamic plan must encode the same requirements, source selectors, and +storage rules. Rust enforces omitted/supplied method availability with trait +bounds; the engine enforces dynamic input presence and validity before key +I/O. Omitted input, explicit zero, and an empty supplied value are distinct. +Go generation can expose convenient signatures using that metadata, but +the guest remains authoritative for runtime requests. + +Implementation proof points: compile-fail required-input calls even for +`C: Default`, reject context for a no-input-only target, preserve optional +extension versus expectation semantics, and execute all nine directional +scenarios through Rust and dynamic plans. Preserve owned-source support +and batching; this trait separation must not add a blanket `Clone` bound. +The final signatures must show where borrowed and owned sources are +supported rather than concealing the existing difference behind the rename. + +## Parallel Go JSON work: PR #1353 + +Read on 2026-10-24: [PR #1353](https://github.com/cipherstash/stack/pull/1353) +is an open documentation-only plan, not an implementation. Its write-up is +[The json_tree capability in the Go SDK](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md). +It tracks [#1347](https://github.com/cipherstash/stack/issues/1347), with +catalog, dynamic execution, queries, Go SDK, and extraction work in +[#1348](https://github.com/cipherstash/stack/issues/1348), +[#1349](https://github.com/cipherstash/stack/issues/1349), +[#1350](https://github.com/cipherstash/stack/issues/1350), +[#1351](https://github.com/cipherstash/stack/issues/1351), and +[#1352](https://github.com/cipherstash/stack/issues/1352). + +Treat that work as an implementation dependency to reconcile, not a parallel +SDK design to recreate. Its engine-owned execution, catalog/generated Go +types, engine-validated options, and cross-language tests fit this plan. +Review the following differences before freezing interfaces: + +- Route its JSON-specific option plumbing through the general target's + typed options/deserialization boundary. Go tags may retain idiomatic + syntax, but lower to target options rather than a second JSON policy path. +- Adapt the proposed `Layout` integration to the configured-target contract; + no second executor or permanently competing public configuration model. +- Reconcile type/registry names and generated query accessors with the shared + glossary and configured query capabilities. Its `At.*` accessor proposal + is useful input, not an accepted Rust or Go naming decision here. +- Carry the agreed context requirements and paired method semantics into + the relevant language interfaces without requiring identical language + syntax. Distinguish JSON `null` as content from nullable SQL fields. +- Evaluate its JSON-text ABI recommendation for numeric fidelity and return + types. This is compatible with direct typed EQL output adaptation: input + transport text and an avoidable encrypted-output Serde round trip are + different boundaries. +- Release scope is now decided: extraction, opening entries and walks ship + in the comprehensive first version alongside writes/queries. The earlier + Go plan's step-5 deferral is superseded by the linked JSON operation + inventory; sequential internal PRs must not become a reduced release. +- Byte-for-byte parity tests need deterministic randomness as that plan + proposes; normal randomized encryption needs cross-opening/query tests. + +These are dependencies and recommendations recorded locally. No changes or +comments have been posted to the parallel PR or its issues. + +## Prior query design: baseline and reconciliation + +**Decision update, 2026-10-25:** ORE does not answer equality in the initial +API, for any plaintext type. Equality uses its declared equality-capable +index. Keep capability selection parameterized by the index, plaintext +encoding/domain and options, so a future explicit exact/injective integer +encoding can opt into equality without changing query syntax or wire +operator identity. Never infer `Eq` merely from ordering support: text and +other lossy encodings cannot establish plaintext equality. A future opt-in +must retain overlap rejection (one selected provider per operator) and +must not silently invalidate existing `(Equality, Ore)` definitions. This +policy does not disable JSON scalar equality, which uses value selectors +rather than ORE/OPE terms. + +Read in full on 2026-10-25: the user-supplied **Stack Encrypt query design**, +dated 2026-10-21, at +`/Users/dan/Downloads/Stack Encrypt query design.md` (654 lines). The source +document was not modified. It is explicitly a discussion draft, based on +an earlier `next` snapshot; its historical code/release claims were not +revalidated as part of this reading. Use it as the query-design baseline, +not as a settled specification or a reason to redesign queries from scratch. + +### Existing proposal to carry forward + +- The caller selects an operator and a typed field/target handle. The field + supplies its own configured index; remove `DefaultIndex` and caller `.via`. +- Each field has at most one index answering each operator. Refuse overlaps + at declaration validation instead of choosing an index by priority. +- Declare operators, index capabilities, and admitted source types once; + derive or verify typed lookup, dynamic lookup, and discovery from that + declaration. Avoid another capability table in EQL. +- Share query derivation and refusals through the same semantic functions. + In particular, tokenize match input once and retain the tokens needed for + probabilistic verification in the query object. +- Retain index/source types in typed field handles. Unsupported operators + are compile errors on the typed path and validated errors dynamically. +- Model analyzer case behavior once in its stages, rather than restating + it in both the match configuration and the operator. + +The draft already proposes the query call +`cipher.query(op::Matches, text).on(&field).await?`, typed +`FieldRef` handles, and a loaded configuration/analyzer owned by the dynamic +plan (`Loaded` / `Arc`). Those are substantive designs to reuse; +they do not yet settle how our output-derived typed factory options are +initialized and shared with convenient `encrypt_into` calls. + +### Reconciliation with this review + +1. **Generate handles from the same output-target declaration.** The query + appendix proposes a plaintext-side `EncryptionPlan` derive and generated + encrypted counterparts. Our agreed starting point is output-side + `EncryptTarget`/`DecryptSource` derives. Recommend emitting typed handles + from that declaration, without separately restating indexes on plaintext. + A source-side convenience derive can later generate the same target. + Hand-written plans remain lower-level composition, not mandatory usage. +2. **Resolve typed factory ownership.** A generic `Match` can fix a policy + identity in its type, but an arbitrary factory can still compute different + values on different calls. Carry a resolved configuration instance through + writes and field handles; settle the owner/initialization API. The draft's + dynamic plan ownership is useful precedent, not proof that all static + factory results are compile-time constants. +3. **Separate portable options from execution types.** JSON's Rust factory + and dynamic deserializer can construct the same `JsonOptions`. The draft's + `Match` and `Match` intentionally have different execution + types to accommodate generic/custom analyzer stages. Preserve shared + validated options and semantic implementations for the portable subset; + do not force all typed analyzers through a serializable stage enum. +4. **Accepted portability scope.** Ship built-in advertised + capabilities in Rust and Go together. Downstream custom Rust stages can + remain explicitly typed-only with `NotPortable` until registered. This + qualifies the draft's blanket "lands in Rust first" rule; it must not + become permission to defer the Go JSON capability under this plan. +5. **Include JSON capabilities before fixing lookup representations.** The + draft's 16-shape dynamic enum covers equality, match, and scalar ordering, + not the new JSON target. Extend operator identity, operand/result shapes, + capability validation, and overlap rules for containment, `AtPath`, and + extraction. Its interim validated index list using one capability table + is a viable starting point; the specific closed enum is not a decision. +6. **Use the agreed context/scope model.** Replace label-only assumptions + with separate ciphertext binding and search scope. Row binding does not + flow into search terms; `PrimaryKey` is context-eligible and `ScopeKey` + is also scope-eligible. Apply the same declared scope to index generation + and queries. Preserve the requirement to return or supply row context + when decrypting selected results. +7. **Accepted validation timing.** All checks independent of keys precede + key resolution/loading, including query-input checks. The draft's possible + named-keyset resolution before those checks is superseded. Reuse the + same validator for Go's `check_query` and execution rather than copying + it or tokenization. Provider facts and cryptographic checks that require + keys remain later stages. Explicit earlier client initialization/keyset + calls are not retroactively prevented by this operation-validation rule. +8. **Apply the user's prototype scope.** Legacy match-option decoding, + deprecation/release-preservation requirements, and frozen Go wire formats + in the older document are not constraints here. Keep new Rust/Go/EQL + consumers aligned. The appendix's `cipher.encrypt(plan, value)` and + plaintext derive are illustrative older proposals, not replacements for + our agreed entry points. + +ORE/Eq is now decided as described above. Retain integration work on +typed field-handle construction/spelling, dynamic +`Index` versus a shared semantic-function boundary, feasibility +of compile-time match-option checks, Go's `check_query`, and ADR updates. +Type-safe stage ordering does not by itself validate runtime numeric values +or string paths; stronger const-check claims require implementation proof. +Use these as specific integration decisions rather than an unbounded query +API task. Coordinate their Go surface with PR #1353. + +## Recommended owner: a reusable typed target definition + +Status: recommendation for review. Use an immutable **target definition** +to own resolved options, prepared indexes/analyzers, context-selection rules, +and operation metadata. The output derive generates this definition's typed +field accessors from the same declaration that generates encryption. It +contains no plaintext, row values, credentials, or selected keyset. A field +handle borrows or shares that definition and retains its plaintext/index +types; it is not an independent name plus a newly constructed default index. + +**Selector naming, 2026-10-25:** the user finds encryption's `using` and +query's `on` confusing and requests considering one name. Recommend +`.using(...)` for both: each selects the definition or field whose declared +policy governs the operation. This spelling is proposed, not yet accepted. +Encryption still accepts the requested target's associated definition type; +query still requires `QueryTarget` on a selected field or +single-value definition. Shared naming does not weaken either bound or +make an entire multi-field record a query target. Older `on` examples in +the supplied query draft describe its earlier spelling. + +Proposed usage, assuming the previously described `EncryptedUser` target: + +```rust +let users = EncryptedUser::definition()?; + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + +let query = cipher + .using(&users.email()) + .query(op::Eq, "alice@example.com") + .keyset(&keyset) + .await?; +``` + +`definition()` obtains the shared resolved definition of that source/target +mapping. Plain `encrypt_into` obtains the same definition internally. It +does not independently invoke the options factory. `.email()` is generated +from the output field declaration, not a hand-maintained second schema. +The method name `definition` is proposed; it names a resolved declaration, +not a builder that users must reconstruct before every operation. + +### Call order and strict typestate + +User requirement, 2026-10-25: use strict typestate for the fluent API. The +user prefers the reading `cipher.using(&field).query(op, input)` and is open +to supporting both call orders. Recommend definition-first as the canonical +documented query form, while supporting operator-first through the same +request states: + +```rust +let a = cipher.using(&users.email()).query(op::Eq, email).await?; +let b = cipher.query(op::Eq, email).using(&users.email()).await?; +``` + +Both are proposed. The current implementation is query-first, followed by +`.on(...)`; this recommendation changes that interface. + +| Starting state | Method | Resulting state | +| --- | --- | --- | +| Cipher | `using(&definition)` | Bound definition, no query selected | +| Bound definition | `query(op, input)` | Query with definition/operator/input known | +| Cipher | `query(op, input)` | Query awaiting a definition | +| Query awaiting a definition | `using(&definition)` | The same query state as the definition-first path | + +The transition that brings the definition and operator/input together has +the `QueryTarget` bound. A definition-first bound handle need not +pretend to know the future operator. Once combined, the index and context +requirements are determined. Only the executable state implements +`IntoFuture`; a bound definition alone, a query missing its definition, or +a query missing required context cannot be awaited. Required context is +completed by a typed context-input transition; optional/no-input cases +start with a valid omitted state. Context completion must not be represented +solely as an unchecked `Option` plus a runtime missing-input error on the +static path. + +Do not expose another `using` transition after a query's definition has +been selected, or another query-selection transition on the request. +Supporting these two orders does not promise every permutation of keyset, +context, and operation selection. Specify those transitions separately, +starting with context supplied after its required type is known. Share +the final query request, validation, and execution implementation rather +than duplicate them per call order. Input-dependent errors such as an +invalid path still need runtime validation; typestate checks completeness +and capability, not arbitrary value semantics. + +The same discipline applies to encryption requests: a fixed-policy +`encrypt_into` has an implicit definition and can be awaited without +`using`; selecting an explicit compatible definition is a one-time +transition before execution. This does not add a new ordinary `encrypt` +receiver or require fixed-policy callers to bind definitions manually. + +Completion tests must include both query orders with identical semantics, +and compile-fail cases for unsupported operators/inputs, premature await, +missing required context, and repeated definition selection. Definition +borrows/shared handles must remain valid for the lifetime of the request. + +### Fixed declarations + +Use lazy, fallible initialization for the fixed definition, retaining each +successful result for reuse. Allow explicit startup initialization via +`EncryptedUser::definition()?`, while the shorthand write initializes it +on demand and returns any error through its awaitable. Factory errors must +not panic or perform key I/O; do not permanently cache initialization +failures. Coordinate concurrent first use so successful initialization is +performed once, and account for reentrant/recursive initialization in the +implementation rather than allowing a deadlock. + +The identity is the complete concrete declaration (including source/target +mapping and generic policy parameters), not merely the ciphertext's output +type. Different instantiations must not share a cache slot by accident. +This is an implementation requirement, not a claim that a static inside a +generic Rust function is automatically per-monomorphization. + +Fixed factories must describe stable application policy; they are not the +interface for reloading environment settings or tenant-specific options. +The cache makes writes and queries in one process share the resolved policy; +it does not prove another process or an older stored row uses that policy. +Per-call context, source values, and keyset choice remain separate inputs. + +### Runtime declarations + +Load/create a distinct definition instance for runtime options. A configured +instance is not stored in the fixed-declaration cache. Use the same type of +definition and field handles, so queries and writes share that instance. + +Proposed Rust spelling for a configurable library target: + +```rust +let payroll = EqlEncryptedJsonTree::configure(options)?; + +let encrypted: EqlEncryptedJsonTree = value + .encrypt_into_with_context(&keyset, column) + .using(&payroll) + .await?; +``` + +Here `using` takes a resolved typed target definition, not a hand-built +plan. This requires a lazy conversion request: select the definition before +building the operation or invoking a default factory. Its await lowers to +the existing `Pending`/batch executor; the shared engine remains unchanged. +The definition's output/source types constrain the conversion. Queries use +the same definition (or its field handles), supplying the required context +and keyset for that call. + +A fixed-policy wrapper such as `EncryptedPayroll` does not expose runtime +override of its declared options. Use the configurable underlying target +for the runtime case, avoiding silent overrides of a fixed policy. Exact +configuration APIs for multi-field runtime definitions are still to be +specified; they must select typed options without introducing another +execution mechanism. + +Go's loaded dynamic plan owns its resolved definitions and generated field +accessors reach those definitions. It has no Rust factory to invoke. The +guest validates and builds the configuration once per loaded declaration, +then writes/queries use the loaded instance. A different runtime policy is +a different instance; it does not mutate one used by in-flight operations. + +### Implementation proof and boundary + +Prove the cache/handle design early with two fixed mappings, multiple generic +instantiations, concurrent first use, initialization failure/retry, and +separate runtime instances. Prove a query and its write use the same options +without calling the factory twice. Preserve source lifetimes, owned-source +support, batch lowering, and no-HTTP/WASM builds. Queries on a field without +the operator must remain compile errors, with dynamic equivalents checked +at declaration/query validation. + +Prefer this generated-definition approach over requiring callers to author +a second plan or putting configuration state in each ciphertext. Plans are +still the lower-level implementation of the definition. Portable option +serialization and index capability discovery remain part of its dynamic +contract; the definition object does not itself need to serialize arbitrary +Rust closures or custom analyzer code. + +## Single-ciphertext representation and plaintext ownership + +Accepted decisions, 2026-10-25: use `OpaqueEncode` / `OpaqueDecode` for +single-ciphertext representation, replacing the working names `EncodeValue` +/ `DecodeValue`. Both direct `encrypt` and target-directed `encrypt_into` +consume their values and take responsibility for owned plaintext cleanup, +including their context-taking variants. These are representation +capabilities; the engine still performs cryptography. `EncryptTarget` / +`DecryptSource` describe how encrypted outputs and layouts compose those +operations. The lifecycle and cleanup scope below are part of the accepted +design. The plaintext cleanup bound is `Zeroize`; implementation remains +to be completed. + +### Owned input is the direct API contract + +`keyset.encrypt(value)` and its context-taking counterpart take an owned +single-value input. No implicit clone and no `Clone` bound. For example, +passing a `String` transfers it; a borrowed `&str` does not transfer the +storage it refers to and must not receive the same cleanup promise. Do not +add a silent borrowed-input fallback to the consuming operation. + +**Decision, 2026-10-25:** require `Zeroize` for plaintext types and use a +library-owned guard. Use the existing ecosystem trait rather than defining +a new cleanup trait or alternative protected-ownership adapter for now. +`Zeroize` supplies the operation; the library must arrange to invoke it on +every supported exit path. A bound alone does not arrange cleanup on drop. + +Acquire protected ownership synchronously when creating the encryption +request, before returning its future/builder. An `async fn` body is not run +until polling, so placing the first guard inside that body would leave an +unpolled request with an ordinarily dropped source. A sync constructor +returning the typed future can establish the guard first. Every subsequent +builder transition moves that guarded ownership rather than making a new +unprotected copy. + +The representation trait may encode through a temporary internal borrow of +the guarded value into a library-owned sensitive sink. Public consumption +does not require giving an encoder unrestricted ownership of the source or +an ordinary growable `Vec` for scratch work. Wipe and release the source +as soon as encoding no longer needs it; keep encoded plaintext guarded until +sealing completes. Preserve the single-leaf representation boundary. + +### Cleanup scope and implementation obligations + +- Cover owned source storage, encoded buffers, codec scratch, and any + library-created plaintext copies. Use established non-elidable wiping + primitives/protected storage rather than `clear()` or ordinary drop alone. +- Cover success, encoding/configuration/KMS/AEAD errors, partially prepared + batches, request cancellation, and an unpolled request being dropped. + Use drop guards for ordinary unwinding too; process abort or a leaked + future is not a destructor guarantee. +- Avoid reallocations that abandon uncleared buffers; growing sensitive + sinks must have a cleanup strategy for old allocations. Prevent plaintext + from escaping in diagnostics, Debug, caches, and instrumentation. +- On decrypt, protect recovered bytes and partial decode state until cleanup + or successful ownership transfer to the caller. Successful returned + plaintext is intentionally retained; its subsequent lifetime is the + caller's responsibility unless the return type itself is protected. +- A Rust move does not prove all physical copies were erased. In particular, + `Copy` inputs leave caller-owned originals usable, and compiler temporaries + may exist. The enforceable API promise concerns supported owned storage + and library-managed artifacts, not caller-retained copies or erasure of + every register/stack remnant. Verify stronger claims with compiler evidence. + +The [zeroize documentation](https://docs.rs/zeroize/latest/zeroize/) +documents drop-based guards, non-elidable wiping, and limits involving +moves, stack copies, and reallocations. The source currently uses +`Protected>` at the AEAD boundary, but that does not establish +end-to-end cleanup of every caller value and intermediate. This discussion +is a design requirement, not a completed source/assembly zeroization audit. + +### Consequences for blobs, layouts, and Go + +`OpaqueBlob` cannot promise source cleanup from `T: Serialize` alone. +Require `T: Serialize + Zeroize` for the proposed Serde-backed blob input; +owned blob decoding requires `T: DeserializeOwned + Zeroize`. Ordinary +application structs may derive `Zeroize`. A +generic Serde implementation can allocate or retain hidden copies, so strong +claims also require a supported codec/implementation contract; wrapping its +final byte buffer does not make arbitrary user serialization code compliant. + +In particular, `T: Zeroize` only supplies cleanup once a `T` exists. If +derived `Deserialize` allocates a `String` field and fails on a later field, +the partially built fields ordinarily drop without invoking `T::zeroize`. +Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix a struct +that was never constructed. Keep this as a concrete decoder design and +acceptance requirement; do not claim the new bound solves partial decoding. + +### Blob codec assessment: Vitamin C frames + +**Priority decision, 2026-10-25:** first define a stable, versioned binary +storage profile. The codec was originally ephemeral transport; stored blobs +need a stronger compatibility commitment. Zeroization and constant-time +implementation improvements can follow, provided they preserve stored +format semantics. Do not defer decisions that affect stored bytes (option +presence, type mapping, versions) or claim timing/cleanup properties not +implemented. The `Zeroize` input bound and ownership direction remain. +Track this action in +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519), created +for this review. It explicitly distinguishes storage stability from the +transport-only rationale in Vitamin C #332. + +**Recommendation, 2026-10-25:** reuse Vitamin C's frame writer/parser as the +foundation for the library-owned blob codec, with a specified blob Serde +mapping. Do not adopt the existing transport Serde mapping unchanged. +The inspected dependency is `vitaminc-aead-frame` 0.8.0; its `serde` module +already provides `to_vec(&T)` and `from_slice::(&[u8])`. Stack currently +uses that module for dynamic control data. Its module documentation +explicitly scopes it to transport data rather than plaintext to seal. + +The implementation already guards the serializer output, temporary variant +buffers, and buffered map keys with `Zeroizing`. The underlying +`vitaminc-aead` frame writer grows into a fresh allocation and wipes the old +one. However, `to_vec` returns an ordinary `Vec` on success. Provide an +internal guarded-output entry point or sensitive sink for the blob path, +keeping the result guarded through sealing. Inspect remaining temporary +storage and custom Serde paths before asserting a complete cleanup contract. + +The current mapping is not lossless for every ordinary Serde type: +`serialize_some` delegates directly to its value, while `None` and unit both +encode as null. Thus `Some(())` reads back as `None`; nested optional values +have the same ambiguity. The blob mapping must preserve option presence +explicitly, using an unambiguous representation, and test these cases. +The current map representation also requires text keys. Specify support or +explicit rejection for non-string map keys, and document enum/buffered Serde +limitations rather than advertising all `Serialize` types as round-trippable. + +Frame parsing validates one complete frame before invoking `Deserialize`, +but that neither prevents type-level decode failures nor cleans up partial +application values. Resolve that cleanup boundary alongside format limits, +authenticated codec versioning, and cross-language fixtures. Reusing the +frame format is not a constant-time guarantee and does not make an opaque +blob a structurally encrypted JSON document. + +[Borsh](https://docs.rs/borsh/latest/borsh/) remains a fallback if the blob +mapping would require disproportionate changes. It introduces +`BorshSerialize` / `BorshDeserialize` instead of the Serde traits, and has +writer-based serialization. Changing serialization libraries does not by +itself establish wiping or a constant-time contract. Prefer evaluating the +existing frame machinery against these requirements first. + +The same consuming contract applies to target-directed `encrypt_into`. +For a structural target, declared passthrough fields intentionally +move into the output; wipe encrypted source fields and temporary copies, +not plaintext fields the caller explicitly chose to retain. Fan-out to +ciphertext and index operations should read a guarded source where possible +instead of requiring a plaintext clone per operation. + +Specify cleanup for dynamic host/guest transport and JSON parsing buffers +as part of the Go implementation. The Rust guest cannot erase Go caller +objects it never owned; host-managed copies need host cleanup and ordinary +Go values do not acquire Rust move semantics. Document that boundary rather +than claiming guest cleanup erases every copy in the calling process. + +Acceptance work: source-cleanup/drop observation on successful and failing +requests, cancellation/unpolled-request tests, nested source/blob cases, +and owned non-Clone inputs. Inspect optimized code for wipe/copy behavior +where necessary; ordinary tests alone cannot prove absence of every residual +copy. Do not infer a current vulnerability or a completed guarantee from +this design discussion. + +## Configured-target implementation sequence and completion criteria + +1. **Normalize the general contract.** Specify typed options, Rust factory + signatures, validated dynamic deserialization, defaults, context + inputs, capability discovery, errors, and the target + object grammar. Create the versioned interoperability specification, + schemas and initial conformance fixtures described below alongside this + work. Build one target-owned validator and lowering path. +2. **Deliver JSON as a complete first implementation.** Add configured EQL + JSON targets, direct typed output/input conversion, catalog/guest + registration, and JSON transport. Keep engine and database encodings aligned. +3. **Expose both declarative front ends.** Add the generic Rust derive + factory hook and Go generator/policy support, including field targets and + standalone Rust wrappers. Neither front end reimplements encryption. +4. **Complete queries and reads in both languages.** Carry the configured + target into query construction, expose extraction results, and standardize + Rust's paired decrypt convenience methods. Settle the role-based JSON + names as part of this public API work. +5. **Verify the complete slice before calling it supported.** Rust-written + EQL JSON must open and query from Go, and vice versa, against PostgreSQL. + Test defaults, ordered paths, both root settings, arrays/scalar roots/null, + numeric boundaries, missing paths, context mismatch, and invalid options. + Assert logical batching and verify fixtures for the chosen format. Compare + semantics rather than ciphertext equality, since encryption is randomized. + Run the shared interoperability conformance suite and gate schema/runtime + drift in CI. Each implemented contract change updates its specification + and fixtures in the same change. + +These are implementation steps, not permission to ship a Rust-only phase. +The Go examples and cross-language acceptance tests are part of the feature. +Do not add the plaintext-configuration wrapper in this first change: the +dynamic target declaration already handles runtime options without adding +another policy location. Reconsider that convenience after both front ends +use the shared contract. + +### Required deliverable: dynamic interoperability specification + +**User requirement:** develop a formal specification of the dynamic +interoperability layer during implementation, not as documentation added +after the Rust and Go APIs have diverged. The current examples remain +proposals until reconciled into that specification; adding this deliverable +does not claim the specification has already been written. + +**Recommended format:** a versioned specification package containing +[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) schemas, +normative semantic and wire-format rules, and executable conformance +fixtures. JSON Schema describes declaration/request/response structure; +it is not the entire behavioral or binary transport specification. Keep +application semantic-type declarations distinct from JSON Schema's own +`type`, `$defs`, and `$ref` vocabulary. + +Required coverage: + +- Definition grammar: built-in and named types, optional `types` section, + inline role wrappers, context selectors, scope eligibility, passthrough + and context storage, targets/options, query capabilities, defaults, and + unknown-field/extension handling. Named ordinary types need no role; + naming a type does not grant scope eligibility. +- Semantic validation: resolve named types and field references, reject + invalid/cyclic references where unsupported, enforce primary-key versus + scope-key capabilities recursively, check source/output compatibility, + and derive write/query scope agreement. Do not imply generic JSON Schema + validators perform the engine's reference resolution or capability checks. +- Operation contracts: input presence and context/scope roles, target and + definition identity, results, stable error codes and locations without + plaintext values, validation stages, and which failures precede key I/O. +- Wire mapping: JSON and frame representations, integer widths/ranges and + lossless numeric transport, bytes/text encodings, missing versus null, + context ordering/grouping, descriptor/AAD/index-scope lowering, and + normative references to the ciphertext/SQL envelope and tag extraction. + Specify the host/guest ABI or reference its versioned specification; + JSON authoring syntax must not silently redefine the binary transport. +- Versioning: explicit protocol/schema identifiers, supported-version and + capability handling, and rejection of unsupported features. Define future + evolution without adding compatibility obligations for the prototype. + +Conformance fixtures include accepted and rejected definitions/requests, +expected error codes/paths, normalized definitions, and deterministic +context/encoding vectors. Exercise the same fixtures through Rust and the +Go guest path, plus SQL extraction where relevant. Randomized ciphertext +comparisons use cross-decryption and query behavior, not byte equality. +Schema validation is one layer of the suite; semantic rejection tests are +required even for structurally valid JSON. + +Keep a single maintained contract: prefer generating structural schemas and +capability metadata from the shared declaration models/catalog where that +faithfully represents the specified grammar, with committed artifacts and +CI drift checks. Review generated schemas against normative rules rather +than treating Serde derives as a complete specification. Do not introduce +OpenAPI or a new transport solely to document this non-HTTP interface. + +## Coverage of the complete review + +The configured-target work above addresses issue 13 in depth. The full API +redesign also requires the following work. Each row identifies a concrete +deliverable; recommendations do not silently settle the open decisions. + +| Work area | Review issues | Required deliverable | +| --- | --- | --- | +| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One signature/import/receiver map, operation-trait design, and dependency migration | +| Context contract | 1, 3, 4, 11, 12, 14 | Scalar and record context rules, explicit optionality, and matching dynamic representation | +| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde/Base64 behavior, and a defined boundary with structural trees | +| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | +| Mixed records and collections | 7, 8, 11 | Passthrough/renaming/context-field derive support and retained collection batching | +| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion and Go coverage | +| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | +| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348–#1352 with this contract; reuse compatible implementation work | +| Prior query design | 16 | Reuse the supplied field-owned query model; reconcile handles, configuration ownership, JSON capabilities, and portability | +| Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | +| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | + +### A. Settle the public method and trait structure first + +Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. +Use the accepted destination-directed decrypt pair. Write +out all signatures, imports, bounds, and return types before renaming traits. +Retain ordinary keyset `decrypt` / `decrypt_with_context` for opaque +ciphertexts, and remove redundant prototype conveniences without +compatibility shims. + +Separate plaintext representation, target operation declarations, and +execution conveniences explicitly. Resolve why `EncryptFrom` declares +`encryption()` while `encrypt_from` lives on `EncryptInto`, and why +`DecryptFrom` supplies methods while `DecryptInto` describes opening. A +rename without an ownership/signature map leaves the discovery problem. + +Recommendation: Stack owns the public value-encryption model; Vitamin C +retains general primitives. Specify the dependency boundary before moving +code, retaining one canonical trait identity and the no-HTTP/WASM builds. +Reassess the original Vitamin C cipher-trait mismatch in that architecture: +do not add a public trait implementation merely for symmetry if it cannot +carry structured KMS context or support its advertised operations. + +Completion: representative scalar, blob, typed record, searchable record, +and JSON calls compile with the documented imports, including generic +callers. The method inventory has no unexplained duplicate execution path. +Compile-fail tests cover genuinely unsupported trait capabilities. + +### B. Turn the context scenarios into a normative contract + +The nine scenarios in the context document are the acceptance matrix, not +optional examples. Cover scalar static context plus caller extension, +source-field selection, caller input, output storage, and decrypt recovery +or expected-context checks. Preserve the accepted `field = "id"` shorthand +as selection on both inputs, without implicit copying. + +Fix composition order and grouping once: literal/field base, composite row +context, and any caller extension. Specify the exact descriptor and AAD +encoding for each case. An expected context checks stored context; it is +not another extension. Never need to open ciphertext to discover the +context required to open that ciphertext. + +Recommendation: represent no input, required input, optional extension, +and optional expectation as explicit capabilities. Do not infer optionality +from the application's `Default` implementation. Settle the wrapper or +trait mechanism alongside the public signatures. Resolve the existing +empty-context mismatch between ordinary scalar encryption and ciphertext +targets without weakening nonempty-context requirements for search targets. + +Give each source/storage/composition rule a dynamic representation and Go +authoring path. Current EQL identifier constraints must be reconciled with +row/caller context explicitly, not bypassed or silently dropped. + +**Next discussion — recommendation, not yet a decision:** distinguish +ciphertext binding from search scope. A row ID may bind a ciphertext to its +record without making every index term row-specific. The target definition +should explicitly describe both scopes; query construction uses the same +declared search scope as index generation. For example, a searchable email +may bind its ciphertext to `(users.email, tenant_id, user_id)` while its +equality index is scoped to `(users.email, tenant_id)`, allowing a query +before the user ID is known. Do not silently remove components from one +generic context argument. Index-key derivation/KMS descriptors must respect +this separation too; changing only leaf AAD is not a complete design. +Finalize the conceptual split before choosing attribute names or exact +tuple encodings, and represent it in the dynamic definition as well. + +**User direction:** allow row-bound ciphertext and document that reads must +provide the row ID to decrypt. Search scope should cover multiple rows, +such as a tenant; automatically including a row ID defeats ordinary search. + +**Misuse-resistant API recommendation (not yet accepted):** + +- Give existing record `context(field = "id")` and its directional variants + ciphertext-binding semantics. Do not implicitly feed those selected fields + or caller extensions into index keys or terms. This is a proposed + clarification for searchable targets; older examples did not distinguish + the roles, and must be updated together before implementation. +- Declare search scope separately on the shared target definition. A named + application scope input (for example `TenantScope`) can come from operation + configuration/caller input. The capability proposal below also permits + record-field selectors when their types explicitly qualify as scopes. + The same typed scope is required for both + writing index terms and constructing queries. A tenant-bound cipher may + supply it only through an explicit declared mapping; never guess it from + unrelated metadata. +- A binding input and a search-scope input are distinct capabilities even + when their underlying values have the same Rust type. Do not provide + blanket implicit conversion between the roles. If one operation needs both + caller inputs, use named members in its typed input rather than one + unlabelled value routed to both operations. +- Missing required scope prevents awaiting a Rust request; dynamic/Go + validation rejects it before key I/O. A default column-wide scope is not + substituted when a tenant scope is missing. Scope-free searches use an + explicitly declared column-wide policy, with column identity still bound. +- The effective ciphertext binding should include the declared search scope + as well as record binding, so changing a supplied tenant also prevents + opening the ciphertext. Index derivation sees column identity plus search + scope only. Query requests never require record binding input. +- Encrypted output must expose whatever the decrypt selector needs or + require it from the caller. Record-projection examples must select the ID; + there is no decrypt fallback which drops row binding. Selecting a field + as context does not implicitly store that field. + +Working declaration spelling: +`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. +The spelling is illustrative; it preserves the agreed directional-context +shorthand and gives search an explicitly separate role. Dynamic definitions +must preserve this separation rather than providing a shared context bag. + +This prevents accidental propagation of row IDs into search terms; it +cannot prove the application chose a scope containing multiple rows. A user +can deliberately put a row ID inside `TenantScope`. Do not claim otherwise, +or attempt to infer semantics from field names such as `id`. Use explicit +types/declarations and documentation for that remaining application choice. + +#### Scope capability and role wrappers — assessment + +**Proposal under discussion:** use both ideas together. Scope is the subset +of context permitted for search terms; semantic wrappers provide positive +trait implementations identifying that subset. Working capability names: +`CiphertextContext` and `SearchScope: CiphertextContext`. These names describe +eligibility, not source selection, optionality, or encoding. Integrate with +the existing context representation rather than introducing a competing +serializer. The previously agreed `ContextInput` capabilities still describe +whether the caller must supply a value. + +For example, `PrimaryKey` implements ciphertext-context eligibility but +not search-scope eligibility. `ScopeKey` implements both, subject to +the inner value's supported context encoding. Generic parameters are Rust +types (`PrimaryKey`); `PrimaryKey(42)` is a value constructor. Prefer +`ScopeKey` as the name for the second wrapper; the user accepted this +name in preference to `Collection`, which can read as a container. +These wrappers need the appropriate `Zeroize` support when they are owned +plaintext fields, consistent with the existing ownership decision. + +Crucially, neither wrapper automatically adds the field to context or scope, +stores it in encrypted output, or changes the plaintext's encryption layout. +The target declaration still selects sources. An illustrative declaration +is `context(field = "id"), search_scope(field = "tenant_id")`. With +`id: PrimaryKey` and `tenant_id: ScopeKey`, the selection is +valid; selecting `id` as search scope is a compile-time error. This makes +typed field selection a reasonable alternative to the earlier recommendation +to accept only caller-supplied search scope. + +Misuse-resistance requirements: + +- No blanket `SearchScope` implementation for every context, serializable + type, raw number, or raw string. Scope use is an explicit classification. +- Composite scopes qualify only when every component is a scope. A tuple + containing a primary key cannot become a scope merely by tuple composition. + Context composites may contain either category. +- Do not forward scope eligibility through `Deref`, `AsRef`, or implicit + conversions that erase a primary-key wrapper. Typed target/query handles + retain the definition's concrete scope type, rather than accepting any + type implementing `SearchScope` interchangeably. Distinct application + newtypes can distinguish tenant and project scopes with equal wire types. +- Permit explicit application opt-in through a documented trait/derive + contract. This is a semantic capability, not an unsafe-memory contract. + Sealing every implementation cannot prove that a value groups multiple + records: a caller can still explicitly rewrap a row identifier as a scope. + Guarantee rejection of accidental role mixing, not business semantics. + +Dynamic definitions must retain field roles and context-versus-scope +selection. Validate those roles before execution: an explicitly declared +primary-key field is not scope-eligible. Plain JSON numbers carry no such +role by themselves; do not infer it from the field name or wire encoding. +Generated Go types can represent the distinction, while the engine validates +dynamic declarations. Rust and dynamic lowering must produce the same +canonical context/scope values; never use Rust type names as wire domains. +Scope classification is not evidence of authorization to access a tenant. + +Acceptance work: valid typed field/caller scopes, rejected primary-key and +mixed-tuple scopes, no automatic selection of marked fields, exact scope +type agreement between write/query, and equivalent dynamic validation. + +#### Dynamic scope roles and validation + +**Recommended design:** declare semantic types and source selectors in the +definition, validate once on load, and supply values separately per operation. +Do not let an individual request assert `scope_key` on an arbitrary value +to override the loaded definition. A definition author can explicitly +classify values, just as a Rust author can construct wrappers; this is +misuse resistance, not authorization or proof of multi-row cardinality. + +**Accepted clarification:** `types` is optional; it holds reusable named +types. Fields may use built-ins or inline role wrappers. Ordinary built-in +or named values need no `kind`. Supported primitives remain eligible for +ciphertext context, but search eligibility requires explicit `scope_key` +classification (or another explicitly supported scope-capable type). + +Illustrative declaration fragment (proposed schema, not today's wire API): + +```json +{ + "types": { + "UserId": { "kind": "primary_key", "value": "u64" }, + "TenantId": { "kind": "scope_key", "value": "u64" } + }, + "fields": { + "id": { "type": "UserId", "passthrough": true }, + "tenant_id": { "type": "TenantId", "passthrough": true } + }, + "context": { "field": "id" }, + "search_scope": { "field": "tenant_id" } +} +``` + +This is the context-related portion of a record target definition; encrypted +email targets and their search options are declared separately as usual. +The shared `field` shorthand selects plaintext fields on encrypt and stored +fields on decrypt; it does not imply a field exists in a query request. +For a query on the derived email handle, the definition exposes a required +`TenantId` scope argument. No `UserId` argument is required or accepted as +an alternative scope. A named/composite scope uses the same ordered schema +on the write and query paths, regardless of their different source selectors. + +Definition loading resolves field references and source/output types, +computes the context/scope capabilities, verifies all composite components, +and rejects a primary-key search selector before key I/O. It also verifies +decrypt selectors are available from stored fields or declared caller input. +Compile the validated definition into an immutable handle shared by encrypt, +decrypt and query operations. No second query-specific scope policy. + +At execution validate presence, nominal type when supplied by a typed handle, +physical representation, range, and composite shape against the selected +definition. Raw wire values obtain their expected semantic type from the +definition; they cannot prove their application provenance. Unknown role +overrides or mismatched nominal value handles are rejected. No fallback to +unscoped querying when a required scope is absent. When JSON is the authoring +or request format, preserve full-width integers through lossless parsing; +the frame transport already distinguishes supported integer widths. + +Recommended Go generation uses distinct named ID types and query methods +with the required scope argument (`TenantID` rather than `uint64` or an +untyped context map). Generated wrappers remain conveniences over the same +validated runtime definition; ungenerated FFI callers receive runtime errors +for invalid declarations/requests. Do not assert Go shares Rust's ownership +or trait-bound guarantees. + +Lower the chosen scope and binding through the existing canonical context +machinery, retaining scalar widths, ordering and grouping. Current dynamic +`context.rs` proves parity between runtime `ContextPiece` trees and Rust +context derivations, but has no primary-key/scope-role distinction. Add role +validation before lowering erases those distinctions, and keep the separate +resolved scope and binding available to their respective operations. +Do not persist role labels in every row just to recover them at runtime; +definition identity and deployment remain explicit concerns of loading the +same policy. Add Rust/Go/dynamic fixtures for valid scopes, primary-key +refusals, nested scopes, missing inputs and identical derived terms/AAD. + +Completion: all nine cases execute through Rust and dynamic declarations; +required input is enforced, altered/wrong context is refused, and existing +context encodings agree across all new implementations. + +### C. Finish the ciphertext and whole-value storage contracts + +Accepted direction: a single-value plaintext encoding produces one opaque +`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode +and seal it as one value, or an explicit target for field-level encryption; +Serde alone does not imply automatic blob serialization. A custom type can +explicitly implement the single-value encoding capability. The wrapper +name and detailed persistence contract remain proposals. +Recommend canonical byte parsing/encoding, human-readable Serde as Base64 +of those bytes, binary Serde as +bytes, and safe Debug behavior. Parsing an envelope does not authenticate +it; authentication occurs during decrypt. + +The existing encrypted-output transcoder can implement the adapter: +`Node::Single(SealedValue)` is consumed by `Visitor::sealed` and moved into +the opaque wrapper. Keep the sealed value internally rather than forcing +a byte serialization at construction. There is no new encryption or KMS +request. On decrypt the wrapper delegates to the same leaf-opening path, +using a single-node wrapper if the internal API requires it. + +Do not convert maps, sequences, passthrough nodes, or sealed empty-container +markers into this type merely because they have a byte representation. +Serializing a whole encrypted tree retains its multiple ciphertexts and +authentication boundaries; it does not produce single-value encryption. +The leaf visitor's unsupported-shape errors are a defensive check, not the +desired public discovery mechanism. Constrain ordinary encryption to +single-value encodable inputs so unsupported structural inputs are rejected +before encryption, statically in Rust where possible and during dynamic +validation. Settle the exact plaintext capability with the representation +traits; do not discover an unsupported shape only after minting keys. + +Keep generic structural encryption and searchable JSON explicit. Specify +whether structural-tree persistence is supported and under what stable +format. Any protocol changes must be coordinated with their consumers. + +#### Typed encodings and extractable ZeroKMS tags + +**Requirements added, 2026-10-25:** ciphertext text conversion returns the +existing concrete encoding wrappers, not `String`. The user also wants to +explore selecting an encoded ciphertext as a field's output type. The +ZeroKMS key tag must be easily extractable from stored ciphertext, including +in PL/pgSQL, to support a separately designed refinable-tag mechanism. +The storage/authentication contract is not yet locked in. + +Current `eql-bindings::encoding::{Base64Text, Base85Text}` validates canonical +padded standard Base64 and the crate's RFC 1924 Base85 encoding respectively. +Their private strings, encoding constructors and validating deserializers +make decoding infallible. They guarantee the text encoding, not that the +bytes form a ciphertext. Reuse these types and add ordinary fallible text +parsing conveniences rather than requiring Serde to construct them from text. + +Recommended ownership: a small Vitamin C encoding component, re-exported by +Stack Encrypt and EQL. These byte/text utilities do not depend on encryption, +SQL or EQL layouts. Keep TypeScript/schema integration optional in the owning +crate, accounting for Rust's orphan rules when moving the existing trait +implementations. Do not introduce a Stack Encrypt dependency on EQL merely +to reuse its text wrappers. Final component placement remains a proposal. + +Proposed ciphertext methods return `Base64Text` / `Base85Text` from +`to_base64` / `to_base85`; corresponding constructors accept the typed text +and still return a ciphertext parse error when its decoded envelope is +invalid. Encoding validation, envelope parsing and authentication are three +different guarantees. Use one canonical byte envelope under both encodings. + +**Accepted, 2026-10-25:** support explicit encoded ciphertext field targets. +Use a ciphertext-specific storage wrapper +(working names `Base64Ciphertext` / `Base85Ciphertext`) over making every +`Base64Text` an encryption target. The wrapper validates both text and +envelope, serializes as the text string, and composes with `EncryptTarget` +and `DecryptSource`: seal once, then encode; parse before opening. These +names remain provisional until the final naming pass; the first-class +encoded target capability is settled. A structural output +can mix `Ciphertext` and encoded ciphertext fields without additional KMS +calls. Do not silently change the prescribed representation of EQL targets. + +The current `SealedValue` v1 envelope already uses a simple prefix: +`version(1) | keyset_id(16) | iv(16) | tag_len(2, LE) | tag | local_ciphertext`. +Its ZeroKMS tag begins at byte offset 35, with length at offsets 33–34. +This is evidence that extraction can be simple, not a commitment to keep +those offsets. Keep the chosen version's layout documented and parseable +without the plaintext codec, key retrieval, or decryption. Prefer a checked +borrowed header/envelope view and an accessor explicitly named `key_tag` +over decomposing the whole ciphertext or conflating it with the AEAD tag. +Extraction does not authenticate the returned tag. + +Ship a matching SQL/PL/pgSQL extraction helper and Rust/Go/SQL fixtures with +the storage format. Check versions, lengths and truncation before slicing. +For text storage prefer Base64 when SQL must inspect the header: +[PostgreSQL](https://www.postgresql.org/docs/current/functions-binarystring.html) +has built-in Base64 decoding and byte slicing, but not Base85 decoding. +If SQL extraction from a Base85 envelope is supported, supply an explicit +decoder; do not leave it as an assumed capability. PostgreSQL Base64 output +also inserts line breaks, unlike the canonical Rust wrapper: normalize SQL +output or validate it against the same canonical rules in cross-language +tests. JSON-tree storage needs its own header adapter: its shared key header +is distinct from each node's ciphertext and parent tag. + +The proposed refinement uses the ZeroKMS tag as an HMAC key and replaces it +with a derived tag. **User clarification:** at decryption, ZeroKMS uses the +refined tag for policy checks and logging, and returns the original tag. +The decrypt path uses that returned original tag to reconstruct leaf AAD. +This resolves the conceptual conflict with binding the original tag into +AEAD: the stored/request tag and the authentication tag input need not be +identical after refinement. Keep those roles explicit in the KMS response +and decrypt implementation. Tag extraction returns the tag actually stored +in the envelope, which may be refined. Protocol details remain deferred; +this is the intended behavior, not a claim that the current client/engine +already implements it. Add refinement/decrypt integration coverage when +that work lands. + +**Decision, 2026-10-25:** direct `encrypt` handles one single-value input; +`EncryptInto` handles collection composition and batching. This supersedes +the earlier open question about automatic collection lifting. The agreed +public examples are: + +```rust +let scalar: Ciphertext = keyset.encrypt(1_i32).await?; + +let batch: Vec = vec![1_i32, 3_i32] + .encrypt_into(&keyset) + .await?; + +let blob: Ciphertext = keyset + .encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])) + .await?; +``` + +The batch produces two independent ciphertexts through the shared batch +executor; the blob encodes and seals the complete collection as one value. +Do not add implicit collection handling to direct `encrypt`, or a separate +batch entry point merely to provide functionality already covered by +`EncryptInto`. Context-taking variants follow the same division. + +Acceptance checks: reject direct `encrypt(Vec)` before execution; +support context-free `Vec` to `Vec`; preserve target-directed +collection opening and batching; and recover the blob as a whole collection. +Existing collection composition is retained, while opaque `Ciphertext` and +its no-context target support are new work. + +The current prototype treats `Vec` as one byte value, but that is not a +compatibility constraint. The latest recommendation is uniform collection +behavior: `Vec` to `Vec` encrypts individual bytes, while +`OpaqueBlob>` encodes the complete vector as one value. Do not add +`Vec` to `Ciphertext` conversion or a bytes wrapper solely to retain +prototype behavior. A distinct raw-byte capability would need its own use +case. The blob codec need not reproduce the prototype's raw-byte encoding. + +For `OpaqueBlob`, select the library-owned codec, authenticated codec/type +discriminator and version, schema-evolution policy, error behavior, and +supported source/decoded types. Authenticate before deserializing plaintext. +Do not silently reinterpret an ordinary byte ciphertext as a different +encoding. Preserve an explicit wrapper rather than implicit Serde fallback. + +The constant-time question needs a concrete conclusion: assess `serdect`, +the codec, both sides of Serde (value traits and serializer/deserializer), +length/shape disclosure, and permitted custom implementations. If arbitrary +Serde types remain supported, do not promise end-to-end constant-time +serialization. Select a restricted profile separately if that property is +required; it cannot be inferred from the wrapper or its encryption primitive. + +Completion: scalar and blob examples persist and reload through a fresh +cipher, reject invalid encodings, and verify fixtures for the selected +format. The supported timing contract and codec version are documented. + +### D. Complete record derives and preserve collection composition + +Implement explicit passthrough and source-field renaming, independently of +context selection and storage. Generate decrypt reconstruction for the +same record. Support nested targets with and without configuration, and +report incompatible context or source types at their declarations. + +Keep the collection behavior already present: `Vec` to `Vec` composes +the target operations into one logical batch, rather than awaiting every +row separately. Preserve document/leaf key granularity; one logical batch +does not promise one physical ZeroKMS request beyond backend limits. + +Completion: the public mixed-record and searchable-record examples run, +including batch encrypt/decrypt, nested targets, renamed passthrough fields, +and stored context. Dynamic/Go declarations produce equivalent semantics. +A plaintext-side derive generating output types is deferred convenience, +not a prerequisite for these workflows. + +### E. Finish the configured policy's lifetime and query surface + +Use the prior query-design baseline and reconciliation above. The query +model is not an empty design space: field-owned index selection, typed +handles, and shared derivation already have a detailed proposal. + +The typed-factory direction is established. The target-definition section +above now proposes its owner: a shared immutable fixed definition, or an +explicit runtime instance, with generated typed field handles. Review that +proposal and prove its initialization/lifetime behavior before treating the +once-only reuse claim as implemented. Multi-field runtime configuration and +the exact generated signatures still require specification. + +Write complete static and Go examples for equality, JSON containment, +ordered paths, and extraction, all using the write policy. Specify query +results and unsupported-operation errors. Keep those examples free of +mandatory hand-written plans. Preserve direct typed EQL conversion and +catalog/guest support as deliverables of the configured-target sequence. + +### F. Make naming and coordinated integration part of completion + +Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for the +proposed role-based JSON names, retained `OpaqueBlob`/`Ciphertext` spellings, +shared format crate and primitive/application split. Retain accepted +declaration/convenience trait names. Recommend renaming the plan builder's +output-selection method to `target`; distinguish it from executing +`EncryptInto`. New spellings and crate moves still require review. + +Remove replaced APIs directly and update EQL, Go, and guest consumers +together. There is no requirement to read prototype ciphertext or preserve +old imports. Specify the selected encoding and cryptographic semantics, +and establish fixtures for that contract going forward. + +Completion: all public examples are executable tests, static/dynamic +cross-opening and querying pass against the selected storage format, batching +checks pass, and the guest builds remain supported. No new API is described +as implemented until its Rust and Go acceptance criteria are met. + +## Recommended order for the remaining design decisions + +### Decision checkpoint, 2026-10-25 + +Settled direction: keyset-directed single-value encryption; target-directed +collection batching; explicit `OpaqueBlob`; conversion methods separate +from `EncryptTarget`/`DecryptSource` declarations; explicit context-input +capabilities instead of `Default`; typed Rust options factories and dynamic +deserialization into the same options; static and dynamic support together; +shared typed definitions and field handles; a consistent `using` selector +with strict typestate and definition-first query spelling; `OpaqueEncode` +/ `OpaqueDecode`; consuming encrypt and encrypt_into with guarded cleanup; +ordinary keyset decrypt alongside the destination-directed decrypt pair, +both with context variants and ordinary owned `T: Zeroize` results; +separate ciphertext context and search scope with `PrimaryKey`/`ScopeKey` +eligibility; definition-time role validation in the dynamic interface, +optional named-type declarations and explicit scope eligibility; and a +versioned interoperability specification with shared conformance coverage; +and no backward compatibility with the prototype. Do not reopen these +choices merely because older examples or work-area descriptions retain +provisional wording. + +The remaining product/API decisions are narrower. Distinguish them from +specification and implementation obligations: + +| Remaining question | Decision required | +| --- | --- | +| JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | +| Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | +| Stable blob profile | Specify supported Serde shapes, lossless options/enums/maps, authenticated codec versions and compatibility fixtures under Vitamin C #519. This is the immediate priority; format-preserving cleanup/timing improvements are follow-up work, with limitations documented honestly. | +| Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | + +Next review the JSON operation document and its static/dynamic examples, +then turn the required inventory into implementation phases and formal +interoperability fixtures. Stable storage-format work is tracked upstream. +Finish remaining query integration details without reopening ORE equality. +The user has now requested the naming/package pass while JSON review +continues; that proposal is ready for review in the packaging companion. + +Specification/implementation work still includes exact context/scope grouping +and encoding, dynamic grammar and version identifiers, error codes, binary +envelope fields, Rust/Go/SQL fixtures, and the refined-versus-original tag +flow. Record candidate choices in the formal specification for review; do +not treat every byte offset or adapter signature as a new architectural +decision. The initial persistence recommendation remains ordinary Serde +composition for explicit layouts, without a new arbitrary-tree wire format. + +Cleanup implementation still needs guarded encoder sinks, partial decode +handling, cancellation/drop checks, and an explicit drop-guarded result +adapter. Under the user's latest sequencing decision, implementation +improvements may follow the stable format; preserve the stored contract and +document actual guarantees instead of claiming the eventual goals are +already satisfied. No completed constant-time/zeroization audit is implied. + +The shared-definition caching, initialization error/retry behavior, generic +identity, and request lifetimes still need concrete signatures and proof. +That is implementation design within the recorded ownership model, not a +reason to reopen the question of whether writes and queries share a policy. + +Passthrough implementation, transcoder adapters, collection composition, +trait relocation, documentation updates, and acceptance tests remain work +items. They are not reasons to characterize the agreed API direction as +undecided. No backward-compatibility phase is needed for the prototype. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index ad6afc78e..f60a088e5 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -7,12 +7,20 @@ Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-e an API design draft covering encryption, persistence, decryption, and queries. The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) explore context sources and storage using Serde-inspired attribute syntax. +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +requirements, recommended decisions, sequencing, and completion criteria. This is a running issue log. Add a numbered section as each issue is found; keep observations, proposed changes, and unresolved decisions distinct. All issues below remain unimplemented. Individual decisions are recorded within each issue; proposed APIs are illustrative, not implemented. +**User clarification, 2026-10-25:** this API has never shipped beyond an +early prototype. Earlier suggestions to preserve old encodings or public +imports are superseded: backward compatibility and prototype data migration +are not requirements. New Rust, Go, guest, and EQL implementations must +agree with each other, but need not retain the prototype's representations. + ## 1. Cipher trait mismatch and explicit context ergonomics **Status:** Open. @@ -130,6 +138,22 @@ the simple scalar case. ### Proposed direction and open decisions +**Subsequent decision:** ordinary encryption should produce one opaque +ciphertext; choose structural encryption through an explicit target, or +encode a composite plaintext first with `OpaqueBlob`. The +[implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends +adapting a native single sealed leaf through the existing transcoder, +without serializing an entire encrypted tree or re-encrypting it. + +**Collection decision, 2026-10-25:** direct `encrypt` handles a single-value +input and does not automatically lift over collections. Retain collection +composition through `EncryptInto`, including +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +Use `OpaqueBlob>` to encode and seal a complete collection as one +ciphertext. This settles the previously reopened collection question. +The spelling for byte buffers (`Vec` versus an explicit bytes wrapper) +remains a separate detail to resolve. + Provide a straightforward opaque ciphertext storage round trip for scalar values, with direct byte conversion and reconstruction. Decide whether this means a dedicated scalar result type, a wrapper, or conversion helpers on @@ -138,8 +162,19 @@ the existing API. Also decide whether Base64 and Serde should be supported conveniences, and whether whole-tree serialization belongs in Stack or remains caller-owned. If it remains caller-owned, make that distinction explicit and keep it out -of the simple scalar path. Preserve the existing canonical leaf encoding; -an ergonomic improvement should not require changing stored ciphertext. +of the simple scalar path. The prototype's canonical leaf encoding is a +useful starting point, not a backward-compatibility obligation. + +**Follow-up, 2026-10-25:** text conversions must use the existing +`Base64Text` / `Base85Text` wrappers from EQL bindings. Explore moving these +general encoding types below Stack/EQL and supporting ciphertext-specific +encoded field targets. The persisted ZeroKMS key tag must be extractable +simply in Rust and PL/pgSQL. A refinable-tag protocol is being designed +separately. The user clarified that ZeroKMS checks policy/logs against the +refined tag and returns the original tag for reconstruction of leaf AAD; +record these distinct roles in the implementation. The +[persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) +record the proposals and the Base85/SQL decoding constraint. **Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) (`StackCipherText`, `SealedValue`) and @@ -540,11 +575,10 @@ become another public API users must understand. Move the value-level backend adapters too, or separate a lower-level core/primitive boundary. - Context composition, type tags, ciphertext framing, index normalization, - and binding transport frames have compatibility consequences. Preserve - their existing encodings when changing ownership; any deliberate semantic - change needs separate versioning and migration treatment. -- Retain one canonical trait/type identity. Compatibility re-exports can - preserve an old path where dependencies permit; copied independent trait + and binding transport frames must agree across the new consumers. They + may change without preserving prototype data or providing migration. +- Retain one canonical trait/type identity. Update consumers directly; + compatibility re-exports are unnecessary. Copied independent trait definitions would require duplicate implementations and adapters. - Account for guest/FFI consumers and EQL target implementations, not just Rust application imports. The no-HTTP/WASM path must remain usable. @@ -682,3 +716,408 @@ for the proposed selectors and input requirements. **Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) (`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). + +## 13. JSON layouts and encrypted-output adaptation + +**Status:** Open; initial assessment of the JSON tree implementation on +`next` at `b1049824b`. No replacement API selected. + +**Agreed follow-up:** investigate replacing the EQL JSON adapter's +serialization round trips with direct typed conversions. This does not yet +select an extended visitor protocol or authorize a particular replacement +API. + +`Layout` describes a target selected by a value carrying runtime options. +It associates a `Target: EncryptFrom` and an `Index`, and supplies an +`encryption(&self)` description plus the configured index for queries. +`EncryptFrom::encryption()` instead selects a description by output type. +For JSON, both routes can produce `JsonDocument`: selecting that type uses +default options, while `Json::new(options)` can specify ordered paths and +whether the root is sealed. A plan retains those options for its queries. + +This is broader than choosing a Rust record's field layout. JSON options +affect cryptographic structure, query capability, and information exposed +to the server. The same output type can represent different configurations; +the type alone does not identify them. The JSON tree belongs to the +searchable-layout workflow, rather than being an opaque serialized blob. + +The existing `target::transcode` protocol serves a different stage: a +`Reader` hands completed ciphertext, structural markers, metadata, and terms +to a destination `Visitor`. `Encryption::transcode()` uses it to adapt +encrypted output without an intermediate serialization tree. It does not +choose JSON indexing options or generate missing search terms. + +The new EQL adapter currently takes another route. `JsonTree` implements +`Layout`, delegates encryption to the +engine's `Json` layout, and wraps its completed `JsonDocument` with EQL +version and column metadata. Its envelope conversion uses a +`serde_json::Value` intermediate; a private helper also named `transcode` +converts query/extraction types through `to_value`/`from_value`. Neither is +the existing `target::transcode` protocol. That protocol currently has no +JSON-document reader or visitor event. The conversion processes encrypted +output and query metadata; it is not a plaintext serialization bypass. + +On decrypt, the EQL target validates its stored column against any supplied +expectation, reconstructs the engine document through the wire form, and +delegates opening to the engine. Its layout object is not required to +perform that conversion. + +### Questions to resolve + +- Keep encryption configuration distinct from encrypted-output adaptation. + Decide whether `Layout` is the clearest public name for a configured + target, especially alongside the earlier use of "layout" for output + records. +- Decide how runtime target options fit the retained `EncryptInto` API; + currently type-directed conversion selects the target's defaults, while + plans accept a configured layout via `encrypt_with_layout`. +- Evaluate a direct typed JSON-document adapter versus extending the + encrypted-output visitor. Avoid forcing the JSON protocol into generic + map/sequence events that lose its header, shared document key, projection, + and extraction semantics. A shared architectural boundary does not + necessarily require one universal visitor. +- Consider whether a general configured-target abstraction should require + an index: today's `Layout` requires one even though encryption-only + targets can also need options. +- Preserve the link between write configuration and query configuration; + a plan's reuse of the configured index is useful independently of naming. + +**Sources:** [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) +(`Layout`, `json_document`, `EncryptFrom for JsonDocument`), +[encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs), +[JSON options and document](../../packages/stack-encrypt/src/json/mod.rs), +and [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) +(`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). + +### Static Rust surface versus runtime configuration + +The intended public Rust experience is traits and derives. Treat plans as +the lower-level composition interface supporting those APIs and the dynamic +FFI interface, rather than requiring Rust callers to write a plan for JSON. +The derive already emits plans; its implementation can retain that shared +execution path without making the plan builder the primary user example. + +JSON already has a type-directed path. Given an initialized `cipher` and +its `keyset`, this uses existing methods and default JSON options: + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use eql_bindings::Identifier; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, EncryptInto}; + +let profile = json!({"name": "Alice", "age": 34}); +let column = Identifier::for_column("users", "profile")?; + +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored: Value = encrypted.decrypt_into(&keyset, column).await?; +assert_eq!(restored, profile); +``` + +This produces searchable JSON with a sealed root and no ordered paths. +The explicit decrypt context checks the expected column against the stored +identifier. The snippet is source-checked, not compiled here. + +The earlier `SearchableEmail` example fixes its recipe in the target type: +ciphertext plus an equality term. `EncryptFrom::encryption()` takes no +configuration value. JSON's default target also fixes a recipe in its type. +What `Layout::encryption(&self)` adds is access to instance configuration: +one `JsonTreeDocument` output type can be produced with `$.age` ordered, +with `$.salary` ordered, or with different root settings, selected at +runtime. Its configured index also supplies the matching query behavior. + +Search metadata itself does not require `Layout`. A custom target could +fix ordered JSON paths in a manual `EncryptFrom` implementation using the +existing JSON operation. A derive could eventually generate such a fixed +policy. Runtime-configured targets need an additional configuration input; +today `encrypt_into_with_context` accepts context but no layout/options, +and custom JSON configurations use the lower-level plan or `keyset.run` +path. Keep those options separate from cryptographic context. + +The cryptographic distinction is separate from the API distinction: JSON +uses a specialized document operation with a shared document data key and +node/extraction semantics. It cannot be implemented merely by adding a +generic equality-term field to the ordinary ciphertext tree. That explains +the JSON operation, but does not by itself require a public `Layout` trait. + +### Naming obscures roles + +| Current type | Role | +| --- | --- | +| `serde_json::Value` | Plaintext input and recovered plaintext | +| `stack_encrypt::json::JsonOptions` | Runtime configuration | +| `stack_encrypt::target::Json` | Configured index; also implements `Layout` | +| `stack_encrypt::json::JsonDocument` | Encrypted engine document, including search metadata | +| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target/layout | +| `eql_bindings::v4::json_tree::JsonTreeDocument` | Encrypted EQL storage envelope | + +Review names that state these roles, using `Encrypted` for ciphertext-bearing +documents, `Index` for index descriptions, and `Target` or `Config` for +configuration, with module paths distinguishing EQL from engine types. +No replacement names are selected. `Json`'s dual index/layout role should +be considered before simply renaming it. + +### Candidate: a fixed-policy wrapper around the EQL output + +**Proposal, not implemented or agreed syntax.** Attach the fixed policy to +an output newtype, while the inner library type owns the database format. +The alias below only clarifies the existing type's role; it is not a +library rename. + +The later [implementation plan](2026-10-24-stack-encrypt-api-plan.md) +recommends a general typed-options factory, `target(options = factory)`, +instead of the JSON-specific attribute below. Dynamic callers deserialize +options into the same concrete type; Rust and Go support ship together. +This sketch is retained to explain the earlier alternative. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use eql_bindings::Identifier; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, DecryptInto, EncryptFrom, EncryptInto}; + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = Value, + transparent, + json_tree(ordering_paths = ["$.salary"]) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); + +// Given a keyset: +let payroll = json!({"name": "Alice", "salary": 120_000}); +let column = Identifier::for_column("employees", "payroll")?; + +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +// The wire representation is the inner EQL envelope, with no extra field. +let wire = serde_json::to_vec(&encrypted)?; +let loaded: EncryptedPayroll = serde_json::from_slice(&wire)?; +let restored: Value = loaded.decrypt_into(&keyset, column).await?; +assert_eq!(restored, payroll); +``` + +The proposed `stash(transparent, json_tree(...))` selects the configured +JSON operation for the inner target and wraps its completed output. It must +not invoke the ordinary scalar/record derivation on the wrapper, nor invoke +the inner target's default encryption and then try to add ordering terms. +It inherits the EQL target's caller-context and decrypt behavior. Serde's +existing `transparent` attribute preserves the EQL envelope; it does not +provide database-driver trait implementations, which remain a separate +integration concern. The root stays sealed under this candidate's defaults. + +The generated declaration must also expose the configured index metadata +for future typed queries, so queries can reuse the policy instead of +repeating paths. The query-facing syntax remains undesigned. The core derive +must delegate configured output construction to a capability of the inner +target, rather than depending directly on the EQL crate. Whether to express +that capability through `Layout` or another trait remains open. + +### Alternative: configuration on a plaintext wrapper + +The user's proposed `JsonPlaintext::new(value).with_ordering_path(path)` can +still yield the exact EQL wire format. Configuration source and output +representation are independent: an explicit `EqlEncryptedJsonTree` target +can accept that input and execute the same configured JSON operation. The +wrapper must be treated as an encryption request, not serialized as part of +the plaintext. The current API does not provide this wrapper. + +This is attractive when paths come from runtime configuration, but repeats +fixed policy at call sites and needs a way to reuse that policy for queries. +Prefer the output newtype for the fixed-policy example. If the runtime +wrapper is added, initially do not implement conversion from it into +fixed-policy targets: avoid silently overriding or merging conflicting +policies. Neither syntax permits callers to redefine EQL's required fields +or its cryptographic wire protocol. + +A fixed Rust type describes the policy for new writes; transparent Serde +deserialization alone does not prove a stored document was produced with +that policy. Ordered paths are not all recorded in the document header. +Policy changes and heterogeneous existing rows need explicit compatibility +and migration semantics before claiming stronger read-time guarantees. + +## 14. Decrypt convenience methods need explicit context variants + +**Required direction:** provide a no-context method and a separate +`_with_context` method, mirroring the encrypt-side distinction. The user +prefers the name `decrypt_from`. + +Today `ciphertext.decrypt_into(cipher, context)` exists on the blanket +`DecryptFrom` extension trait. `Plaintext::decrypt_from(ciphertext, cipher)` +and `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)` also +exist on that trait. `DecryptInto

` separately declares the opening +operation. These method/trait names are easy to confuse. + +**Recommendation:** use the destination-directed pair in the preferred +public examples. `User::decrypt_from(encrypted, &keyset)` follows Rust's +From convention; `encrypted.decrypt_from(...)` would reverse it. The +receiver and migration of the competing method remain recommendations to +confirm, not settled user decisions. Do not change the declaration trait +merely because its convenience spelling changes. Address trait discoverability +with issue 5 and optional-context capability with issue 12. + +## 15. Coordinate the Go JSON work with the shared API redesign + +The user identified [PR #1353](https://github.com/cipherstash/stack/pull/1353) +as parallel agent work. The PR is currently a Go JSON design document with +implementation tasks #1348 through #1352, tracked by #1347. Its dynamic +JSON/options path, public names, context behavior, and staged extraction +support must be reconciled with the configured-target and trait/context +contracts developed here. Preserve compatible catalog, transport, engine, +and testing work rather than designing or implementing it twice. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +the concrete reconciliation points after reading the PR write-up. This is +a local coordination item; no message or review was posted to that agent +or PR. + +## 16. Integrate the prior query design rather than reopening it + +The user supplied **Stack Encrypt query design** (2026-10-21), read in full +on 2026-10-25 from `/Users/dan/Downloads/Stack Encrypt query design.md`. +It proposes field-owned index selection, one answering index per operator, +typed field handles, a single capability declaration, and shared query +derivation/refusals. It is explicitly a discussion draft with open questions. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses that +draft as the baseline and records concrete reconciliation points: our +output-side derives and configured options factories, JSON operators and +extraction, context inputs, static/dynamic portability, and validation +timing. Older compatibility requirements are superseded by the user's +prototype clarification. The source document was left unchanged. + +## 17. Use consistent definition-selection names for encrypt and query + +The user finds `.using(...)` for encryption and `.on(...)` for queries +confusing and asks to consider one name for both. Recommend `.using(...)`: +the selected definition or field supplies the operation's policy. Final +spelling is not yet accepted. Retain the operation-specific bounds: +encryption selects the output target's associated definition; queries +require a field or single-value target implementing `QueryTarget`. +This is a naming change, not a merger of those capabilities. + +The user also prefers `cipher.using(&field).query(op, input)` and explicitly +requires strict typestate. Supporting both orders is acceptable to them. +The plan recommends definition-first as canonical, with operator-first +converging to the same typed request. Missing definition/required context +must prevent awaiting on the static path; repeated selection must not +silently overwrite a request's definition. The current `.query(...).on(...)` +implementation is not claimed to already provide this API. + +## 18. Single-ciphertext representation needs consuming, cleanup-aware inputs + +**Decision, 2026-10-25:** use `OpaqueEncode` / `OpaqueDecode` for single-value +encoding/decoding, distinguishing one opaque ciphertext from target layouts. +Both direct `encrypt` and target-directed `encrypt_into` consume input and +are responsible for cleanup of owned plaintext artifacts, including the +context-taking variants. Explicit passthrough fields move to the output. +The plan records source cleanup capabilities, +immediate guarding before returning an unpolled request, cleanup of encoded +buffers/scratch on all ordinary exit paths, and boundaries for caller copies, +custom serialization, passthrough output, and Go host/guest ownership. +`Serialize` alone cannot establish cleanup of an `OpaqueBlob` source. +Consumption and zeroization are distinct; no completed audit is claimed. + +**Follow-up decision:** plaintext types must implement the existing +`Zeroize` trait; do not introduce our own cleanup trait for now. For the +proposed Serde blob, this means `Serialize + Zeroize` on input and +`DeserializeOwned + Zeroize` on owned decode. Library guards must actually +invoke cleanup; partially constructed Serde values need separate treatment. + +**Decrypt decision:** retain `keyset.decrypt` / `decrypt_with_context` for +ordinary opaque ciphertext and `T::decrypt_from` / +`decrypt_from_with_context` for declared targets. Return an ordinary owned +`T: Zeroize` on success, transferring cleanup responsibility to the caller. +Stack guards plaintext intermediates while it owns them. Explicit +drop-guarded results should be supported without becoming the default; +`Zeroize` alone does not imply cleanup on drop. This does not resolve the +separate partial-deserialization cleanup requirement. + +**Codec recommendation:** reuse Vitamin C frame machinery with a blob +mapping. The 0.8.0 transport Serde implementation already exists and guards +serialization buffers, and its frame writer wipes allocations on growth. +Do not reuse that mapping unchanged: option presence can be lost +(`Some(())` and `None` both encode as null), maps require text keys, and +successful `to_vec` returns an unguarded vector. See the +[codec assessment](2026-10-24-stack-encrypt-api-plan.md#blob-codec-assessment-vitamin-c-frames) +for the required work and Borsh fallback. Neither codec choice nor the +`Zeroize` bound establishes constant-time behavior or partial-decode cleanup. + +**Priority update:** first specify stable, versioned storage bytes; the +existing frame codec was ephemeral transport. Further zeroization and timing +implementation improvements may follow without changing stored format. +The upstream action is +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). + +## 19. First-release completeness and remaining review decisions + +**Accepted, 2026-10-25:** encoded ciphertext wrappers are first-class field +targets; final wrapper names/package placement come last. ORE does not +answer equality initially, even for integers. Preserve a future explicit +capability for equality-preserving encodings; never infer exact equality +from ordering support for text or other lossy encodings. Keep operator +provider overlap validation and avoid silently changing existing definitions. + +The JSON first version must be comprehensive across Rust and dynamic/Go, +including extraction, opening entries, children/counts and subtree walks. +Claude's earlier Go plan deferred walks; the new scope supersedes that +deferral. The [operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) +lists required operations, current evidence, proposed Rust/dynamic surfaces, +SQL use, missing-path/null semantics and implementation acceptance gates. +Names and packaging are explicitly the last design activity. + +**Further accepted decisions:** custom Rust stages can remain explicitly +Rust-only with `NotPortable` for unsupported dynamic export; built-ins retain +Rust/Go parity. All key-independent checks precede key resolution/loading. +The detailed JSON operation contract is still under user review. + +## 20. Plaintext-derived keysets and optional provider policies + +The user requested routing a plaintext record to a keyset by ID/name from +one of its fields; assessment of Vitamin C's unused `AsyncCipher`; and +ZeroKMS lock context including principal JWT claim selectors. Keyset and +lock-context capabilities must remain optional for future AWS KMS/Vault +backends. See the [provider-capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) +for current source evidence, proposed static/dynamic declarations and +acceptance work. + +Current facts: Stack resolves keysets by ID/name but generation batches use +one preselected keyset. Stack deliberately orchestrates provider requests +outside Vitamin C's leaf walk and already implements synchronous `Cipher`. +Its provider payload layer supports lock context but Stack Encrypt sends +empty lock context. The existing data/index key abstractions still embed +ZeroKMS protocol types; future providers need a neutral base plus optional +capabilities, not no-op implementations of mandatory ZeroKMS features. + +Recommendations, not yet accepted: derive key routing as an explicit target +selector and resolve it in the client executor; partition batches by resolved +key/provider/principal requirements; carry lock context as a separate typed +provider policy on both generation and retrieval; reject unsupported required +features before I/O. Keep key routing, ciphertext context, search scope and +principal policy separate. Revisit cipher adapter cleanup during packaging +without discarding cross-record batching or native/WASM support. + +## 21. Packaging, naming and retirement of AsyncCipher + +The user favors removing Vitamin C `AsyncCipher` and moving application +encryption behavior into Stack Encrypt. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes +Stack ownership of value traversal, declarations, target/index adapters and +execution; a small shared `stack-encrypt-format` crate for codecs and typed +encodings; a ZeroKMS-specific client crate; and reusable primitives remaining +in Vitamin C. It includes an acyclic dependency map and public name glossary. + +Stack does not use `AsyncCipher`, but Vitamin C's async helpers and tests do. +Retire that execution path with its consumers and move relevant coverage; +do not treat deletion of the trait alone as complete. Normalization/PRF/ORE +adapters also depend on the existing value model and need separation from +their primitives. Moving the frame implementation refines the earlier codec +ownership recommendation; coordinate stable storage work under Vitamin C +#519. Names and package moves remain proposals, not implementation changes. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index da361fc04..ee700cabc 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -5,6 +5,12 @@ Date: 2026-10-24. Companion to the The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) explore directional source selection, storing context, and decrypt inputs across nine scenarios. Their attributes are proposals, not implemented APIs. +The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) +covers the comprehensive first-release JSON workflows with Rust, dynamic +and SQL examples, including extraction and walks. + +The user clarified on 2026-10-25 that this API is an early prototype: +backward compatibility with its old APIs and encodings is not required. This document tests the proposed public API against three complete workflows: encrypt, persist, load, and decrypt a user; the searchable workflow also @@ -21,6 +27,36 @@ the examples really encode, write, read, and decode their stored forms. ## What is agreed, and what these examples propose +**Ownership decision (2026-10-25):** `encrypt` and `encrypt_into` consume +their inputs, including context-taking variants. Their implementations must +guard owned plaintext and temporary buffers for cleanup on success, error, +and request drop; passthrough fields intentionally transfer to the output. +The single-ciphertext capabilities are named `OpaqueEncode` / `OpaqueDecode`. +Plaintext types must implement `Zeroize`; no new cleanup trait is planned. +The candidate programs below predate that bound: update their source derives +and any reuse of consumed input when making them executable. +`Serialize` alone does not establish the cleanup contract for `OpaqueBlob`. + +**Decrypt decision (2026-10-25):** retain direct `keyset.decrypt` and +`decrypt_with_context` for opaque ciphertext, alongside destination-directed +`T::decrypt_from` and `decrypt_from_with_context` for declared targets. +Successful calls return ordinary owned `T: Zeroize`; the caller then owns +cleanup responsibility. Stack guards intermediate plaintext. Support +explicit drop-guarded results without imposing a wrapper by default; +`Zeroize` alone does not wipe returned values on drop. These are accepted +API contracts, not a claim that the prototype already implements them. + +**Collection decision (2026-10-25):** direct `keyset.encrypt(value)` handles +one single-value input and produces `Ciphertext`; it does not automatically +encrypt collections element by element. Collection batching uses +`EncryptInto`, for example +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` instead produces +one ciphertext for the encoded collection. The latest recommendation +applies the same rule to byte vectors, with no wrapper or special case +needed to preserve prototype behavior. These calls require the proposed +opaque ciphertext and no-context target support. + | Surface | Status in this review | | --- | --- | | `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | @@ -32,10 +68,11 @@ the examples really encode, write, read, and decode their stored forms. | `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | | `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | | `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | -| `Ciphertext` as an encryption/decryption target | Proposed scalar target, including support for `String` and `Vec`; does not silently flatten a struct into a blob | +| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64/Base85 storage compose with encrypt/decrypt targets; final names remain for the naming pass | +| `Ciphertext` as an encryption/decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | | Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | -| `cipher.decrypt_with_context` | Proposed companion to `encrypt_with_context`; ordinary decrypt naming remains a review decision | -| `User::decrypt_from` / `decrypt_from_with_context` | Existing `DecryptFrom` extension conveniences, retained here pending the decrypt naming discussion | +| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair, symmetric with direct encryption | +| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair on `DecryptFrom` | | `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or @@ -46,8 +83,8 @@ subject to review where marked proposed or provisional. The proposed opaque ciphertext is intentionally untyped: the decrypt call's expected type selects decoding. A typed wrapper remains an open alternative. Its byte decoder parses a stored envelope; authentication happens on decrypt. -For a derived field it must preserve the existing context and leaf encoding, -not introduce a new cryptographic format just to improve ergonomics. +Context and leaf encodings must agree across the new Rust and dynamic +implementations; preserving the prototype's encoding is not required. The trait name `DecryptFrom` is somewhat surprising: it supplies the convenience functions, while `#[derive(DecryptInto)]` describes how an @@ -84,8 +121,8 @@ application to select and invoke a codec. The proposed wrapper owns the input; operation. The context identifies the application's document schema and binds this document to an externally known record id. -For this draft the wrapper accepts Serde `Serialize` on write and -`DeserializeOwned` on read. That input-trait choice is provisional: it does +For this draft the wrapper accepts `Serialize + Zeroize` on write and +`DeserializeOwned + Zeroize` on read. This does not claim that arbitrary Serde implementations can satisfy a constant-time contract. The initial wrapper has one library-selected, versioned binary codec, not a caller-selected codec parameter. Selecting and specifying that @@ -163,7 +200,7 @@ raw bytes. Application schema evolution remains explicit: a codec version does not automatically migrate changes to `User`. **Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` -form. A matching no-context `reader.decrypt(loaded)` is proposed for symmetry; +form. A matching no-context `reader.decrypt(loaded)` is accepted for symmetry; it is not the signature of today's two-argument decrypt method. This document uses explicit context in the full example so the saved format is unambiguous. @@ -547,7 +584,7 @@ separate adapter example to add once that boundary is settled. | Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | The principal unresolved choices exposed by the examples are the opaque -ciphertext name and storage contract, the public decrypt spelling, the blob +ciphertext name and storage contract, the blob codec and timing contract, EQL context adaptation, and optional source-side output generation. None requires reintroducing public `encrypt_as`. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md new file mode 100644 index 000000000..2ede5410a --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -0,0 +1,570 @@ +# JSON tree: first-release operation contract and public examples + +Date: 2026-10-25. Companion to the +[API plan](2026-10-24-stack-encrypt-api-plan.md) and +[public workflow examples](2026-10-24-stack-encrypt-public-examples.md). + +## Status and scope + +The customer requires a comprehensive first version. This document defines +the required operation inventory for Rust and the dynamic interface used by +Go. Extraction, opening extracted values, and subtree walks are included; +they are not a post-release phase. Internal implementation can still be +split into PRs. The feature is complete only when every required row below +has Rust, dynamic/Go, and applicable PostgreSQL coverage. + +This is a target API contract, not a claim that these public signatures +already exist. Rust and JSON examples below propose the redesigned surface. +Final names/package paths are deliberately left to the last naming pass. +The [packaging glossary](2026-10-25-stack-encrypt-packaging.md#json-names-by-role) +now proposes that pass; this document retains its working names until review. +They are candidates for executable documentation during implementation; +they have not been compiled against today's API. Existing SQL spellings and +current behavior are separately identified. Do not publish these as current +SDK instructions until the acceptance suite runs them. + +Sources reviewed: + +- [Claude's Go JSON plan, PR #1353](https://github.com/cipherstash/stack/pull/1353), + head `65ece530934d367e30ea99d1f149f76117191739`, still a draft when read. + Tracking #1347; steps #1348–#1352. Its step-5 deferral is superseded here. +- [JSON tree design](2026-10-17-json-tree-index.md), including the Phase 4 + notes on extracted-path checking and stored order. +- [Engine query implementation](../../packages/stack-encrypt/src/json/query.rs), + [path grammar](../../packages/stack-encrypt/src/json/path.rs), + [derivation/opening](../../packages/stack-encrypt/src/json/derive.rs). +- [EQL target and result types](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs), + [SQL functions](../../packages/eql/src/v3/json_tree/functions.sql), + [operators](../../packages/eql/src/v3/json_tree/operators.sql), + [refused operations](../../packages/eql/src/v3/json_tree/blockers.sql), + and [leakage contract](../../packages/eql/docs/reference/json-tree-leakage.md). + +The local source baseline is the reviewed `next` commit `b1049824b` and +this documentation branch. Untracked experimental EQL files are not used +as evidence of implemented JSON-tree behavior. + +## Roles and shared policy + +| Working name | Role | +| --- | --- | +| `JsonPlaintext` | Owned JSON input/output with `Zeroize`; an adapter around the engine's JSON value model, not ciphertext or a policy | +| `JsonOptions` | Validated target options: `sealed_root` and `ordered` paths | +| `EncryptedProfile` | Application-defined fixed-policy output wrapper | +| `JsonTreeDocument` | Existing EQL encrypted document envelope, stored in one column | +| `JsonTreeQuery` | Containment/path-equality query operand; cannot decrypt | +| `JsonTreeLeafQuery` | Ordering query operand; cannot decrypt | +| `JsonTreeSelector` | Encoded path selector, not a plaintext path or ciphertext | +| `JsonTreeEntry` | Extracted encrypted node with the metadata needed to open it | +| `JsonSubtreeKey` | Client-held walk request state: token, gates, requested path and policy binding | +| `JsonTreeSubtree` | Encrypted walk rows paired with their original request state | +| `JsonSelection` | Proposed owned plaintext result distinguishing missing, one value and multiple matches; implements `Zeroize` | + +`JsonPlaintext` is a proposed library adapter, not a claim that +`serde_json::Value` implements `Zeroize`. It accepts JSON text without +requiring callers to implement their own cleanup trait. Returned plaintext +is owned by the caller. Future zeroization/timing work must not change the +stored representation. + +One immutable definition supplies options, column identity, search scope +and ciphertext binding to every operation. Rust's derive builds it from a +typed options factory; dynamic loading deserializes the same options and +validates the same constraints. Query code never supplies a second options +object that can disagree with the write policy. + +For a row-bound, tenant-scoped field: + +- Ciphertext binding: declared column + tenant scope + row ID. +- Search selectors, value terms, ordering terms and walk-token derivation: + declared column + tenant scope, with the operation's domain separation. +- Queries do not require row ID. Decrypting a selected document, entry or + subtree requires it; SELECT examples must return that ID. +- Preserve JSON node selector/ordinal authentication in addition to these + outer bindings. The engine currently combines context roles; implementing + their separation is required, not just changing a derive attribute. + +## Definitive first-release inventory + +`q` below means the typed query result, not manually assembled EQL JSON. +SQL functions shown with `eql_v4` use the existing EQL v4 names. Parameter +types are explicit to avoid PostgreSQL selecting native jsonb operators. + +| ID | Required operation | Rust surface / result | Dynamic operation / result | Database use | +| --- | --- | --- | --- | --- | +| J01 | Encrypt document: object, array, scalar or JSON null | `value.encrypt_into` → encrypted target | `encrypt` → typed EQL document | Store `eql_v4_json_tree` | +| J02 | Decrypt complete document | `JsonPlaintext::decrypt_from` → JSON | `decrypt` → JSON text | Read full column | +| J03 | Batch write/read | `Vec<_>` conversion composition | `encrypt_many`, `decrypt_many` | Preserve row/result ordering; shared executor | +| J04 | Contains | `query(op::Contains, value)` → `JsonTreeQuery` | `contains` → same EQL operand | `profile @> $1::eql_v4.query_json_tree` | +| J05 | Contained by | `query(op::ContainedBy, value)` → `JsonTreeQuery` | `contained_by` | `profile <@ $1::eql_v4.query_json_tree` | +| J06 | Scalar equality at path | `query(op::AtPath(op::Eq), (path, scalar))` → `JsonTreeQuery` | `eq_at` | Same containment predicate as J04 | +| J07 | Four ordering predicates | `AtPath(Lt/Lte/Gt/Gte)` → `JsonTreeLeafQuery` | `lt_at`, `lte_at`, `gt_at`, `gte_at` | `(profile -> $1::text) < $2::eql_v4.query_json_tree_leaf`, etc. | +| J08 | Inclusive range | `AtPath(Between)` → two leaf operands | `between_at` → lower/upper operands | `>= lower AND <= upper` | +| J09 | Sort ascending/descending | `Selector` query → selector; no bound value | `selector` | `ORDER BY eql_v4.ord_term(profile -> $1::text) ASC/DESC` | +| J10 | Path exists | `Selector` query | `selector` | `json_tree_path_exists(profile::jsonb, $1::text)` | +| J11 | Extract first match | `Selector` query → optional encrypted entry from DB | `selector`; result accepted by `decrypt_entry` | `profile -> $1::text` or `json_tree_path_query_first` | +| J12 | Extract first match as text | Same selector; parse text into `JsonTreeEntry` | Same selector; parse encrypted entry text | `profile ->> $1::text`; never plaintext | +| J13 | Extract all matches | Same selector → encrypted entry rows | Same selector; `decrypt_entries` | `json_tree_path_query(profile::jsonb, $1::text)` | +| J14 | Open extracted scalar(s) | `decrypt_from` on source bound to selection request | `decrypt_entry`, `decrypt_entries` with selection handle | No server decrypt; container markers refused | +| J15 | Prepare subtree/positional request | `query(op::Subtree, path)` → `JsonSubtreeKey` | `subtree_key` → handle + SQL parameters | Token/gates, derived under field scope/options | +| J16 | Fetch/open subtree | Request-bound `JsonTreeSubtree`, then `decrypt_from` | `decrypt_subtree` with walk handle | `json_tree_subtree` / `json_tree_subtree_at` | +| J17 | Enumerate immediate children | J15 key, encrypted child rows | J15 key; opening scalar children uses J14 with walk validation | `json_tree_children(profile, token, gates)` | +| J18 | Count immediate children | J15 key; result is integer | J15 key; database returns integer | `json_tree_child_count(profile, token, gates)` | +| J19 | Walk budget and sealed-root fallback | Same J16 plaintext result regardless of path taken | Same handle and result shape | Default budget 64; zero walks without fallback | +| J20 | Compare stored documents/ordered entries within one scope | Existing typed EQL operands; no new plaintext term | Existing typed EQL operands | Document `@>`/`<@`; entry `< <= > >=`; same policy/scope required | + +J09–J13 and J17–J18 are SQL actions using a derived selector/key, not new +cryptographic primitives or SDK database execution methods. The dynamic +engine derives operands and opens results; the caller runs SQL. The +definition's capability discovery must describe both operand preparation +and the supported database use, including ordering-path restrictions. + +J20 does not authorize whole-document `=` or equality via an ordering +term. Initial ORE support never answers `Eq`. JSON scalar equality remains +available through its independent exact value-selector mechanism. Any +future ORE equality capability must be explicit and limited to encodings +that preserve equality; text/lossy encodings remain ineligible. + +## Path, null and result semantics + +- Grammar: `$`, `.member`, `["member"]`, nonnegative `[n]`, and `[*]`. + No filters, recursive descent or slices in this release. +- Selector-based operations address a path class across array elements. + Existing selectors erase positions (`[2]` has the same selector as + `[*]`). **Proposed first-class API rule:** reject positional paths for + `EqAt`, selector extraction/existence and selector ordering rather than + silently broadening them. Use the positional subtree route for exact + `[n]` retrieval. This is a deliberate stricter wrapper than today's engine. +- Wildcard equality means at least one matching element has that scalar + value. Containment preserves grouping within array elements; it is not + an ungrouped bag of independent terms. Multiple independent EqAt SQL + predicates do not imply the values belong to the same array element. + Use one containment needle to require that correlation. +- Equality accepts null, boolean, string and supported JSON number values; + objects/arrays are refused. Containers use containment, not an operator + labelled equality. At `$`, equality asks for that scalar root, not an + array containing it (which is containment semantics). +- Ordering is opt-in per path and accepts supported numbers/strings only. + No implicit ordering at undeclared paths. A wildcard ordering policy can + supply terms to many entries: `->` compares/sorts the first matching + entry, while `json_tree_path_query` can enumerate all and an explicit + SQL `EXISTS` chooses any-match semantics. Never present first-match as + any-match, or term equality as exact plaintext equality. +- A missing path gives no extraction row (or SQL NULL for first-match). + A present JSON null gives an encrypted entry that opens to JSON null. + SQL NULL for an absent document is a separate field-null policy, not an + encrypted JSON null. An empty object/array is neither missing nor null. +- JSON inputs cross the dynamic ABI as UTF-8 JSON text. No intermediate + conversion of JSON numbers to Go `float64`; use raw JSON/number-preserving + decoding. Define accepted numeric range and canonicalization from the + engine, test `2`/`2.0`, negative zero and large integers. Reject values + outside the supported representation rather than silently rounding them. +- SQL returns encrypted data, even with `->>`. A scalar node opens directly; + an ordinary container node is a marker, not a serialized subtree. Use a + subtree walk to reconstruct containers. The sealed root holds the whole + document and has an explicit whole-document/fallback decode path. +- Selection/open requests retain expected selector, field definition and + scope. Check the returned selector before opening: current code opens + under server-supplied selectors, which is insufficient to establish the + requested path. Row provenance requires explicit row binding. Walks must + validate the path/selector relationships needed for their requested result. + Successful decryption does not prove query completeness or authenticate + every search/index metadata field. +- Requests for `$` (or root-array wildcards requiring a root token) under + `sealed_root=true` use whole-document fetch/decrypt and local selection; + never fabricate a nonexistent root token. For a missing/empty/scalar + subtree where a descendants-only walk cannot distinguish the cases, use + the selected start-node metadata or full-document fallback explicitly; + do not treat an empty row set as an empty object by assumption. + +## Rust write/read example + +Proposed public code. `target(column = ...)` is illustrative declaration +syntax for an explicit EQL column identity; the exact attribute spelling +belongs to the forthcoming specification/naming pass. The options factory +is typed Rust, not JSON hidden inside an attribute. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use serde::{Deserialize, Serialize}; +use stack_encrypt::{DecryptFrom, DecryptSource, EncryptInto, EncryptTarget, StackCipher}; +use stack_encrypt::json::{JsonOptions, JsonPlaintext}; +use stack_encrypt::sem::TermError; +use std::error::Error; +use zeroize::Zeroize; + +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].date"]) +} + +#[derive(Serialize, Deserialize, EncryptTarget, DecryptSource)] +#[serde(transparent)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("users", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); + +#[tokio::main] +async fn main() -> Result<(), Box> { + let client = StackCipher::new().await?; + let keyset = client.default_keyset(); + let _definition = EncryptedProfile::definition()?; + + let profile = JsonPlaintext::parse( + r#"{"age":42,"name":"Alice","visits":[{"date":"2026-10-01"}]}"#, + )?; + let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; + + // These bytes can instead be bound to the EQL PostgreSQL column. + let stored = serde_json::to_vec(&encrypted)?; + let loaded: EncryptedProfile = serde_json::from_slice(&stored)?; + let mut opened = JsonPlaintext::decrypt_from(loaded, &keyset).await?; + // The caller now owns the returned plaintext and its cleanup. + opened.zeroize(); + + let profiles = vec![ + JsonPlaintext::parse(r#"{"age":42}"#)?, + JsonPlaintext::parse(r#"{"age":27}"#)?, + ]; + let encrypted: Vec = profiles.encrypt_into(&keyset).await?; + let mut opened = Vec::::decrypt_from(encrypted, &keyset).await?; + assert_eq!(opened.len(), 2); + opened.zeroize(); + Ok(()) +} +``` + +This example explicitly declares column-wide scope and no row binding by +its fixed standalone definition; the normative schema must express that +choice rather than silently replacing missing required tenant scope. +For an output-derived `EncryptedUser` with `id: PrimaryKey` and +`tenant_id: ScopeKey`, select those fields through the context/scope +attributes described in the API plan. Obtain its JSON field handle from +`EncryptedUser::definition()?.profile()` and require `TenantId` on query +requests. Store/return both ID fields for decrypt. If caller input is used, +use the agreed `_with_context` variants with a typed input separating +binding and scope. No row ID is ever an index/query input implicitly. + +## Rust query and extraction recipes + +These snippets share `keyset` and a resolved definition `profile` from the +write example. Add `use stack_encrypt::{op, json::JsonPath};`. The `using` +selector chooses that definition before the operation. `Selector` and +`Subtree` are proposed operation markers integrating today's separate +`json_selector`/`json_subtree_key` helpers into the shared definition API. +All query operands below use the same plaintext JSON adapter as writes. + +```rust +let profile = EncryptedProfile::definition()?; +let contains = keyset.using(&profile) + .query(op::Contains, JsonPlaintext::parse(r#"{"name":"Alice"}"#)?) + .await?; +let contained_by = keyset.using(&profile) + .query(op::ContainedBy, JsonPlaintext::parse(r#"{"age":42,"name":"Alice"}"#)?) + .await?; +let eq = keyset.using(&profile) + .query(op::AtPath(op::Eq), ("$.name", JsonPlaintext::parse(r#""Alice""#)?)) + .await?; + +let lt = keyset.using(&profile) + .query(op::AtPath(op::Lt), ("$.age", JsonPlaintext::parse("65")?)).await?; +let lte = keyset.using(&profile) + .query(op::AtPath(op::Lte), ("$.age", JsonPlaintext::parse("65")?)).await?; +let gt = keyset.using(&profile) + .query(op::AtPath(op::Gt), ("$.age", JsonPlaintext::parse("18")?)).await?; +let gte = keyset.using(&profile) + .query(op::AtPath(op::Gte), ("$.age", JsonPlaintext::parse("18")?)).await?; +let range = keyset.using(&profile) + .query(op::AtPath(op::Between), ( + "$.age", (JsonPlaintext::parse("18")?, JsonPlaintext::parse("65")?), + )).await?; + +let age = keyset.using(&profile) + .query(op::Selector, JsonPath::parse("$.age")?).await?; +let visit_dates = keyset.using(&profile) + .query(op::Selector, JsonPath::parse("$.visits[*].date")?).await?; +``` + +The results carry named typed EQL payloads. For example, `lt.selector()` +and `lt` bind the selector and leaf operand in the ordering SQL below; +`range.lower()`/`upper()` name the two bounds (proposed accessors). +For static policies, generated ordered-path handles can add compile-time +checks; runtime paths still validate against the same definition before +term derivation. Missing definition/scope makes a Rust request unawaitable. + +```sql +-- Containment/equality operands: bind the named query payload, not plaintext. +SELECT id, tenant_id, profile FROM users +WHERE profile @> $1::eql_v4.query_json_tree; + +SELECT id, tenant_id, profile FROM users +WHERE profile <@ $1::eql_v4.query_json_tree; + +-- Lower and upper bounds; use <, <=, >, >= individually for J07. +SELECT id, tenant_id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_tree_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_tree_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Existence, first encrypted entry, first entry as encrypted JSON text. +SELECT id, tenant_id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, tenant_id, profile -> $1::text AS entry FROM users; +SELECT id, tenant_id, profile ->> $1::text AS entry_text FROM users; + +-- All matching entries: group results by document before opening them. +SELECT u.id, u.tenant_id, e.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS e(entry); + +-- Stored-to-stored comparisons require matching policy and search scope. +SELECT a.id, b.id FROM users AS a JOIN users AS b + ON a.tenant_id = b.tenant_id AND a.profile @> b.profile; +SELECT a.id, b.id FROM users AS a JOIN users AS b + ON a.tenant_id = b.tenant_id + AND (a.profile -> $1::text) < (b.profile -> $1::text); +``` + +Add the application's tenant filter where appropriate; cryptographic scope +does not replace SQL authorization. A selector identifies a path class; +the SQL above never receives `$.age` or another plaintext JSON path. + +Proposed request-bound opening preserves the expected path and avoids the +current server-supplied-selector ambiguity. Here `entry` and `entries` are +typed database results for the corresponding selection request, and each +batch belongs to the identified document/context: + +```rust +let value = JsonPlaintext::decrypt_from(age.bind(entry)?, &keyset).await?; +let values = Vec::::decrypt_from( + visit_dates.bind_many(entries)?, &keyset, +).await?; +``` + +`bind` is a proposed checked source adapter, not a second decrypt method. +For row-bound policies, call `decrypt_from_with_context` with that result +row's expected ID and required scope. Missing first-match rows stay absent; +do not manufacture an encrypted null. Containers use the walk route below. + +## Walks, children, positional retrieval and fallback + +```rust +let walk = keyset.using(&profile) + .query(op::Subtree, JsonPath::parse("$.visits[0]")?).await?; + +// After the SQL below returns all rows for one document: +let source = walk.bind_subtree(rows)?; +let visits = JsonSelection::decrypt_from(source, &keyset).await?; +``` + +`walk` retains the requested path, resolved scope and definition; only the +token/gates go to SQL. Exact positions are supported here, including the +necessary selector gate to distinguish an array element from an object +member at the same ordinal. The final specification must normalize result +cardinality through `JsonSelection`: a concrete path yields `Missing` or +`One(JsonPlaintext)`; wildcard paths yield `Many(Vec)` in +document order. Missing and an explicit JSON null remain distinguishable. +`bind_subtree`/opening must preserve that distinction. `JsonSelection` is a +working type name and is imported from the proposed JSON module. + +```sql +-- $1: token; $2: gates; $3: budget (64 default, 0 disables fallback). +SELECT u.id, u.tenant_id, w.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree_at( + u.profile, $1::text, $2::text[], $3::integer +) AS w(entry); + +-- With no gates, the dedicated form is also supported. +SELECT u.id, u.tenant_id, w.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text, $2::integer +) AS w(entry); + +-- Immediate children and counts use the same token/gates. +SELECT u.id, u.tenant_id, c.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children( + u.profile, $1::text, $2::text[] +) AS c(entry); +SELECT id, tenant_id, + eql_v4.json_tree_child_count(profile, $1::text, $2::text[]) +FROM users; +``` + +Use one request-bound source per document; do not merge entries from several +rows into a subtree. Immediate scalar children can use checked entry +opening; container children need their complete descendants or an explicit +full-document fallback. The SDK must expose this distinction rather than +pretend that a container marker decrypts to its children. + +Budget 64/0 are current SQL semantics to retain in the first public scope. +When `sealed_root=true`, an over-budget walk returns the sealed root instead +of partial data; the client opens it and selects the original requested +path. With `sealed_root=false`, walk the full result or report an explicit +resource limit, never silently truncate it. The same requested value(s) +must result from budget 0 and fallback. Tokens reveal subtree structure +within their search scope and are not decryption keys; document their +existing leakage and lifetime with these examples. + +## Dynamic definition and operation examples + +These are logical JSON messages for the specification, not a promise to +replace the frame ABI with JSON. The host/guest transport encodes their +typed equivalents. `using` identifies a validated field definition; the +engine, not Go, creates EQL envelopes and query terms. Naming the definition +in a request must resolve to the same options and scope contract as writes. + +The fixed-column example's dynamic counterpart is: + +```json +{ + "version": 1, + "name": "users.profile", + "plaintext": "json", + "target": { + "name": "JsonTreeDocument", + "column": { "table": "users", "column": "profile" }, + "options": { "sealed_root": true, "ordered": ["$.age", "$.visits[*].date"] } + }, + "search_scope": { "kind": "column" }, + "context": { "kind": "none" } +} +``` + +`version: 1` is an illustrative schema version, to allocate in the formal +specification. Named types are optional. For tenant/row policies use the +accepted `ScopeKey`/`PrimaryKey` declarations and source selectors; the +resolved query input requires the tenant scope but never the row ID. + +Request corpus (each array element is an independent example, not one +batch of mixed operations). JSON text is deliberately explicit so no host +float conversion precedes the engine parser: + +```json +[ + { "operation": "encrypt", "using": "users.profile", "value": { "json_text": "{\"age\":42}" } }, + { "operation": "encrypt_many", "using": "users.profile", "values": [{ "json_text": "{\"age\":42}" }, { "json_text": "{\"age\":27}" }] }, + { "operation": "query", "using": "users.profile", "operator": "contains", "input": { "json_text": "{\"age\":42}" } }, + { "operation": "query", "using": "users.profile", "operator": "contained_by", "input": { "json_text": "{\"age\":42,\"name\":\"Alice\"}" } }, + { "operation": "query", "using": "users.profile", "operator": "eq_at", "input": { "path": "$.name", "json_text": "\"Alice\"" } }, + { "operation": "query", "using": "users.profile", "operator": "lt_at", "input": { "path": "$.age", "json_text": "65" } }, + { "operation": "query", "using": "users.profile", "operator": "lte_at", "input": { "path": "$.age", "json_text": "65" } }, + { "operation": "query", "using": "users.profile", "operator": "gt_at", "input": { "path": "$.age", "json_text": "18" } }, + { "operation": "query", "using": "users.profile", "operator": "gte_at", "input": { "path": "$.age", "json_text": "18" } }, + { "operation": "query", "using": "users.profile", "operator": "between_at", "input": { "path": "$.age", "lower": { "json_text": "18" }, "upper": { "json_text": "65" } } }, + { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.age" } }, + { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.visits[*].date" } }, + { "operation": "query", "using": "users.profile", "operator": "subtree_key", "input": { "path": "$.visits[0]" } } +] +``` + +Responses carry a result kind and typed payload: document, containment +operand, leaf operand, range, selection, or walk request. SQL preparation +returns the named domain and encoded payload (or typed token/selector +parameters); it does not concatenate SQL. Selectors serve J09–J13, and walk +keys serve J16–J19, so those SQL uses do not need redundant engine opcodes. + +For decode operations, this table specifies the dynamic signatures. Values +such as `document` and `entries` mean complete validated EQL payloads, not +literal placeholder strings to parse as ciphertext: + +| Operation | Required inputs | Result | +| --- | --- | --- | +| `decrypt` | `using`, `document`, required context/scope | JSON text | +| `decrypt_many` | `using`, documents with per-record context/scope | JSON texts in input order | +| `decrypt_entry` | `using`, `selection_handle`, entry, required context/scope | Scalar JSON text | +| `decrypt_entries` | `using`, selection/walk handle, entries grouped by record and context | Ordered scalar JSON texts | +| `decrypt_subtree` | `using`, `walk_handle`, complete rows for one document, required context/scope | Explicit missing / one JSON value / multiple matches | + +The response to `selector`/`subtree_key` includes an opaque request handle +in addition to SQL parameters. The handle retains the path and resolved +definition/scope in the engine, so callers cannot accidentally substitute +arbitrary path/options during decode. Specify handle lifetime and release +in the guest ABI; handles are not persistent database values. If the host +must recreate one after restart, derive it again from the original request, +not from server-returned metadata. Stale or mismatched handles are errors. +An equivalent Rust source adapter retains the same state directly. + +Example host orchestration, using proposed dynamic method names rather +than inventing ciphertext strings in a JSON example: + +```rust +let loaded = dynamic.load_definition(definition_json)?; +let encrypted = loaded.encrypt(json_text, inputs).await?; +let stored = database.store(encrypted).await?; +let opened = loaded.decrypt(stored, decrypt_inputs).await?; + +let selection = loaded.selector("$.age", scope).await?; +let entry = database.select_first(selection.sql_parameters()).await?; +if let Some(entry) = entry { + let age = loaded.decrypt_entry(&selection, entry, row_inputs).await?; +} + +let walk = loaded.subtree_key("$.visits[0]", scope).await?; +let rows = database.subtree(walk.sql_parameters(), 64).await?; +let result = loaded.decrypt_subtree(&walk, rows, row_inputs).await?; +``` + +This is an orchestration recipe: `database` represents application SQL, +not a new Stack Encrypt database driver; the input variables are supplied +by that application. Generated Go methods should hide these message details, +use named results, and call the same validated dynamic operations. JSON +marshalling helpers must not force a containment needle into a whole record +struct whose zero-valued fields accidentally narrow the query. + +## Explicitly unsupported operations + +The comprehensive first release covers the inventory, not all PostgreSQL +JSON operators. Reject whole-document `=`/`<>` as plaintext equality, +entry equality through ordering terms, plaintext path SQL operands, +JSONPath filters/recursive descent/slices, and in-place encrypted document +patch/delete/concatenation. Updates read/decrypt, modify, then encrypt the +complete document. Projection extraction does not confer the ability to +write a standalone node back into an existing tree. + +Reject malformed JSON/path/options; unsupported numeric values; container +EqAt operands; boolean/null ordering operands; undeclared ordering paths; +selector-position ambiguity; missing scope/context; unknown/expired request +handles; and wrong definition/scope/path during opening. Preserve useful +reason codes and structural locations without echoing plaintext values. +Structural validation that needs no key must precede key I/O. The current +engine refusal names include `JsonValueNotScalar`, `JsonLeafNotOrdered`, +`JsonPathNotOrdered`, `JsonOrderingAtPosition`, `JsonTooDeep`, +`JsonTooWide`, `JsonSealedRoot`, and `OpenedType`; final dynamic codes must +map deliberately rather than exposing unstable Rust enum spellings. + +## Implementation gaps and release gates + +1. **Shared definitions and scope separation.** Implement fixed/dynamic + target resolution, typed options, scope-role validation, row-binding + inputs and matching write/query context derivation. Generate capability + metadata from one registry, including JSON-specific operand/result kinds. +2. **Catalog/guest coverage.** Complete the JSON tree catalog, target table, + guest registration, dynamic index execution and JSON-text record kind. + Current Rust helpers/EQL types are not evidence that Go supports them. +3. **Direct EQL adapters.** Replace the existing serialization round trip + with typed conversions for documents, entries and operands. +4. **Selectors and opening.** Add the request-bound source adapters and + dynamic handles, validate expected selectors, preserve required row + context, and test mismatched results and missing headers/ordinals. +5. **Walk completeness.** Cover children/counts, concrete positions, + wildcards, empty containers, missing paths, scalar starts, root requests, + sealed-root on/off and budgets 0/64. Resolve reconstruction ambiguities + explicitly, including returning/fetching the start node when needed. + Do not ship opening/walks later under the earlier PR #1353 scope. +6. **Conformance/documentation.** Turn J01–J20 into identified test cases + and the examples into checked docs. Run Rust writes → Go reads/queries + and Go writes → Rust reads/queries against PostgreSQL. Include negative + cases, exact number handling, null distinctions, multi-element correlated + containment, and batch ordering. Test SQL GIN/btree usage on representative + containment/range queries. Randomized ciphertexts need semantic parity; + deterministic fixtures are only for explicitly seeded test builds. + +The current wire order/leakage and extraction trust boundaries must be +documented accurately. This operation inventory does not claim a new +authenticated-query-completeness protocol or a storage-order leakage fix. +Stable blob framing is a separate deliverable tracked in +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); it must +not be confused with the EQL JSON-tree storage format. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md new file mode 100644 index 000000000..7b4e749d3 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -0,0 +1,253 @@ +# Stack Encrypt packaging and public names + +Date: 2026-10-25. Recommendation for review, accompanying the +[implementation plan](2026-10-24-stack-encrypt-api-plan.md). +These are proposed boundaries and names, not implemented APIs or accepted +renames. Existing companion examples retain their working names until this +proposal is accepted. The detailed JSON operation contract remains under +separate user review. + +Source baseline: the current documentation branch and inspected local Vitamin +C 0.8.0 sources. Stack's +[cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) +explain its existing batching boundary. The +[EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) establishes +the optional downstream engine dependency, and +[encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) +are the concrete shared-type extraction candidates. This is a source-level +design assessment, not an audit of every external Vitamin C consumer. + +## Ownership rule + +Stack Encrypt owns the application value model and its execution: turning +owned Rust/dynamic values into ciphertext, structured targets and search +operands, and reconstructing plaintext. Vitamin C owns reusable cryptographic +and protected-memory primitives. A normal Stack Encrypt user should not need +to import Vitamin C encryption traits to implement an application type. + +Moving code should remove duplicate abstractions, not preserve both public +surfaces under new names. The user's prototype compatibility exemption +applies to this API; review other Vitamin C consumers before deleting their +dependencies upstream. + +## Proposed packages + +| Package | Responsibility | +| --- | --- | +| `stack-encrypt` | Public Rust API, value traversal/reconstruction, target declarations and definitions, context/scope roles, normalization and searchable-target adapters, plans, batching, JSON-tree execution, and dynamic definition validation/execution | +| `stack-encrypt-derive` | Derives generating the same declarations and definitions available by hand; re-export derives from `stack-encrypt` | +| `stack-encrypt-format` — new | Pure data representations/codecs shared by the engine and bindings: frame primitives and required scalar representations, stable blob profile, ciphertext envelope parsing/encoding, and validated Base64/Base85 text; no key requests or execution runtime | +| `stack-zerokms` — rename current `stack-kms` | ZeroKMS protocol client, authentication integration, native keysets, lock-context request types, and responses; it is not the neutral multi-provider abstraction | +| `eql-bindings` | EQL versioned storage/query types, SQL-compatible JSON serialization, catalog and Stack target adapters; stays downstream of the engine | +| Vitamin C crates | Protected memory, randomness, byte-level AEAD/key operations, HMAC/PRF and ORE algorithm primitives, and reusable canonical context encoding | + +Keep `stack-auth` and `stack-profile` separate: they have consumers outside +encryption. Keep `stack-guest-abi` for the host/guest calling convention. +Neither the ABI nor the Go host should own a second target-policy model. +Put the formal dynamic contract, schemas and conformance fixtures beside +Stack Encrypt; its implementation uses the same definition types and +validation as static Rust. + +Use modules for the remaining boundaries initially. In particular, do not +create separate execution, JSON-policy, query, context or provider-traits +crates merely to mirror each concept. + +The format crate has a concrete consumer justification: EQL's wire types +currently work without its optional `stack-encrypt` integration. They need +shared encoding types without an encryption client/runtime dependency. Make +Serde, schema and TypeScript support feature-scoped where practical; define +implementations alongside the owned types so downstream orphan rules do not +force duplicate wrappers. Re-export the ordinary format types through Stack +Encrypt, so users normally need only its dependency. + +The format crate is a single implementation with distinct profiles. Blob +storage, host/guest transport, and ciphertext envelopes have different +contracts and versioning requirements; sharing code must not silently make +the ephemeral transport Serde mapping the persisted blob format. This does +not introduce persistence for arbitrary internal `Node` trees: structured +outputs still use their explicit target layout and ordinary Serde composition. + +## Dependency direction + +```mermaid +flowchart TD + App[Application / Rust or guest adapter] --> EQL[eql-bindings] + App --> Engine[stack-encrypt] + EQL -->|optional target integration| Engine + EQL --> Format[stack-encrypt-format] + Engine --> Format + Engine --> Derive[stack-encrypt-derive] + Engine -->|optional ZeroKMS adapter| KMS[stack-zerokms] + Engine --> Vitamin[Vitamin C primitives] +``` + +The derive emits Stack paths but does not depend on the engine as a normal +runtime dependency. No Vitamin C primitive depends back on Stack Encrypt. +`stack-zerokms` does not depend on the engine. Provider traits belong to +`stack_encrypt::provider`; a Stack-owned adapter implements them using the +ZeroKMS client. This avoids an engine/client dependency cycle. + +Make ZeroKMS an optional provider feature independently of the HTTP feature, +so a different provider does not pull its protocol into the core contract. +Preserve injected providers and guest/native builds. Data-key support is the +base capability; index-key support, named-keyset resolution and lock context +remain explicit optional capabilities. Unsupported required capabilities +fail before key I/O. Future provider adapters can become separate downstream +crates when needed; do not design fictitious AWS/Vault keysets now. + +Do not re-export EQL from Stack Encrypt: EQL's target integration already +depends on Stack Encrypt. Applications explicitly import their EQL types. + +## What moves out of Vitamin C + +Move application-facing plaintext/ciphertext visitors, structural nodes, +value conversion traits and derives, tree walking, collection composition, +and application encryption helpers into Stack's value/target/plan layers. +Keep advanced extension points public only where target implementers need +them; they are not the normal root API. Internal nodes do not become a +second public opaque ciphertext type. + +Move the generic frame machinery and codecs needed by Stack into the format +crate. Stack owns the stored blob profile and its compatibility promise. +This refines the earlier recommendation to reuse Vitamin C's frame codec: +reuse its implementation, with a lossless storage mapping, while changing +its long-term ownership. Coordinate this handoff under +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); retain +the issue's format specification and fixture work. No upstream issue or +repository has been changed by this proposal. + +Also move normalization and PRF/ORE adapters that understand Stack's logical +values, traversal, targets or query semantics. Keep independent cryptographic +algorithms below them. The current `vitaminc-normalize`, `vitaminc-prf` and +`vitaminc-ore` manifests depend on `vitaminc-aead`; normalization even uses +its physical scalar types. Moving only the encryption traits would therefore +leave a dependency back to the old value model. Separate primitive inputs +from application adapters as part of the move; do not add a reverse +Vitamin C dependency on Stack to make it compile. + +Retain canonical context byte construction in `vitaminc-context` where it +serves independent algorithms. Stack owns the semantic roles and selection +rules (`PrimaryKey`, `ScopeKey`, ciphertext context, query scope and caller +input capabilities), reusing that canonical encoding instead of inventing +an incompatible second one. Provider policy remains a separate role. + +## Retire AsyncCipher + +Recommend removing Vitamin C's `AsyncCipher` and its associated application +execution helpers as the Stack executor takes ownership. Do not introduce a +replacement public catch-all async cipher trait in Stack. + +Stack already plans and groups remote key operations, then performs local +synchronous cryptography. Its current `MintedKeys`/`BoundKeys` implementations +use synchronous leaf cipher operations; `AsyncCipher` models a different +place to put asynchronous work. The meaningful async boundary is the provider +and the public pending operation, with batching controlled by Stack. + +It is unused by Stack, but not unused inside Vitamin C: `aead/src/seal.rs`, +`context.rs`, `typed.rs`, `walk.rs` and `normalize/src/terms.rs` expose or use +async helpers, and their tests implement it. Migrate the relevant batching, +ordering, failure and reconstruction coverage with the code, then delete +the retired trait/helper surface. Audit external consumers before the +upstream release. Separate internal leaf seal/open capabilities where useful +instead of retaining implementations that reject half of a combined trait. + +## Recommended public glossary + +| Name/path | Meaning | +| --- | --- | +| `stack_encrypt::Cipher` | Provider-backed execution handle; replaces `StackCipher` | +| `KeysetCipher` | Execution handle bound to a resolved keyset, available for providers supporting keysets | +| `Ciphertext` | One opaque encrypted value with a specified storage envelope; replaces the public use of `StackCipherText = Node` | +| `OpaqueBlob` | Explicit owned whole-value serialization adapter | +| `EncryptInto`, `DecryptFrom` | Target-directed encryption and destination-directed decryption convenience traits, with context variants | +| `OpaqueEncode`, `OpaqueDecode` | Single-ciphertext representation traits | +| `target::{EncryptTarget, DecryptSource}` | Declaration traits; matching derives are re-exported at the crate root | +| `target::TargetDefinition` | Validated immutable definition shared by static and dynamic paths | +| `context::{PrimaryKey, ScopeKey}` | Eligibility wrappers; neither automatically selects a context or keyset | +| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes; owned in the format crate and re-exported here | +| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated encoded opaque-ciphertext representations and supported field targets; encoding validity alone is insufficient | +| `plan::Plan` | Explicit low-level composition interface; output selection is `.target::()` | +| `provider` | Neutral provider contracts and capability bounds | +| `provider::zerokms` | ZeroKMS adapter and policy extension surface, including `LockContext` | +| `dynamic` | Versioned definition loading, capability discovery and execution through the common engine | + +Keep the already-agreed `.using(...)`, `definition()`, `encrypt`, +`encrypt_with_context`, `encrypt_into`, `encrypt_into_with_context`, +`decrypt_from` and `decrypt_from_with_context` spellings. Keep direct +`cipher.decrypt`/`decrypt_with_context`. Remove plaintext `.encrypt`, +redundant `encrypt_as`, and public `decrypt_into` as already agreed. + +Use `TargetDefinition` for the configured public contract and `Layout` only +for the advanced internal composition concept where that distinction is +needed. A layout does not name a plaintext document or a stored ciphertext. +Name low-level operations `seal`/`open` where they act on bytes; reserve the +application encrypt/decrypt vocabulary for the public value operations. + +Keep ciphertext representations in the format crate independent of traits; +Stack implements its own target traits for those types. Plain encoded text +is not itself an encryption target. Construction must validate both layers +for the ciphertext wrappers; avoid an unchecked public string constructor. + +## JSON names by role + +Keep the existing EQL package and version modules. Avoid importing low-level +engine JSON documents into normal application examples alongside EQL output +types. Recommend this glossary for the first-class public surface: + +| Working/current name | Recommended name | Role / home | +| --- | --- | --- | +| `JsonPlaintext` | `JsonPlaintext` | Owned plaintext adapter, `stack_encrypt::json` | +| `JsonOptions` | `JsonTreeOptions` | Validated tree-target policy, `stack_encrypt::json` | +| `JsonSelection` | `JsonSelection` | Proposed plaintext selection result, `stack_encrypt::json` | +| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document, `eql_bindings::v4` | +| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry, `eql_bindings::v4` | +| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result, `eql_bindings::v4` | +| `JsonTreeQuery` | `JsonContainmentQuery` | Containment/path-value query operand, `eql_bindings::v4` | +| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Ordering query operand, `eql_bindings::v4` | +| `JsonTreeSelector` | `JsonSelector` | Encoded path selector, `eql_bindings::v4` | +| `JsonSubtreeKey` | `JsonWalkRequest` | Client-held walk token and request binding; engine type with a typed EQL adapter | + +`JsonContainmentQuery` can still be the physical operand for a path equality +operation where the agreed SQL recipe uses containment; operation and wire +operand need not have the same name. `JsonWalkRequest` describes the actual +role better than "key": it carries sensitive capability/request state, +not the data-encryption key. Exact operation behavior remains governed by +the separate JSON contract review. + +Rename the internal target index `Json` to `JsonTreeIndex`. Keep the internal +engine `JsonDocument` out of the ordinary facade, with a role-qualified name +such as `EncryptedJsonTree` if adapter authors require public access. +`EncryptedJson` always denotes the EQL-compatible stored form, not any +arbitrary JSON serialization of encrypted nodes. An application's +`EncryptedUser` remains its own declared record layout. + +Dynamic target/operation identifiers are explicitly versioned contract +identifiers, not Rust type names serialized into JSON. Select their exact +spellings in the interoperability specification and use the same glossary +in Go. Rust renames alone must not accidentally redefine stored SQL formats +or dynamic discriminants. + +## Implementation order and acceptance + +1. Approve this glossary and ownership map; update all public example imports + and type names together without implying the examples already compile. +2. Specify the stable format profiles and conformance fixtures, coordinating + #519. Establish the small format crate and move the encoding wrappers and + frame foundation; preserve distinct transport/storage semantics. +3. Move the value model, traversal and dependent normalization/index adapters + into Stack, separating primitive inputs so the dependency graph stays + acyclic. Move relevant tests with their implementations. +4. Introduce neutral provider capabilities and the ZeroKMS adapter, rename + the protocol crate, then retire Vitamin C's async application execution + surface. Keep mixed-keyset/auth-policy batching and request isolation. +5. Apply the agreed method/derive/JSON names across Rust, EQL, dynamic catalog, + guest ABI adapters, Go generation and examples. Preserve the comprehensive + first-release JSON inventory rather than treating packaging as a scope cut. + +Acceptance includes an acyclic Cargo graph; an EQL wire-only build without +the engine; engine builds without ZeroKMS and without HTTP; native/guest +builds; compiled public examples; the moved behavior tests; and shared +Rust/Go/SQL format/operation fixtures. Full timing and memory-cleanup +improvements remain follow-up work under the accepted format-first priority; +record actual guarantees honestly. No implementation changes, tests of the +proposed API, code moves, or upstream removals are claimed by this document. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md new file mode 100644 index 000000000..eeaf0afef --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -0,0 +1,342 @@ +# Key selection, lock context and provider capabilities + +Date: 2026-10-25. Assessment accompanying the +[API implementation plan](2026-10-24-stack-encrypt-api-plan.md). + +Status: recommendations for review, not accepted public signatures or an +implementation. The user requested plaintext-derived ZeroKMS keyset +selection, review of Vitamin C `AsyncCipher`, and optional lock-context +support without requiring future AWS KMS/Vault backends to emulate it. +The JSON operation contract remains under user review independently. + +## Distinct roles + +| Role | Selects/binds | Example | +| --- | --- | --- | +| Key selection | Provider key domain used to encrypt and derive index terms | ZeroKMS keyset ID or name selected from `tenant_id` | +| Ciphertext context | Application binding verified when opening ciphertext | Column + tenant scope + row ID | +| Search scope | Domain in which index/query terms compare | Column + tenant ID | +| Provider policy | Conditions the provider applies to key generation/retrieval | ZeroKMS lock context selecting authenticated JWT `sub` | + +These are separate logical inputs, even where the protocol binds several +into its tag/AAD. A `ScopeKey` does not select a ZeroKMS keyset by itself; +a keyset selector does not insert its source field into AAD; lock context +is not an alias for either ciphertext context or search scope. Sharing a +source value requires explicit declaration, not inference from a name. + +## Current implementation evidence + +- [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts + `IdentifiedBy` by UUID or name, resolves the keyset and its index key, + and returns a bound `KeysetCipher`. The client has a bounded keyset cache + and name-resolution freshness policy. `.default_keyset()` is already + bound at initialization. Current initialization requires both + `DataKeySource` and `IndexKeySource`. +- A sealed leaf carries the resolved keyset UUID. Client-scoped decrypt + groups retrieval by the UUIDs in the leaves; a bound keyset can constrain + which keyset is permitted. Names are lookup conveniences, not persistent + key identities. See [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). +- [Pending dispatch](../../packages/stack-encrypt/src/target/pending.rs) + currently takes one selected keyset for generates and groups retrieves by + their stored keyset IDs. It constructs generate/retrieve payloads without + lock context and passes no unverified request context. +- [Key-source traits](../../packages/stack-kms/src/key_source.rs) and + [payloads](../../packages/stack-kms/src/payload.rs) already model ZeroKMS + lock-context lists, optional decryption policies and unverified context. + They are ZeroKMS-specific abstractions used for provider injection/testing, + not yet a neutral contract for arbitrary KMS backends. Returned key types + also reflect the existing ZeroKMS/recipher integration. +- [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) + accepts `.withLockContext({ identityClaim: ["sub"] })`. Its documented + behavior is server resolution of claim values from the authenticating + principal. Encrypt/decrypt operations pass that selector to protect-ffi; + audit metadata travels separately as `unverifiedContext`. +- The pinned `zerokms-protocol` 0.12.31 `Context` enum includes `Tag`, + `Value` and `IdentityClaim`. Its `RetrievedKey` currently exposes key + material only; `stack-kms::DataKey` contains IV and key, not the original + authentication tag returned by the proposed refinement flow. Supporting + that flow remains a protocol/client response integration requirement, + consistent with the user's earlier explanation of intended behavior. + +## 1. Plaintext-derived keyset selection + +Add a key-selection declaration to the existing target definition, parallel +to context/scope source selectors. A useful minimal portable form is an +explicit ID or name selector over a plaintext field: + +```rust +// Proposed attribute fragment. tenant_id contains the actual keyset name. +#[stash(keyset(name(field = "tenant_id")))] +``` + +```rust +// Alternative: the selected field contains a keyset UUID, not an arbitrary +// application UUID assumed to be a keyset. +#[stash(keyset(id(field = "keyset_id")))] +``` + +Neither selector creates a keyset. Missing, unknown, unauthorized, disabled +or incorrectly typed keysets are errors; never fall back to the default. +A business tenant ID need not be a keyset name/UUID. Where they differ, +configure an explicit tenant-to-key mapping/resolver. Start with direct +selectors and a shared registered resolver contract; do not encode arbitrary +Rust callbacks in dynamic definitions. Pure mappings can return a selector; +network lookup belongs to the execution/provider layer, not the derive. + +The existing manually selected form already fits the agreed API: + +```rust +let tenant = cipher.keyset(IdentifiedBy::Name(user.tenant_id.clone().into())).await?; +let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; +``` + +The proposed derived route removes that repetition: + +```rust +// EncryptedUser's definition declares how to select the keyset. +let encrypted: EncryptedUser = user.encrypt_into(&cipher).await?; +let encrypted: Vec = users.encrypt_into(&cipher).await?; +``` + +`&cipher` is the client/provider executor which can resolve keysets; it is +not an unscoped request that silently encrypts under the default. Extend the +conversion executor capability to accept either a resolved keyset or a +client plus a declared selection recipe. The target's representation and +encryption operation description remain unchanged. Direct +`tenant_keyset.encrypt(value)` retains its accepted meaning. + +Execution order: + +1. Acquire guarded source ownership at request construction as already + required. Validate target/options, input shape, selectors, role/capability + requirements and every key-independent value check before provider I/O. +2. Evaluate the key-selection recipe from the guarded plaintext. Resolve + selectors through the configured provider; pin each resolved identity + for the duration of the request, including index derivation. +3. Partition a batch by resolved key identity, provider/authentication + session and compatible request-level options. Preserve original result + ordering; many tenants do not imply one remote call for the whole batch. +4. Use the selected keyset's index material for its searchable fields and + mint data keys in the selected keyset. Persist the resolved identity, + not a mutable name, in the ciphertext's provider metadata. + +Define conflict behavior: a preselected keyset cannot silently be overridden +by a target's route. If both are provided, validate that the resolved route +matches the bound keyset, or reject the combination explicitly. No implicit +precedence or fallback. Errors requiring name resolution can only occur +after that resolution; the accepted preflight rule covers key-independent +checks, not provider facts unavailable locally. + +For queries there is no plaintext user to inspect. The same target policy +must expose a caller-supplied tenant routing input or use an explicitly +selected keyset. A typed input may deliberately supply both key routing and +search scope; declaration determines both uses. Do not derive a keyset from +the email search value. Searching across multiple keysets is explicit +fan-out with a term per keyset, not one universally comparable query term. + +Decrypt uses the stable key identity stored with the ciphertext, as today. +Do not re-resolve a historical keyset name and substitute its current target. +An expected tenant/keyset restriction can constrain retrieval, with any +historical key mapping supplied explicitly. Merely trusting a tenant field +returned alongside a ciphertext does not authenticate application tenancy; +row binding or trusted expected context supplies that separate check. +Likewise, choosing a keyset from caller-controlled plaintext is routing, +not permission to use every tenant keyset accessible to a service account. + +## 2. Vitamin C AsyncCipher + +**Packaging follow-up:** the user favors removal. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md#retire-asynccipher) +now recommends retiring this trait and its application helpers as execution +moves into Stack, rather than adding the optional adapter considered below. +Vitamin C still has helper/test consumers to migrate; no removal has been +implemented. + +The absence of `AsyncCipher` is intentional in the current design, not an +unexplained omission. [cipher.rs](../../packages/stack-encrypt/src/cipher.rs) +documents the reason, and implements the synchronous Vitamin C `Cipher` +through `MintedKeys` (seal) and `BoundKeys` (open). + +Vitamin C 0.8.0 `AsyncCipher` takes leaf plaintext and derived AAD, with +`seal_many`/`open_many` operating on the leaves of one document. Stack's +executor retains the original provider descriptor and combines requests +across fields/documents/records before calling the leaf cipher. Leaf AAD +alone is not a reversible representation of those execution inputs. +Future routed keys and lock policies add more execution metadata rather +than removing the mismatch. The trait also requires `Sync` and `Send` +futures; Stack's provider traits use `MaybeSend` for native/WASM support. + +Recommendation: retain Stack's batch/provider orchestration and Vitamin C's +leaf cryptography. Do not implement `AsyncCipher` on `StackCipher` merely to +claim trait symmetry, encode provider policy secretly into AAD, or lose +cross-record batching. An explicitly configured adapter could implement the +trait for a restricted use case with provider descriptor/key route/policy +already supplied, but it is not required for the public redesign and must +advertise its limitations. A general refactor would need a richer request +contract and evidence that batching/metadata/WASM behavior remain intact. + +There is a smaller cleanup opportunity: each synchronous adapter currently +implements an unused half of `Cipher` by returning an error. Consider +separate seal/open capabilities or an internal adapter improvement during +the final dependency/packaging pass. Do not conflate that with requiring the +public Stack client to implement the async trait. + +## 3. Lock context as an optional provider policy + +Use a dedicated, typed ZeroKMS policy input, not the ordinary +`encrypt_with_context` argument and not a `SearchScope`. Proposed spelling: + +```rust +use stack_encrypt::providers::zerokms::{LockContext, LockContextExt}; + +let lock = LockContext::identity_claim("sub"); +let encrypted = user.encrypt_into(&tenant_keyset) + .with_lock_context(&lock) + .await?; +let user = User::decrypt_from(encrypted, &tenant_keyset) + .with_lock_context(&lock) + .await?; +``` + +This is illustrative module/method syntax pending the naming pass. The +operation builder retains owned plaintext guards while policy is attached. +The extension is present for a provider with the required capability; a +runtime-erased provider checks the equivalent capability before key I/O. +Ordinary encryption without this optional policy works on providers that +do not support it. A request/definition requiring it must fail on those +providers, never discard it or substitute literal AAD. + +The input selects a claim **name**. ZeroKMS verifies the authenticating +principal and resolves its claim value; do not decode an unverified JWT in +the application and turn its `sub` into a caller assertion. Authentication +belongs to the configured provider/session. Use explicit, immutable +request/session binding when identities differ concurrently; avoid mutating +a global client's identity between operations. + +Support explicit literal tag/value items through the same provider policy, +while distinguishing those caller assertions from principal-derived claims. +Missing claims or provider policy failures are errors with no retry without +the lock. Forward policy on generate and retrieve. A required policy can +live in the target definition so callers cannot accidentally omit it; an +operation may supply an undeclared optional policy, but cannot replace or +weaken a definition-required one. Define composition/conflict rules rather +than last-call-wins behavior. Policy selectors may need to be available on +reads; define their storage or required caller input without serializing +JWTs, credentials or claim values resolved by the provider. + +Keep audit metadata (`UnverifiedContext`) separate: it is not trusted +identity, ciphertext context or a policy assertion. Preserve distinctions +between lock context and the protocol's separate decryption-policy feature; +do not silently reinterpret one as the other. + +Index/query derivation may use an already-loaded deterministic index key +without generating/retrieving a data key. Therefore lock context on those +data-key endpoints does not by itself restrict who can derive search terms +or view query results. If search authorization is required, define that +capability separately; do not infer that the TypeScript query method's +acceptance of `withLockContext` proves the new index path is protected. +Do not implicitly make search scope principal-specific to emulate a lock. + +Batching must preserve per-item lock inputs and separate incompatible +principal/provider sessions. If a provider supports per-payload policy, +retain it in each payload; if an option is request-wide, partition by it. +Any future data-key cache must respect policy/principal checking semantics; +a cache hit must not silently replace a required provider authorization or +audit event. This is a design requirement, not a claim of an existing cache +vulnerability. Do not broaden the current keyset/index-key cache into an +unqualified data-key authorization cache. + +## Provider-neutral base and progressive capabilities + +The public value/target model needs extensions, not replacement. The larger +refactor is the provider boundary, which currently imports ZeroKMS payloads, +UUID keyset IDs, policies, errors and returned key types directly. + +Recommended capability split (working names): + +| Capability | Required for | +| --- | --- | +| Data-key generation/recovery and protected key material | Ordinary opaque and structural encryption/decryption | +| Index-key support | Searchable targets and query derivation | +| Keyset resolution by ID/name | ZeroKMS-style keyset selectors; not every provider | +| Lock-context support, including claim selectors | ZeroKMS policy extension; not every provider | + +Define a resolved provider key reference and versioned opaque recovery +metadata. Do not force every provider to supply a ZeroKMS UUID, IV/tag or +recipher response. A caller selects/configures a provider; ciphertext +metadata identifies a registered provider/key format, never arbitrary +network endpoints or credentials to execute. Keep ZeroKMS's key tag simple +to extract within its documented envelope profile for Rust/SQL consumers. +Other providers' metadata need not invent such a tag. Stable blob payload +encoding is independent of these provider-specific envelope profiles. + +Future AWS KMS/Vault integrations implement the minimal capabilities they +offer. They need not implement lock context, named keysets or searchable +index-key management merely to encrypt an opaque value. The application +can configure its own tenant-to-provider-key mapping without representing +it as a native ZeroKMS keyset. A searchable target requires an explicitly +supported index-key strategy; the engine must not assume every data-key +backend supplies deterministic search keys automatically. + +Refined-tag support also needs distinct stored/request tag and returned +original authentication-tag input. Preserve the user's intended ZeroKMS +flow when designing neutral responses; do not require that every provider +implements refinable tags or that a generic KMS returns that concept. + +## Dynamic interoperability + +Provider capabilities join target capabilities in discovery and definition +validation. Example provider-specific extension (proposed logical schema): + +```json +{ + "provider": "zerokms", + "key_selection": { + "keyset": { "by": "name", "encrypt": { "field": "tenant_id" } } + }, + "provider_policy": { + "zerokms": { + "lock_context": [{ "identity_claim": "sub" }] + } + } +} +``` + +The field's declared type must admit the chosen selector; missing fields +are definition/input errors before provider resolution. Query routing +declares an equivalent caller input because it has no source record. +Providers and extensions are registered, versioned capabilities. Unsupported +required key selection/policy is an error during loading or binding to an +execution provider, before key I/O. Do not accept arbitrary extension maps +and ignore unfamiliar entries. Auth credentials stay in provider setup, not +in this declaration or persisted policy metadata. + +Go generation uses named selectors/policies and the same validated dynamic +definition; provider-specific methods/capabilities are explicit. The +interoperability specification must include mixed-keyset batches, policy +inputs, capability refusals, key identity persistence and the distinctions +between authenticated claims and caller-supplied audit metadata. + +## Implementation and acceptance work + +1. Specify neutral key references/recovery metadata and optional capability + traits without altering the accepted encrypt/decrypt conversion roles. +2. Add definition selectors and client-scoped routed execution. Cover ID + versus name, explicit mappings, conflict errors and no default fallback. +3. Extend request planning/grouping to carry resolved route and provider + policy. Preserve per-record ordering and guard lifetimes across async + resolution/cancellation. +4. Wire ZeroKMS lock context through both generate/retrieve, including + literal and principal-claim forms, and test required-policy enforcement. +5. Verify mixed-tenant batches, matching write/query keys, historical ID + decrypt, wrong expected keysets, missing claims, unsupported-provider + failures before I/O, and concurrent principal isolation. A minimal fake + provider without keysets/locks/index keys must still encrypt opaque values. +6. Resolve refined-tag response support with the ZeroKMS client/protocol. + Add integration fixtures without assuming the pinned API already returns + the original tag. +7. Defer the `AsyncCipher` adapter/trait cleanup decision to the packaging + pass; preserve the documented batching and WASM constraints meanwhile. + +No implementation changes, protocol guarantees for future providers, or +completed security audit are claimed by this assessment. From c5efc48837d3cb48bc7e168f780fd616a4f4845e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 22:41:49 +1100 Subject: [PATCH 03/42] docs: prioritize Go contract and stable storage --- .../2026-10-24-stack-encrypt-api-plan.md | 101 ++++++++++-------- .../2026-10-24-stack-encrypt-api-review.md | 23 ++++ ...026-10-25-stack-encrypt-json-operations.md | 10 ++ .../2026-10-25-stack-encrypt-packaging.md | 36 ++++--- 4 files changed, 113 insertions(+), 57 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 600b52091..2109fe49f 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -10,6 +10,15 @@ criteria. User requirements are distinguished from recommendations that have not yet been accepted. Examples below are proposed APIs, not working code for the current release. +**Customer delivery priority, 2026-10-25:** the JSON-tree stack +headed by #1363 has merged; next deliver Go JSON and configurable Bloom search, including +the short-input requirement under investigation. Stabilize the Go contract +and persisted encoding/search semantics before customer release. Adopt the +Vitamin C codec in its existing repository now; defer crate moves and the +standalone Rust facade. The customer-first sequence below supersedes +earlier implementation ordering where they conflict. The short-gram design is +a release dependency, not a feature silently deferred to a later version. + **Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets are accepted. The comprehensive JSON first-release contract and Rust/dynamic examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), @@ -43,6 +52,9 @@ aliases, deprecation period, or data migration is required. Choose the coherent new design and update its Rust, Go, guest, and EQL consumers together. Engine output must still agree with the SQL that consumes it; that is current interoperability, not a legacy-format requirement. +This exemption ends for the customer profiles when released: subsequent +Rust API/package refactors must preserve the Go interface and stored +data/search compatibility. ## Requirements established in the review @@ -59,8 +71,10 @@ that is current interoperability, not a legacy-format requirement. prototype-compatibility reason to add a raw-byte exception. - Rust traits and derives are the primary static interface. Plans support those declarations and dynamic/FFI lowering. -- Design and deliver static and dynamic support together. Go support is an - acceptance criterion, not a later follow-up. +- Design static and dynamic support against the same engine/definition + contract. Deliver the customer Go interface first; polished public Rust + traits/derives may follow. Engine-level Rust interoperability tests remain + required, so this does not introduce a second Go encryption implementation. - Provide decrypt entry points with and without explicit context. The no-context variant must not invent a default required application value. - `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation @@ -881,38 +895,32 @@ this design discussion. ## Configured-target implementation sequence and completion criteria -1. **Normalize the general contract.** Specify typed options, Rust factory - signatures, validated dynamic deserialization, defaults, context - inputs, capability discovery, errors, and the target - object grammar. Create the versioned interoperability specification, - schemas and initial conformance fixtures described below alongside this - work. Build one target-owned validator and lowering path. -2. **Deliver JSON as a complete first implementation.** Add configured EQL - JSON targets, direct typed output/input conversion, catalog/guest - registration, and JSON transport. Keep engine and database encodings aligned. -3. **Expose both declarative front ends.** Add the generic Rust derive - factory hook and Go generator/policy support, including field targets and - standalone Rust wrappers. Neither front end reimplements encryption. -4. **Complete queries and reads in both languages.** Carry the configured - target into query construction, expose extraction results, and standardize - Rust's paired decrypt convenience methods. Settle the role-based JSON - names as part of this public API work. -5. **Verify the complete slice before calling it supported.** Rust-written - EQL JSON must open and query from Go, and vice versa, against PostgreSQL. - Test defaults, ordered paths, both root settings, arrays/scalar roots/null, - numeric boundaries, missing paths, context mismatch, and invalid options. - Assert logical batching and verify fixtures for the chosen format. Compare - semantics rather than ciphertext equality, since encryption is randomized. - Run the shared interoperability conformance suite and gate schema/runtime - drift in CI. Each implemented contract change updates its specification - and fixtures in the same change. - -These are implementation steps, not permission to ship a Rust-only phase. -The Go examples and cross-language acceptance tests are part of the feature. -Do not add the plaintext-configuration wrapper in this first change: the -dynamic target declaration already handles runtime options without adding -another policy location. Reconsider that convenience after both front ends -use the shared contract. +Customer commitments supersede the earlier Rust/Go front-end lockstep sequence: + +1. **JSON engine baseline is landed:** #1345, #1358, #1361 and #1363 have + merged. Build Go support against that baseline; revise the older #1353 plan. +2. **Settle Go/dynamic contracts and storage profiles:** target options, + context/scope, errors, query composition and stable Vitamin C encodings. + Vitamin C #520 and #526 implement the Rust/Go storage profiles and await + colleague review and release; adoption remains a Stack integration task. Start short-gram research immediately; + its storage/query implications must resolve before that feature freezes. +3. **Deliver Go JSON and configurable match search:** use the landed engine + and SQL contract, implement analyzer/filter definitions and supported + two-character behavior, and preserve the comprehensive JSON operation + inventory. Carry through the numeric round-trip fix from #1366 (#1364). +4. **Freeze and release the customer surface:** Go generation/runtime tests, + Rust-engine/Go cross-opening, PostgreSQL queries, schema/semantic + conformance, and stable storage/search vectors are release gates. + The final Rust derive facade is not. +5. **Complete Rust ergonomics and packaging:** implement public traits, + derives and examples and move crates behind the frozen Go and data + contracts. Internal plans may change; customer data must remain usable. + +Keep shared validation and one execution engine throughout. Final static +Rust examples become executable acceptance tests in step 5; they need not +delay step 4. Each delivered contract change updates its specification and +fixtures in the same change. Do not add a second policy location through +an ad hoc plaintext-configuration wrapper. ### Required deliverable: dynamic interoperability specification @@ -990,7 +998,12 @@ deliverable; recommendations do not silently settle the open decisions. | Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | | Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | -### A. Settle the public method and trait structure first +### A. Consolidate the public Rust method and trait structure + +This is the Rust-facade workstream. Under the latest customer ordering, do +only the engine/contract changes needed by Go before its release; defer +standalone Rust ergonomics and trait relocation. The sections below organize +the full review scope, not the customer critical path. Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. Use the accepted destination-directed decrypt pair. Write @@ -1462,6 +1475,8 @@ Completion: all public examples are executable tests, static/dynamic cross-opening and querying pass against the selected storage format, batching checks pass, and the guest builds remain supported. No new API is described as implemented until its Rust and Go acceptance criteria are met. +These are full redesign completion criteria. The earlier Go release uses +the facade-independent Go, engine and storage gates listed above. ## Recommended order for the remaining design decisions @@ -1492,15 +1507,14 @@ specification and implementation obligations: | --- | --- | | JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | | Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | -| Stable blob profile | Specify supported Serde shapes, lossless options/enums/maps, authenticated codec versions and compatibility fixtures under Vitamin C #519. This is the immediate priority; format-preserving cleanup/timing improvements are follow-up work, with limitations documented honestly. | +| Customer storage profiles | Stabilize the Vitamin C encodings and semantics used by Go/JSON/search before release, coordinating #519 upstream. Full Rust blob conveniences can follow; unsupported shapes must remain explicitly unsupported. Crate relocation is deferred. | +| Configurable match and short input | Whole-word semantics are confirmed. Define the analyzer/token-filter catalog and named indexes/Boolean composition; choose the short-word mechanism only after leakage/performance assessment. Select the storage/query scheme before release. | | Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | -Next review the JSON operation document and its static/dynamic examples, -then turn the required inventory into implementation phases and formal -interoperability fixtures. Stable storage-format work is tracked upstream. -Finish remaining query integration details without reopening ORE equality. -The user has now requested the naming/package pass while JSON review -continues; that proposal is ready for review in the packaging companion. +Next reconcile the operation document and #1353 with the merged JSON +stack. Settle the Go contract and adopt the upcoming Vitamin C storage +release. Use the customer-facing conformance gates above for release. Packaging remains a proposed destination after Go +delivery; finish query integration without reopening ORE equality. Specification/implementation work still includes exact context/scope grouping and encoding, dynamic grammar and version identifiers, error codes, binary @@ -1525,4 +1539,5 @@ reason to reopen the question of whether writes and queries share a policy. Passthrough implementation, transcoder adapters, collection composition, trait relocation, documentation updates, and acceptance tests remain work items. They are not reasons to characterize the agreed API direction as -undecided. No backward-compatibility phase is needed for the prototype. +undecided. No backward-compatibility phase is needed for the prototype; +the customer release establishes a new Go and persisted-data commitment. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index f60a088e5..f72bf39e4 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1121,3 +1121,26 @@ adapters also depend on the existing value model and need separation from their primitives. Moving the frame implementation refines the earlier codec ownership recommendation; coordinate stable storage work under Vitamin C #519. Names and package moves remain proposals, not implementation changes. + +## 22. Customer-driven sequencing: Go JSON and match search first + +The customer needs Go JSON support and robust Bloom-filter handling with +tokenizers/token filters, including two-character input. Standalone Rust +release quality is secondary. Claude's JSON-tree PR stack headed by #1363 has now merged. Revise the +older Go plan in #1353 against it and the agreed API direction. Vitamin C +#520 and #526 implement the storage profiles; colleague review and release +are pending. Stack still needs to adopt that release. + +Freeze the public Go contract and stored binary/search semantics before the +customer release. Internal plans may change behind that interface. Adopt and +stabilize the Vitamin C codec in its current repository now; defer physical +crate moves, AsyncCipher retirement and final Rust derives. The earlier +prototype compatibility exemption does not permit breaking customer data +after this release. + +The supplied `Short-ngrams.md` explores short whole-word filters and variant +queries, which must be distinguished from arbitrary two-character substring +search. The current tokenizer permits length 2, but the current one-match +output and one/range-query-term shapes need expansion for named filters and +Boolean alternatives. Resolve the scheme's leakage, SQL representation and +performance before release; do not silently defer the customer requirement. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 2ede5410a..7ff0d6fd3 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -13,6 +13,16 @@ they are not a post-release phase. Internal implementation can still be split into PRs. The feature is complete only when every required row below has Rust, dynamic/Go, and applicable PostgreSQL coverage. +**Customer sequencing update:** the PR stack headed by +[#1363](https://github.com/cipherstash/stack/pull/1363) has merged. Deliver Go +against that baseline. Its token lists, self tags, wildcard probes and result semantics +supersede the older gated-walk baseline used in parts of this document. +Reconcile those recipes before implementation and format freeze; the +operation inventory remains required. Engine-level Rust interoperability +coverage is a release gate; the final public Rust facade may follow +customer delivery. Retain regression coverage for the numeric round-trip +fix now merged in #1366 (closing #1364). + This is a target API contract, not a claim that these public signatures already exist. Rust and JSON examples below propose the redesigned surface. Final names/package paths are deliberately left to the last naming pass. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 7b4e749d3..9c7d50e7c 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -7,6 +7,12 @@ renames. Existing companion examples retain their working names until this proposal is accepted. The detailed JSON operation contract remains under separate user review. +**Sequencing update:** customer Go delivery supersedes the original +move-first sequence. The JSON stack headed by #1363 has merged. Vitamin C +#520 and #526 address storage and await review/release. Adopt those encodings +and deliver Go JSON/match search first. The package map below is the intended destination; neither a new +format crate nor public Rust facade completion is a customer release gate. + Source baseline: the current documentation branch and inspected local Vitamin C 0.8.0 sources. Stack's [cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) @@ -229,20 +235,22 @@ or dynamic discriminants. ## Implementation order and acceptance -1. Approve this glossary and ownership map; update all public example imports - and type names together without implying the examples already compile. -2. Specify the stable format profiles and conformance fixtures, coordinating - #519. Establish the small format crate and move the encoding wrappers and - frame foundation; preserve distinct transport/storage semantics. -3. Move the value model, traversal and dependent normalization/index adapters - into Stack, separating primitive inputs so the dependency graph stays - acyclic. Move relevant tests with their implementations. -4. Introduce neutral provider capabilities and the ZeroKMS adapter, rename - the protocol crate, then retire Vitamin C's async application execution - surface. Keep mixed-keyset/auth-policy batching and request isolation. -5. Apply the agreed method/derive/JSON names across Rust, EQL, dynamic catalog, - guest ABI adapters, Go generation and examples. Preserve the comprehensive - first-release JSON inventory rather than treating packaging as a scope cut. +1. Use the merged JSON stack and specify Go-visible names/contracts and stable + profiles needed by the customer. Keep codec work in Vitamin C, coordinating + #519. Preserve distinct transport/storage semantics and freeze search + semantics as well as byte layouts. +2. Deliver and validate Go JSON, configurable match search and the selected + short-input solution under the customer-first sequence. Record frozen Go/data + fixtures; leave internal Rust plan ergonomics free to change. +3. After delivery, establish the shared format crate and move the value model, + traversal and dependent normalization/index adapters into Stack. Preserve + frozen encodings and an acyclic graph; move tests with their code. +4. Complete neutral provider extraction and protocol-package renaming, then + retire Vitamin C's async application execution surface. Required Go-facing + policy and routing contracts must already have been handled before release. +5. Complete public Rust trait/derive names, imports and executable examples. + These changes must not rename the shipped Go API, reinterpret stored + profiles or require customer re-encryption/reindexing. Acceptance includes an acyclic Cargo graph; an EQL wire-only build without the engine; engine builds without ZeroKMS and without HTTP; native/guest From cddddda12c2a3d4f2dc2fbeab68cf5e28365e8d1 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 27 Oct 2026 13:03:54 +1100 Subject: [PATCH 04/42] docs: reconcile API plan with merged Go work --- .../2026-10-24-stack-encrypt-api-plan.md | 260 ++++--- .../2026-10-24-stack-encrypt-api-review.md | 30 + ...026-10-25-stack-encrypt-json-operations.md | 715 +++++------------- .../2026-10-25-stack-encrypt-packaging.md | 37 +- ...-25-stack-encrypt-provider-capabilities.md | 9 + 5 files changed, 395 insertions(+), 656 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 2109fe49f..00969faf7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -10,14 +10,28 @@ criteria. User requirements are distinguished from recommendations that have not yet been accepted. Examples below are proposed APIs, not working code for the current release. -**Customer delivery priority, 2026-10-25:** the JSON-tree stack -headed by #1363 has merged; next deliver Go JSON and configurable Bloom search, including -the short-input requirement under investigation. Stabilize the Go contract -and persisted encoding/search semantics before customer release. Adopt the -Vitamin C codec in its existing repository now; defer crate moves and the -standalone Rust facade. The customer-first sequence below supersedes -earlier implementation ordering where they conflict. The short-gram design is -a release dependency, not a feature silently deferred to a later version. +**Implementation reconciliation, 2026-10-27:** reviewed merged `next` at +`ffeed7adb3f7d8739cda53261046147852bffc82`. Go JSON catalog, configured +targets, queries, request-bound reads, typed documents and executable docs +have landed (#1372, #1373, #1378, #1402). Opaque Go storage adoption has +landed (#1369/#1370). The active customer work is now contract conformance, +remaining read/configuration gaps and release readiness, not implementing +those foundations again. See the refreshed +[Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and [JSON inventory](2026-10-25-stack-encrypt-json-operations.md). + +#1379 added cascade authentication and randomized storage/projection order. +Wildcard `->` now selects an unspecified element; old first-in-document-order +recipes are invalid. Freeze the post-cascade format and compare query +semantics rather than randomized byte ordering. #1397 implements short +whole-word terms in the shared Bloom filter; configurable native Go match +options and broader tokenizer/filter work remain. This is implementation +status, not acceptance of a stronger privacy claim for those terms. + +Go's Vitamin C storage modules are temporarily pinned to `06568ba93bd9`. +The final-release guard requires replacing pseudo-versions with released +module tags; prereleases may carry them. Rust remains on crates.io 0.8.0. +Keep crate moves and the final Rust facade behind customer delivery. **Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets are accepted. The comprehensive JSON first-release contract and Rust/dynamic @@ -52,7 +66,9 @@ aliases, deprecation period, or data migration is required. Choose the coherent new design and update its Rust, Go, guest, and EQL consumers together. Engine output must still agree with the SQL that consumes it; that is current interoperability, not a legacy-format requirement. -This exemption ends for the customer profiles when released: subsequent +The merged Go implementation nevertheless retains a legacy JSON opaque +reader; record it as existing behavior, not a new requirement to add +prototype migration machinery. This exemption ends for customer profiles when released: subsequent Rust API/package refactors must preserve the Go interface and stored data/search compatibility. @@ -202,15 +218,15 @@ particular, the full ordered-path policy is not stored in the JSON header. ## Candidate dynamic representation and Go requirements -Extend the existing string target with an object form, using the current -type name provisionally rather than deciding a registry rename here: +Implemented by #1372: string targets and configured object targets share +the native options validator. Use the landed `JsonSearch` identifier: ```json { "payroll": { "context": ["employees", "payroll"], "target": { - "name": "JsonTreeDocument", + "name": "JsonSearch", "options": { "ordered": ["$.salary"] } } } @@ -218,7 +234,7 @@ type name provisionally rather than deciding a registry rename here: ``` Keep `"target": "TextEq"` as shorthand for that target's default options. -The object form is new: current dynamic targets accept only a string. +The object form and JSON target options are now implemented. Reject unknown options, incompatible options, unsupported target names, and invalid paths during declaration validation. Do not silently fall back to defaults. Specify one consistent JSON wire format and context encoding @@ -236,13 +252,13 @@ serialize its resolved options, not its function identifier. This contract does not require all Rust-only targets to depend on Serde; data conversion is required for targets registered with the dynamic interface. -Current gaps: JSON tree is absent from EQL's catalog/target table, the -dynamic JSON index is refused with `JsonIndexPending`, target fields carry -only a name, and Go's generated target declarations carry no target options. -Supporting an index's option syntax in Go does not establish executable -JSON-tree support. Include JSON input transport for objects, arrays, scalar -roots, and null; preserve the engine's numeric semantics and reject values -the cross-language contract cannot faithfully represent. +The catalog/guest, JSON value kind, strict JSON text parser and generated +Go target options are now implemented. #1373 adds operation input shapes and +operand encodings, and #1378 adds typed Go document adapters. Remaining +work includes the formal versioned specification, generalized definition +lifecycle/caching, direct typed EQL conversion and context-role separation. +The current data-plan grammar is still a field map; the illustrative +versioned schema below is not already accepted by the runtime. Writes and queries must reuse the normalized configured target in Rust and Go. An ordering query on an undeclared path must fail before key I/O. @@ -408,48 +424,34 @@ and batching; this trait separation must not add a blanket `Clone` bound. The final signatures must show where borrowed and owned sources are supported rather than concealing the existing difference behind the rename. -## Parallel Go JSON work: PR #1353 - -Read on 2026-10-24: [PR #1353](https://github.com/cipherstash/stack/pull/1353) -is an open documentation-only plan, not an implementation. Its write-up is -[The json_tree capability in the Go SDK](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md). -It tracks [#1347](https://github.com/cipherstash/stack/issues/1347), with -catalog, dynamic execution, queries, Go SDK, and extraction work in -[#1348](https://github.com/cipherstash/stack/issues/1348), -[#1349](https://github.com/cipherstash/stack/issues/1349), -[#1350](https://github.com/cipherstash/stack/issues/1350), -[#1351](https://github.com/cipherstash/stack/issues/1351), and -[#1352](https://github.com/cipherstash/stack/issues/1352). - -Treat that work as an implementation dependency to reconcile, not a parallel -SDK design to recreate. Its engine-owned execution, catalog/generated Go -types, engine-validated options, and cross-language tests fit this plan. -Review the following differences before freezing interfaces: - -- Route its JSON-specific option plumbing through the general target's - typed options/deserialization boundary. Go tags may retain idiomatic - syntax, but lower to target options rather than a second JSON policy path. -- Adapt the proposed `Layout` integration to the configured-target contract; - no second executor or permanently competing public configuration model. -- Reconcile type/registry names and generated query accessors with the shared - glossary and configured query capabilities. Its `At.*` accessor proposal - is useful input, not an accepted Rust or Go naming decision here. -- Carry the agreed context requirements and paired method semantics into - the relevant language interfaces without requiring identical language - syntax. Distinguish JSON `null` as content from nullable SQL fields. -- Evaluate its JSON-text ABI recommendation for numeric fidelity and return - types. This is compatible with direct typed EQL output adaptation: input - transport text and an avoidable encrypted-output Serde round trip are - different boundaries. -- Release scope is now decided: extraction, opening entries and walks ship - in the comprehensive first version alongside writes/queries. The earlier - Go plan's step-5 deferral is superseded by the linked JSON operation - inventory; sequential internal PRs must not become a reduced release. -- Byte-for-byte parity tests need deterministic randomness as that plan - proposes; normal randomized encryption needs cross-opening/query tests. - -These are dependencies and recommendations recorded locally. No changes or -comments have been posted to the parallel PR or its issues. +## Go JSON implementation and PR #1353 + +The [Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +has been updated against merged `next`. Tracking issue #1347 and steps +#1348–#1352 are closed. #1372, #1373 and #1378 implement the catalog, +configured JSON targets, dynamic execution, operation-specific operands, +request-bound opening and generated typed Go frontend; #1402 supplies +current docs and the executable database example. + +The shipped target identifier is `JsonSearch`; Go uses `eql.JSONSearch`, +`JSONSearchQuery`, `JSONSearchLeafQuery` and `JSONEntry`. Do not impose the +proposed `EncryptedJson` glossary as a pre-release rewrite. Preserve the +role distinctions and freeze actual customer names before later Rust renames. +Prepared Go reads retain their cipher and use `Open(ctx, rows)`. The engine +rederives trusted selectors/read keys from the original request. No new +persistent guest-handle protocol is needed to implement that verification. + +Residual requirements must be explicit: SQL children/counts exist but a +parent-bound plaintext child opener does not; selector opening accepts at +most one entry per document group. Use subtree reads for all-match plaintext +retrieval. General scope/row-binding inputs remain unsupported for EQL +targets. The existing column/keyset profile can ship without claiming them. + +The JSON contract now uses randomized projection order and cascade-bound +ciphertexts. Semantic parity and cross-opening are required; raw term byte +equality is meaningful only with controlled test randomness. Go typed JSON +adapters follow `encoding/json`, while raw JSON goes through the strict +engine parser. Do not claim arbitrary Go encoders preserve every input byte. ## Prior query design: baseline and reconciliation @@ -834,44 +836,34 @@ Track this action in for this review. It explicitly distinguishes storage stability from the transport-only rationale in Vitamin C #332. -**Recommendation, 2026-10-25:** reuse Vitamin C's frame writer/parser as the -foundation for the library-owned blob codec, with a specified blob Serde -mapping. Do not adopt the existing transport Serde mapping unchanged. -The inspected dependency is `vitaminc-aead-frame` 0.8.0; its `serde` module -already provides `to_vec(&T)` and `from_slice::(&[u8])`. Stack currently -uses that module for dynamic control data. Its module documentation -explicitly scopes it to transport data rather than plaintext to seal. - -The implementation already guards the serializer output, temporary variant -buffers, and buffered map keys with `Zeroizing`. The underlying -`vitaminc-aead` frame writer grows into a fresh allocation and wipes the old -one. However, `to_vec` returns an ordinary `Vec` on success. Provide an -internal guarded-output entry point or sensitive sink for the blob path, -keeping the result guarded through sealing. Inspect remaining temporary -storage and custom Serde paths before asserting a complete cleanup contract. - -The current mapping is not lossless for every ordinary Serde type: -`serialize_some` delegates directly to its value, while `None` and unit both -encode as null. Thus `Some(())` reads back as `None`; nested optional values -have the same ambiguity. The blob mapping must preserve option presence -explicitly, using an unambiguous representation, and test these cases. -The current map representation also requires text keys. Specify support or -explicit rejection for non-string map keys, and document enum/buffered Serde -limitations rather than advertising all `Serialize` types as round-trippable. - -Frame parsing validates one complete frame before invoking `Deserialize`, -but that neither prevents type-level decode failures nor cleans up partial -application values. Resolve that cleanup boundary alongside format limits, -authenticated codec versioning, and cross-language fixtures. Reusing the -frame format is not a constant-time guarantee and does not make an opaque -blob a structurally encrypted JSON document. - -[Borsh](https://docs.rs/borsh/latest/borsh/) remains a fallback if the blob -mapping would require disproportionate changes. It introduces -`BorshSerialize` / `BorshDeserialize` instead of the Serde traits, and has -writer-based serialization. Changing serialization libraries does not by -itself establish wiping or a constant-time contract. Prefer evaluating the -existing frame machinery against these requirements first. +**Implemented upstream and in Go, 2026-10-27:** Vitamin C's dedicated +storage profile and Go reader/writer address the format work from #519 +(#520/#526, carried onto upstream main via #544). Stack #1369/#1370 adopt +the Go codec: opaque generated structs call `vcffi.MarshalStorage` and +`UnmarshalStorage`. The profile is versioned (`VCS` plus version), supports +explicit optional values and typed map keys, and is separate from transport. +The transport mapping's `Some(())` ambiguity and text-only map-key rules +must not be reported as defects of this new storage profile. + +Go generated shapes use `vc` tags and `GeneratedVersion2`. A reader rejects +unsupported future blob versions distinctly and retains the old JSON +fallback. Go storage mapping has documented qualifications: nil containers +become empty, supported nonfinite floats round-trip, and unsupported opaque +types are refused by generation. Go pointers and Rust `Option` do not always +write identical bytes; cross-reading and the normative corpus matter. + +Rust workspaces still use crates.io Vitamin C 0.8.0; the final proposed Rust +`OpaqueBlob` facade has not thereby been implemented. Adopt the dedicated +storage entry points when that facade lands rather than repurposing the +transport Serde codec. Go's temporary pseudo-version dependency must move +to released module tags for a final Go release. A new serializer evaluation +is not on the customer critical path. + +Stable bytes do not establish constant-time behavior or complete cleanup. +Retain the remaining guarded-output, temporary-buffer, partial decode and +caller-owned-value work, without changing stored format. JSON tree retains +its own node/envelope encoding; its post-cascade compatibility profile must +be frozen separately from opaque blobs. The same consuming contract applies to target-directed `encrypt_into`. For a structural target, declared passthrough fields intentionally @@ -895,32 +887,34 @@ this design discussion. ## Configured-target implementation sequence and completion criteria -Customer commitments supersede the earlier Rust/Go front-end lockstep sequence: - -1. **JSON engine baseline is landed:** #1345, #1358, #1361 and #1363 have - merged. Build Go support against that baseline; revise the older #1353 plan. -2. **Settle Go/dynamic contracts and storage profiles:** target options, - context/scope, errors, query composition and stable Vitamin C encodings. - Vitamin C #520 and #526 implement the Rust/Go storage profiles and await - colleague review and release; adoption remains a Stack integration task. Start short-gram research immediately; - its storage/query implications must resolve before that feature freezes. -3. **Deliver Go JSON and configurable match search:** use the landed engine - and SQL contract, implement analyzer/filter definitions and supported - two-character behavior, and preserve the comprehensive JSON operation - inventory. Carry through the numeric round-trip fix from #1366 (#1364). -4. **Freeze and release the customer surface:** Go generation/runtime tests, - Rust-engine/Go cross-opening, PostgreSQL queries, schema/semantic - conformance, and stable storage/search vectors are release gates. - The final Rust derive facade is not. -5. **Complete Rust ergonomics and packaging:** implement public traits, - derives and examples and move crates behind the frozen Go and data - contracts. Internal plans may change; customer data must remain usable. - -Keep shared validation and one execution engine throughout. Final static -Rust examples become executable acceptance tests in step 5; they need not -delay step 4. Each delivered contract change updates its specification and -fixtures in the same change. Do not add a second policy location through -an ad hoc plaintext-configuration wrapper. +Customer commitments supersede the earlier Rust/Go front-end lockstep sequence. +The foundational Go JSON slice and opaque storage adoption are now merged. + +1. **Specify the implemented contract.** Capture target options, operator + input/result encodings, grouped reads, errors and the column/keyset + profile in the formal interoperability spec and conformance fixtures. + Keep proposed schema envelopes distinct from the current field-map grammar. +2. **Close actual remaining customer gaps.** Parent-bound child opening, + complex ordered-path declarations if needed, configurable Go match + options (#1112) and the tokenizer/filter pipeline. #1397 already supplies + short whole-word matching through EQL; confirm its accepted leakage and + freeze its index semantics rather than planning it as unimplemented. +3. **Validate and freeze the release candidate.** Run the existing engine, + Go/guest and PostgreSQL suites against the selected bundle. Preserve + numeric/null behavior, cascade tamper checks, wildcard/fallback semantics, + known omission limitations and post-cascade stored vectors. Tests already + landed are not proof this review reran a release build. +4. **Release the dependencies and customer surface.** Replace Vitamin C's + temporary Go pseudo-version pins for a final release; the existing guard + allows prereleases. Opaque storage is already integrated; keep it separate + from the JSON-tree storage profile. Build production guests and matching SQL. +5. **Complete Rust ergonomics and packaging.** Implement public traits, + derives, definition lifecycle and crate moves behind the frozen Go/data + contract. Direct EQL conversion is a format-preserving optimization. + +Use one execution engine throughout. The final Rust facade and package +layout are not gates for the existing Go JSON profile. Do not reimplement +configured JSON options or request-bound reads that have already landed. ### Required deliverable: dynamic interoperability specification @@ -1505,16 +1499,16 @@ specification and implementation obligations: | Remaining question | Decision required | | --- | --- | -| JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | +| JSON contract completion | Main Go surface is implemented. Close parent-bound child opening and formal conformance gaps; use current cascade/random-order semantics. Do not reopen operator design or rename shipped Go APIs for cosmetic consistency. | | Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | -| Customer storage profiles | Stabilize the Vitamin C encodings and semantics used by Go/JSON/search before release, coordinating #519 upstream. Full Rust blob conveniences can follow; unsupported shapes must remain explicitly unsupported. Crate relocation is deferred. | -| Configurable match and short input | Whole-word semantics are confirmed. Define the analyzer/token-filter catalog and named indexes/Boolean composition; choose the short-word mechanism only after leakage/performance assessment. Select the storage/query scheme before release. | +| Customer storage profiles | Go opaque storage is integrated. Replace temporary Go dependency pins for final release and freeze post-cascade JSON/search profiles. Rust blob conveniences and crate relocation may follow. | +| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenizers/filters remain, including #1112. Reconcile the earlier privacy concern explicitly; merging support is not proof of unchanged leakage. | | Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | -Next reconcile the operation document and #1353 with the merged JSON -stack. Settle the Go contract and adopt the upcoming Vitamin C storage -release. Use the customer-facing conformance gates above for release. Packaging remains a proposed destination after Go -delivery; finish query integration without reopening ORE equality. +Next complete the supported-contract specification and remaining customer +configuration/read gaps, then validate the candidate and resolve final-release +pins. The original JSON vertical slice is already implemented. Packaging +remains deferred; keep initial ORE equality policy unchanged. Specification/implementation work still includes exact context/scope grouping and encoding, dynamic grammar and version identifiers, error codes, binary diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index f72bf39e4..4fa7e186d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1144,3 +1144,33 @@ search. The current tokenizer permits length 2, but the current one-match output and one/range-query-term shapes need expansion for named filters and Boolean alternatives. Resolve the scheme's leakage, SQL representation and performance before release; do not silently defer the customer requirement. + +### Merged implementation reconciliation, 2026-10-27 + +Reviewed `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` against the +previous `ebdb5de0a` baseline. #1372/#1373/#1378/#1402 deliver the Go JSON +catalog, configured targets, operators, request-bound opening, typed document +adapters and executable documentation. #1347–#1352 are closed. #1369/#1370 +integrate Go opaque storage; final Go releases still require replacing the +temporary Vitamin C pseudo-version pins. Rust dependencies remain on 0.8.0. + +#1379 adds cascade authentication and randomized stored/projection order. +Wildcard `->` returns an unspecified match, not the first document element. +Post-cascade data is the format to freeze; unchanged header version 1 does +not imply compatibility with the old prototype AAD. Read verification still +does not authenticate result completeness, absence or database row identity. + +#1397 implements deterministic, domain-separated short-word terms inside +the existing Bloom filter, enabled by EQL's pinned match profile. This avoids +a separate short-word index but is not the random-variant mechanism explored +earlier, and it does not by itself resolve the user's frequency/leakage +concern. Native Go index options still refuse configuration (#1112). The +uncommitted ngram research and Go delivery document are unchanged. + +The refreshed API plan and JSON operation inventory mark landed work as +implemented. Remaining items include the formal interoperability spec and +conformance corpus, parent-bound child opening, configurable Go match +pipelines, direct typed EQL conversion and later context-role separation. +The customer release does not wait for final Rust traits or crate moves. +This review inspected source and existing tests; runtime/database suites +were not rerun and no production code was changed. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 7ff0d6fd3..dc6d2e7e0 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -1,189 +1,186 @@ -# JSON tree: first-release operation contract and public examples - -Date: 2026-10-25. Companion to the -[API plan](2026-10-24-stack-encrypt-api-plan.md) and -[public workflow examples](2026-10-24-stack-encrypt-public-examples.md). - -## Status and scope - -The customer requires a comprehensive first version. This document defines -the required operation inventory for Rust and the dynamic interface used by -Go. Extraction, opening extracted values, and subtree walks are included; -they are not a post-release phase. Internal implementation can still be -split into PRs. The feature is complete only when every required row below -has Rust, dynamic/Go, and applicable PostgreSQL coverage. - -**Customer sequencing update:** the PR stack headed by -[#1363](https://github.com/cipherstash/stack/pull/1363) has merged. Deliver Go -against that baseline. Its token lists, self tags, wildcard probes and result semantics -supersede the older gated-walk baseline used in parts of this document. -Reconcile those recipes before implementation and format freeze; the -operation inventory remains required. Engine-level Rust interoperability -coverage is a release gate; the final public Rust facade may follow -customer delivery. Retain regression coverage for the numeric round-trip -fix now merged in #1366 (closing #1364). - -This is a target API contract, not a claim that these public signatures -already exist. Rust and JSON examples below propose the redesigned surface. -Final names/package paths are deliberately left to the last naming pass. -The [packaging glossary](2026-10-25-stack-encrypt-packaging.md#json-names-by-role) -now proposes that pass; this document retains its working names until review. -They are candidates for executable documentation during implementation; -they have not been compiled against today's API. Existing SQL spellings and -current behavior are separately identified. Do not publish these as current -SDK instructions until the acceptance suite runs them. - -Sources reviewed: - -- [Claude's Go JSON plan, PR #1353](https://github.com/cipherstash/stack/pull/1353), - head `65ece530934d367e30ea99d1f149f76117191739`, still a draft when read. - Tracking #1347; steps #1348–#1352. Its step-5 deferral is superseded here. -- [JSON tree design](2026-10-17-json-tree-index.md), including the Phase 4 - notes on extracted-path checking and stored order. -- [Engine query implementation](../../packages/stack-encrypt/src/json/query.rs), - [path grammar](../../packages/stack-encrypt/src/json/path.rs), - [derivation/opening](../../packages/stack-encrypt/src/json/derive.rs). -- [EQL target and result types](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs), - [SQL functions](../../packages/eql/src/v3/json_tree/functions.sql), - [operators](../../packages/eql/src/v3/json_tree/operators.sql), - [refused operations](../../packages/eql/src/v3/json_tree/blockers.sql), - and [leakage contract](../../packages/eql/docs/reference/json-tree-leakage.md). - -The local source baseline is the reviewed `next` commit `b1049824b` and -this documentation branch. Untracked experimental EQL files are not used -as evidence of implemented JSON-tree behavior. - -## Roles and shared policy - -| Working name | Role | +# JSON tree: operation inventory and release contract + +Updated: 2026-10-27 against merged `next` +`ffeed7adb3f7d8739cda53261046147852bffc82`. Companion to the +[API plan](2026-10-24-stack-encrypt-api-plan.md). + +The Go JSON feature is implemented through #1372/#1373/#1378/#1402. This +inventory distinguishes current support from remaining requirements. The +public Rust derive/facade example below remains proposed, not executable +code for the current engine. The +[refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and its engine/Go findings document the exact source evidence. + +## Current names and policy + +| Role | Current public Go / dynamic / SQL name | | --- | --- | -| `JsonPlaintext` | Owned JSON input/output with `Zeroize`; an adapter around the engine's JSON value model, not ciphertext or a policy | -| `JsonOptions` | Validated target options: `sealed_root` and `ordered` paths | -| `EncryptedProfile` | Application-defined fixed-policy output wrapper | -| `JsonTreeDocument` | Existing EQL encrypted document envelope, stored in one column | -| `JsonTreeQuery` | Containment/path-equality query operand; cannot decrypt | -| `JsonTreeLeafQuery` | Ordering query operand; cannot decrypt | -| `JsonTreeSelector` | Encoded path selector, not a plaintext path or ciphertext | -| `JsonTreeEntry` | Extracted encrypted node with the metadata needed to open it | -| `JsonSubtreeKey` | Client-held walk request state: token, gates, requested path and policy binding | -| `JsonTreeSubtree` | Encrypted walk rows paired with their original request state | -| `JsonSelection` | Proposed owned plaintext result distinguishing missing, one value and multiple matches; implements `Zeroize` | - -`JsonPlaintext` is a proposed library adapter, not a claim that -`serde_json::Value` implements `Zeroize`. It accepts JSON text without -requiring callers to implement their own cleanup trait. Returned plaintext -is owned by the caller. Future zeroization/timing work must not change the -stored representation. - -One immutable definition supplies options, column identity, search scope -and ciphertext binding to every operation. Rust's derive builds it from a -typed options factory; dynamic loading deserializes the same options and -validates the same constraints. Query code never supplies a second options -object that can disagree with the write policy. - -For a row-bound, tenant-scoped field: - -- Ciphertext binding: declared column + tenant scope + row ID. -- Search selectors, value terms, ordering terms and walk-token derivation: - declared column + tenant scope, with the operation's domain separation. -- Queries do not require row ID. Decrypting a selected document, entry or - subtree requires it; SELECT examples must return that ID. -- Preserve JSON node selector/ordinal authentication in addition to these - outer bindings. The engine currently combines context roles; implementing - their separation is required, not just changing a derive attribute. - -## Definitive first-release inventory - -`q` below means the typed query result, not manually assembled EQL JSON. -SQL functions shown with `eql_v4` use the existing EQL v4 names. Parameter -types are explicit to avoid PostgreSQL selecting native jsonb operators. - -| ID | Required operation | Rust surface / result | Dynamic operation / result | Database use | -| --- | --- | --- | --- | --- | -| J01 | Encrypt document: object, array, scalar or JSON null | `value.encrypt_into` → encrypted target | `encrypt` → typed EQL document | Store `eql_v4_json_tree` | -| J02 | Decrypt complete document | `JsonPlaintext::decrypt_from` → JSON | `decrypt` → JSON text | Read full column | -| J03 | Batch write/read | `Vec<_>` conversion composition | `encrypt_many`, `decrypt_many` | Preserve row/result ordering; shared executor | -| J04 | Contains | `query(op::Contains, value)` → `JsonTreeQuery` | `contains` → same EQL operand | `profile @> $1::eql_v4.query_json_tree` | -| J05 | Contained by | `query(op::ContainedBy, value)` → `JsonTreeQuery` | `contained_by` | `profile <@ $1::eql_v4.query_json_tree` | -| J06 | Scalar equality at path | `query(op::AtPath(op::Eq), (path, scalar))` → `JsonTreeQuery` | `eq_at` | Same containment predicate as J04 | -| J07 | Four ordering predicates | `AtPath(Lt/Lte/Gt/Gte)` → `JsonTreeLeafQuery` | `lt_at`, `lte_at`, `gt_at`, `gte_at` | `(profile -> $1::text) < $2::eql_v4.query_json_tree_leaf`, etc. | -| J08 | Inclusive range | `AtPath(Between)` → two leaf operands | `between_at` → lower/upper operands | `>= lower AND <= upper` | -| J09 | Sort ascending/descending | `Selector` query → selector; no bound value | `selector` | `ORDER BY eql_v4.ord_term(profile -> $1::text) ASC/DESC` | -| J10 | Path exists | `Selector` query | `selector` | `json_tree_path_exists(profile::jsonb, $1::text)` | -| J11 | Extract first match | `Selector` query → optional encrypted entry from DB | `selector`; result accepted by `decrypt_entry` | `profile -> $1::text` or `json_tree_path_query_first` | -| J12 | Extract first match as text | Same selector; parse text into `JsonTreeEntry` | Same selector; parse encrypted entry text | `profile ->> $1::text`; never plaintext | -| J13 | Extract all matches | Same selector → encrypted entry rows | Same selector; `decrypt_entries` | `json_tree_path_query(profile::jsonb, $1::text)` | -| J14 | Open extracted scalar(s) | `decrypt_from` on source bound to selection request | `decrypt_entry`, `decrypt_entries` with selection handle | No server decrypt; container markers refused | -| J15 | Prepare subtree/positional request | `query(op::Subtree, path)` → `JsonSubtreeKey` | `subtree_key` → handle + SQL parameters | Token/gates, derived under field scope/options | -| J16 | Fetch/open subtree | Request-bound `JsonTreeSubtree`, then `decrypt_from` | `decrypt_subtree` with walk handle | `json_tree_subtree` / `json_tree_subtree_at` | -| J17 | Enumerate immediate children | J15 key, encrypted child rows | J15 key; opening scalar children uses J14 with walk validation | `json_tree_children(profile, token, gates)` | -| J18 | Count immediate children | J15 key; result is integer | J15 key; database returns integer | `json_tree_child_count(profile, token, gates)` | -| J19 | Walk budget and sealed-root fallback | Same J16 plaintext result regardless of path taken | Same handle and result shape | Default budget 64; zero walks without fallback | -| J20 | Compare stored documents/ordered entries within one scope | Existing typed EQL operands; no new plaintext term | Existing typed EQL operands | Document `@>`/`<@`; entry `< <= > >=`; same policy/scope required | - -J09–J13 and J17–J18 are SQL actions using a derived selector/key, not new -cryptographic primitives or SDK database execution methods. The dynamic -engine derives operands and opens results; the caller runs SQL. The -definition's capability discovery must describe both operand preparation -and the supported database use, including ordering-path restrictions. - -J20 does not authorize whole-document `=` or equality via an ordering -term. Initial ORE support never answers `Eq`. JSON scalar equality remains -available through its independent exact value-selector mechanism. Any -future ORE equality capability must be explicit and limited to encodings -that preserve equality; text/lossy encodings remain ineligible. - -## Path, null and result semantics - -- Grammar: `$`, `.member`, `["member"]`, nonnegative `[n]`, and `[*]`. - No filters, recursive descent or slices in this release. -- Selector-based operations address a path class across array elements. - Existing selectors erase positions (`[2]` has the same selector as - `[*]`). **Proposed first-class API rule:** reject positional paths for - `EqAt`, selector extraction/existence and selector ordering rather than - silently broadening them. Use the positional subtree route for exact - `[n]` retrieval. This is a deliberate stricter wrapper than today's engine. -- Wildcard equality means at least one matching element has that scalar - value. Containment preserves grouping within array elements; it is not - an ungrouped bag of independent terms. Multiple independent EqAt SQL - predicates do not imply the values belong to the same array element. - Use one containment needle to require that correlation. -- Equality accepts null, boolean, string and supported JSON number values; - objects/arrays are refused. Containers use containment, not an operator - labelled equality. At `$`, equality asks for that scalar root, not an - array containing it (which is containment semantics). -- Ordering is opt-in per path and accepts supported numbers/strings only. - No implicit ordering at undeclared paths. A wildcard ordering policy can - supply terms to many entries: `->` compares/sorts the first matching - entry, while `json_tree_path_query` can enumerate all and an explicit - SQL `EXISTS` chooses any-match semantics. Never present first-match as - any-match, or term equality as exact plaintext equality. -- A missing path gives no extraction row (or SQL NULL for first-match). - A present JSON null gives an encrypted entry that opens to JSON null. - SQL NULL for an absent document is a separate field-null policy, not an - encrypted JSON null. An empty object/array is neither missing nor null. -- JSON inputs cross the dynamic ABI as UTF-8 JSON text. No intermediate - conversion of JSON numbers to Go `float64`; use raw JSON/number-preserving - decoding. Define accepted numeric range and canonicalization from the - engine, test `2`/`2.0`, negative zero and large integers. Reject values - outside the supported representation rather than silently rounding them. -- SQL returns encrypted data, even with `->>`. A scalar node opens directly; - an ordinary container node is a marker, not a serialized subtree. Use a - subtree walk to reconstruct containers. The sealed root holds the whole - document and has an explicit whole-document/fallback decode path. -- Selection/open requests retain expected selector, field definition and - scope. Check the returned selector before opening: current code opens - under server-supplied selectors, which is insufficient to establish the - requested path. Row provenance requires explicit row binding. Walks must - validate the path/selector relationships needed for their requested result. - Successful decryption does not prove query completeness or authenticate - every search/index metadata field. -- Requests for `$` (or root-array wildcards requiring a root token) under - `sealed_root=true` use whole-document fetch/decrypt and local selection; - never fabricate a nonexistent root token. For a missing/empty/scalar - subtree where a descendants-only walk cannot distinguish the cases, use - the selected start-node metadata or full-document fallback explicitly; - do not treat an empty row set as an empty object by assumption. +| Configured target | `JsonSearch` | +| Encrypted document | `eql.JSONSearch`, `public.eql_v4_json_search` | +| Containment / EqAt operand | `eql.JSONSearchQuery`, `eql_v4.query_json` | +| Ordering operand | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | +| Encrypted entry | `eql.JSONEntry`, `public.eql_v4_json_entry` | +| Prepared selector / subtree | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | +| Singular plaintext result | `JSONValue{Found, JSON}` | + +Handwritten Rust EQL adapters retain their `JsonTree*` names; the internal +engine `JsonDocument` is not the Go ciphertext type. `eql.JSON` is the +storage-only type. Do not use the old `query_json_tree` / `eql_v4_json_tree` +domains in current recipes. + +A field's configured target options are resolved into native `JsonOptions` +and reused for writes, queries and reads. The shipped profile uses column +identity and keyset selection. Separate row binding and search scope remain +accepted future API work, not supported EQL context inputs. Reject those +unsupported declarations; do not append row IDs to the existing shared +context. Queries do not gain row isolation merely because their returned +ciphertexts authenticate. + +## Required inventory and current support + +| ID | Operation | Current behavior / remaining work | +| --- | --- | --- | +| J01 | Encrypt object/array/scalar/null | Implemented; raw JSON supports every root, typed adapters support declared structs/maps/slices | +| J02 | Decrypt complete document | Implemented; raw JSON text or declared Go type | +| J03 | Batch write/read | Implemented; shared executor and provider chunking | +| J04 | Contains | Implemented `Contains`, `contains`, `@>` with `query_json`; generated needle guards apply | +| J05 | Contained by | Implemented `ContainedBy`, `contained_by`, `<@` | +| J06 | Scalar equality at path | Implemented `EqAt`, `eq_at`; value selectors, not ordering equality | +| J07 | Four ordering predicates | Implemented runtime paths and `At.` accessors; ordered number/string paths only | +| J08 | Inclusive range | Implemented `Between`, `between_at`, named lower/upper operands | +| J09 | Sort | Selector plus explicit SQL; wildcard sort needs defined reduction, not an assumed first array element | +| J10 | Path exists | Selector plus `json_tree_path_exists` | +| J11 | Extract one match | Selector plus `->` / path-query-first; wildcard choice is unspecified | +| J12 | Extract as encrypted text | `->>`; still encrypted entry JSON, not plaintext | +| J13 | Enumerate all matches | SQL path query exists; bounded wildcard subtree is the supported grouped plaintext route | +| J14 | Open extracted scalars | `JSONSelection.Open` handles one optional entry per document group. Multi-entry selector groups are refused; do not label them supported | +| J15 | Prepare positional/wildcard read | Implemented `Subtree`, `subtree`; token list plus retained trusted request | +| J16 | Fetch/open subtree | `json_tree_subtree(document, tokens, budget)`, `JSONRead.Open` / `OpenOne` | +| J17 | Immediate children | `Children` prepares a concrete token; SQL returns child entries. Parent-bound child-scalar opening remains unimplemented | +| J18 | Count children | `ChildCount` token and SQL nullable count; not authenticated completeness | +| J19 | Budget/fallback | Implemented, with original requested selection applied after root fallback | +| J20 | Stored-value comparisons | Existing document containment and entry ordering, within compatible column/keyset/profile | + +The comprehensive release requirement remains. Do not equate closed tracking +issues with implementation of a parent-bound child opener, or feed child +rows into an unrelated selector/subtree request. Complete that opener for +the J17 plaintext-child requirement; until then, fetch and open the parent's +subtree to inspect its children, documenting the different transfer/leakage +cost. SQL all-match enumeration is distinct from the supported bounded +subtree plaintext interface. + +## Semantics that changed or are now implemented + +- #1379 randomizes stored entries, object ordinals and containment projection + order. `->` at a wildcard selects an unspecified element. Use all matches + with an explicit `EXISTS`, `MIN` or `MAX` rule for ordering/sorting. +- Cascade authentication commits containers to their child ciphertext tags. + Nonempty standalone container entries fail `JsonNotALeaf`, also used for + alteration; empty container entries authenticate but fail `OpenedType`. + All ordinary containers need a subtree read. A requested sealed root can + still open as a complete document. +- The dynamic engine now rejects positional selector/ordering/EqAt paths. + Concrete `[n]` positions belong to subtree reads; wildcard selectors address + a path class. Independent EqAt predicates need not match one array element; + one containment needle preserves their correlation. +- The strict Rust parser now rejects duplicate keys and out-of-range integer + literals before key I/O. Typed Go encoding occurs first and follows + `encoding/json`, including invalid-UTF8 replacement. Raw JSON is checked + directly. Neither path promises byte-for-byte JSON text preservation. +- SQL NULL, missing selection, JSON null and empty containers are distinct. + `JSONValue.Found` carries selection presence. `OpenOne` rejects wildcards; + plural subtree reads return requested values in document order. +- Prepared reads retain the cipher, field definition, path and wildcard + bound. `Open(ctx, rows)` asks Rust to rederive and verify the request. + No serialized guest-local key handle is required. Rows cannot supply their + own expected path. Keep one group per database document. +- Cascade strengthens verification beneath a returned root. It does not + authenticate the whole result set, absence, freshness or the intended + database row. Whole results/probes can still be withheld. +- Wildcard bound defaults to 64 with boundary probes and an 8,192-token cap. + Walk budget independently defaults to 64; zero disables root fallback. + An open-root document cannot use sealed-root fallback. The budget is not + a universal response-size cap. +- Header version remains 1 despite the changed cascade/AAD semantics. Freeze + the post-cascade profile explicitly; the unchanged header is not evidence + of compatibility with older prototype ciphertexts. Randomized query + projection order requires canonical semantic comparisons in parity tests. + +## Current SQL recipes + +The examples assume `profile public.eql_v4_json_search`. Parameters are the +SDK's generated query/selector/token values, never plaintext paths. Add the +application's authorization and tenant predicates as usual. + +```sql +-- Contains / contained by / EqAt operands. +SELECT id, profile FROM users WHERE profile @> $1::eql_v4.query_json; +SELECT id, profile FROM users WHERE profile <@ $1::eql_v4.query_json; + +-- A single-valued ordered path: selector, lower bound, upper bound. +SELECT id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Wildcard ordering: explicitly existential, not an unspecified -> match. +SELECT u.id FROM users AS u +WHERE EXISTS ( + SELECT 1 + FROM eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS x(entry) + WHERE x.entry > $2::eql_v4.query_json_leaf +); + +SELECT id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, profile -> $1::text AS entry FROM users; +SELECT id, profile ->> $1::text AS encrypted_entry_text FROM users; + +-- Token list, then walk budget. Group entries by id before opening. +SELECT u.id, w.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text[], $2::integer +) AS w(entry); + +-- Children/count take one concrete node token, with no gates parameter. +SELECT u.id, c.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children(u.profile, $1::text) AS c(entry); +SELECT id, eql_v4.json_tree_child_count(profile, $1::text) FROM users; +``` + +Child count is zero for scalars/empty containers and NULL for a missing node +or the sealed root. Do not convert NULL to zero. The former token/gates and +`json_tree_subtree_at` recipes are removed. Current full Go programs and +parameter binding examples live in +[the JSON example](../../languages/golang/encrypt/example/json/main.go). + +## Current dynamic grammar and results + +This is implemented data-plan syntax, without an illustrative versioned +outer envelope: + +```json +{ + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "JsonSearch", + "options": {"sealed_root": true, "ordered": ["$.age"]} + } + } +} +``` + +Operations select the field through the ABI. Their named inputs include +`{"json"}`, `{"path","json"}`, `{"path","low","high"}`, `{"path"}` +and `{"path","wildcard_bound"}`. JSON values are UTF-8 JSON text. Query +results identify `encoding` (`eql`, `text`, `text_array`, or scalar `bytes`), +SQL domain and operands. Capabilities identify input shapes. Opening uses +`json_open` / `se_json_open` with the original `selector` or `subtree` +request and grouped encrypted rows. The general versioned definition/schema +package remains required work; it is not the current runtime grammar. ## Rust write/read example @@ -255,326 +252,28 @@ requests. Store/return both ID fields for decrypt. If caller input is used, use the agreed `_with_context` variants with a typed input separating binding and scope. No row ID is ever an index/query input implicitly. -## Rust query and extraction recipes - -These snippets share `keyset` and a resolved definition `profile` from the -write example. Add `use stack_encrypt::{op, json::JsonPath};`. The `using` -selector chooses that definition before the operation. `Selector` and -`Subtree` are proposed operation markers integrating today's separate -`json_selector`/`json_subtree_key` helpers into the shared definition API. -All query operands below use the same plaintext JSON adapter as writes. - -```rust -let profile = EncryptedProfile::definition()?; -let contains = keyset.using(&profile) - .query(op::Contains, JsonPlaintext::parse(r#"{"name":"Alice"}"#)?) - .await?; -let contained_by = keyset.using(&profile) - .query(op::ContainedBy, JsonPlaintext::parse(r#"{"age":42,"name":"Alice"}"#)?) - .await?; -let eq = keyset.using(&profile) - .query(op::AtPath(op::Eq), ("$.name", JsonPlaintext::parse(r#""Alice""#)?)) - .await?; - -let lt = keyset.using(&profile) - .query(op::AtPath(op::Lt), ("$.age", JsonPlaintext::parse("65")?)).await?; -let lte = keyset.using(&profile) - .query(op::AtPath(op::Lte), ("$.age", JsonPlaintext::parse("65")?)).await?; -let gt = keyset.using(&profile) - .query(op::AtPath(op::Gt), ("$.age", JsonPlaintext::parse("18")?)).await?; -let gte = keyset.using(&profile) - .query(op::AtPath(op::Gte), ("$.age", JsonPlaintext::parse("18")?)).await?; -let range = keyset.using(&profile) - .query(op::AtPath(op::Between), ( - "$.age", (JsonPlaintext::parse("18")?, JsonPlaintext::parse("65")?), - )).await?; - -let age = keyset.using(&profile) - .query(op::Selector, JsonPath::parse("$.age")?).await?; -let visit_dates = keyset.using(&profile) - .query(op::Selector, JsonPath::parse("$.visits[*].date")?).await?; -``` - -The results carry named typed EQL payloads. For example, `lt.selector()` -and `lt` bind the selector and leaf operand in the ordering SQL below; -`range.lower()`/`upper()` name the two bounds (proposed accessors). -For static policies, generated ordered-path handles can add compile-time -checks; runtime paths still validate against the same definition before -term derivation. Missing definition/scope makes a Rust request unawaitable. - -```sql --- Containment/equality operands: bind the named query payload, not plaintext. -SELECT id, tenant_id, profile FROM users -WHERE profile @> $1::eql_v4.query_json_tree; - -SELECT id, tenant_id, profile FROM users -WHERE profile <@ $1::eql_v4.query_json_tree; - --- Lower and upper bounds; use <, <=, >, >= individually for J07. -SELECT id, tenant_id, profile FROM users -WHERE (profile -> $1::text) >= $2::eql_v4.query_json_tree_leaf - AND (profile -> $1::text) <= $3::eql_v4.query_json_tree_leaf -ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; - --- Existence, first encrypted entry, first entry as encrypted JSON text. -SELECT id, tenant_id FROM users -WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); -SELECT id, tenant_id, profile -> $1::text AS entry FROM users; -SELECT id, tenant_id, profile ->> $1::text AS entry_text FROM users; - --- All matching entries: group results by document before opening them. -SELECT u.id, u.tenant_id, e.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS e(entry); - --- Stored-to-stored comparisons require matching policy and search scope. -SELECT a.id, b.id FROM users AS a JOIN users AS b - ON a.tenant_id = b.tenant_id AND a.profile @> b.profile; -SELECT a.id, b.id FROM users AS a JOIN users AS b - ON a.tenant_id = b.tenant_id - AND (a.profile -> $1::text) < (b.profile -> $1::text); -``` - -Add the application's tenant filter where appropriate; cryptographic scope -does not replace SQL authorization. A selector identifies a path class; -the SQL above never receives `$.age` or another plaintext JSON path. - -Proposed request-bound opening preserves the expected path and avoids the -current server-supplied-selector ambiguity. Here `entry` and `entries` are -typed database results for the corresponding selection request, and each -batch belongs to the identified document/context: - -```rust -let value = JsonPlaintext::decrypt_from(age.bind(entry)?, &keyset).await?; -let values = Vec::::decrypt_from( - visit_dates.bind_many(entries)?, &keyset, -).await?; -``` - -`bind` is a proposed checked source adapter, not a second decrypt method. -For row-bound policies, call `decrypt_from_with_context` with that result -row's expected ID and required scope. Missing first-match rows stay absent; -do not manufacture an encrypted null. Containers use the walk route below. - -## Walks, children, positional retrieval and fallback - -```rust -let walk = keyset.using(&profile) - .query(op::Subtree, JsonPath::parse("$.visits[0]")?).await?; - -// After the SQL below returns all rows for one document: -let source = walk.bind_subtree(rows)?; -let visits = JsonSelection::decrypt_from(source, &keyset).await?; -``` - -`walk` retains the requested path, resolved scope and definition; only the -token/gates go to SQL. Exact positions are supported here, including the -necessary selector gate to distinguish an array element from an object -member at the same ordinal. The final specification must normalize result -cardinality through `JsonSelection`: a concrete path yields `Missing` or -`One(JsonPlaintext)`; wildcard paths yield `Many(Vec)` in -document order. Missing and an explicit JSON null remain distinguishable. -`bind_subtree`/opening must preserve that distinction. `JsonSelection` is a -working type name and is imported from the proposed JSON module. - -```sql --- $1: token; $2: gates; $3: budget (64 default, 0 disables fallback). -SELECT u.id, u.tenant_id, w.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_subtree_at( - u.profile, $1::text, $2::text[], $3::integer -) AS w(entry); - --- With no gates, the dedicated form is also supported. -SELECT u.id, u.tenant_id, w.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_subtree( - u.profile, $1::text, $2::integer -) AS w(entry); - --- Immediate children and counts use the same token/gates. -SELECT u.id, u.tenant_id, c.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_children( - u.profile, $1::text, $2::text[] -) AS c(entry); -SELECT id, tenant_id, - eql_v4.json_tree_child_count(profile, $1::text, $2::text[]) -FROM users; -``` - -Use one request-bound source per document; do not merge entries from several -rows into a subtree. Immediate scalar children can use checked entry -opening; container children need their complete descendants or an explicit -full-document fallback. The SDK must expose this distinction rather than -pretend that a container marker decrypts to its children. - -Budget 64/0 are current SQL semantics to retain in the first public scope. -When `sealed_root=true`, an over-budget walk returns the sealed root instead -of partial data; the client opens it and selects the original requested -path. With `sealed_root=false`, walk the full result or report an explicit -resource limit, never silently truncate it. The same requested value(s) -must result from budget 0 and fallback. Tokens reveal subtree structure -within their search scope and are not decryption keys; document their -existing leakage and lifetime with these examples. - -## Dynamic definition and operation examples - -These are logical JSON messages for the specification, not a promise to -replace the frame ABI with JSON. The host/guest transport encodes their -typed equivalents. `using` identifies a validated field definition; the -engine, not Go, creates EQL envelopes and query terms. Naming the definition -in a request must resolve to the same options and scope contract as writes. - -The fixed-column example's dynamic counterpart is: - -```json -{ - "version": 1, - "name": "users.profile", - "plaintext": "json", - "target": { - "name": "JsonTreeDocument", - "column": { "table": "users", "column": "profile" }, - "options": { "sealed_root": true, "ordered": ["$.age", "$.visits[*].date"] } - }, - "search_scope": { "kind": "column" }, - "context": { "kind": "none" } -} -``` - -`version: 1` is an illustrative schema version, to allocate in the formal -specification. Named types are optional. For tenant/row policies use the -accepted `ScopeKey`/`PrimaryKey` declarations and source selectors; the -resolved query input requires the tenant scope but never the row ID. - -Request corpus (each array element is an independent example, not one -batch of mixed operations). JSON text is deliberately explicit so no host -float conversion precedes the engine parser: - -```json -[ - { "operation": "encrypt", "using": "users.profile", "value": { "json_text": "{\"age\":42}" } }, - { "operation": "encrypt_many", "using": "users.profile", "values": [{ "json_text": "{\"age\":42}" }, { "json_text": "{\"age\":27}" }] }, - { "operation": "query", "using": "users.profile", "operator": "contains", "input": { "json_text": "{\"age\":42}" } }, - { "operation": "query", "using": "users.profile", "operator": "contained_by", "input": { "json_text": "{\"age\":42,\"name\":\"Alice\"}" } }, - { "operation": "query", "using": "users.profile", "operator": "eq_at", "input": { "path": "$.name", "json_text": "\"Alice\"" } }, - { "operation": "query", "using": "users.profile", "operator": "lt_at", "input": { "path": "$.age", "json_text": "65" } }, - { "operation": "query", "using": "users.profile", "operator": "lte_at", "input": { "path": "$.age", "json_text": "65" } }, - { "operation": "query", "using": "users.profile", "operator": "gt_at", "input": { "path": "$.age", "json_text": "18" } }, - { "operation": "query", "using": "users.profile", "operator": "gte_at", "input": { "path": "$.age", "json_text": "18" } }, - { "operation": "query", "using": "users.profile", "operator": "between_at", "input": { "path": "$.age", "lower": { "json_text": "18" }, "upper": { "json_text": "65" } } }, - { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.age" } }, - { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.visits[*].date" } }, - { "operation": "query", "using": "users.profile", "operator": "subtree_key", "input": { "path": "$.visits[0]" } } -] -``` - -Responses carry a result kind and typed payload: document, containment -operand, leaf operand, range, selection, or walk request. SQL preparation -returns the named domain and encoded payload (or typed token/selector -parameters); it does not concatenate SQL. Selectors serve J09–J13, and walk -keys serve J16–J19, so those SQL uses do not need redundant engine opcodes. - -For decode operations, this table specifies the dynamic signatures. Values -such as `document` and `entries` mean complete validated EQL payloads, not -literal placeholder strings to parse as ciphertext: - -| Operation | Required inputs | Result | -| --- | --- | --- | -| `decrypt` | `using`, `document`, required context/scope | JSON text | -| `decrypt_many` | `using`, documents with per-record context/scope | JSON texts in input order | -| `decrypt_entry` | `using`, `selection_handle`, entry, required context/scope | Scalar JSON text | -| `decrypt_entries` | `using`, selection/walk handle, entries grouped by record and context | Ordered scalar JSON texts | -| `decrypt_subtree` | `using`, `walk_handle`, complete rows for one document, required context/scope | Explicit missing / one JSON value / multiple matches | - -The response to `selector`/`subtree_key` includes an opaque request handle -in addition to SQL parameters. The handle retains the path and resolved -definition/scope in the engine, so callers cannot accidentally substitute -arbitrary path/options during decode. Specify handle lifetime and release -in the guest ABI; handles are not persistent database values. If the host -must recreate one after restart, derive it again from the original request, -not from server-returned metadata. Stale or mismatched handles are errors. -An equivalent Rust source adapter retains the same state directly. - -Example host orchestration, using proposed dynamic method names rather -than inventing ciphertext strings in a JSON example: - -```rust -let loaded = dynamic.load_definition(definition_json)?; -let encrypted = loaded.encrypt(json_text, inputs).await?; -let stored = database.store(encrypted).await?; -let opened = loaded.decrypt(stored, decrypt_inputs).await?; - -let selection = loaded.selector("$.age", scope).await?; -let entry = database.select_first(selection.sql_parameters()).await?; -if let Some(entry) = entry { - let age = loaded.decrypt_entry(&selection, entry, row_inputs).await?; -} - -let walk = loaded.subtree_key("$.visits[0]", scope).await?; -let rows = database.subtree(walk.sql_parameters(), 64).await?; -let result = loaded.decrypt_subtree(&walk, rows, row_inputs).await?; -``` - -This is an orchestration recipe: `database` represents application SQL, -not a new Stack Encrypt database driver; the input variables are supplied -by that application. Generated Go methods should hide these message details, -use named results, and call the same validated dynamic operations. JSON -marshalling helpers must not force a containment needle into a whole record -struct whose zero-valued fields accidentally narrow the query. - -## Explicitly unsupported operations - -The comprehensive first release covers the inventory, not all PostgreSQL -JSON operators. Reject whole-document `=`/`<>` as plaintext equality, -entry equality through ordering terms, plaintext path SQL operands, -JSONPath filters/recursive descent/slices, and in-place encrypted document -patch/delete/concatenation. Updates read/decrypt, modify, then encrypt the -complete document. Projection extraction does not confer the ability to -write a standalone node back into an existing tree. - -Reject malformed JSON/path/options; unsupported numeric values; container -EqAt operands; boolean/null ordering operands; undeclared ordering paths; -selector-position ambiguity; missing scope/context; unknown/expired request -handles; and wrong definition/scope/path during opening. Preserve useful -reason codes and structural locations without echoing plaintext values. -Structural validation that needs no key must precede key I/O. The current -engine refusal names include `JsonValueNotScalar`, `JsonLeafNotOrdered`, -`JsonPathNotOrdered`, `JsonOrderingAtPosition`, `JsonTooDeep`, -`JsonTooWide`, `JsonSealedRoot`, and `OpenedType`; final dynamic codes must -map deliberately rather than exposing unstable Rust enum spellings. - -## Implementation gaps and release gates - -1. **Shared definitions and scope separation.** Implement fixed/dynamic - target resolution, typed options, scope-role validation, row-binding - inputs and matching write/query context derivation. Generate capability - metadata from one registry, including JSON-specific operand/result kinds. -2. **Catalog/guest coverage.** Complete the JSON tree catalog, target table, - guest registration, dynamic index execution and JSON-text record kind. - Current Rust helpers/EQL types are not evidence that Go supports them. -3. **Direct EQL adapters.** Replace the existing serialization round trip - with typed conversions for documents, entries and operands. -4. **Selectors and opening.** Add the request-bound source adapters and - dynamic handles, validate expected selectors, preserve required row - context, and test mismatched results and missing headers/ordinals. -5. **Walk completeness.** Cover children/counts, concrete positions, - wildcards, empty containers, missing paths, scalar starts, root requests, - sealed-root on/off and budgets 0/64. Resolve reconstruction ambiguities - explicitly, including returning/fetching the start node when needed. - Do not ship opening/walks later under the earlier PR #1353 scope. -6. **Conformance/documentation.** Turn J01–J20 into identified test cases - and the examples into checked docs. Run Rust writes → Go reads/queries - and Go writes → Rust reads/queries against PostgreSQL. Include negative - cases, exact number handling, null distinctions, multi-element correlated - containment, and batch ordering. Test SQL GIN/btree usage on representative - containment/range queries. Randomized ciphertexts need semantic parity; - deterministic fixtures are only for explicitly seeded test builds. - -The current wire order/leakage and extraction trust boundaries must be -documented accurately. This operation inventory does not claim a new -authenticated-query-completeness protocol or a storage-order leakage fix. -Stable blob framing is a separate deliverable tracked in -[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); it must -not be confused with the EQL JSON-tree storage format. +## Remaining work and release gates + +1. Capture the implemented wire grammar, errors, presence states and context + profile in the promised formal specification and conformance corpus. + General Rust definitions/caching and additional context roles can follow. +2. Close the explicit J14/J17 scope gaps. The supported all-match plaintext + route is subtree reads; a parent-bound child opener is separate engine + work. Do not add generic raw-entry decryption as a shortcut. +3. Replace the internal EQL serialize-to-Value/deserialize bridge with typed + conversion while preserving the frozen profile. This is an optimization, + not a missing Go JSON execution path. +4. Run the existing Rust/guest/Go/PostgreSQL suites on the release candidate, + mapping tests to J01–J20. Include typed/raw inputs, numeric/null cases, + both root policies, wildcard bounds, fallback, cascade tampering and known + omission limitations. Inspect real SQL parameter casts and index use. +5. Freeze post-cascade storage/search vectors and the matching EQL bundle. + Compare canonical projection semantics and cross-opening; ordinary + randomized bytes need not match. Replace temporary Go Vitamin C pins + before a final release. Opaque storage adoption is already done and is + a separate format from JSON-tree storage. + +This refresh is source and documentation review, not a fresh runtime test +run or a complete cryptographic audit. Final Rust example compilation, +package extraction and stronger zeroization/constant-time work remain later +API-plan activities; they must preserve the released Go/data contract. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 9c7d50e7c..09c7bd248 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -7,11 +7,12 @@ renames. Existing companion examples retain their working names until this proposal is accepted. The detailed JSON operation contract remains under separate user review. -**Sequencing update:** customer Go delivery supersedes the original -move-first sequence. The JSON stack headed by #1363 has merged. Vitamin C -#520 and #526 address storage and await review/release. Adopt those encodings -and deliver Go JSON/match search first. The package map below is the intended destination; neither a new -format crate nor public Rust facade completion is a customer release gate. +**Implementation update, 2026-10-27:** Go JSON and opaque storage have +landed on `next` through #1369/#1370/#1372/#1373/#1378/#1402; #1397 adds +short whole-word match terms. The customer path is remaining configuration +and conformance work plus release readiness, including replacing temporary +Vitamin C Go module pins. Rust stays on crates.io Vitamin C 0.8.0. Neither +crate extraction nor final Rust facade completion is a customer release gate. Source baseline: the current documentation branch and inspected local Vitamin C 0.8.0 sources. Stack's @@ -196,9 +197,16 @@ for the ciphertext wrappers; avoid an unchecked public string constructor. ## JSON names by role -Keep the existing EQL package and version modules. Avoid importing low-level -engine JSON documents into normal application examples alongside EQL output -types. Recommend this glossary for the first-class public surface: +Keep the existing EQL package and version modules. The merged public Go +surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery` and +`JSONEntry`, configured through `JsonSearch`; SQL uses `eql_v4_json_search`, +`query_json`, `query_json_leaf` and `eql_v4_json_entry`. Prepared Go objects +are `encrypt.JSONSelection` and `JSONRead`. + +The table below remains a proposed Rust facade glossary. Its `JsonTree*` +names describe handwritten Rust adapters, not the registered Go target. +Do not rename the working Go/SQL surface to this glossary as a customer +release prerequisite. Later Rust renames must preserve frozen dynamic IDs. | Working/current name | Recommended name | Role / home | | --- | --- | --- | @@ -235,13 +243,12 @@ or dynamic discriminants. ## Implementation order and acceptance -1. Use the merged JSON stack and specify Go-visible names/contracts and stable - profiles needed by the customer. Keep codec work in Vitamin C, coordinating - #519. Preserve distinct transport/storage semantics and freeze search - semantics as well as byte layouts. -2. Deliver and validate Go JSON, configurable match search and the selected - short-input solution under the customer-first sequence. Record frozen Go/data - fixtures; leave internal Rust plan ergonomics free to change. +1. Specify and validate the now-implemented Go JSON and opaque storage + contracts. Freeze post-cascade JSON/search semantics and exact fixtures. + Replace temporary Vitamin C Go module pins for a final release. +2. Complete outstanding Go match configuration and JSON read/conformance + gaps. Short whole-word matching is implemented in EQL; its leakage and + profile compatibility still require explicit release acceptance. 3. After delivery, establish the shared format crate and move the value model, traversal and dependent normalization/index adapters into Stack. Preserve frozen encodings and an acyclic graph; move tests with their code. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index eeaf0afef..0ce88185e 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -26,6 +26,15 @@ source value requires explicit declaration, not inference from a name. ## Current implementation evidence +**Update, 2026-10-27:** #1379 makes open-root JSON verification load the +index keyset as well as the data key. `Pending` settlement and generic +execution entry points now require `DataKeySource + IndexKeySource`. +The future neutral provider split below must preserve that verification +requirement for JSON while allowing data-key-only capabilities where an +operation genuinely needs no index key. This change does not implement +plaintext-derived keyset routing or ZeroKMS lock-context plumbing. + + - [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts `IdentifiedBy` by UUID or name, resolves the keyset and its index key, and returns a bound `KeysetCipher`. The client has a bounded keyset cache From 84bf574d6b2e26aa5c0e3560416801e0b11f8093 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 11:36:11 +1100 Subject: [PATCH 05/42] docs: rewrite the Stack Encrypt API plans in plain language Rewrite the seven plan documents to the plain-language guide: each document starts with its decision or status, defines its terms where they first appear, and uses headings that name their topics. Facts, links, issue numbers, and code blocks are unchanged. --- .../2026-10-24-stack-encrypt-api-plan.md | 2900 ++++++++++------- .../2026-10-24-stack-encrypt-api-review.md | 2214 ++++++++----- ...26-10-24-stack-encrypt-context-examples.md | 567 ++-- ...026-10-24-stack-encrypt-public-examples.md | 845 +++-- ...026-10-25-stack-encrypt-json-operations.md | 404 ++- .../2026-10-25-stack-encrypt-packaging.md | 575 ++-- ...-25-stack-encrypt-provider-capabilities.md | 701 ++-- 7 files changed, 4956 insertions(+), 3250 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 00969faf7..5188a4bbf 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1,177 +1,300 @@ # Stack Encrypt API implementation plan -Date: 2026-10-24. Status: proposed implementation direction, developed from -the [review log](2026-10-24-stack-encrypt-api-review.md), -[public examples](2026-10-24-stack-encrypt-public-examples.md), and -[context examples](2026-10-24-stack-encrypt-context-examples.md). - -This document turns the review into implementation choices and completion -criteria. User requirements are distinguished from recommendations that -have not yet been accepted. Examples below are proposed APIs, not working -code for the current release. - -**Implementation reconciliation, 2026-10-27:** reviewed merged `next` at -`ffeed7adb3f7d8739cda53261046147852bffc82`. Go JSON catalog, configured -targets, queries, request-bound reads, typed documents and executable docs -have landed (#1372, #1373, #1378, #1402). Opaque Go storage adoption has -landed (#1369/#1370). The active customer work is now contract conformance, -remaining read/configuration gaps and release readiness, not implementing -those foundations again. See the refreshed -[Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) -and [JSON inventory](2026-10-25-stack-encrypt-json-operations.md). - -#1379 added cascade authentication and randomized storage/projection order. -Wildcard `->` now selects an unspecified element; old first-in-document-order -recipes are invalid. Freeze the post-cascade format and compare query -semantics rather than randomized byte ordering. #1397 implements short -whole-word terms in the shared Bloom filter; configurable native Go match -options and broader tokenizer/filter work remain. This is implementation -status, not acceptance of a stronger privacy claim for those terms. - -Go's Vitamin C storage modules are temporarily pinned to `06568ba93bd9`. -The final-release guard requires replacing pseudo-versions with released -module tags; prereleases may carry them. Rust remains on crates.io 0.8.0. -Keep crate moves and the final Rust facade behind customer delivery. - -**Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets -are accepted. The comprehensive JSON first-release contract and Rust/dynamic -examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), -including extraction and walks. ORE does not provide equality initially; -retain a future explicit capability for equality-preserving encodings. -Prioritize stable blob storage bytes via -[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); -zeroization/timing implementation improvements may follow without changing -stored format. The naming/package review is now in progress. These decisions supersede -earlier sequencing recommendations below where they conflict. - -**Follow-up decisions:** custom Rust query stages may be explicitly -Rust-only, with `NotPortable` on dynamic export until registered; built-in -capabilities retain Rust/Go parity. Every validation independent of keys -runs before key resolution/loading. The user is still reviewing the JSON -operation contract; its detailed API proposals are not collectively -approved. New keyset-routing, lock-context and provider-capability questions -are assessed in [provider capabilities](2026-10-25-stack-encrypt-provider-capabilities.md). - -**Packaging review:** the user favors retiring Vitamin C `AsyncCipher` and -moving application encryption behavior into Stack. The -[packaging and naming recommendation](2026-10-25-stack-encrypt-packaging.md) -defines crate ownership, an acyclic dependency graph, the proposed shared -format crate, role-based public names and coordinated migration work. It is -a proposal for review, not an accepted rename or an implementation. - -**Compatibility scope (user clarification, 2026-10-25):** this API has only -existed as an early prototype. Do not preserve its APIs, stored ciphertexts, -or encodings for backward compatibility. No legacy decoders, compatibility -aliases, deprecation period, or data migration is required. Choose the -coherent new design and update its Rust, Go, guest, and EQL consumers -together. Engine output must still agree with the SQL that consumes it; -that is current interoperability, not a legacy-format requirement. -The merged Go implementation nevertheless retains a legacy JSON opaque -reader; record it as existing behavior, not a new requirement to add -prototype migration machinery. This exemption ends for customer profiles when released: subsequent -Rust API/package refactors must preserve the Go interface and stored -data/search compatibility. +First written on 24 October 2026. Checked against the merged code on 27 +October 2026. + +Status: proposed implementation direction. This plan develops the [API +review log](2026-10-24-stack-encrypt-api-review.md), the [public +examples](2026-10-24-stack-encrypt-public-examples.md), and the [context +examples](2026-10-24-stack-encrypt-context-examples.md) into implementation +choices, and sets the criteria for finishing each one. + +The plan marks which statements are the user's requirements and which are +recommendations that nobody has accepted yet. "The user" is the person who +asked for the review and who decides its open questions. The code examples +show proposed APIs, not working code in the current release. + +## Summary + +- The Go JSON foundations and Go opaque storage are merged. The customer + work that remains is conformance to the contract, closing the remaining + gaps in reads and configuration, and release readiness. See [Status on 27 + October 2026](#status-on-27-october-2026). +- The Go customer release comes first. The final Rust facade and the crate + moves come after it, and do not hold it up. +- Until the customer release, nothing needs to stay compatible with the + prototype API or its stored data. After it, the Go interface and the + stored data must stay compatible. +- The [decision checkpoint](#decision-checkpoint-on-25-october-2026) lists + the settled directions. The [remaining questions](#remaining-questions) + table lists what is still open. + +## Terms used in this plan + +| Term | Meaning | +| --- | --- | +| AAD (additional authenticated data) | Data that encryption binds to a ciphertext without encrypting it. Decryption fails unless the same AAD is supplied again | +| AEAD (authenticated encryption with associated data) | The cipher that seals each value and authenticates its AAD | +| Blob | A value that `OpaqueBlob` encodes as a whole and seals as one ciphertext | +| Bloom filter | The bit array that the match index stores. Each term sets some bits, so a query can test whether a value may contain the term | +| Cascaded tags | Each JSON container's authentication tag covers the tags of its children. #1379 added them | +| Column and keyset profile | The released configuration. It uses the column identity and the keyset selection, and accepts no row binding or search scope input | +| Configured target | A target together with typed options, such as a JSON target with a list of ordered paths | +| Context | Values bound into a ciphertext, such as a column name or a row ID. Decryption fails unless the same context is supplied again | +| Dynamic interface | The interface that takes declarations and requests as data instead of Rust types. Go uses it through the guest | +| EQL (Encrypt Query Language) | The SQL types and functions in `packages/eql` that store and query encrypted values in PostgreSQL | +| Facade | The proposed public Rust API of `stack-encrypt`: its traits, derives, and convenience methods | +| FFI (foreign function interface) | The boundary through which another language calls the Rust engine | +| Guest | The Rust engine compiled to WebAssembly, which the Go SDK runs | +| Key I/O | Any request to ZeroKMS to resolve, load, or derive keys | +| Keyset | A set of keys in ZeroKMS, the CipherStash key management service (KMS) | +| Open and seal | To seal a value is to encrypt and authenticate it. To open it is to authenticate and decrypt it | +| ORE (order-revealing encryption) | Index terms that reveal the order of the plaintexts, for range queries and sorting | +| OPE (order-preserving encryption) | Encryption whose ciphertexts keep the order of the plaintexts | +| Passthrough field | A field copied unencrypted from the plaintext type to the encrypted output | +| Plan | The lower-level description of which fields a conversion encrypts, indexes, or passes through. Derives and dynamic declarations compile to plans | +| Post-cascade profile | The JSON storage and search format as it is after #1379 added cascaded tags and randomised order. This is the format to freeze | +| Pseudo-version | A Go module version that names a commit instead of a released tag | +| Reads that Rust verifies against the original request | Go prepared reads. When the rows are opened, Rust derives the request again and checks every row against it | +| Row binding | Context that ties a ciphertext to one row, such as the row ID, so that it opens only for that row. Also called ciphertext binding | +| Search scope | The part of the context that also goes into index keys and search terms, such as a tenant ID. One query matches every row in the scope | +| Target | A type or dynamic name that says how a value is encrypted and stored, such as `JsonSearch` or `TextEq` | +| Target definition | The resolved, immutable form of a target's declaration: options, prepared indexes, context rules, and operation metadata | +| Typestate | A builder whose type changes with each call, so the compiler refuses to run an incomplete request | +| Vitamin C | CipherStash's cryptography library, `vitaminc`. Its codec encodes values as binary frames | + +## Status on 27 October 2026 + +### Merged Go work + +On 27 October 2026, this plan was checked against merged `next` at commit +`ffeed7adb3f7d8739cda53261046147852bffc82`. This Go work is merged: + +- the Go JSON catalog, configured targets, queries, reads that Rust verifies + against the original request, typed documents, and executable + documentation (#1372, #1373, #1378, and #1402); +- Go adoption of opaque storage (#1369 and #1370). + +The active customer work is now conformance to the contract, closing the +remaining gaps in reads and configuration, and release readiness. That work +does not implement those foundations again. See the refreshed [Go JSON +plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and the [JSON operations +inventory](2026-10-25-stack-encrypt-json-operations.md). + +### JSON storage and match changes in #1379 and #1397 + +Pull request #1379 added cascaded tags, and randomised the order of stored +entries and query projections. A wildcard `->` now selects an unspecified +element, so old SQL recipes that relied on the first element in document +order are invalid. Freeze the post-cascade profile. Compare query semantics +in tests, not the randomised byte order. + +Pull request #1397 implements short whole-word terms in the shared Bloom +filter. Configurable native Go match options and broader tokeniser and +filter work remain. This paragraph records implementation status only. It +does not accept a stronger privacy claim for those terms. + +### Temporary Vitamin C pins + +Go's Vitamin C storage modules are temporarily pinned to pseudo-versions at +commit `06568ba93bd9`. The guard for a final release requires each +pseudo-version to be replaced with a released module tag. Prereleases can +keep pseudo-versions. Rust remains on Vitamin C 0.8.0 from crates.io. +Schedule the crate moves and the final Rust facade after the customer +release. + +### Decisions made on 25 October 2026 + +The latest decisions about what to release: + +- Encoded ciphertext field targets are accepted. These are fields whose + stored output is the ciphertext as text, such as Base64. +- The comprehensive JSON contract for the first release, with Rust and + dynamic examples, is in [JSON + operations](2026-10-25-stack-encrypt-json-operations.md). The contract + includes extraction and walks. +- ORE does not provide equality at first. Keep the option of a future + explicit capability for encodings that preserve equality. +- Make stable blob storage bytes the priority, through [Vitamin + C #519](https://github.com/cipherstash/vitaminc/issues/519). Improvements + to zeroisation and timing can follow without changing the stored format. +- The review of names and packages is now in progress. + +Where these decisions conflict with earlier sequencing recommendations in +this plan, these decisions apply. + +### Follow-up decisions + +- Custom Rust query stages can be explicitly Rust-only. They export to the + dynamic interface as `NotPortable` until they are registered. Built-in + capabilities keep parity between Rust and Go. +- Every validation that does not need keys runs before keys are resolved or + loaded. +- The user is still reviewing the JSON operation contract. Its detailed API + proposals are not approved as a set. +- The [provider + capabilities](2026-10-25-stack-encrypt-provider-capabilities.md) companion + assesses the new questions about keyset routing, lock context, and + provider capabilities. + +### Packaging review + +The user favours retiring Vitamin C's `AsyncCipher`, and moving application +encryption behaviour into Stack. The [packaging and naming +recommendation](2026-10-25-stack-encrypt-packaging.md) defines: + +- crate ownership; +- a dependency graph with no cycles; +- the proposed shared format crate; +- role-based public names; +- coordinated migration work. + +The recommendation is a proposal for review. Nobody has accepted the +renames. None of the proposal is implemented. + +### No compatibility with the prototype + +On 25 October 2026, the user clarified the compatibility scope. This API has +existed only as an early prototype. Do not preserve its APIs, stored +ciphertexts, or encodings for backward compatibility. No legacy decoders, +compatibility aliases, deprecation period, or data migration is required. +Choose the coherent new design, and update its Rust, Go, guest, and EQL +consumers together. + +Engine output must still agree with the SQL that consumes it. That is +current interoperability, not a requirement to support a legacy format. The +merged Go implementation does still keep a reader for the legacy JSON +encoding of opaque values. Record that reader as existing behaviour, not as +a new requirement to build machinery for migrating prototype data. + +This exemption ends for the customer profiles when they are released. After +that, later refactors of the Rust API and packages must keep the Go +interface, and keep stored data and search compatible. ## Requirements established in the review -- Ordinary encryption is keyset-directed, with `encrypt` and - `encrypt_with_context`. Keep the separate target-directed `EncryptInto` - conversion interface; remove redundant public `encrypt_as`. -- Single-value encryption produces opaque ciphertext; `OpaqueBlob` +- The keyset directs ordinary encryption through `encrypt` and + `encrypt_with_context`. Keep `EncryptInto` as a separate conversion + interface directed by the target. Remove the redundant public + `encrypt_as`. +- Encrypting a single value produces an opaque ciphertext. `OpaqueBlob` explicitly encodes a composite value before sealing it as one value. - Direct `encrypt` does not automatically lift over collections. Use - `EncryptInto` for collection batching, including `Vec` into - `Vec` without caller context. This division is agreed as of - 2026-10-25. The current byte-vector recommendation follows that same - collection rule and uses `OpaqueBlob` for a whole vector; there is no - prototype-compatibility reason to add a raw-byte exception. -- Rust traits and derives are the primary static interface. Plans support - those declarations and dynamic/FFI lowering. -- Design static and dynamic support against the same engine/definition - contract. Deliver the customer Go interface first; polished public Rust - traits/derives may follow. Engine-level Rust interoperability tests remain - required, so this does not introduce a second Go encryption implementation. + Direct `encrypt` does not automatically apply itself to each item of a + collection. +- Use `EncryptInto` to batch a collection, including `Vec` into + `Vec` without caller context. This division was agreed on 25 + October 2026. The current recommendation for byte vectors follows the same + collection rule, and uses `OpaqueBlob` for a whole vector. Compatibility + with the prototype gives no reason to add an exception for raw bytes. +- Rust traits and derives are the main static interface. Plans support those + declarations, and their lowering to the dynamic interface and FFI. +- Design static and dynamic support against the same contract for the engine + and definitions. Deliver the Go customer interface first. Polished public + Rust traits and derives can follow. Interoperability tests at the level of + the Rust engine are still required, so this order does not introduce a + second Go encryption implementation. - Provide decrypt entry points with and without explicit context. The - no-context variant must not invent a default required application value. -- `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation - capabilities. Both `encrypt` and `encrypt_into`, including their - context-taking variants, consume input and take responsibility for - cleaning up owned plaintext and plaintext artifacts. Ownership alone is - not a zeroization implementation; require `Zeroize` on plaintext types - and enforce cleanup through guarded storage and lifecycle checks. Deliberate passthrough - fields transfer to the output rather than being erased. -- Keep EQL's engine/SQL contract coherent and retain shared batching/execution - behavior; prototype representations need not be preserved. -- Investigate and replace the EQL JSON adapter's serialization round trips - with direct typed conversion, maintaining engine/SQL agreement. + variant without context must not invent a default value for a required + application input. +- `OpaqueEncode` and `OpaqueDecode` name the capabilities that represent a + value as a single ciphertext. `encrypt` and `encrypt_into`, including + their context-taking variants, consume their input. Both operations take + responsibility for cleaning up owned plaintext and plaintext artefacts. +- Ownership alone does not implement zeroisation. Require `Zeroize` on + plaintext types, and enforce cleanup through guarded storage and lifecycle + checks. Deliberate passthrough fields move to the output instead of being + erased. +- Keep EQL's contract between the engine and the SQL coherent, and keep the + shared batching and execution behaviour. Prototype representations need + not be preserved. +- Investigate the serialisation round trips in the EQL JSON adapter, and + replace them with direct typed conversion. Keep the engine and the SQL in + agreement. ## Recommended decision: general configured targets -Use a general target-plus-typed-options contract. Do not add a public -`json_tree(...)` derive construct. Rust constructs concrete options directly; -the dynamic interface deserializes its options into that same type. JSON -supplies the options type, validation, and implementation behind the general -mechanism. Sharing a contract does not require sharing input syntax or -making static Rust configuration pass through deserialization. +Use a general contract: a target plus typed options. Do not add a public +`json_tree(...)` derive construct. Rust constructs concrete options +directly. The dynamic interface deserialises its options into that same +type. + +JSON supplies the options type, the validation, and the implementation +behind the general mechanism. Sharing a contract does not require shared +input syntax, or static Rust configuration that passes through +deserialisation. Three approaches were considered: | Approach | Benefit | Limitation | Recommendation | | --- | --- | --- | --- | -| Typed Rust options factory | Compiler checks the options type; normal Rust APIs and tooling | The function itself is not portable, but its concrete output has a data form | Preferred static surface | -| Named policy type holding options | Reusable and discoverable | Adds a provider trait and marker type where a function can suffice | Optional later convenience | -| Target plus data options | Expressible in Go declarations and dynamic data | Requires deserialization and semantic validation | Dynamic surface, producing the same typed options | - -This supersedes the initial recommendation to put option field literals in -Rust attributes. Rejecting a Rust factory because its code cannot cross FFI -was unnecessarily restrictive: the common boundary is its typed result, -not the code used to construct it. No arbitrary encryption callback is -introduced; the factory only supplies configuration. - -The target owns its source capabilities, canonical output, options/defaults, -validation, encryption description, index/query capabilities, and opening -behavior. Context remains a separate input. Configuration does not supply -arbitrary execution callbacks or let a caller redefine EQL's wire fields. - -Implement configuration as a target capability (working name: -`ConfigurableTarget`), lowering into the existing operation descriptions. -Its associated options type has a data representation. Rust factories -return that type directly; registered dynamic targets deserialize into it. -Validate and normalize once when a reusable declaration is built. Both -paths use the same target-owned semantic validator, before key I/O. Use -validated deserialization, for example a wire helper plus `TryFrom`, rather -than a derived deserializer that bypasses checked builders. Keep any raw -options fields private if the public type promises validated invariants. +| Typed Rust options factory | The compiler checks the options type, and normal Rust APIs and tools work | The function itself is not portable, but its concrete output has a data form | Preferred static interface | +| Named policy type that holds options | Reusable and easy to discover | Adds a provider trait and a marker type where a function is enough | Optional later convenience | +| Target plus options as data | Can be written in Go declarations and dynamic data | Needs deserialisation and semantic validation | Dynamic interface, producing the same typed options | + +The first recommendation put option field literals in Rust attributes, and +this recommendation replaces it. The first recommendation rejected a Rust +factory because its code cannot cross FFI. The rejection was too +restrictive: the common boundary is the factory's typed result, not the code +that builds it. This plan introduces no arbitrary encryption callback. The +factory only supplies configuration. + +The target owns: + +- its source capabilities, canonical output, and options and defaults; +- validation, the encryption description, and index and query capabilities; +- how its output is opened. + +Context remains a separate input. Configuration does not supply arbitrary +execution callbacks. Configuration also does not let a caller redefine EQL's +wire fields. + +Implement configuration as a target capability, with the working name +`ConfigurableTarget`. The capability lowers into the existing operation +descriptions. Its associated options type has a data representation. Rust +factories return that type directly. Registered dynamic targets deserialise +into it. + +Validate and normalise the options once, when a reusable declaration is +built. Both paths run the target's own semantic validator before key I/O. +Use validated deserialisation, for example a wire helper plus `TryFrom`. Do +not use a derived deserialiser that bypasses the checked builders. If the +public type promises validated invariants, keep any raw options fields +private. Compile-time guarantees cover factory resolution and signatures, the -target/options association, and typed builder arguments. They do not imply -that a string is a valid JSON path or that the factory evaluates at compile -time. JSON path construction is currently fallible. Begin with a fallible -factory contract, `fn() -> Result`, with errors converted -to the engine's configuration error path. Propagate errors without panics -or `unwrap`; a later literal-path macro can add stronger static validation. - -Resolve the factory once per reusable configured declaration and reuse its -normalized options for writes and queries. A factory can read mutable -external state, so its type alone cannot prove a fixed policy; document -that fixed-policy providers must be stable. Do not independently reevaluate -the provider when generating a query. Intentionally variable configuration -belongs in an explicitly constructed configured-target instance. - -Unlike today's `Layout`, the general capability must not require one index: -targets may have none or multiple. Adapt the existing JSON layout internally -while migrating; do not expose two competing configured-target concepts. -The target catalog resolves registered targets for Go. Custom Rust targets -remain possible without registration, but cannot be dynamically addressed -until their adapter and descriptor are included in the guest build. +association between target and options, and typed builder arguments. They do +not mean that a string is a valid JSON path, or that the factory runs at +compile time. Building a JSON path can currently fail. + +Start with a fallible factory contract, +`fn() -> Result`, with errors converted to the engine's +configuration error path. Propagate errors without panics or `unwrap`. A +later macro for literal paths can add stronger static validation. + +Resolve the factory once for each reusable configured declaration, and reuse +its normalised options for writes and queries. A factory can read mutable +external state, so its type alone cannot prove a fixed policy. Document that +providers of a fixed policy must be stable. Do not evaluate the provider +again when generating a query. Configuration that is meant to vary belongs +in a configured-target instance that the caller constructs explicitly. + +Unlike today's `Layout`, the general capability must not require exactly one +index: a target can have none, or several. Adapt the existing JSON layout +internally during the migration. Do not expose two competing concepts of a +configured target. + +For Go, the target catalog resolves registered targets. Custom Rust targets +remain possible without registration. The dynamic interface can address them +only once the guest build includes their adapter and descriptor. ## Candidate static syntax `target(options = payroll_options)` names a Rust factory for the inner target of a transparent newtype. The inner Rust type selects the output -representation and associated options type. The spelling is proposed; -typed factories for Rust and typed deserialization for dynamic callers are -the revised recommendation. +representation and the associated options type. The spelling is a proposal. +The revised recommendation is typed factories for Rust, and typed +deserialisation for dynamic callers. ```rust use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; @@ -192,15 +315,17 @@ fn payroll_options() -> Result { struct EncryptedPayroll(EqlEncryptedJsonTree); ``` -The factory body uses today's `JsonOptions` builder; the derive support is -proposed. The derive calls the factory and type-checks its result against -the inner target's associated options type. It knows neither JSON path -semantics nor JSON-specific option names, and does not serialize or -deserialize the factory's result. The configured target constructs its -output, which the derive wraps. Serde transparency preserves that output's -wire envelope; SQL driver traits are a separate integration requirement. -The same factory attribute should work on a field of a derived output -record, with existing context selection rules. +The factory body uses today's `JsonOptions` builder. The derive support is +proposed. The derive calls the factory, and type-checks its result against +the inner target's associated options type. The derive does not depend on +the meaning of JSON paths, or on JSON-specific option names. It does not +serialise or deserialise the factory's result. + +The configured target constructs its output, and the derive wraps it. Serde +transparency keeps that output's wire envelope. SQL driver traits are a +separate integration requirement. The same factory attribute should work on +a field of a derived output record, under the existing rules for selecting +context. ```rust let encrypted: EncryptedPayroll = payroll @@ -212,14 +337,16 @@ let restored = serde_json::Value::decrypt_from_with_context( ).await?; ``` -Fixed policy is a recipe for writes and queries. Transparent deserialization -does not attest that an existing row was created with that policy. In -particular, the full ordered-path policy is not stored in the JSON header. +A fixed policy is a recipe for writes and queries. Transparent +deserialisation does not prove that an existing row was created with that +policy. In particular, the JSON header does not store the full policy of +ordered paths. ## Candidate dynamic representation and Go requirements -Implemented by #1372: string targets and configured object targets share -the native options validator. Use the landed `JsonSearch` identifier: +Pull request #1372 implements this: string targets and configured object +targets share the native options validator. Use the merged `JsonSearch` +identifier: ```json { @@ -234,44 +361,57 @@ the native options validator. Use the landed `JsonSearch` identifier: ``` Keep `"target": "TextEq"` as shorthand for that target's default options. -The object form and JSON target options are now implemented. -Reject unknown options, incompatible options, unsupported target names, and -invalid paths during declaration validation. Do not silently fall back to -defaults. Specify one consistent JSON wire format and context encoding -across the Rust, dynamic, Go, and database paths. +The object form and the JSON target options are now implemented. Reject +unknown options, incompatible options, unsupported target names, and invalid +paths during declaration validation. Do not fall back to defaults silently. +Specify one consistent JSON wire format and context encoding across the +Rust, dynamic, Go, and database paths. Go's generator and policy API must represent these same options, emit the -same declaration, and obtain the EQL value from the guest. Go must not -assemble the EQL envelope or reproduce JSON indexing. Discovery must expose -the options contract and capabilities; extend the existing target catalog -and resolver rather than installing a parallel JSON registry. -The dynamic registry invokes the concrete target options deserializer; it -does not look up or invoke the Rust application's factory by name. A Go -caller supplies equivalent options data. If a Rust policy must be exported, -serialize its resolved options, not its function identifier. This contract -does not require all Rust-only targets to depend on Serde; data conversion -is required for targets registered with the dynamic interface. - -The catalog/guest, JSON value kind, strict JSON text parser and generated -Go target options are now implemented. #1373 adds operation input shapes and -operand encodings, and #1378 adds typed Go document adapters. Remaining -work includes the formal versioned specification, generalized definition -lifecycle/caching, direct typed EQL conversion and context-role separation. -The current data-plan grammar is still a field map; the illustrative -versioned schema below is not already accepted by the runtime. - -Writes and queries must reuse the normalized configured target in Rust and +same declaration, and get the EQL value from the guest. Go must not assemble +the EQL envelope or reproduce JSON indexing. Discovery must expose the +options contract and the capabilities. Extend the existing target catalog +and resolver instead of installing a parallel JSON registry. + +The dynamic registry calls the concrete target's options deserialiser. It +does not look up or call the Rust application's factory by name. A Go caller +supplies equivalent options as data. If a Rust policy must be exported, +serialise its resolved options, not the function's identifier. This contract +does not require every Rust-only target to depend on Serde: only targets +registered with the dynamic interface need data conversion. + +These are now implemented: + +- the catalog and guest; +- the JSON value kind; +- the strict JSON text parser; +- the generated Go target options. + +Pull request #1373 adds operation input shapes and operand encodings. Pull +request #1378 adds typed Go document adapters. Remaining work includes: + +- the formal versioned specification; +- a general lifecycle and caching for definitions; +- direct typed EQL conversion; +- separating the roles within context. + +The current grammar for data plans is still a map of fields. The runtime +does not yet accept the illustrative versioned schema later in this plan. + +Writes and queries must reuse the normalised configured target in Rust and Go. An ordering query on an undeclared path must fail before key I/O. -Containment, path equality, ordering, and extraction need capability metadata -that can express their different operand/result shapes, not a single -assumed scalar query twin. Decryption must use stored cryptographic metadata -and context, not require the caller to remember the write's ordered paths. -## Accepted decrypt surface and return ownership +Containment, path equality, ordering, and extraction need capability +metadata that can express their different shapes of operand and result. The +metadata cannot assume a single scalar query counterpart. Decryption must +use the stored cryptographic metadata and context. It must not require the +caller to remember the write's ordered paths. -**Decision, 2026-10-25:** retain ordinary keyset-directed decryption for one -opaque ciphertext and destination-directed `DecryptFrom` for declared -targets/layouts. Both paths have context-free and context-taking variants: +## Accepted decrypt interface and return ownership + +Decided on 25 October 2026: keep ordinary keyset-directed decryption for one +opaque ciphertext. Keep destination-directed `DecryptFrom` for declared +targets and layouts. Both paths have variants with and without context: ```rust let value: i32 = keyset.decrypt(encrypted).await?; @@ -282,44 +422,52 @@ let plaintext = User::decrypt_from_with_context(encrypted, &keyset, context) .await?; ``` -These are independent call examples, not sequential reuse of a consumed -ciphertext. Direct decrypt mirrors direct encrypt; `DecryptFrom` mirrors -target-directed `EncryptInto`. The existing input capabilities still govern -whether omission of context is available. Do not add a competing public -ciphertext-receiver `decrypt_into` method. +These are separate call examples. They do not reuse one consumed ciphertext +in sequence. Direct decrypt mirrors direct encrypt, and `DecryptFrom` +mirrors the target-directed `EncryptInto`. The existing input capabilities +still decide whether a caller can leave out context. Do not add a competing +public `decrypt_into` method that takes the ciphertext as its receiver. Successful decryption returns an ordinary owned `T: Zeroize`. Stack guards -recovered bytes and intermediate plaintext while it owns them; successful -return transfers responsibility for the value to the caller. `Zeroize` does -not automatically wipe that returned value on drop. Support explicitly -requested drop-guarded results without requiring a wrapper around every -decrypted application struct; the precise opt-in adapter is implementation -design within this decision. Guarding the completed value does not resolve -partial-deserialization cleanup, which remains required work. - -This reads as constructing `User` from ciphertext. Calling -`encrypted.decrypt_from(...)` would conflict with the ordinary From/Into -receiver convention; retain the agreed destination receiver. -The destination-directed pair already exists on the blanket `DecryptFrom` -extension trait. Today's ciphertext method is -`encrypted.decrypt_into(cipher, context)`, also on `DecryptFrom`, and has -no separate context-free overload. The current declaration trait -`DecryptInto

` is another layer; do not rename it accidentally while -settling method ergonomics. Consolidate declaration and convenience-trait -naming with review issue 5. The public examples use the destination-directed -pair; remove the competing ciphertext method when implementing this surface. +the recovered bytes and intermediate plaintext for as long as they belong to +Stack. A successful return transfers responsibility for the value to the +caller. `Zeroize` does not automatically wipe that returned value on drop. + +Support results with a drop guard when the caller explicitly asks for one, +without requiring a wrapper around every decrypted application struct. The +exact opt-in adapter is implementation design within this decision. Guarding +the completed value does not solve cleanup after a partial deserialisation. +That cleanup remains required work. + +`User::decrypt_from(...)` reads as constructing a `User` from a ciphertext. +Calling `encrypted.decrypt_from(...)` would conflict with the usual From and +Into receiver convention, so keep the agreed destination receiver. The +destination-directed pair already exists on the blanket `DecryptFrom` +extension trait. + +Today's ciphertext method is `encrypted.decrypt_into(cipher, context)`, also +on `DecryptFrom`. That method has no separate overload without context. The +current declaration trait `DecryptInto

` is a different layer, so do not +rename it by accident while settling how the methods are called. + +Settle the names of declaration traits and convenience traits together with +review issue 5. The public examples use the destination-directed pair. +Remove the competing ciphertext method when implementing this interface. ## Recommended public traits and context input capabilities -Status: direction accepted by the user on 2026-10-24: separate conversion -methods from `EncryptTarget`/`DecryptSource` declarations, and use explicit -caller-context capabilities instead of `Default`. Implementation, complete -signatures, and call-site updates remain pending. This section resolves the -direction of work areas A and B below; earlier examples retain historical -derive names until their migration. +Status: the user accepted this direction on 24 October 2026. Separate the +conversion methods from the `EncryptTarget` and `DecryptSource` +declarations. Use explicit capabilities for caller context instead of +`Default`. Implementation, complete signatures, and updates to call sites +remain pending. + +This section settles the direction of work areas A and B, later in this +plan. Earlier examples keep their historical derive names until they are +migrated. -Keep two user-facing conversion traits, implemented by blanket library -implementations: +Keep two user-facing conversion traits, both provided by blanket +implementations in the library: | Trait | Public methods | Receiver | | --- | --- | --- | @@ -327,13 +475,14 @@ implementations: | `DecryptFrom` | `decrypt_from`, `decrypt_from_with_context` | Plaintext destination type; ciphertext supplied as an argument | These methods run declarations through the common engine and return the -existing batched awaitable. Target implementers do not override execution. -Ordinary `keyset.encrypt` and `encrypt_with_context` remain inherent methods -and do not gain a plaintext-side `encrypt` synonym. Their accepted decrypt -counterparts are `keyset.decrypt` and `decrypt_with_context`. +existing batched awaitable value. Target implementers do not override +execution. The ordinary `keyset.encrypt` and `encrypt_with_context` remain +inherent methods, and do not gain an `encrypt` synonym on the plaintext. +Their accepted decrypt counterparts are `keyset.decrypt` and +`decrypt_with_context`. -Give the declaration traits role names, rather than From/Into names that -promise conversion methods they do not define: +Give the declaration traits names that state their role. From and Into names +would promise conversion methods that these traits do not define: | Proposed declaration | Implemented on | Describes | | --- | --- | --- | @@ -341,59 +490,70 @@ promise conversion methods they do not define: | `target::DecryptSource

` | Recoverable encrypted output | How this stored value yields `P` | The corresponding derives should use the same names. This replaces today's -declaration traits `EncryptFrom

` and `DecryptInto

`; it does not remove -the retained `EncryptInto` extension trait. A term-only target implements -only `EncryptTarget`: no decrypt method that always fails. Plaintext visitor -traits remain a separate Stack-owned representation layer; moving/renaming -them must retain a single canonical trait implementation and a consistent -wire contract across the new consumers; prototype compatibility is not needed. -Configured-target options are another associated capability, not context. - -Declaration methods continue returning core-owned operation descriptions. -Encryption declarations do not receive the plaintext and cipher together; -decrypt declarations inspect stored output and select its opening, without -performing key I/O. Plans, derives, and dynamic resolution reuse those -descriptions. Keep the declaration module public for custom targets and -keep normal method imports/derives available from the Stack entry point. - -Use `ContextInput` as the associated-type name on each declaration trait. -It describes only what the caller must or may supply. It is not the complete -cryptographic context, which can also include a prefix and fields selected -from plaintext or stored output. Encrypt and decrypt declare this type -independently. - -Recommend library-owned input wrappers with explicit capabilities: - -| Proposed input | Omitted argument | Supplied argument | Meaning | +declaration traits `EncryptFrom

` and `DecryptInto

`. It does not remove +the retained `EncryptInto` extension trait. A target that only produces +terms implements only `EncryptTarget`, so it has no decrypt method that +always fails. + +Plaintext visitor traits remain a separate representation layer that Stack +owns. Moving or renaming them must keep a single canonical trait +implementation, and a consistent wire contract across the new consumers. +Compatibility with the prototype is not needed. Configured-target options +are another associated capability, not context. + +Declaration methods continue to return operation descriptions that the core +owns. Encryption declarations do not receive the plaintext and the cipher +together. Decrypt declarations inspect the stored output and select how to +open it, without any key I/O. Plans, derives, and dynamic resolution reuse +those descriptions. Keep the declaration module public for custom targets, +and keep the normal method imports and derives available from the Stack +entry point. + +Use `ContextInput` as the name of the associated type on each declaration +trait. `ContextInput` describes only what the caller must or can supply. It +is not the complete cryptographic context. The complete context can also +include a prefix, and fields selected from the plaintext or the stored +output. Encrypt and decrypt declare this type independently. + +Recommended: input wrappers that the library owns, each with explicit +capabilities: + +| Proposed input | Argument left out | Argument supplied | Meaning | | --- | --- | --- | --- | -| `NoCallerContext` | Allowed | Not accepted | Declaration derives all context and exposes no caller input | -| `RequiredContext` | Unavailable | Required `C` | Supplies context or the missing component selected by the declaration | -| `OptionalContext` | Allowed | Accepted `C` | For ordinary ciphertext: omission means empty context; supplied context is used directly | -| `ContextExtension` | Allowed | Accepted `C` | Omission preserves the declared base; supply extends it | -| `ExpectedContext` | Allowed | Accepted `C` | Omission reconstructs stored context; supply additionally checks it | - -`OptionalContext` does not append a synthetic empty prefix, and omission -of an extension does not append `()`. `ExpectedContext` never extends AAD. -The table specifies behavior; names may be refined without changing it. -Nonempty requirements belong to the accepted argument type or validated -resolved context, not to whether an input is optional. Search targets still -require a nonempty effective context. Ordinary scalar ciphertext should -accept the same empty context through both keyset and target-directed APIs. - -No-context methods require an explicit, sealed `AllowsNoContext` capability; -`_with_context` methods require a corresponding `AcceptsContext` capability -with its accepted argument type. Neither is inferred from `C: Default`. -`RequiredContext` consequently remains required even though `u64` -defaults to zero. The helpers accept ordinary argument values (or ergonomic -checked conversions); users do not manually wrap `RequiredContext` or -`ExpectedContext` at every call. Conversion errors propagate before key I/O. - -The core-owned wrappers carry presence and intent; target declarations -perform source selection and context composition. For composite required -input, `C` may be a tuple or named struct; it does not have to be the full -logical context. Preserve existing grouping rules and context encodings. - -Examples of independent encrypt/decrypt requirements: +| `NoCallerContext` | Allowed | Not accepted | The declaration derives all context and takes no caller input | +| `RequiredContext` | Not available | Required `C` | Supplies the context, or the missing component that the declaration selects | +| `OptionalContext` | Allowed | Accepted `C` | For an ordinary ciphertext, leaving it out means empty context, and a supplied context is used directly | +| `ContextExtension` | Allowed | Accepted `C` | Leaving it out keeps the declared base, and supplying it extends the base | +| `ExpectedContext` | Allowed | Accepted `C` | Leaving it out rebuilds the stored context, and supplying it also checks that context | + +`OptionalContext` does not append a synthetic empty prefix, and leaving out +an extension does not append `()`. `ExpectedContext` never extends the AAD. +The table specifies behaviour, and the names can change without changing it. +A requirement for nonempty input belongs to the accepted argument type or to +the validated resolved context, not to whether the input is optional. + +Search targets still require a nonempty effective context. Ordinary scalar +ciphertext should accept the same empty context through both the keyset API +and the target-directed API. + +Methods without context require an explicit, sealed `AllowsNoContext` +capability. `_with_context` methods require a matching `AcceptsContext` +capability with its accepted argument type. Neither capability is inferred +from `C: Default`. So `RequiredContext` stays required, even though +`u64` defaults to zero. + +The helper methods accept ordinary argument values, or convenient checked +conversions. Users do not wrap values in `RequiredContext` or +`ExpectedContext` by hand at every call. Conversion errors propagate before +key I/O. + +The wrappers that the core owns record presence and intent. Target +declarations select the sources and compose the context. For composite +required input, `C` can be a tuple or a named struct. It does not have to be +the full logical context. Keep the existing grouping rules and context +encodings. + +Examples where encrypt and decrypt have different requirements: | Declaration | Encrypt input | Decrypt input | | --- | --- | --- | @@ -402,183 +562,247 @@ Examples of independent encrypt/decrypt requirements: | Source row ID not recoverable before decrypt | `NoCallerContext` | `RequiredContext` | | Static field base with optional unstored extension | `ContextExtension` | `ContextExtension` | -In the last row, decrypting an extended ciphertext requires supplying the -same extension; availability of the omitted method is not a promise that -it opens every ciphertext of that Rust type. A derive can expose an optional -extension instead of `NoCallerContext` where its declaration explicitly -allows that behavior. Expected-context checks do not authenticate stored -data themselves: successful cryptographic opening remains necessary. +In the last row, decrypting a ciphertext that has an extension requires the +same extension. That the method without context is available does not +promise that it opens every ciphertext of that Rust type. Where a +declaration explicitly allows it, a derive can expose an optional extension +instead of `NoCallerContext`. Expected-context checks do not authenticate +stored data themselves. Successful cryptographic opening is still necessary. The dynamic plan must encode the same requirements, source selectors, and -storage rules. Rust enforces omitted/supplied method availability with trait -bounds; the engine enforces dynamic input presence and validity before key -I/O. Omitted input, explicit zero, and an empty supplied value are distinct. -Go generation can expose convenient signatures using that metadata, but -the guest remains authoritative for runtime requests. - -Implementation proof points: compile-fail required-input calls even for -`C: Default`, reject context for a no-input-only target, preserve optional -extension versus expectation semantics, and execute all nine directional -scenarios through Rust and dynamic plans. Preserve owned-source support -and batching; this trait separation must not add a blanket `Clone` bound. -The final signatures must show where borrowed and owned sources are -supported rather than concealing the existing difference behind the rename. +storage rules. Rust uses trait bounds to enforce which methods are +available, with or without context. The engine enforces the presence and +validity of dynamic input before key I/O. Omitted input, an explicit zero, +and an empty supplied value are three different things. Go generation can +expose convenient signatures from that metadata, but the guest stays +authoritative for runtime requests. + +The implementation must prove these points: + +- a call that leaves out required input fails to compile, even when + `C: Default`; +- a target that accepts only no input rejects context; +- an optional extension and an expected context keep their different + meanings; +- all nine directional scenarios in the [context + examples](2026-10-24-stack-encrypt-context-examples.md) run through Rust + and through dynamic plans. + +Keep support for owned sources, and keep batching. This separation of traits +must not add a blanket `Clone` bound. The final signatures must show the +support for borrowed sources and for owned sources. They must not hide the +existing difference behind the rename. ## Go JSON implementation and PR #1353 -The [Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) -has been updated against merged `next`. Tracking issue #1347 and steps -#1348–#1352 are closed. #1372, #1373 and #1378 implement the catalog, -configured JSON targets, dynamic execution, operation-specific operands, -request-bound opening and generated typed Go frontend; #1402 supplies -current docs and the executable database example. - -The shipped target identifier is `JsonSearch`; Go uses `eql.JSONSearch`, -`JSONSearchQuery`, `JSONSearchLeafQuery` and `JSONEntry`. Do not impose the -proposed `EncryptedJson` glossary as a pre-release rewrite. Preserve the -role distinctions and freeze actual customer names before later Rust renames. -Prepared Go reads retain their cipher and use `Open(ctx, rows)`. The engine -rederives trusted selectors/read keys from the original request. No new -persistent guest-handle protocol is needed to implement that verification. - -Residual requirements must be explicit: SQL children/counts exist but a -parent-bound plaintext child opener does not; selector opening accepts at -most one entry per document group. Use subtree reads for all-match plaintext -retrieval. General scope/row-binding inputs remain unsupported for EQL -targets. The existing column/keyset profile can ship without claiming them. - -The JSON contract now uses randomized projection order and cascade-bound -ciphertexts. Semantic parity and cross-opening are required; raw term byte -equality is meaningful only with controlled test randomness. Go typed JSON -adapters follow `encoding/json`, while raw JSON goes through the strict -engine parser. Do not claim arbitrary Go encoders preserve every input byte. +The [Go JSON +plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +has been updated against merged `next`. Tracking issue #1347 and steps #1348 +to #1352 are closed. #1372, #1373, and #1378 implement: + +- the catalog; +- configured JSON targets; +- dynamic execution; +- operands specific to each operation; +- opening that Rust verifies against the original request; +- the generated typed Go front end. + +Pull request #1402 supplies current documentation and the executable +database example. + +The released target identifier is `JsonSearch`. Go uses `eql.JSONSearch`, +`JSONSearchQuery`, `JSONSearchLeafQuery`, and `JSONEntry`. Do not impose the +proposed `EncryptedJson` glossary as a rewrite before release. Keep the +distinctions between roles, and freeze the actual customer names before +later Rust renames. + +Prepared Go reads keep their cipher and use `Open(ctx, rows)`. The engine +derives the trusted selectors and read keys again from the original request. +No new protocol for persistent guest handles is needed to implement that +verification. + +State the remaining requirements explicitly: + +- SQL for children and child counts exists, but there is no opener for + plaintext children that is bound to their parent. +- Opening by selector accepts at most one entry for each document group. +- Use subtree reads to retrieve plaintext for all matches. +- General inputs for search scope and row binding remain unsupported for EQL + targets. The existing column and keyset profile can be released without + claiming them. + +The JSON contract now uses randomised projection order and ciphertexts bound +by cascaded tags. Semantic parity and cross-opening are required. Comparing +raw term bytes is meaningful only with controlled test randomness. Go typed +JSON adapters follow `encoding/json`, while raw JSON goes through the +engine's strict parser. Do not claim that arbitrary Go encoders preserve +every input byte. ## Prior query design: baseline and reconciliation -**Decision update, 2026-10-25:** ORE does not answer equality in the initial -API, for any plaintext type. Equality uses its declared equality-capable -index. Keep capability selection parameterized by the index, plaintext -encoding/domain and options, so a future explicit exact/injective integer -encoding can opt into equality without changing query syntax or wire -operator identity. Never infer `Eq` merely from ordering support: text and -other lossy encodings cannot establish plaintext equality. A future opt-in -must retain overlap rejection (one selected provider per operator) and -must not silently invalidate existing `(Equality, Ore)` definitions. This -policy does not disable JSON scalar equality, which uses value selectors -rather than ORE/OPE terms. - -Read in full on 2026-10-25: the user-supplied **Stack Encrypt query design**, -dated 2026-10-21, at -`/Users/dan/Downloads/Stack Encrypt query design.md` (654 lines). The source -document was not modified. It is explicitly a discussion draft, based on -an earlier `next` snapshot; its historical code/release claims were not -revalidated as part of this reading. Use it as the query-design baseline, -not as a settled specification or a reason to redesign queries from scratch. +### No ORE equality in the first API + +Decided on 25 October 2026: ORE does not support equality in the first API, +for any plaintext type. Equality uses the field's declared index that +supports equality. Parameterise capability selection by the index, by the +encoding or domain of the plaintext, and by the options. A future explicit +exact or injective integer encoding can then opt in to equality, without +changing the query syntax or the operator's wire identity. + +Never infer `Eq` only from support for ordering. Text and other lossy +encodings cannot establish plaintext equality. A future opt-in must keep the +rejection of overlaps, with one selected provider for each operator. It must +not silently invalidate existing `(Equality, Ore)` definitions. + +This policy does not disable JSON scalar equality. JSON scalar equality uses +value selectors, not ORE or OPE terms. + +### The query design draft of 21 October 2026 + +On 25 October 2026, the whole user-supplied **Stack Encrypt query design** +draft was read. The draft is dated 21 October 2026. It is 654 lines long, at +`/Users/dan/Downloads/Stack Encrypt query design.md`. The source document +was not changed. + +The document is explicitly a discussion draft, based on an earlier snapshot +of `next`. Its historical claims about code and releases were not checked +again as part of this reading. + +Use the draft as the baseline for query design. It is not a settled +specification, or a reason to redesign queries from scratch. ### Existing proposal to carry forward -- The caller selects an operator and a typed field/target handle. The field - supplies its own configured index; remove `DefaultIndex` and caller `.via`. -- Each field has at most one index answering each operator. Refuse overlaps - at declaration validation instead of choosing an index by priority. -- Declare operators, index capabilities, and admitted source types once; - derive or verify typed lookup, dynamic lookup, and discovery from that +- The caller selects an operator and a typed field or target handle. The + field supplies its own configured index. Remove `DefaultIndex` and the + caller's `.via`. +- Each field has at most one index that supports each operator. Refuse + overlaps at declaration validation, instead of choosing an index by + priority. +- Declare operators, index capabilities, and admitted source types once. + Derive or verify typed lookup, dynamic lookup, and discovery from that declaration. Avoid another capability table in EQL. - Share query derivation and refusals through the same semantic functions. - In particular, tokenize match input once and retain the tokens needed for + In particular, tokenise match input once, and keep the tokens needed for probabilistic verification in the query object. -- Retain index/source types in typed field handles. Unsupported operators - are compile errors on the typed path and validated errors dynamically. -- Model analyzer case behavior once in its stages, rather than restating - it in both the match configuration and the operator. - -The draft already proposes the query call -`cipher.query(op::Matches, text).on(&field).await?`, typed -`FieldRef` handles, and a loaded configuration/analyzer owned by the dynamic -plan (`Loaded` / `Arc`). Those are substantive designs to reuse; -they do not yet settle how our output-derived typed factory options are -initialized and shared with convenient `encrypt_into` calls. +- Keep index and source types in typed field handles. Unsupported operators + are compile errors on the typed path, and validation errors on the dynamic + path. +- Model the analyser's handling of letter case once, in its stages, instead + of stating it in both the match configuration and the operator. + +The draft already proposes three designs to reuse: + +- the query call `cipher.query(op::Matches, text).on(&field).await?`; +- typed `FieldRef` handles; +- a loaded configuration and analyser that the dynamic plan owns (`Loaded` + and `Arc`). + +These designs do not yet settle how our typed factory options, derived from +the output, are initialised and shared with convenient `encrypt_into` calls. ### Reconciliation with this review 1. **Generate handles from the same output-target declaration.** The query - appendix proposes a plaintext-side `EncryptionPlan` derive and generated - encrypted counterparts. Our agreed starting point is output-side - `EncryptTarget`/`DecryptSource` derives. Recommend emitting typed handles - from that declaration, without separately restating indexes on plaintext. - A source-side convenience derive can later generate the same target. - Hand-written plans remain lower-level composition, not mandatory usage. -2. **Resolve typed factory ownership.** A generic `Match` can fix a policy - identity in its type, but an arbitrary factory can still compute different - values on different calls. Carry a resolved configuration instance through - writes and field handles; settle the owner/initialization API. The draft's - dynamic plan ownership is useful precedent, not proof that all static - factory results are compile-time constants. + appendix proposes an `EncryptionPlan` derive on the plaintext side, and + generated encrypted counterparts. Our agreed starting point is the + output-side `EncryptTarget` and `DecryptSource` derives. Recommended: + emit typed handles from that declaration, without restating indexes + separately on the plaintext. A convenience derive on the source side can + later generate the same target. Hand-written plans remain lower-level + composition, not required usage. +2. **Settle who owns the results of typed factories.** A generic `Match` + can fix a policy identity in its type. An arbitrary factory can still + compute different values on different calls. Carry a resolved + configuration instance through writes and field handles, and settle the + API for its owner and its initialisation. The draft's ownership in the + dynamic plan is useful precedent. That precedent does not prove that all + static factory results are compile-time constants. 3. **Separate portable options from execution types.** JSON's Rust factory - and dynamic deserializer can construct the same `JsonOptions`. The draft's - `Match` and `Match` intentionally have different execution - types to accommodate generic/custom analyzer stages. Preserve shared - validated options and semantic implementations for the portable subset; - do not force all typed analyzers through a serializable stage enum. -4. **Accepted portability scope.** Ship built-in advertised - capabilities in Rust and Go together. Downstream custom Rust stages can - remain explicitly typed-only with `NotPortable` until registered. This - qualifies the draft's blanket "lands in Rust first" rule; it must not - become permission to defer the Go JSON capability under this plan. + and its dynamic deserialiser can construct the same `JsonOptions`. To + allow generic and custom analyser stages, the draft's `Match` and + `Match` deliberately have different execution types. Keep shared + validated options and semantic implementations for the portable subset. + Do not force all typed analysers through a serialisable enum of stages. +4. **Accepted portability scope.** Release built-in advertised capabilities + in Rust and Go together. Downstream custom Rust stages can stay + explicitly typed-only, with `NotPortable`, until they are registered. + This qualifies the draft's blanket "lands in Rust first" rule. This + qualification must not become permission to defer the Go JSON capability + under this plan. 5. **Include JSON capabilities before fixing lookup representations.** The - draft's 16-shape dynamic enum covers equality, match, and scalar ordering, - not the new JSON target. Extend operator identity, operand/result shapes, - capability validation, and overlap rules for containment, `AtPath`, and - extraction. Its interim validated index list using one capability table - is a viable starting point; the specific closed enum is not a decision. -6. **Use the agreed context/scope model.** Replace label-only assumptions - with separate ciphertext binding and search scope. Row binding does not - flow into search terms; `PrimaryKey` is context-eligible and `ScopeKey` - is also scope-eligible. Apply the same declared scope to index generation - and queries. Preserve the requirement to return or supply row context - when decrypting selected results. -7. **Accepted validation timing.** All checks independent of keys precede - key resolution/loading, including query-input checks. The draft's possible - named-keyset resolution before those checks is superseded. Reuse the - same validator for Go's `check_query` and execution rather than copying - it or tokenization. Provider facts and cryptographic checks that require - keys remain later stages. Explicit earlier client initialization/keyset - calls are not retroactively prevented by this operation-validation rule. -8. **Apply the user's prototype scope.** Legacy match-option decoding, - deprecation/release-preservation requirements, and frozen Go wire formats - in the older document are not constraints here. Keep new Rust/Go/EQL - consumers aligned. The appendix's `cipher.encrypt(plan, value)` and - plaintext derive are illustrative older proposals, not replacements for - our agreed entry points. - -ORE/Eq is now decided as described above. Retain integration work on -typed field-handle construction/spelling, dynamic -`Index` versus a shared semantic-function boundary, feasibility -of compile-time match-option checks, Go's `check_query`, and ADR updates. -Type-safe stage ordering does not by itself validate runtime numeric values -or string paths; stronger const-check claims require implementation proof. -Use these as specific integration decisions rather than an unbounded query -API task. Coordinate their Go surface with PR #1353. + draft's dynamic enum of 16 shapes covers equality, match, and scalar + ordering, but not the new JSON target. For containment, `AtPath`, and + extraction, extend the operator identity, operand shapes, result shapes, + capability validation, and overlap rules. The draft's interim validated + index list, using one capability table, is a workable starting point. Its + specific closed enum is not a decision. +6. **Use the agreed model for context and scope.** Replace the assumption + that context is only a label with separate ciphertext binding and search + scope. Row binding does not flow into search terms. `PrimaryKey` is + eligible for context only. `ScopeKey` is eligible for both context and + scope. Apply the same declared scope to index generation and to queries. + Keep the requirement to return or supply row context when decrypting + selected results. +7. **Accepted validation timing.** All checks that do not need keys come + before keys are resolved or loaded, including checks on query input. This + replaces the draft's possible resolution of a named keyset before those + checks. Reuse the same validator for Go's `check_query` and for + execution, instead of copying it or the tokenisation. Provider facts, and + cryptographic checks that need keys, remain later stages. This rule for + validating operations does not forbid explicit earlier calls that + initialise a client or keyset. +8. **Apply the user's prototype scope.** These parts of the older document + are not constraints here: decoding legacy match options, deprecation + requirements, release-preservation requirements, and frozen Go wire + formats. Keep the new Rust, Go, and EQL consumers aligned. The appendix's + `cipher.encrypt(plan, value)` and plaintext derive are illustrative older + proposals. They do not replace our agreed entry points. + +ORE equality is now decided, as described in [No ORE equality in the first +API](#no-ore-equality-in-the-first-api). Integration work remains on: + +- how typed field handles are constructed and spelled; +- dynamic `Index` compared with a shared boundary of semantic + functions; +- the feasibility of checking match options at compile time; +- Go's `check_query`; +- updates to the architecture decision records (ADRs). + +Type-safe ordering of stages does not by itself validate runtime numeric +values or string paths. Stronger claims about const checks need proof in the +implementation. Treat these items as specific integration decisions, not as +an open-ended query API task. Coordinate their Go interface with PR #1353. ## Recommended owner: a reusable typed target definition -Status: recommendation for review. Use an immutable **target definition** -to own resolved options, prepared indexes/analyzers, context-selection rules, -and operation metadata. The output derive generates this definition's typed -field accessors from the same declaration that generates encryption. It -contains no plaintext, row values, credentials, or selected keyset. A field -handle borrows or shares that definition and retains its plaintext/index -types; it is not an independent name plus a newly constructed default index. - -**Selector naming, 2026-10-25:** the user finds encryption's `using` and -query's `on` confusing and requests considering one name. Recommend -`.using(...)` for both: each selects the definition or field whose declared -policy governs the operation. This spelling is proposed, not yet accepted. -Encryption still accepts the requested target's associated definition type; -query still requires `QueryTarget` on a selected field or -single-value definition. Shared naming does not weaken either bound or -make an entire multi-field record a query target. Older `on` examples in -the supplied query draft describe its earlier spelling. +Status: recommendation for review. Use an immutable target definition to +own: + +- the resolved options; +- prepared indexes and analysers; +- the rules for selecting context; +- the operation metadata. + +The output derive generates the definition's typed field accessors from the +same declaration that generates encryption. The definition contains no +plaintext, row values, credentials, or selected keyset. A field handle +borrows or shares that definition, and keeps its plaintext and index types. +It is not an independent name plus a newly constructed default index. + +### One selector name for encryption and queries + +On 25 October 2026, the user said that encryption's `using` and the query's +`on` are confusing, and asked for one name to be considered. Recommended: +`.using(...)` for both. In each case it selects the definition or field +whose declared policy governs the operation. This spelling is proposed, not +yet accepted. + +Encryption still accepts the requested target's associated definition type. +A query still requires `QueryTarget` on a selected field or +single-value definition. Sharing one name does not weaken either bound, or +make a whole multi-field record a query target. The older `on` examples in +the supplied query draft show its earlier spelling. Proposed usage, assuming the previously described `EncryptedUser` target: @@ -594,96 +818,111 @@ let query = cipher .await?; ``` -`definition()` obtains the shared resolved definition of that source/target -mapping. Plain `encrypt_into` obtains the same definition internally. It -does not independently invoke the options factory. `.email()` is generated -from the output field declaration, not a hand-maintained second schema. -The method name `definition` is proposed; it names a resolved declaration, -not a builder that users must reconstruct before every operation. +`definition()` gets the shared resolved definition of that mapping from +source to target. Plain `encrypt_into` gets the same definition internally, +and does not call the options factory separately. `.email()` is generated +from the declaration of the output field, not from a second schema that +someone maintains by hand. The method name `definition` is proposed. It +names a resolved declaration, not a builder that users must reconstruct +before every operation. ### Call order and strict typestate -User requirement, 2026-10-25: use strict typestate for the fluent API. The -user prefers the reading `cipher.using(&field).query(op, input)` and is open -to supporting both call orders. Recommend definition-first as the canonical -documented query form, while supporting operator-first through the same -request states: +On 25 October 2026, the user required strict typestate for the fluent API. +The user prefers the reading `cipher.using(&field).query(op, input)`, and is +open to supporting both call orders. Recommended: document the +definition-first form as the canonical query form, and support the +operator-first form through the same request states: ```rust let a = cipher.using(&users.email()).query(op::Eq, email).await?; let b = cipher.query(op::Eq, email).using(&users.email()).await?; ``` -Both are proposed. The current implementation is query-first, followed by -`.on(...)`; this recommendation changes that interface. +Both forms are proposed. The current implementation is query-first, followed +by `.on(...)`. This recommendation changes that interface. | Starting state | Method | Resulting state | | --- | --- | --- | -| Cipher | `using(&definition)` | Bound definition, no query selected | -| Bound definition | `query(op, input)` | Query with definition/operator/input known | -| Cipher | `query(op, input)` | Query awaiting a definition | -| Query awaiting a definition | `using(&definition)` | The same query state as the definition-first path | - -The transition that brings the definition and operator/input together has -the `QueryTarget` bound. A definition-first bound handle need not -pretend to know the future operator. Once combined, the index and context -requirements are determined. Only the executable state implements -`IntoFuture`; a bound definition alone, a query missing its definition, or -a query missing required context cannot be awaited. Required context is -completed by a typed context-input transition; optional/no-input cases -start with a valid omitted state. Context completion must not be represented -solely as an unchecked `Option` plus a runtime missing-input error on the -static path. - -Do not expose another `using` transition after a query's definition has -been selected, or another query-selection transition on the request. +| Cipher | `using(&definition)` | Definition bound, no query selected | +| Definition bound | `query(op, input)` | Query with its definition, operator, and input known | +| Cipher | `query(op, input)` | Query waiting for a definition | +| Query waiting for a definition | `using(&definition)` | The same query state as the definition-first path | + +The transition that brings the definition together with the operator and +input has the `QueryTarget` bound. A handle bound in the +definition-first order does not need to fix the future operator in advance. +Combining the two determines the index. It also determines the context +requirements. + +Only the executable state implements `IntoFuture`. A bound definition alone, +a query missing its definition, and a query missing required context cannot +be awaited. + +A typed context-input transition completes required context. Cases with +optional input or no input start in a valid state with context left out. On +the static path, completing context must not be represented only as an +unchecked `Option` plus a runtime error for missing input. + +Do not expose another `using` transition after a query's definition has been +selected, or another transition on the request that selects a query. Supporting these two orders does not promise every permutation of keyset, context, and operation selection. Specify those transitions separately, -starting with context supplied after its required type is known. Share -the final query request, validation, and execution implementation rather -than duplicate them per call order. Input-dependent errors such as an -invalid path still need runtime validation; typestate checks completeness -and capability, not arbitrary value semantics. - -The same discipline applies to encryption requests: a fixed-policy -`encrypt_into` has an implicit definition and can be awaited without -`using`; selecting an explicit compatible definition is a one-time -transition before execution. This does not add a new ordinary `encrypt` -receiver or require fixed-policy callers to bind definitions manually. - -Completion tests must include both query orders with identical semantics, -and compile-fail cases for unsupported operators/inputs, premature await, -missing required context, and repeated definition selection. Definition -borrows/shared handles must remain valid for the lifetime of the request. +starting with context supplied after its required type is known. + +Share the final query request, validation, and execution implementation, +instead of duplicating them for each call order. Errors that depend on +input, such as an invalid path, still need runtime validation. Typestate +checks completeness and capability, not the meaning of arbitrary values. + +The same rules apply to encryption requests. A fixed-policy `encrypt_into` +has an implicit definition, so a caller can await it without `using`. +Selecting an explicit compatible definition is a single transition before +execution. This does not add a new ordinary `encrypt` receiver, or require +callers with a fixed policy to bind definitions by hand. + +Completion tests must include: + +- both query orders, with identical semantics; +- compile-fail cases for unsupported operators and inputs, awaiting too + early, missing required context, and selecting a definition twice. + +Borrowed definitions and shared handles must stay valid for the lifetime of +the request. ### Fixed declarations -Use lazy, fallible initialization for the fixed definition, retaining each -successful result for reuse. Allow explicit startup initialization via -`EncryptedUser::definition()?`, while the shorthand write initializes it -on demand and returns any error through its awaitable. Factory errors must -not panic or perform key I/O; do not permanently cache initialization -failures. Coordinate concurrent first use so successful initialization is -performed once, and account for reentrant/recursive initialization in the -implementation rather than allowing a deadlock. - -The identity is the complete concrete declaration (including source/target -mapping and generic policy parameters), not merely the ciphertext's output -type. Different instantiations must not share a cache slot by accident. -This is an implementation requirement, not a claim that a static inside a -generic Rust function is automatically per-monomorphization. - -Fixed factories must describe stable application policy; they are not the -interface for reloading environment settings or tenant-specific options. -The cache makes writes and queries in one process share the resolved policy; -it does not prove another process or an older stored row uses that policy. -Per-call context, source values, and keyset choice remain separate inputs. +Initialise the fixed definition lazily, in a way that can fail, and keep +each successful result for reuse. Allow explicit initialisation at startup +through `EncryptedUser::definition()?`. The shorthand write initialises the +definition on demand, and returns any error through its awaitable value. +Factory errors must not panic or perform key I/O. + +Do not cache initialisation failures permanently. Coordinate concurrent +first use so that successful initialisation happens once. Handle reentrant +and recursive initialisation in the implementation, instead of allowing a +deadlock. + +The identity of a definition is the complete concrete declaration, including +the mapping from source to target and any generic policy parameters. It is +not only the ciphertext's output type. Different instantiations must not +share a cache slot by accident. This is an implementation requirement. It +does not claim that a static inside a generic Rust function is automatically +one per monomorphisation. + +Fixed factories must describe stable application policy. They are not the +interface for reloading environment settings, or for options that differ for +each tenant. The cache makes writes and queries in one process share the +resolved policy. It does not prove that another process, or an older stored +row, uses that policy. Per-call context, source values, and keyset choice +remain separate inputs. ### Runtime declarations -Load/create a distinct definition instance for runtime options. A configured -instance is not stored in the fixed-declaration cache. Use the same type of -definition and field handles, so queries and writes share that instance. +Load or create a separate definition instance for runtime options. A +configured instance is not stored in the fixed-declaration cache. Use the +same type of definition and field handles, so that queries and writes share +that instance. Proposed Rust spelling for a configurable library target: @@ -698,495 +937,625 @@ let encrypted: EqlEncryptedJsonTree = value Here `using` takes a resolved typed target definition, not a hand-built plan. This requires a lazy conversion request: select the definition before -building the operation or invoking a default factory. Its await lowers to -the existing `Pending`/batch executor; the shared engine remains unchanged. -The definition's output/source types constrain the conversion. Queries use -the same definition (or its field handles), supplying the required context -and keyset for that call. - -A fixed-policy wrapper such as `EncryptedPayroll` does not expose runtime -override of its declared options. Use the configurable underlying target -for the runtime case, avoiding silent overrides of a fixed policy. Exact -configuration APIs for multi-field runtime definitions are still to be -specified; they must select typed options without introducing another -execution mechanism. - -Go's loaded dynamic plan owns its resolved definitions and generated field -accessors reach those definitions. It has no Rust factory to invoke. The -guest validates and builds the configuration once per loaded declaration, -then writes/queries use the loaded instance. A different runtime policy is -a different instance; it does not mutate one used by in-flight operations. +building the operation or calling a default factory. Awaiting the request +lowers it to the existing `Pending` and batch executor, and the shared +engine does not change. The definition's output and source types constrain +the conversion. Queries use the same definition, or its field handles, and +supply the required context and keyset for that call. + +A fixed-policy wrapper such as `EncryptedPayroll` does not allow its +declared options to be overridden at runtime. Use the configurable +underlying target for the runtime case, so that nothing silently overrides a +fixed policy. The exact configuration APIs for runtime definitions with +several fields are still to be specified. They must select typed options +without introducing another execution mechanism. + +Go's loaded dynamic plan owns its resolved definitions, and generated field +accessors reach those definitions. Go has no Rust factory to call. The guest +validates and builds the configuration once for each loaded declaration, and +then writes and queries use the loaded instance. A different runtime policy +is a different instance. It does not change an instance that operations in +progress are using. ### Implementation proof and boundary -Prove the cache/handle design early with two fixed mappings, multiple generic -instantiations, concurrent first use, initialization failure/retry, and -separate runtime instances. Prove a query and its write use the same options -without calling the factory twice. Preserve source lifetimes, owned-source -support, batch lowering, and no-HTTP/WASM builds. Queries on a field without -the operator must remain compile errors, with dynamic equivalents checked -at declaration/query validation. - -Prefer this generated-definition approach over requiring callers to author -a second plan or putting configuration state in each ciphertext. Plans are -still the lower-level implementation of the definition. Portable option -serialization and index capability discovery remain part of its dynamic -contract; the definition object does not itself need to serialize arbitrary -Rust closures or custom analyzer code. +Prove the cache and handle design early with: + +- two fixed mappings; +- several generic instantiations; +- concurrent first use; +- an initialisation failure followed by a retry; +- separate runtime instances. + +Prove that a query and its write use the same options without calling the +factory twice. Keep the source lifetimes, the support for owned sources, +batch lowering, and the builds without HTTP and for WebAssembly. A query on +a field without the operator must remain a compile error, with dynamic +equivalents checked at declaration or query validation. + +Prefer this approach, with generated definitions, over requiring callers to +write a second plan or putting configuration state in each ciphertext. Plans +are still the lower-level implementation of the definition. Portable +serialisation of options and discovery of index capabilities remain part of +its dynamic contract. The definition object does not itself need to +serialise arbitrary Rust closures or custom analyser code. ## Single-ciphertext representation and plaintext ownership -Accepted decisions, 2026-10-25: use `OpaqueEncode` / `OpaqueDecode` for -single-ciphertext representation, replacing the working names `EncodeValue` -/ `DecodeValue`. Both direct `encrypt` and target-directed `encrypt_into` -consume their values and take responsibility for owned plaintext cleanup, -including their context-taking variants. These are representation -capabilities; the engine still performs cryptography. `EncryptTarget` / -`DecryptSource` describe how encrypted outputs and layouts compose those -operations. The lifecycle and cleanup scope below are part of the accepted -design. The plaintext cleanup bound is `Zeroize`; implementation remains -to be completed. +Accepted on 25 October 2026: + +- Use `OpaqueEncode` and `OpaqueDecode` to represent a value as a single + ciphertext. They replace the working names `EncodeValue` and + `DecodeValue`. +- Direct `encrypt` and target-directed `encrypt_into`, including their + context-taking variants, consume their values. Both operations take + responsibility for cleaning up owned plaintext. +- These are representation capabilities, and the engine still performs the + cryptography. `EncryptTarget` and `DecryptSource` describe how encrypted + outputs and layouts compose those operations. +- The lifecycle and cleanup scope below are part of the accepted design. +- The bound for plaintext cleanup is `Zeroize`. The implementation is not + finished. ### Owned input is the direct API contract `keyset.encrypt(value)` and its context-taking counterpart take an owned -single-value input. No implicit clone and no `Clone` bound. For example, -passing a `String` transfers it; a borrowed `&str` does not transfer the -storage it refers to and must not receive the same cleanup promise. Do not -add a silent borrowed-input fallback to the consuming operation. +single-value input. There is no implicit clone and no `Clone` bound. -**Decision, 2026-10-25:** require `Zeroize` for plaintext types and use a -library-owned guard. Use the existing ecosystem trait rather than defining -a new cleanup trait or alternative protected-ownership adapter for now. -`Zeroize` supplies the operation; the library must arrange to invoke it on -every supported exit path. A bound alone does not arrange cleanup on drop. +For example, passing a `String` transfers it. A borrowed `&str` does not +transfer the storage it refers to. So it must not receive the same cleanup +promise. Do not add a silent fallback to borrowed input in the consuming +operation. + +Decided on 25 October 2026: require `Zeroize` for plaintext types, and use a +guard that the library owns. For now, use that existing ecosystem trait, +instead of defining a new cleanup trait or another adapter for protected +ownership. `Zeroize` supplies the wiping operation, and the library must +arrange to call it on every supported exit path. A bound alone does not +arrange cleanup on drop. Acquire protected ownership synchronously when creating the encryption -request, before returning its future/builder. An `async fn` body is not run -until polling, so placing the first guard inside that body would leave an -unpolled request with an ordinarily dropped source. A sync constructor -returning the typed future can establish the guard first. Every subsequent -builder transition moves that guarded ownership rather than making a new -unprotected copy. - -The representation trait may encode through a temporary internal borrow of -the guarded value into a library-owned sensitive sink. Public consumption -does not require giving an encoder unrestricted ownership of the source or -an ordinary growable `Vec` for scratch work. Wipe and release the source -as soon as encoding no longer needs it; keep encoded plaintext guarded until -sealing completes. Preserve the single-leaf representation boundary. +request, before returning its future or builder. The body of an `async fn` +does not run until the future is polled. If the first guard were inside that +body, a request that is never polled would drop its source in the ordinary +way. A synchronous constructor that returns the typed future can set up the +guard first. Every later builder transition moves that guarded ownership, +instead of making a new unprotected copy. + +The representation trait can encode through a temporary internal borrow of +the guarded value, into a sensitive sink that the library owns. Public +consumption does not require giving an encoder unrestricted ownership of the +source, or an ordinary growable `Vec` for scratch work. Wipe and release +the source as soon as encoding no longer needs it. Keep the encoded +plaintext guarded until sealing completes. Keep the boundary of a +representation as a single leaf. ### Cleanup scope and implementation obligations -- Cover owned source storage, encoded buffers, codec scratch, and any - library-created plaintext copies. Use established non-elidable wiping - primitives/protected storage rather than `clear()` or ordinary drop alone. -- Cover success, encoding/configuration/KMS/AEAD errors, partially prepared - batches, request cancellation, and an unpolled request being dropped. - Use drop guards for ordinary unwinding too; process abort or a leaked - future is not a destructor guarantee. -- Avoid reallocations that abandon uncleared buffers; growing sensitive - sinks must have a cleanup strategy for old allocations. Prevent plaintext - from escaping in diagnostics, Debug, caches, and instrumentation. -- On decrypt, protect recovered bytes and partial decode state until cleanup - or successful ownership transfer to the caller. Successful returned - plaintext is intentionally retained; its subsequent lifetime is the - caller's responsibility unless the return type itself is protected. -- A Rust move does not prove all physical copies were erased. In particular, - `Copy` inputs leave caller-owned originals usable, and compiler temporaries - may exist. The enforceable API promise concerns supported owned storage - and library-managed artifacts, not caller-retained copies or erasure of - every register/stack remnant. Verify stronger claims with compiler evidence. +- Cover owned source storage, encoded buffers, codec scratch space, and any + plaintext copies that the library creates. Use established wiping + primitives that the compiler cannot remove, or protected storage. Do not + rely on `clear()` or an ordinary drop alone. +- Cover success, errors in encoding, configuration, KMS, and AEAD, partly + prepared batches, cancelled requests, and a request dropped before it is + polled. Use drop guards for ordinary unwinding too. A process abort or a + leaked future does not run destructors, so neither is covered. +- Avoid reallocations that leave uncleared buffers behind. Sensitive sinks + that grow must have a cleanup strategy for their old allocations. Prevent + plaintext from escaping in diagnostics, `Debug` output, caches, and + instrumentation. +- On decrypt, protect the recovered bytes and partial decode state until + cleanup, or until ownership passes successfully to the caller. + Successfully returned plaintext is kept on purpose. After the return, its + lifetime is the caller's responsibility, unless the return type itself is + protected. +- A Rust move does not prove that all physical copies were erased. In + particular, `Copy` inputs leave the caller's originals usable, and + compiler temporaries may exist. The API promise that can be enforced + covers supported owned storage and the artefacts that the library manages. + It does not cover copies that the caller keeps, or every remnant in + registers and on the stack. Check stronger claims with evidence from the + compiled code. The [zeroize documentation](https://docs.rs/zeroize/latest/zeroize/) -documents drop-based guards, non-elidable wiping, and limits involving -moves, stack copies, and reallocations. The source currently uses -`Protected>` at the AEAD boundary, but that does not establish -end-to-end cleanup of every caller value and intermediate. This discussion -is a design requirement, not a completed source/assembly zeroization audit. +describes guards based on drop, wiping that the compiler cannot remove, and +the limits involving moves, stack copies, and reallocations. The source +currently uses `Protected>` at the AEAD boundary. That does not +establish cleanup of every caller value and intermediate from end to end. +This discussion is a design requirement, not a completed audit of +zeroisation in the source or the assembly. ### Consequences for blobs, layouts, and Go -`OpaqueBlob` cannot promise source cleanup from `T: Serialize` alone. -Require `T: Serialize + Zeroize` for the proposed Serde-backed blob input; -owned blob decoding requires `T: DeserializeOwned + Zeroize`. Ordinary -application structs may derive `Zeroize`. A -generic Serde implementation can allocate or retain hidden copies, so strong -claims also require a supported codec/implementation contract; wrapping its -final byte buffer does not make arbitrary user serialization code compliant. - -In particular, `T: Zeroize` only supplies cleanup once a `T` exists. If -derived `Deserialize` allocates a `String` field and fails on a later field, -the partially built fields ordinarily drop without invoking `T::zeroize`. -Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix a struct -that was never constructed. Keep this as a concrete decoder design and -acceptance requirement; do not claim the new bound solves partial decoding. +`OpaqueBlob` cannot promise to clean up the source from `T: Serialize` +alone. Require `T: Serialize + Zeroize` for the proposed blob input that +uses Serde. Owned blob decoding requires `T: DeserializeOwned + Zeroize`. +Ordinary application structs can derive `Zeroize`. + +A generic Serde implementation can allocate or keep hidden copies, so strong +claims also need a supported contract for the codec and implementation. +Wrapping the final byte buffer does not make arbitrary user serialisation +code compliant. + +In particular, `T: Zeroize` only supplies cleanup once a `T` exists. Suppose +a derived `Deserialize` allocates a `String` field and then fails on a later +field. The fields built so far drop in the ordinary way, without calling +`T::zeroize`. Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix +a struct that was never constructed. Keep this as a concrete requirement for +decoder design and acceptance, and do not claim that the new bound solves +partial decoding. ### Blob codec assessment: Vitamin C frames -**Priority decision, 2026-10-25:** first define a stable, versioned binary -storage profile. The codec was originally ephemeral transport; stored blobs -need a stronger compatibility commitment. Zeroization and constant-time -implementation improvements can follow, provided they preserve stored -format semantics. Do not defer decisions that affect stored bytes (option -presence, type mapping, versions) or claim timing/cleanup properties not -implemented. The `Zeroize` input bound and ownership direction remain. -Track this action in -[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519), created -for this review. It explicitly distinguishes storage stability from the -transport-only rationale in Vitamin C #332. - -**Implemented upstream and in Go, 2026-10-27:** Vitamin C's dedicated -storage profile and Go reader/writer address the format work from #519 -(#520/#526, carried onto upstream main via #544). Stack #1369/#1370 adopt -the Go codec: opaque generated structs call `vcffi.MarshalStorage` and -`UnmarshalStorage`. The profile is versioned (`VCS` plus version), supports -explicit optional values and typed map keys, and is separate from transport. -The transport mapping's `Some(())` ambiguity and text-only map-key rules -must not be reported as defects of this new storage profile. +#### Stable storage format first + +Decided on 25 October 2026, as the priority: first define a stable, +versioned binary storage profile. The codec was first designed for +short-lived transport, and stored blobs need a stronger compatibility +commitment. Improvements to zeroisation and constant-time implementation can +follow, provided they keep the meaning of the stored format. + +Do not defer decisions that affect stored bytes, such as option presence, +type mapping, and versions. Do not claim timing or cleanup properties that +are not implemented. The `Zeroize` input bound and the ownership direction +remain. [Vitamin +C #519](https://github.com/cipherstash/vitaminc/issues/519), created for +this review, tracks this action. The issue explicitly separates storage +stability from the transport-only reasons in Vitamin C #332. + +#### Storage profile implemented in Vitamin C and Go + +By 27 October 2026, Vitamin C's dedicated storage profile, and its Go reader +and writer, addressed the format work from #519. That work is in Vitamin +C #520 and #526, carried onto upstream main by #544. Stack #1369 and #1370 +adopt the Go codec: generated opaque structs call `vcffi.MarshalStorage` and +`UnmarshalStorage`. + +The profile is versioned (`VCS` plus a version). It supports explicit +optional values and typed map keys, and it is separate from transport. Do +not report these properties of the transport mapping as defects of the new +storage profile: + +- the ambiguity of `Some(())`; +- the rules that allow only text map keys. Go generated shapes use `vc` tags and `GeneratedVersion2`. A reader rejects -unsupported future blob versions distinctly and retains the old JSON -fallback. Go storage mapping has documented qualifications: nil containers -become empty, supported nonfinite floats round-trip, and unsupported opaque -types are refused by generation. Go pointers and Rust `Option` do not always -write identical bytes; cross-reading and the normative corpus matter. - -Rust workspaces still use crates.io Vitamin C 0.8.0; the final proposed Rust -`OpaqueBlob` facade has not thereby been implemented. Adopt the dedicated -storage entry points when that facade lands rather than repurposing the -transport Serde codec. Go's temporary pseudo-version dependency must move -to released module tags for a final Go release. A new serializer evaluation -is not on the customer critical path. - -Stable bytes do not establish constant-time behavior or complete cleanup. -Retain the remaining guarded-output, temporary-buffer, partial decode and -caller-owned-value work, without changing stored format. JSON tree retains -its own node/envelope encoding; its post-cascade compatibility profile must -be frozen separately from opaque blobs. - -The same consuming contract applies to target-directed `encrypt_into`. -For a structural target, declared passthrough fields intentionally -move into the output; wipe encrypted source fields and temporary copies, -not plaintext fields the caller explicitly chose to retain. Fan-out to -ciphertext and index operations should read a guarded source where possible -instead of requiring a plaintext clone per operation. - -Specify cleanup for dynamic host/guest transport and JSON parsing buffers -as part of the Go implementation. The Rust guest cannot erase Go caller -objects it never owned; host-managed copies need host cleanup and ordinary -Go values do not acquire Rust move semantics. Document that boundary rather -than claiming guest cleanup erases every copy in the calling process. - -Acceptance work: source-cleanup/drop observation on successful and failing -requests, cancellation/unpolled-request tests, nested source/blob cases, -and owned non-Clone inputs. Inspect optimized code for wipe/copy behavior -where necessary; ordinary tests alone cannot prove absence of every residual -copy. Do not infer a current vulnerability or a completed guarantee from -this design discussion. +unsupported future blob versions with a distinct error, and still falls back +to the old JSON encoding. Go storage mapping has these documented +qualifications: + +- nil containers become empty; +- supported nonfinite floats round-trip; +- generation refuses unsupported opaque types. + +Go pointers and Rust `Option` do not always write identical bytes, so +cross-reading tests and the normative corpus matter. + +#### Remaining Rust and release work + +Rust workspaces still use Vitamin C 0.8.0 from crates.io. So this work does +not implement the final proposed Rust `OpaqueBlob` facade. When that facade +is built, adopt the dedicated storage entry points, instead of reusing the +transport Serde codec for storage. Go's temporary pseudo-version dependency +must move to released module tags for a final Go release. A new evaluation +of serialisers is not on the critical path for customers. + +Stable bytes do not establish constant-time behaviour or complete cleanup. +Keep the remaining work on guarded output, temporary buffers, partial +decoding, and values that the caller owns, without changing the stored +format. The JSON tree keeps its own node and envelope encoding. Freeze its +post-cascade profile separately from opaque blobs. + +### Cleanup in target-directed encryption and at the Go boundary + +The same consuming contract applies to target-directed `encrypt_into`. For a +structural target, declared passthrough fields move into the output on +purpose. Wipe encrypted source fields and temporary copies, not plaintext +fields that the caller explicitly chose to keep. Where possible, the fan-out +to ciphertext and index operations should read a guarded source, instead of +needing a plaintext clone for each operation. + +As part of the Go implementation, specify cleanup for dynamic transport +between host and guest, and for JSON parsing buffers. The Rust guest cannot +erase Go caller objects that it never owned. Copies that the host manages +need cleanup by the host, and ordinary Go values do not gain Rust move +semantics. Document that boundary, instead of claiming that guest cleanup +erases every copy in the calling process. + +Acceptance work: + +- observe source cleanup and drop on successful and failing requests; +- test cancellation, and requests dropped before they are polled; +- cover nested source and blob cases; +- cover owned inputs that are not `Clone`. + +Where necessary, inspect optimised code for wiping and copying behaviour. +Ordinary tests alone cannot prove that no copy remains. Do not infer a +current vulnerability, or a completed guarantee, from this design +discussion. ## Configured-target implementation sequence and completion criteria -Customer commitments supersede the earlier Rust/Go front-end lockstep sequence. -The foundational Go JSON slice and opaque storage adoption are now merged. - -1. **Specify the implemented contract.** Capture target options, operator - input/result encodings, grouped reads, errors and the column/keyset - profile in the formal interoperability spec and conformance fixtures. - Keep proposed schema envelopes distinct from the current field-map grammar. -2. **Close actual remaining customer gaps.** Parent-bound child opening, - complex ordered-path declarations if needed, configurable Go match - options (#1112) and the tokenizer/filter pipeline. #1397 already supplies - short whole-word matching through EQL; confirm its accepted leakage and - freeze its index semantics rather than planning it as unimplemented. -3. **Validate and freeze the release candidate.** Run the existing engine, - Go/guest and PostgreSQL suites against the selected bundle. Preserve - numeric/null behavior, cascade tamper checks, wildcard/fallback semantics, - known omission limitations and post-cascade stored vectors. Tests already - landed are not proof this review reran a release build. -4. **Release the dependencies and customer surface.** Replace Vitamin C's - temporary Go pseudo-version pins for a final release; the existing guard - allows prereleases. Opaque storage is already integrated; keep it separate - from the JSON-tree storage profile. Build production guests and matching SQL. -5. **Complete Rust ergonomics and packaging.** Implement public traits, - derives, definition lifecycle and crate moves behind the frozen Go/data - contract. Direct EQL conversion is a format-preserving optimization. +Commitments to customers replace the earlier sequence, in which the Rust and +Go front ends moved in step. The foundational Go JSON work and the adoption +of opaque storage are now merged. + +1. **Specify the implemented contract.** Keep the proposed schema envelopes + distinct from the current field-map grammar. Record these in the formal + interoperability specification and conformance fixtures: + - target options; + - operator input and result encodings; + - grouped reads; + - errors; + - the column and keyset profile. +2. **Close the actual remaining customer gaps.** The gaps are opening a + child bound to its parent, and complex ordered-path declarations if + needed. The gaps also include configurable Go match options (#1112), and + the tokeniser and filter pipeline. #1397 already supplies short + whole-word matching through EQL. Confirm the leakage accepted for it, and + freeze its index semantics, instead of planning it as unimplemented. +3. **Validate and freeze the release candidate.** Run the existing suites + for the engine, for Go with the guest, and for PostgreSQL against the + selected bundle. Keep numeric and null behaviour, tamper checks on + cascaded tags, wildcard semantics, fallback semantics, known omission + limitations, and post-cascade stored vectors. Tests that are already + merged do not prove that this review ran a release build again. +4. **Release the dependencies and the customer interface.** For a final + release, replace Vitamin C's temporary Go pseudo-version pins. The + existing guard allows them in prereleases. Opaque storage is already + integrated. Keep it separate from the JSON-tree storage profile. Build + production guests and the matching SQL. +5. **Complete the Rust convenience APIs and packaging.** Implement public + traits, derives, the definition lifecycle, and crate moves within the + frozen Go and data contract. Direct EQL conversion is an optimisation + that keeps the format. Use one execution engine throughout. The final Rust facade and package -layout are not gates for the existing Go JSON profile. Do not reimplement -configured JSON options or request-bound reads that have already landed. +layout do not gate the existing Go JSON profile. Do not reimplement +configured JSON options, or reads that Rust verifies against the original +request: both are already merged. ### Required deliverable: dynamic interoperability specification -**User requirement:** develop a formal specification of the dynamic -interoperability layer during implementation, not as documentation added -after the Rust and Go APIs have diverged. The current examples remain -proposals until reconciled into that specification; adding this deliverable -does not claim the specification has already been written. +The user requires a formal specification of the dynamic interoperability +layer, developed during implementation. Do not write it afterwards as +documentation, after the Rust API has diverged from the Go API. The current +examples remain proposals until they are reconciled into that specification. +Adding this deliverable does not claim that the specification has been +written. + +Recommended format: a versioned specification package that contains: -**Recommended format:** a versioned specification package containing -[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) schemas, -normative semantic and wire-format rules, and executable conformance -fixtures. JSON Schema describes declaration/request/response structure; -it is not the entire behavioral or binary transport specification. Keep -application semantic-type declarations distinct from JSON Schema's own -`type`, `$defs`, and `$ref` vocabulary. +- [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) + schemas; +- normative rules for semantics and wire format; +- executable conformance fixtures. + +JSON Schema describes the structure of declarations, requests, and +responses. It is not the whole specification of behaviour or of the binary +transport. Keep application declarations of semantic types distinct from +JSON Schema's own `type`, `$defs`, and `$ref` vocabulary. Required coverage: -- Definition grammar: built-in and named types, optional `types` section, - inline role wrappers, context selectors, scope eligibility, passthrough - and context storage, targets/options, query capabilities, defaults, and - unknown-field/extension handling. Named ordinary types need no role; - naming a type does not grant scope eligibility. -- Semantic validation: resolve named types and field references, reject - invalid/cyclic references where unsupported, enforce primary-key versus - scope-key capabilities recursively, check source/output compatibility, - and derive write/query scope agreement. Do not imply generic JSON Schema +- **Definition grammar.** Named ordinary types need no role. Naming a type + does not make it eligible for scope. The grammar covers: + - built-in and named types; + - the optional `types` section; + - inline role wrappers; + - context selectors and scope eligibility; + - passthrough and context storage; + - targets and options; + - query capabilities and defaults; + - handling of unknown fields and extensions. +- **Semantic validation.** Resolve named types and field references, and + reject invalid or cyclic references where they are unsupported. Enforce + the capabilities of primary keys and scope keys recursively. Check that + sources and outputs are compatible, and derive agreement between the write + scope and the query scope. Do not imply that generic JSON Schema validators perform the engine's reference resolution or capability checks. -- Operation contracts: input presence and context/scope roles, target and - definition identity, results, stable error codes and locations without - plaintext values, validation stages, and which failures precede key I/O. -- Wire mapping: JSON and frame representations, integer widths/ranges and - lossless numeric transport, bytes/text encodings, missing versus null, - context ordering/grouping, descriptor/AAD/index-scope lowering, and - normative references to the ciphertext/SQL envelope and tag extraction. - Specify the host/guest ABI or reference its versioned specification; - JSON authoring syntax must not silently redefine the binary transport. -- Versioning: explicit protocol/schema identifiers, supported-version and - capability handling, and rejection of unsupported features. Define future - evolution without adding compatibility obligations for the prototype. - -Conformance fixtures include accepted and rejected definitions/requests, -expected error codes/paths, normalized definitions, and deterministic -context/encoding vectors. Exercise the same fixtures through Rust and the -Go guest path, plus SQL extraction where relevant. Randomized ciphertext -comparisons use cross-decryption and query behavior, not byte equality. -Schema validation is one layer of the suite; semantic rejection tests are -required even for structurally valid JSON. - -Keep a single maintained contract: prefer generating structural schemas and -capability metadata from the shared declaration models/catalog where that -faithfully represents the specified grammar, with committed artifacts and -CI drift checks. Review generated schemas against normative rules rather -than treating Serde derives as a complete specification. Do not introduce -OpenAPI or a new transport solely to document this non-HTTP interface. +- **Operation contracts.** The contracts cover: + - input presence; + - the roles of context and scope; + - target and definition identity; + - results; + - stable error codes and locations, without plaintext values; + - validation stages; + - which failures come before key I/O. +- **Wire mapping.** Specify the application binary interface (ABI) between + host and guest, or reference its versioned specification. JSON authoring + syntax must not silently redefine the binary transport. The mapping + covers: + - JSON and frame representations; + - integer widths and ranges, and lossless numeric transport; + - encodings of bytes and text; + - missing compared with null; + - context ordering and grouping; + - the lowering of descriptors, AAD, and index scope; + - normative references to the ciphertext and SQL envelope, and to tag + extraction. +- **Versioning.** Explicit protocol and schema identifiers, handling of + supported versions and capabilities, and rejection of unsupported + features. Define future evolution without adding compatibility obligations + for the prototype. + +Conformance fixtures include accepted and rejected definitions and requests, +expected error codes and paths, normalised definitions, and deterministic +vectors for context and encoding. Run the same fixtures through Rust and +through the Go guest path, and through SQL extraction where relevant. +Compare randomised ciphertexts by cross-decryption and query behaviour, not +byte equality. Schema validation is one layer of the suite. Semantic +rejection tests are required even for structurally valid JSON. + +Keep a single maintained contract. Prefer generating structural schemas and +capability metadata from the shared declaration models and catalog, where +that faithfully represents the specified grammar. Commit the generated +artefacts and check them for drift in CI. + +Review generated schemas against the normative rules, instead of treating +Serde derives as a complete specification. This interface does not use HTTP. +Do not introduce OpenAPI, or a new transport, only to document it. ## Coverage of the complete review -The configured-target work above addresses issue 13 in depth. The full API -redesign also requires the following work. Each row identifies a concrete -deliverable; recommendations do not silently settle the open decisions. +The configured-target work above covers review issue 13 in depth. The full +API redesign also needs the work in this table. Each row names a concrete +deliverable. Recommendations do not silently settle the open decisions. | Work area | Review issues | Required deliverable | | --- | --- | --- | -| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One signature/import/receiver map, operation-trait design, and dependency migration | -| Context contract | 1, 3, 4, 11, 12, 14 | Scalar and record context rules, explicit optionality, and matching dynamic representation | -| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde/Base64 behavior, and a defined boundary with structural trees | +| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One map of signatures, imports, and receivers; operation-trait design; and migration of dependencies | +| Context contract | 1, 3, 4, 11, 12, 14 | Context rules for scalars and records, explicit optionality, and a matching dynamic representation | +| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde and Base64 behaviour, and a defined boundary with structural trees | | Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | -| Mixed records and collections | 7, 8, 11 | Passthrough/renaming/context-field derive support and retained collection batching | -| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion and Go coverage | +| Mixed records and collections | 7, 8, 11 | Derive support for passthrough, renaming, and context fields, and retained collection batching | +| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion, and Go coverage | | Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | -| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348–#1352 with this contract; reuse compatible implementation work | -| Prior query design | 16 | Reuse the supplied field-owned query model; reconcile handles, configuration ownership, JSON capabilities, and portability | -| Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | -| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | +| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348 to #1352 with this contract, and reuse compatible implementation work | +| Prior query design | 16 | Reuse the supplied field-owned query model, and reconcile handles, configuration ownership, JSON capabilities, and portability | +| Consistent definition selector | 17 | Choose one method name for encrypt and query. Recommended: `using`, with bounds specific to each operation | +| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode` and `OpaqueDecode`, consuming `encrypt` and `encrypt_into`, guarded lifetimes for sources and buffers, and an explicit cleanup scope | ### A. Consolidate the public Rust method and trait structure -This is the Rust-facade workstream. Under the latest customer ordering, do -only the engine/contract changes needed by Go before its release; defer -standalone Rust ergonomics and trait relocation. The sections below organize -the full review scope, not the customer critical path. +This is the workstream for the Rust facade. Under the latest customer +ordering, make only the engine and contract changes that Go needs before its +release. Defer standalone Rust convenience work and the move of traits. The +sections below organise the full scope of the review, not the critical path +for customers. -Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. -Use the accepted destination-directed decrypt pair. Write -out all signatures, imports, bounds, and return types before renaming traits. -Retain ordinary keyset `decrypt` / `decrypt_with_context` for opaque -ciphertexts, and remove redundant prototype conveniences without -compatibility shims. +Keep the agreed ordinary keyset receiver and the `EncryptInto` conversion +forms. Use the accepted destination-directed decrypt pair. Write out all +signatures, imports, bounds, and return types before renaming traits. Keep +the ordinary keyset `decrypt` and `decrypt_with_context` for opaque +ciphertexts. Remove redundant prototype conveniences without compatibility +shims. Separate plaintext representation, target operation declarations, and -execution conveniences explicitly. Resolve why `EncryptFrom` declares -`encryption()` while `encrypt_from` lives on `EncryptInto`, and why -`DecryptFrom` supplies methods while `DecryptInto` describes opening. A -rename without an ownership/signature map leaves the discovery problem. - -Recommendation: Stack owns the public value-encryption model; Vitamin C -retains general primitives. Specify the dependency boundary before moving -code, retaining one canonical trait identity and the no-HTTP/WASM builds. -Reassess the original Vitamin C cipher-trait mismatch in that architecture: -do not add a public trait implementation merely for symmetry if it cannot -carry structured KMS context or support its advertised operations. - -Completion: representative scalar, blob, typed record, searchable record, -and JSON calls compile with the documented imports, including generic -callers. The method inventory has no unexplained duplicate execution path. -Compile-fail tests cover genuinely unsupported trait capabilities. +execution conveniences explicitly. Resolve two inconsistencies: + +- `EncryptFrom` declares `encryption()`, while `encrypt_from` is on + `EncryptInto`; +- `DecryptFrom` supplies methods, while `DecryptInto` describes opening. + +A rename without a map of ownership and signatures leaves the discovery +problem unsolved. + +Recommended: Stack owns the public model for encrypting values, and Vitamin +C keeps general primitives. Specify the dependency boundary before moving +code. Keep one canonical identity for each trait, and keep the builds +without HTTP and for WebAssembly. Reassess the original mismatch with +Vitamin C's cipher trait in that architecture. Do not add a public trait +implementation only for symmetry, if it cannot pass structured KMS context +or support the operations it advertises. + +Completion criteria for work area A: representative scalar, blob, typed +record, searchable record, and JSON calls compile with the documented +imports, including from generic callers. The method inventory has no +unexplained duplicate execution path. Compile-fail tests cover the trait +capabilities that are not supported. ### B. Turn the context scenarios into a normative contract -The nine scenarios in the context document are the acceptance matrix, not -optional examples. Cover scalar static context plus caller extension, -source-field selection, caller input, output storage, and decrypt recovery -or expected-context checks. Preserve the accepted `field = "id"` shorthand -as selection on both inputs, without implicit copying. - -Fix composition order and grouping once: literal/field base, composite row -context, and any caller extension. Specify the exact descriptor and AAD -encoding for each case. An expected context checks stored context; it is -not another extension. Never need to open ciphertext to discover the -context required to open that ciphertext. - -Recommendation: represent no input, required input, optional extension, -and optional expectation as explicit capabilities. Do not infer optionality -from the application's `Default` implementation. Settle the wrapper or -trait mechanism alongside the public signatures. Resolve the existing -empty-context mismatch between ordinary scalar encryption and ciphertext -targets without weakening nonempty-context requirements for search targets. - -Give each source/storage/composition rule a dynamic representation and Go -authoring path. Current EQL identifier constraints must be reconciled with -row/caller context explicitly, not bypassed or silently dropped. - -**Next discussion — recommendation, not yet a decision:** distinguish -ciphertext binding from search scope. A row ID may bind a ciphertext to its -record without making every index term row-specific. The target definition -should explicitly describe both scopes; query construction uses the same -declared search scope as index generation. For example, a searchable email -may bind its ciphertext to `(users.email, tenant_id, user_id)` while its -equality index is scoped to `(users.email, tenant_id)`, allowing a query -before the user ID is known. Do not silently remove components from one -generic context argument. Index-key derivation/KMS descriptors must respect -this separation too; changing only leaf AAD is not a complete design. -Finalize the conceptual split before choosing attribute names or exact -tuple encodings, and represent it in the dynamic definition as well. - -**User direction:** allow row-bound ciphertext and document that reads must -provide the row ID to decrypt. Search scope should cover multiple rows, -such as a tenant; automatically including a row ID defeats ordinary search. - -**Misuse-resistant API recommendation (not yet accepted):** - -- Give existing record `context(field = "id")` and its directional variants - ciphertext-binding semantics. Do not implicitly feed those selected fields - or caller extensions into index keys or terms. This is a proposed - clarification for searchable targets; older examples did not distinguish - the roles, and must be updated together before implementation. -- Declare search scope separately on the shared target definition. A named - application scope input (for example `TenantScope`) can come from operation - configuration/caller input. The capability proposal below also permits - record-field selectors when their types explicitly qualify as scopes. - The same typed scope is required for both - writing index terms and constructing queries. A tenant-bound cipher may - supply it only through an explicit declared mapping; never guess it from - unrelated metadata. -- A binding input and a search-scope input are distinct capabilities even - when their underlying values have the same Rust type. Do not provide +The nine scenarios in the [context +examples](2026-10-24-stack-encrypt-context-examples.md) are the acceptance +matrix, not optional examples. Cover: + +- a scalar's static context plus a caller extension; +- selecting a source field; +- caller input; +- storing context in the output; +- recovering context, or checking expected context, on decrypt. + +Keep the accepted `field = "id"` shorthand as a selection on both inputs: +the plaintext on encrypt, and the stored output on decrypt. The shorthand +does not copy anything implicitly. + +Fix the composition order and grouping once: the literal or field base, then +the composite row context, then any caller extension. Specify the exact +descriptor and AAD encoding for each case. An expected context checks the +stored context, and is not another extension. A caller must never need to +open a ciphertext to discover the context required to open that ciphertext. + +Recommended: represent no input, required input, optional extension, and +optional expectation as explicit capabilities. Do not infer optionality from +the application's `Default` implementation. Settle the wrapper or trait +mechanism together with the public signatures. Resolve the existing mismatch +in empty context between ordinary scalar encryption and ciphertext targets. +Do not weaken the requirement for nonempty context on search targets while +doing so. + +Give each rule for source, storage, and composition a dynamic representation +and a way to write it in Go. Reconcile the current EQL identifier +constraints with row and caller context explicitly. Do not bypass those +constraints, or drop context silently. + +#### Separate ciphertext binding from search scope + +Status: recommendation for the next discussion, not yet a decision. + +Distinguish ciphertext binding from search scope. A row ID can bind a +ciphertext to its record without making every index term specific to that +row. The target definition should describe both scopes explicitly. Query +construction uses the same declared search scope as index generation. + +For example, a searchable email can bind its ciphertext to +`(users.email, tenant_id, user_id)`, while its equality index is scoped to +`(users.email, tenant_id)`. A query can then run before the user ID is +known. Do not silently remove components from one generic context argument. +Index-key derivation and KMS descriptors must respect this separation too. +Changing only the leaf AAD is not a complete design. + +Settle the conceptual split before choosing attribute names or exact tuple +encodings, and represent it in the dynamic definition as well. + +The user's direction: allow ciphertext bound to a row, and document that +reads must supply the row ID to decrypt. Search scope should cover several +rows, such as a tenant. Including a row ID automatically would defeat +ordinary search. + +#### Misuse-resistant API (recommendation, not accepted) + +- Give the existing record `context(field = "id")` and its directional + variants the meaning of ciphertext binding. Do not implicitly feed those + selected fields or caller extensions into index keys or terms. This is a + proposed clarification for searchable targets. Older examples did not + distinguish the roles. Update them all together before implementation. +- Declare search scope separately, on the shared target definition. A named + application scope input, such as `TenantScope`, can come from operation + configuration or caller input. The capability proposal below also allows + record-field selectors, when their types explicitly qualify as scopes. +- Writing index terms and constructing queries both require the same typed + scope. A cipher bound to a tenant can supply the scope only through an + explicit declared mapping. Never guess the scope from unrelated metadata. +- A binding input and a search-scope input are distinct capabilities, even + when their underlying values have the same Rust type. Do not provide a blanket implicit conversion between the roles. If one operation needs both - caller inputs, use named members in its typed input rather than one - unlabelled value routed to both operations. -- Missing required scope prevents awaiting a Rust request; dynamic/Go - validation rejects it before key I/O. A default column-wide scope is not - substituted when a tenant scope is missing. Scope-free searches use an - explicitly declared column-wide policy, with column identity still bound. + caller inputs, use named members in its typed input, not one unlabelled + value routed to both operations. +- Missing required scope prevents awaiting a Rust request. Dynamic and Go + validation reject it before key I/O. A default column-wide scope is not + substituted when a tenant scope is missing. Searches without scope use an + explicitly declared column-wide policy, and still bind the column + identity. - The effective ciphertext binding should include the declared search scope - as well as record binding, so changing a supplied tenant also prevents - opening the ciphertext. Index derivation sees column identity plus search - scope only. Query requests never require record binding input. -- Encrypted output must expose whatever the decrypt selector needs or - require it from the caller. Record-projection examples must select the ID; - there is no decrypt fallback which drops row binding. Selecting a field - as context does not implicitly store that field. + as well as the record binding. Then changing a supplied tenant also + prevents opening the ciphertext. Index derivation uses only the column + identity plus the search scope. Query requests never require + record-binding input. +- Encrypted output must expose whatever the decrypt selector needs, or + require it from the caller. Examples that project records must select the + ID. No decrypt fallback drops row binding. Selecting a field as context + does not implicitly store that field. Working declaration spelling: -`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. -The spelling is illustrative; it preserves the agreed directional-context -shorthand and gives search an explicitly separate role. Dynamic definitions -must preserve this separation rather than providing a shared context bag. - -This prevents accidental propagation of row IDs into search terms; it -cannot prove the application chose a scope containing multiple rows. A user -can deliberately put a row ID inside `TenantScope`. Do not claim otherwise, -or attempt to infer semantics from field names such as `id`. Use explicit -types/declarations and documentation for that remaining application choice. - -#### Scope capability and role wrappers — assessment - -**Proposal under discussion:** use both ideas together. Scope is the subset -of context permitted for search terms; semantic wrappers provide positive -trait implementations identifying that subset. Working capability names: -`CiphertextContext` and `SearchScope: CiphertextContext`. These names describe -eligibility, not source selection, optionality, or encoding. Integrate with -the existing context representation rather than introducing a competing -serializer. The previously agreed `ContextInput` capabilities still describe -whether the caller must supply a value. - -For example, `PrimaryKey` implements ciphertext-context eligibility but -not search-scope eligibility. `ScopeKey` implements both, subject to -the inner value's supported context encoding. Generic parameters are Rust -types (`PrimaryKey`); `PrimaryKey(42)` is a value constructor. Prefer -`ScopeKey` as the name for the second wrapper; the user accepted this -name in preference to `Collection`, which can read as a container. -These wrappers need the appropriate `Zeroize` support when they are owned -plaintext fields, consistent with the existing ownership decision. - -Crucially, neither wrapper automatically adds the field to context or scope, -stores it in encrypted output, or changes the plaintext's encryption layout. -The target declaration still selects sources. An illustrative declaration -is `context(field = "id"), search_scope(field = "tenant_id")`. With +`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. The +spelling is illustrative. The spelling keeps the agreed directional-context +shorthand, and gives search an explicitly separate role. Dynamic definitions +must keep this separation, instead of providing one shared collection of +context values. + +This design stops row IDs from spreading into search terms by accident. The +design cannot prove that the application chose a scope that contains several +rows. A user can deliberately put a row ID inside `TenantScope`. Do not +claim otherwise, or try to infer meaning from field names such as `id`. Use +explicit types, declarations, and documentation for that remaining +application choice. + +#### Scope capability and role wrappers + +Status: proposal under discussion. + +Use both ideas together. Scope is the subset of context that search terms +are allowed to use. Semantic wrappers provide positive trait implementations +that identify that subset. The working capability names are +`CiphertextContext`, and `SearchScope: CiphertextContext`. These names +describe eligibility, not source selection, optionality, or encoding. + +Integrate these capabilities with the existing context representation, +instead of introducing a competing serialiser. The previously agreed +`ContextInput` capabilities still describe whether the caller must supply a +value. + +For example, `PrimaryKey` implements eligibility for ciphertext context, +but not for search scope. `ScopeKey` implements both, subject to the +inner value's supported context encoding. Generic parameters are Rust types, +such as `PrimaryKey`. By contrast, `PrimaryKey(42)` is a value +constructor. + +The name `ScopeKey` is preferred for the second wrapper. The user +accepted it in place of `Collection`, because `Collection` can read as +a container. These wrappers need the appropriate `Zeroize` support when they +are owned plaintext fields, consistent with the existing ownership decision. + +Neither wrapper automatically adds the field to context or scope, stores it +in the encrypted output, or changes the plaintext's encryption layout. The +target declaration still selects sources. An illustrative declaration is +`context(field = "id"), search_scope(field = "tenant_id")`. With `id: PrimaryKey` and `tenant_id: ScopeKey`, the selection is -valid; selecting `id` as search scope is a compile-time error. This makes -typed field selection a reasonable alternative to the earlier recommendation -to accept only caller-supplied search scope. - -Misuse-resistance requirements: - -- No blanket `SearchScope` implementation for every context, serializable - type, raw number, or raw string. Scope use is an explicit classification. -- Composite scopes qualify only when every component is a scope. A tuple - containing a primary key cannot become a scope merely by tuple composition. - Context composites may contain either category. -- Do not forward scope eligibility through `Deref`, `AsRef`, or implicit - conversions that erase a primary-key wrapper. Typed target/query handles - retain the definition's concrete scope type, rather than accepting any - type implementing `SearchScope` interchangeably. Distinct application - newtypes can distinguish tenant and project scopes with equal wire types. -- Permit explicit application opt-in through a documented trait/derive +valid, and selecting `id` as search scope is a compile-time error. This +makes typed field selection a reasonable alternative to the earlier +recommendation to accept only search scope supplied by the caller. + +Requirements for misuse resistance: + +- No blanket `SearchScope` implementation for every context type, + serialisable type, raw number, or raw string. Using a value as scope is an + explicit classification. +- A composite scope qualifies only when every component is a scope. A tuple + that contains a primary key cannot become a scope only because it is a + tuple. Context composites can contain either category. +- Do not pass scope eligibility through `Deref`, `AsRef`, or implicit + conversions that remove a primary-key wrapper. Typed target and query + handles keep the definition's concrete scope type, and do not accept any + other type that implements `SearchScope` in its place. Distinct + application newtypes can tell tenant and project scopes apart, even when + their wire types are equal. +- Allow explicit opt-in by applications through a documented trait or derive contract. This is a semantic capability, not an unsafe-memory contract. - Sealing every implementation cannot prove that a value groups multiple - records: a caller can still explicitly rewrap a row identifier as a scope. - Guarantee rejection of accidental role mixing, not business semantics. - -Dynamic definitions must retain field roles and context-versus-scope -selection. Validate those roles before execution: an explicitly declared -primary-key field is not scope-eligible. Plain JSON numbers carry no such -role by themselves; do not infer it from the field name or wire encoding. -Generated Go types can represent the distinction, while the engine validates -dynamic declarations. Rust and dynamic lowering must produce the same -canonical context/scope values; never use Rust type names as wire domains. -Scope classification is not evidence of authorization to access a tenant. - -Acceptance work: valid typed field/caller scopes, rejected primary-key and -mixed-tuple scopes, no automatic selection of marked fields, exact scope -type agreement between write/query, and equivalent dynamic validation. + Sealing every implementation cannot prove that a value groups several + records, because a caller can still explicitly rewrap a row identifier as + a scope. Guarantee that accidental mixing of roles is rejected, not + business semantics. + +Dynamic definitions must keep field roles, and the choice between context +and scope for each selection. Validate those roles before execution: a field +explicitly declared as a primary key is not eligible for scope. A plain JSON +number has no role by itself. Do not infer a role from the field name or the +wire encoding. Generated Go types can represent the distinction, while the +engine validates dynamic declarations. + +Rust and dynamic lowering must produce the same canonical context and scope +values. Never use Rust type names as wire domains. Classifying a value as +scope is not evidence of authorisation to access a tenant. + +Acceptance work: + +- valid typed field scopes and caller scopes; +- rejected primary-key scopes and mixed-tuple scopes; +- no automatic selection of marked fields; +- exact agreement of scope type between write and query; +- equivalent dynamic validation. #### Dynamic scope roles and validation -**Recommended design:** declare semantic types and source selectors in the -definition, validate once on load, and supply values separately per operation. -Do not let an individual request assert `scope_key` on an arbitrary value -to override the loaded definition. A definition author can explicitly -classify values, just as a Rust author can construct wrappers; this is -misuse resistance, not authorization or proof of multi-row cardinality. +Recommended design: declare semantic types and source selectors in the +definition, validate them once on load, and supply values separately for +each operation. Do not let an individual request assert `scope_key` on an +arbitrary value to override the loaded definition. A definition author can +explicitly classify values, in the same way that a Rust author can construct +wrappers. This resists misuse. It is not authorisation, or proof that a +scope covers several rows. -**Accepted clarification:** `types` is optional; it holds reusable named -types. Fields may use built-ins or inline role wrappers. Ordinary built-in -or named values need no `kind`. Supported primitives remain eligible for -ciphertext context, but search eligibility requires explicit `scope_key` -classification (or another explicitly supported scope-capable type). +Accepted clarification: `types` is optional, and contains reusable named +types. Fields can use built-in types or inline role wrappers. Ordinary +built-in or named values need no `kind`. Supported primitives remain +eligible for ciphertext context. Eligibility for search requires explicit +`scope_key` classification, or another explicitly supported type that can +act as a scope. -Illustrative declaration fragment (proposed schema, not today's wire API): +Illustrative declaration fragment. This is a proposed schema, not today's +wire API: ```json { @@ -1203,167 +1572,212 @@ Illustrative declaration fragment (proposed schema, not today's wire API): } ``` -This is the context-related portion of a record target definition; encrypted -email targets and their search options are declared separately as usual. -The shared `field` shorthand selects plaintext fields on encrypt and stored -fields on decrypt; it does not imply a field exists in a query request. +This is the context-related part of a record target definition. Encrypted +email targets and their search options are declared separately, as usual. +The shared `field` shorthand selects plaintext fields on encrypt, and stored +fields on decrypt. It does not imply that the field exists in a query +request. + For a query on the derived email handle, the definition exposes a required -`TenantId` scope argument. No `UserId` argument is required or accepted as -an alternative scope. A named/composite scope uses the same ordered schema -on the write and query paths, regardless of their different source selectors. - -Definition loading resolves field references and source/output types, -computes the context/scope capabilities, verifies all composite components, -and rejects a primary-key search selector before key I/O. It also verifies -decrypt selectors are available from stored fields or declared caller input. -Compile the validated definition into an immutable handle shared by encrypt, -decrypt and query operations. No second query-specific scope policy. - -At execution validate presence, nominal type when supplied by a typed handle, -physical representation, range, and composite shape against the selected -definition. Raw wire values obtain their expected semantic type from the -definition; they cannot prove their application provenance. Unknown role -overrides or mismatched nominal value handles are rejected. No fallback to -unscoped querying when a required scope is absent. When JSON is the authoring -or request format, preserve full-width integers through lossless parsing; -the frame transport already distinguishes supported integer widths. - -Recommended Go generation uses distinct named ID types and query methods -with the required scope argument (`TenantID` rather than `uint64` or an +`TenantId` scope argument. No `UserId` argument is required, or accepted as +an alternative scope. A named or composite scope uses the same ordered +schema on the write and query paths, even though their source selectors +differ. + +Loading a definition: + +- resolves field references, and source and output types; +- computes the context and scope capabilities; +- verifies all composite components; +- rejects a primary-key search selector before key I/O; +- verifies that decrypt selectors are available from stored fields or + declared caller input. + +Compile the validated definition into an immutable handle that encrypt, +decrypt, and query operations share. There is no second scope policy +specific to queries. + +At execution, validate these against the selected definition: presence, +nominal type when a typed handle supplies it, physical representation, +range, and composite shape. Raw wire values get their expected semantic type +from the definition. A raw value cannot prove where it came from in the +application. Unknown role overrides and mismatched nominal value handles are +rejected. A query never falls back to unscoped querying when a required +scope is absent. + +When JSON is the authoring or request format, keep full-width integers +through lossless parsing. The frame transport already distinguishes the +supported integer widths. + +Recommended Go generation uses distinct named ID types, and query methods +that take the required scope argument (`TenantID`, not `uint64` or an untyped context map). Generated wrappers remain conveniences over the same -validated runtime definition; ungenerated FFI callers receive runtime errors -for invalid declarations/requests. Do not assert Go shares Rust's ownership -or trait-bound guarantees. +validated runtime definition. FFI callers that do not use generated code get +runtime errors for invalid declarations and requests. Do not claim that Go +shares Rust's ownership or trait-bound guarantees. Lower the chosen scope and binding through the existing canonical context -machinery, retaining scalar widths, ordering and grouping. Current dynamic -`context.rs` proves parity between runtime `ContextPiece` trees and Rust -context derivations, but has no primary-key/scope-role distinction. Add role -validation before lowering erases those distinctions, and keep the separate -resolved scope and binding available to their respective operations. -Do not persist role labels in every row just to recover them at runtime; -definition identity and deployment remain explicit concerns of loading the -same policy. Add Rust/Go/dynamic fixtures for valid scopes, primary-key -refusals, nested scopes, missing inputs and identical derived terms/AAD. - -Completion: all nine cases execute through Rust and dynamic declarations; -required input is enforced, altered/wrong context is refused, and existing -context encodings agree across all new implementations. +machinery, keeping scalar widths, ordering, and grouping. The current +dynamic `context.rs` proves parity between runtime `ContextPiece` trees and +Rust context derivations. It does not distinguish the primary-key role from +the scope role. Add role validation before lowering removes those +distinctions. Keep the resolved scope and the resolved binding available +separately to their own operations. + +Do not store role labels in every row only to recover them at runtime. +Definition identity and deployment remain explicit concerns of loading the +same policy. Add Rust, Go, and dynamic fixtures for valid scopes, +primary-key refusals, nested scopes, missing inputs, and identical derived +terms and AAD. + +Completion criteria for work area B: all nine cases run through Rust and +dynamic declarations. Required input is enforced, altered or wrong context +is refused, and existing context encodings agree across all new +implementations. ### C. Finish the ciphertext and whole-value storage contracts Accepted direction: a single-value plaintext encoding produces one opaque -`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode -and seal it as one value, or an explicit target for field-level encryption; -Serde alone does not imply automatic blob serialization. A custom type can -explicitly implement the single-value encoding capability. The wrapper -name and detailed persistence contract remain proposals. -Recommend canonical byte parsing/encoding, human-readable Serde as Base64 -of those bytes, binary Serde as -bytes, and safe Debug behavior. Parsing an envelope does not authenticate -it; authentication occurs during decrypt. - -The existing encrypted-output transcoder can implement the adapter: -`Node::Single(SealedValue)` is consumed by `Visitor::sealed` and moved into -the opaque wrapper. Keep the sealed value internally rather than forcing -a byte serialization at construction. There is no new encryption or KMS -request. On decrypt the wrapper delegates to the same leaf-opening path, -using a single-node wrapper if the internal API requires it. - -Do not convert maps, sequences, passthrough nodes, or sealed empty-container -markers into this type merely because they have a byte representation. -Serializing a whole encrypted tree retains its multiple ciphertexts and -authentication boundaries; it does not produce single-value encryption. -The leaf visitor's unsupported-shape errors are a defensive check, not the -desired public discovery mechanism. Constrain ordinary encryption to -single-value encodable inputs so unsupported structural inputs are rejected -before encryption, statically in Rust where possible and during dynamic -validation. Settle the exact plaintext capability with the representation -traits; do not discover an unsupported shape only after minting keys. +`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode and +seal it as one value, or an explicit target for field-level encryption. +Serde alone does not imply automatic blob serialisation. A custom type can +explicitly implement the single-value encoding capability. The wrapper name +and the detailed persistence contract remain proposals. + +Recommended persistence behaviour: + +- canonical byte parsing and encoding; +- human-readable Serde as Base64 of those bytes; +- binary Serde as bytes; +- safe `Debug` behaviour. + +Parsing an envelope does not authenticate it. Authentication happens during +decryption. + +The existing transcoder for encrypted output can implement the adapter. +`Visitor::sealed` consumes `Node::Single(SealedValue)` and moves it into the +opaque wrapper. Keep the sealed value internally, instead of forcing a byte +serialisation at construction. There is no new encryption or KMS request. On +decrypt, the wrapper delegates to the same leaf-opening path, using a +single-node wrapper if the internal API requires one. + +Do not convert maps, sequences, passthrough nodes, or sealed markers for +empty containers into this type only because they have a byte +representation. Serialising a whole encrypted tree keeps its several +ciphertexts and authentication boundaries, and does not produce single-value +encryption. The leaf visitor's errors for unsupported shapes are a defensive +check, not the intended way for users to discover the rule. + +Constrain ordinary encryption to inputs that can be encoded as a single +value. Unsupported structural inputs are then rejected before encryption: +statically in Rust where possible, and during dynamic validation. Settle the +exact plaintext capability together with the representation traits. Do not +discover an unsupported shape only after keys have been minted. Keep generic structural encryption and searchable JSON explicit. Specify -whether structural-tree persistence is supported and under what stable -format. Any protocol changes must be coordinated with their consumers. +whether persistence of structural trees is supported, and in what stable +format. Coordinate any protocol changes with their consumers. #### Typed encodings and extractable ZeroKMS tags -**Requirements added, 2026-10-25:** ciphertext text conversion returns the -existing concrete encoding wrappers, not `String`. The user also wants to -explore selecting an encoded ciphertext as a field's output type. The -ZeroKMS key tag must be easily extractable from stored ciphertext, including -in PL/pgSQL, to support a separately designed refinable-tag mechanism. -The storage/authentication contract is not yet locked in. +Requirements added on 25 October 2026: -Current `eql-bindings::encoding::{Base64Text, Base85Text}` validates canonical -padded standard Base64 and the crate's RFC 1924 Base85 encoding respectively. -Their private strings, encoding constructors and validating deserializers -make decoding infallible. They guarantee the text encoding, not that the -bytes form a ciphertext. Reuse these types and add ordinary fallible text -parsing conveniences rather than requiring Serde to construct them from text. +- Converting a ciphertext to text returns the existing concrete encoding + wrappers, not `String`. +- The user also wants to explore selecting an encoded ciphertext as a + field's output type. +- The ZeroKMS key tag must be easy to extract from a stored ciphertext, + including in PL/pgSQL. This supports a refinable-tag mechanism that is + designed separately. + +The storage and authentication contract is not yet fixed. + +The current `eql-bindings::encoding::{Base64Text, Base85Text}` validate +canonical padded standard Base64, and the crate's RFC 1924 Base85 encoding, +respectively. Their private strings, encoding constructors, and validating +deserialisers make decoding infallible. They guarantee the text encoding, +not that the bytes form a ciphertext. Reuse these types, and add ordinary +fallible conveniences for parsing text, instead of requiring Serde to +construct them from text. Recommended ownership: a small Vitamin C encoding component, re-exported by -Stack Encrypt and EQL. These byte/text utilities do not depend on encryption, -SQL or EQL layouts. Keep TypeScript/schema integration optional in the owning -crate, accounting for Rust's orphan rules when moving the existing trait -implementations. Do not introduce a Stack Encrypt dependency on EQL merely -to reuse its text wrappers. Final component placement remains a proposal. - -Proposed ciphertext methods return `Base64Text` / `Base85Text` from -`to_base64` / `to_base85`; corresponding constructors accept the typed text -and still return a ciphertext parse error when its decoded envelope is -invalid. Encoding validation, envelope parsing and authentication are three +Stack Encrypt and EQL. These byte and text utilities do not depend on +encryption, SQL, or EQL layouts. Keep TypeScript and schema integration +optional in the owning crate, and account for Rust's orphan rules when +moving the existing trait implementations. Do not make Stack Encrypt depend +on EQL only to reuse its text wrappers. Final placement of the component +remains a proposal. + +Proposed ciphertext methods `to_base64` and `to_base85` return `Base64Text` +and `Base85Text`. Matching constructors accept the typed text, and still +return a ciphertext parse error when the decoded envelope is invalid. +Encoding validation, envelope parsing, and authentication are three different guarantees. Use one canonical byte envelope under both encodings. -**Accepted, 2026-10-25:** support explicit encoded ciphertext field targets. -Use a ciphertext-specific storage wrapper -(working names `Base64Ciphertext` / `Base85Ciphertext`) over making every -`Base64Text` an encryption target. The wrapper validates both text and -envelope, serializes as the text string, and composes with `EncryptTarget` -and `DecryptSource`: seal once, then encode; parse before opening. These -names remain provisional until the final naming pass; the first-class -encoded target capability is settled. A structural output -can mix `Ciphertext` and encoded ciphertext fields without additional KMS +Accepted on 25 October 2026: support explicit encoded ciphertext field +targets. Use a storage wrapper specific to ciphertexts, with the working +names `Base64Ciphertext` and `Base85Ciphertext`, instead of making every +`Base64Text` an encryption target. The wrapper validates both the text and +the envelope, and serialises as the text string. The wrapper composes with +`EncryptTarget` and `DecryptSource`: seal once and then encode, and parse +before opening. + +These names stay provisional until the final naming pass, but the encoded +target, as a capability in its own right, is settled. A structural output +can mix `Ciphertext` fields and encoded ciphertext fields without extra KMS calls. Do not silently change the prescribed representation of EQL targets. The current `SealedValue` v1 envelope already uses a simple prefix: `version(1) | keyset_id(16) | iv(16) | tag_len(2, LE) | tag | local_ciphertext`. -Its ZeroKMS tag begins at byte offset 35, with length at offsets 33–34. -This is evidence that extraction can be simple, not a commitment to keep -those offsets. Keep the chosen version's layout documented and parseable -without the plaintext codec, key retrieval, or decryption. Prefer a checked -borrowed header/envelope view and an accessor explicitly named `key_tag` -over decomposing the whole ciphertext or conflating it with the AEAD tag. -Extraction does not authenticate the returned tag. - -Ship a matching SQL/PL/pgSQL extraction helper and Rust/Go/SQL fixtures with -the storage format. Check versions, lengths and truncation before slicing. -For text storage prefer Base64 when SQL must inspect the header: -[PostgreSQL](https://www.postgresql.org/docs/current/functions-binarystring.html) -has built-in Base64 decoding and byte slicing, but not Base85 decoding. -If SQL extraction from a Base85 envelope is supported, supply an explicit -decoder; do not leave it as an assumed capability. PostgreSQL Base64 output -also inserts line breaks, unlike the canonical Rust wrapper: normalize SQL -output or validate it against the same canonical rules in cross-language -tests. JSON-tree storage needs its own header adapter: its shared key header -is distinct from each node's ciphertext and parent tag. - -The proposed refinement uses the ZeroKMS tag as an HMAC key and replaces it -with a derived tag. **User clarification:** at decryption, ZeroKMS uses the -refined tag for policy checks and logging, and returns the original tag. -The decrypt path uses that returned original tag to reconstruct leaf AAD. -This resolves the conceptual conflict with binding the original tag into -AEAD: the stored/request tag and the authentication tag input need not be -identical after refinement. Keep those roles explicit in the KMS response -and decrypt implementation. Tag extraction returns the tag actually stored -in the envelope, which may be refined. Protocol details remain deferred; -this is the intended behavior, not a claim that the current client/engine -already implements it. Add refinement/decrypt integration coverage when -that work lands. - -**Decision, 2026-10-25:** direct `encrypt` handles one single-value input; -`EncryptInto` handles collection composition and batching. This supersedes -the earlier open question about automatic collection lifting. The agreed -public examples are: +The envelope's ZeroKMS tag begins at byte offset 35, with its length at +offsets 33 and 34. This shows that extraction can be simple. It is not a +commitment to keep those offsets. + +Document the chosen version's layout, and keep it parseable without the +plaintext codec, key retrieval, or decryption. Prefer a checked borrowed +view of the header or envelope, with an accessor explicitly named `key_tag`. +Do not decompose the whole ciphertext, or confuse the key tag with the AEAD +tag. Extraction does not authenticate the returned tag. + +Release a matching SQL and PL/pgSQL extraction helper, and Rust, Go, and SQL +fixtures, with the storage format. Check versions, lengths, and truncation +before slicing. + +For text storage, prefer Base64 when SQL must inspect the header. +[PostgreSQL's binary string +functions](https://www.postgresql.org/docs/current/functions-binarystring.html) +include Base64 decoding and byte slicing, but not Base85 decoding. If SQL +extraction from a Base85 envelope is supported, supply an explicit decoder. +Do not leave Base85 decoding as an assumed capability. + +PostgreSQL's Base64 output also inserts line breaks, unlike the canonical +Rust wrapper. Normalise the SQL output, or validate it against the same +canonical rules in cross-language tests. JSON-tree storage needs its own +header adapter, because its shared key header is distinct from each node's +ciphertext and parent tag. + +The proposed refinement uses the ZeroKMS tag as an HMAC (hash-based message +authentication code) key, and replaces it with a derived tag. The user +clarified the decryption behaviour: ZeroKMS uses the refined tag for policy +checks and logging, and returns the original tag. The decrypt path uses that +returned original tag to rebuild the leaf AAD. + +This resolves the conceptual conflict with binding the original tag into the +AEAD. After refinement, the stored or request tag and the tag used as +authentication input need not be identical. Keep those roles explicit in the +KMS response and the decrypt implementation. Tag extraction returns the tag +actually stored in the envelope. That tag may be the refined one. + +Protocol details remain deferred. This is the intended behaviour, not a +claim that the current client or engine already implements it. Add +integration coverage for refinement and decryption when that work is merged. + +#### Single values for encrypt, collections for EncryptInto + +Decided on 25 October 2026: direct `encrypt` handles one single-value input, +and `EncryptInto` handles collection composition and batching. This replaces +the earlier open question about applying encryption to collections +automatically. The agreed public examples are: ```rust let scalar: Ciphertext = keyset.encrypt(1_i32).await?; @@ -1378,160 +1792,230 @@ let blob: Ciphertext = keyset ``` The batch produces two independent ciphertexts through the shared batch -executor; the blob encodes and seals the complete collection as one value. -Do not add implicit collection handling to direct `encrypt`, or a separate -batch entry point merely to provide functionality already covered by -`EncryptInto`. Context-taking variants follow the same division. +executor. The blob encodes and seals the complete collection as one value. +Do not add implicit collection handling to direct `encrypt`. Do not add a +separate batch entry point only to provide what `EncryptInto` already +covers. Context-taking variants follow the same division. + +Acceptance checks: + +- reject direct `encrypt(Vec)` before execution; +- support `Vec` to `Vec` without context; +- keep target-directed opening and batching of collections; +- recover the blob as a whole collection. -Acceptance checks: reject direct `encrypt(Vec)` before execution; -support context-free `Vec` to `Vec`; preserve target-directed -collection opening and batching; and recover the blob as a whole collection. -Existing collection composition is retained, while opaque `Ciphertext` and -its no-context target support are new work. +Existing collection composition is kept. Opaque `Ciphertext`, and its target +support without context, are new work. The current prototype treats `Vec` as one byte value, but that is not a compatibility constraint. The latest recommendation is uniform collection -behavior: `Vec` to `Vec` encrypts individual bytes, while -`OpaqueBlob>` encodes the complete vector as one value. Do not add -`Vec` to `Ciphertext` conversion or a bytes wrapper solely to retain -prototype behavior. A distinct raw-byte capability would need its own use -case. The blob codec need not reproduce the prototype's raw-byte encoding. - -For `OpaqueBlob`, select the library-owned codec, authenticated codec/type -discriminator and version, schema-evolution policy, error behavior, and -supported source/decoded types. Authenticate before deserializing plaintext. -Do not silently reinterpret an ordinary byte ciphertext as a different -encoding. Preserve an explicit wrapper rather than implicit Serde fallback. - -The constant-time question needs a concrete conclusion: assess `serdect`, -the codec, both sides of Serde (value traits and serializer/deserializer), -length/shape disclosure, and permitted custom implementations. If arbitrary -Serde types remain supported, do not promise end-to-end constant-time -serialization. Select a restricted profile separately if that property is -required; it cannot be inferred from the wrapper or its encryption primitive. - -Completion: scalar and blob examples persist and reload through a fresh -cipher, reject invalid encodings, and verify fixtures for the selected -format. The supported timing contract and codec version are documented. +behaviour. `Vec` to `Vec` encrypts individual bytes, while +`OpaqueBlob>` encodes the complete vector as one value. Do not add a +conversion from `Vec` to `Ciphertext`, or a bytes wrapper, only to keep +prototype behaviour. A distinct raw-byte capability would need its own use +case, and the blob codec need not reproduce the prototype's raw-byte +encoding. + +#### Blob codec, format, and timing requirements + +For `OpaqueBlob`, select: + +- the codec, owned by the library; +- an authenticated codec or type discriminator, and a version; +- the schema-evolution policy; +- the error behaviour; +- the supported source and decoded types. + +Authenticate before deserialising plaintext. Do not silently reinterpret an +ordinary byte ciphertext as a different encoding. Keep an explicit wrapper, +not an implicit Serde fallback. + +The constant-time question needs a concrete conclusion. Assess: + +- `serdect`; +- the codec; +- both sides of Serde: the value traits, and the serialiser and + deserialiser; +- disclosure of length and shape; +- which custom implementations are allowed. + +If arbitrary Serde types remain supported, do not promise constant-time +serialisation from end to end. If that property is required, select a +restricted profile separately. Constant-time behaviour cannot be inferred +from the wrapper or its encryption primitive. + +Completion criteria for work area C: scalar and blob examples persist, and +reload through a new cipher instance. The examples reject invalid encodings +and verify fixtures for the selected format. The supported timing contract +and the codec version are documented. ### D. Complete record derives and preserve collection composition -Implement explicit passthrough and source-field renaming, independently of -context selection and storage. Generate decrypt reconstruction for the -same record. Support nested targets with and without configuration, and -report incompatible context or source types at their declarations. +Implement explicit passthrough and renaming of source fields, independently +of context selection and storage. Generate decrypt reconstruction for the +same record. Support nested targets with and without configuration. Report +incompatible context or source types at their declarations. + +Keep the existing collection behaviour: `Vec` to `Vec` composes the +target operations into one logical batch, instead of awaiting each row +separately. Keep the key granularity per document and per leaf. One logical +batch does not promise one physical ZeroKMS request beyond the backend's +limits. + +Completion criteria for work area D: the public examples of mixed records +and searchable records run. These examples include batch encrypt and +decrypt, nested targets, renamed passthrough fields, and stored context. +Dynamic and Go declarations produce equivalent semantics. A plaintext-side +derive that generates output types is a deferred convenience, not a +prerequisite for these workflows. + +### E. Finish the configured policy's lifetime and query interface + +Use the query-design baseline and reconciliation in [Prior query +design](#prior-query-design-baseline-and-reconciliation). The query model is +not an empty design space. A detailed proposal already exists for +field-owned index selection, typed handles, and shared derivation. + +The direction of typed factories is established. The [target definition +section](#recommended-owner-a-reusable-typed-target-definition) now proposes +their owner. The owner is a shared immutable fixed definition or an explicit +runtime instance, with generated typed field handles. Review that proposal, +and prove its initialisation and lifetime behaviour, before treating the +claim of reuse after one resolution as implemented. Runtime configuration +with several fields, and the exact generated signatures, still need to be +specified. -Keep the collection behavior already present: `Vec` to `Vec` composes -the target operations into one logical batch, rather than awaiting every -row separately. Preserve document/leaf key granularity; one logical batch -does not promise one physical ZeroKMS request beyond backend limits. +Write complete static and Go examples for equality, JSON containment, +ordered paths, and extraction, all using the write policy. Specify query +results and the errors for unsupported operations. Keep those examples free +of mandatory hand-written plans. Keep direct typed EQL conversion, and +support in the catalog and guest, as deliverables of the configured-target +sequence. -Completion: the public mixed-record and searchable-record examples run, -including batch encrypt/decrypt, nested targets, renamed passthrough fields, -and stored context. Dynamic/Go declarations produce equivalent semantics. -A plaintext-side derive generating output types is deferred convenience, -not a prerequisite for these workflows. +### F. Make naming and coordinated integration part of completion -### E. Finish the configured policy's lifetime and query surface +Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for: -Use the prior query-design baseline and reconciliation above. The query -model is not an empty design space: field-owned index selection, typed -handles, and shared derivation already have a detailed proposal. +- the proposed role-based JSON names; +- the retained `OpaqueBlob` and `Ciphertext` spellings; +- the shared format crate; +- the split between primitives and the application layer. -The typed-factory direction is established. The target-definition section -above now proposes its owner: a shared immutable fixed definition, or an -explicit runtime instance, with generated typed field handles. Review that -proposal and prove its initialization/lifetime behavior before treating the -once-only reuse claim as implemented. Multi-field runtime configuration and -the exact generated signatures still require specification. +Keep the accepted names of the declaration and convenience traits. +Recommended: rename the plan builder's output-selection method to `target`, +and distinguish it from executing `EncryptInto`. New spellings and crate +moves still need review. -Write complete static and Go examples for equality, JSON containment, -ordered paths, and extraction, all using the write policy. Specify query -results and unsupported-operation errors. Keep those examples free of -mandatory hand-written plans. Preserve direct typed EQL conversion and -catalog/guest support as deliverables of the configured-target sequence. +Remove replaced APIs directly, and update the EQL, Go, and guest consumers +together. There is no requirement to read prototype ciphertext or keep old +imports. Specify the selected encoding and cryptographic semantics, and set +up fixtures for that contract from now on. -### F. Make naming and coordinated integration part of completion +Completion criteria for work area F: + +- all public examples are executable tests; +- cross-opening and querying between the static and dynamic paths pass + against the selected storage format; +- batching checks pass; +- the guest builds remain supported. -Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for the -proposed role-based JSON names, retained `OpaqueBlob`/`Ciphertext` spellings, -shared format crate and primitive/application split. Retain accepted -declaration/convenience trait names. Recommend renaming the plan builder's -output-selection method to `target`; distinguish it from executing -`EncryptInto`. New spellings and crate moves still require review. - -Remove replaced APIs directly and update EQL, Go, and guest consumers -together. There is no requirement to read prototype ciphertext or preserve -old imports. Specify the selected encoding and cryptographic semantics, -and establish fixtures for that contract going forward. - -Completion: all public examples are executable tests, static/dynamic -cross-opening and querying pass against the selected storage format, batching -checks pass, and the guest builds remain supported. No new API is described -as implemented until its Rust and Go acceptance criteria are met. -These are full redesign completion criteria. The earlier Go release uses -the facade-independent Go, engine and storage gates listed above. +No new API is described as implemented until it meets its Rust and Go +acceptance criteria. These are the completion criteria for the full +redesign. The earlier Go release uses the gates for Go, the engine, and +storage in the [implementation +sequence](#configured-target-implementation-sequence-and-completion-criteria). +Those gates do not depend on the facade. ## Recommended order for the remaining design decisions -### Decision checkpoint, 2026-10-25 - -Settled direction: keyset-directed single-value encryption; target-directed -collection batching; explicit `OpaqueBlob`; conversion methods separate -from `EncryptTarget`/`DecryptSource` declarations; explicit context-input -capabilities instead of `Default`; typed Rust options factories and dynamic -deserialization into the same options; static and dynamic support together; -shared typed definitions and field handles; a consistent `using` selector -with strict typestate and definition-first query spelling; `OpaqueEncode` -/ `OpaqueDecode`; consuming encrypt and encrypt_into with guarded cleanup; -ordinary keyset decrypt alongside the destination-directed decrypt pair, -both with context variants and ordinary owned `T: Zeroize` results; -separate ciphertext context and search scope with `PrimaryKey`/`ScopeKey` -eligibility; definition-time role validation in the dynamic interface, -optional named-type declarations and explicit scope eligibility; and a -versioned interoperability specification with shared conformance coverage; -and no backward compatibility with the prototype. Do not reopen these -choices merely because older examples or work-area descriptions retain -provisional wording. - -The remaining product/API decisions are narrower. Distinguish them from -specification and implementation obligations: +### Decision checkpoint on 25 October 2026 + +These directions are settled: + +- single-value encryption directed by the keyset; +- collection batching directed by the target; +- explicit `OpaqueBlob`; +- conversion methods separate from the `EncryptTarget` and `DecryptSource` + declarations; +- explicit context-input capabilities instead of `Default`; +- typed Rust options factories, and dynamic deserialisation into the same + options; +- static and dynamic support together; +- shared typed definitions and field handles; +- a consistent `using` selector, with strict typestate and definition-first + query spelling; +- `OpaqueEncode` and `OpaqueDecode`; +- consuming `encrypt` and `encrypt_into`, with guarded cleanup; +- ordinary keyset decrypt alongside the destination-directed decrypt pair, + both with context variants, and both returning ordinary owned `T: Zeroize` + results; +- separate ciphertext context and search scope, with `PrimaryKey` and + `ScopeKey` eligibility; +- validation of roles when a dynamic definition is loaded, optional + named-type declarations, and explicit scope eligibility; +- a versioned interoperability specification with shared conformance + coverage; +- no backward compatibility with the prototype. + +Do not reopen these choices only because older examples or descriptions of +work areas still use provisional wording. + +### Remaining questions + +The remaining product and API decisions are narrower. Keep them separate +from the obligations to specify and implement: | Remaining question | Decision required | | --- | --- | -| JSON contract completion | Main Go surface is implemented. Close parent-bound child opening and formal conformance gaps; use current cascade/random-order semantics. Do not reopen operator design or rename shipped Go APIs for cosmetic consistency. | -| Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | -| Customer storage profiles | Go opaque storage is integrated. Replace temporary Go dependency pins for final release and freeze post-cascade JSON/search profiles. Rust blob conveniences and crate relocation may follow. | -| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenizers/filters remain, including #1112. Reconcile the earlier privacy concern explicitly; merging support is not proof of unchanged leakage. | -| Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | - -Next complete the supported-contract specification and remaining customer -configuration/read gaps, then validate the candidate and resolve final-release -pins. The original JSON vertical slice is already implemented. Packaging -remains deferred; keep initial ORE equality policy unchanged. - -Specification/implementation work still includes exact context/scope grouping -and encoding, dynamic grammar and version identifiers, error codes, binary -envelope fields, Rust/Go/SQL fixtures, and the refined-versus-original tag -flow. Record candidate choices in the formal specification for review; do -not treat every byte offset or adapter signature as a new architectural -decision. The initial persistence recommendation remains ordinary Serde -composition for explicit layouts, without a new arbitrary-tree wire format. - -Cleanup implementation still needs guarded encoder sinks, partial decode -handling, cancellation/drop checks, and an explicit drop-guarded result -adapter. Under the user's latest sequencing decision, implementation -improvements may follow the stable format; preserve the stored contract and -document actual guarantees instead of claiming the eventual goals are -already satisfied. No completed constant-time/zeroization audit is implied. - -The shared-definition caching, initialization error/retry behavior, generic -identity, and request lifetimes still need concrete signatures and proof. -That is implementation design within the recorded ownership model, not a -reason to reopen the question of whether writes and queries share a policy. +| JSON contract completion | The main Go interface is implemented. Close the gaps in opening a child bound to its parent, and in formal conformance. Use the current semantics of cascaded tags and random order. Do not reopen operator design, or rename released Go APIs for cosmetic consistency | +| Key selection and optional provider policy | Review keyset selection derived from plaintext, routed execution scoped to a client, and ZeroKMS lock context, as optional provider capabilities. In the provider-capabilities companion, assess the neutral boundary between data keys and index keys, and the deliberate non-use of Vitamin C's `AsyncCipher` | +| Customer storage profiles | Go opaque storage is integrated. For the final release, replace the temporary Go dependency pins, and freeze the post-cascade JSON and search profiles. Rust blob conveniences and crate moves can follow | +| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenisers and filters remain, including #1112. Reconcile the earlier privacy concern explicitly. Merging support does not prove that leakage is unchanged | +| Packaging and names (review in progress) | Review the packaging companion's shared format crate, value execution owned by Stack, a Vitamin C boundary of primitives only, retirement of `AsyncCipher`, the ZeroKMS client rename, and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again | + +### Next steps + +Next, complete the specification of the supported contract, and close the +remaining customer gaps in configuration and reads. Then validate the +release candidate, and resolve the pins for the final release. The original +end-to-end JSON implementation is already done. Packaging remains deferred. +Keep the initial ORE equality policy unchanged. + +Specification and implementation work still includes: + +- exact grouping and encoding of context and scope; +- the dynamic grammar and version identifiers; +- error codes; +- binary envelope fields; +- Rust, Go, and SQL fixtures; +- the flow of the refined tag and the original tag. + +Record candidate choices in the formal specification for review. Do not +treat every byte offset or adapter signature as a new architectural +decision. The initial recommendation for persistence remains ordinary Serde +composition for explicit layouts, without a new wire format for arbitrary +trees. + +Cleanup implementation still needs: + +- guarded encoder sinks; +- handling of partial decoding; +- checks for cancellation and drop; +- an explicit adapter for drop-guarded results. + +Under the user's latest sequencing decision, implementation improvements can +follow the stable format. Keep the stored contract, and document the actual +guarantees, instead of claiming that the eventual goals are already met. +This plan does not imply a completed constant-time or zeroisation audit. + +The caching of shared definitions, the behaviour on initialisation error and +retry, generic identity, and request lifetimes still need concrete +signatures and proof. That is implementation design within the recorded +ownership model. It is not a reason to reopen whether writes and queries +share a policy. Passthrough implementation, transcoder adapters, collection composition, -trait relocation, documentation updates, and acceptance tests remain work -items. They are not reasons to characterize the agreed API direction as -undecided. No backward-compatibility phase is needed for the prototype; -the customer release establishes a new Go and persisted-data commitment. +moving traits, documentation updates, and acceptance tests remain work +items. These items are not reasons to describe the agreed API direction as +undecided. The prototype needs no backward-compatibility phase. The customer +release creates a new commitment for Go and for persisted data. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index 4fa7e186d..ef9bc092c 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1,65 +1,102 @@ # Stack Encrypt API review -Started: 2026-10-24. Scope: the current Rust `stack-encrypt` API, starting -with encrypting one `i32` into opaque ciphertext and storing the result. - -Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md), -an API design draft covering encryption, persistence, decryption, and queries. -The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) -explore context sources and storage using Serde-inspired attribute syntax. -The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records -requirements, recommended decisions, sequencing, and completion criteria. - -This is a running issue log. Add a numbered section as each issue is found; -keep observations, proposed changes, and unresolved decisions distinct. -All issues below remain unimplemented. Individual decisions are recorded -within each issue; proposed APIs are illustrative, not implemented. - -**User clarification, 2026-10-25:** this API has never shipped beyond an -early prototype. Earlier suggestions to preserve old encodings or public -imports are superseded: backward compatibility and prototype data migration -are not requirements. New Rust, Go, guest, and EQL implementations must -agree with each other, but need not retain the prototype's representations. - -## 1. Cipher trait mismatch and explicit context ergonomics +Started on 24 October 2026. This review covers the current Rust +`stack-encrypt` API. It started with one task: encrypt one `i32` into +opaque ciphertext and store the result. + +This document is a running log of the issues found. Each numbered section +records one issue: what the code does now, what changes were proposed, and +which decisions are made or still open. The proposed APIs are illustrations, +not implemented code. All the issues below are still unimplemented. Add a +numbered section for each new issue, and keep observations, proposed +changes, and open decisions apart. + +Three companion documents go with this log: + +- [Complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md) + is a draft API design that covers encryption, storage, decryption, and + queries. +- [Directional context examples](2026-10-24-stack-encrypt-context-examples.md) + explores where context comes from and where it is stored, using attribute + syntax modelled on Serde. +- [The implementation plan](2026-10-24-stack-encrypt-api-plan.md) records the + requirements, the recommended decisions, the order of work, and when the + work is complete. + +## No backward compatibility with the prototype + +On 25 October 2026 the user clarified that this API has never shipped beyond +an early prototype. Backward compatibility, and migration of prototype data, +are not requirements. This replaces any earlier suggestion in this log to +keep old encodings or public imports. The new Rust, Go, guest, and EQL +(Encrypt Query Language) implementations must agree with each other. They do +not need to keep the prototype's representations. + +Issue 22 limits this exemption: once the customer release ships, customer +data must not be broken. + +## Terms used in this log + +These terms appear in many issues. Terms used in only one issue are defined +where they first appear. + +| Term | Meaning | +| --- | --- | +| Vitamin C | CipherStash's Rust cryptography library, published as the `vitaminc-*` crates. Stack Encrypt uses version 0.8.0 | +| ZeroKMS | CipherStash's key management service, which issues and retrieves data keys | +| KMS | Key management service | +| EQL | Encrypt Query Language, the SQL and storage format for searchable encrypted data in PostgreSQL | +| AEAD | Authenticated encryption with associated data | +| AAD | Additional authenticated data: data that the ciphertext is bound to, but that is not encrypted | +| Context | The application data a caller binds a ciphertext to, such as a column name. Stack Encrypt derives each leaf's AAD from it, and sends ZeroKMS a text descriptor of it, such as `users/age` | +| Leaf | One sealed value, `SealedValue`, in Vitamin C's ciphertext tree | +| Node | One element of Vitamin C's ciphertext tree, `Node`: a single leaf, a sequence, a map, an empty container, or a passthrough value | +| Passthrough | A field that is copied into the output in plaintext, without encryption | +| Target | An output type that declares how a plaintext is encrypted into it, through `EncryptFrom` | +| Plan | The lower-level builder, `Plan`, that composes encryption operations. The derives generate plans, and the dynamic interface uses them | +| Dynamic interface | The interface driven by data instead of Rust types, used through FFI (foreign function interface) and by the Go SDK | +| Guest | The WebAssembly build of the Rust engine that the Go SDK runs | +| ORE | Order-revealing encryption, the index used for range queries and sorting | + +## Summary of status + +| Issue | Status on the last update | +| --- | --- | +| 1. Cipher trait mismatch | Open. The user asked for separate `encrypt` and `encrypt_with_context` methods | +| 2. Opaque ciphertext storage | Open. Decisions recorded for ordinary encryption, collections, and text encodings | +| 3. Plaintext-side encryption | Open. Receiver decided on 24 October 2026 | +| 4. Fixed context on the scalar derive | Direction agreed. Implementation pending | +| 5. `EncryptFrom` has no `encrypt_from` | Open | +| 6. Plan builder `encrypt_into` | Open. Revisit during the plan API discussion | +| 7. Mixed encrypted and plaintext fields | Passthrough attribute accepted in principle. Implementation pending | +| 8. Whole-value encryption and layouts | Open architectural discussion | +| 9. Ownership across Stack and Vitamin C | Open architectural discussion. Moving the traits is proposed, not authorised | +| 10. Public `encrypt_as` | Decision agreed. Implementation pending | +| 11. Context from an input struct | Open | +| 12. `Default` compared with optional context | Open | +| 13. JSON layouts and output adaptation | Open. One follow-up agreed | +| 14. Decrypt context variants | Direction required by the user. Receiver still to confirm | +| 15. Go JSON work in pull request #1353 | Coordination item | +| 16. Prior query design | Adopted as the baseline | +| 17. One name for definition selection | Recommendation. Final spelling not accepted | +| 18. Consuming inputs with cleanup | Decisions made on 25 October 2026 | +| 19. First-release completeness | Decisions accepted on 25 October 2026. JSON contract under user review | +| 20. Keysets chosen from plaintext | Recommendations, not accepted | +| 21. Packaging and `AsyncCipher` | Proposals | +| 22. Customer-driven order of work | Order of work set by the customer's needs. Merged work reconciled on 27 October 2026 | + +## 1. Cipher trait mismatch and the required context argument **Status:** Open. -### Current behavior - -`KeysetCipher::encrypt` is an inherent method, not a trait method: - -```rust -let ciphertext = keyset.encrypt(42_i32, ()).await?; -let ciphertext = keyset.encrypt(42_i32, "users/age").await?; -``` - -Vitamin C (`vitaminc-aead` 0.8.0) provides both `Cipher` and `AsyncCipher`. -Stack implements the synchronous `Cipher` trait on internal adapters after -fetching keys: `MintedKeys` seals leaves and `BoundKeys` opens them. Neither -public cipher implements `AsyncCipher`, so callers cannot use -`KeysetCipher` directly with generic Vitamin C code requiring that trait. - -Each internal adapter implements only half of `Cipher`: `MintedKeys::open` -and `BoundKeys::seal` always return errors. - -### Why the mismatch exists - -`AsyncCipher` already supports batching through `seal_many` and `open_many`; -batching alone does not explain the separate API. Its methods receive each -leaf's derived AAD, while ZeroKMS needs the original structured context's -descriptor. That original context is not carried by the derived leaf AAD. -Stack's pending-operation layer also combines key requests across documents, -fields, and rows, beyond the trait's per-document batch interface. - -These constraints need to survive any trait integration. The current -separation is deliberate, but leaves an interoperability gap and internal -implementations with unsupported operations. +The user asked for two separate methods, so that the common case needs no +empty context argument. Questions remain about how Stack's ciphers relate +to Vitamin C's cipher traits. ### Requested API direction -Requiring an empty tuple for the common case of no application AAD is -unnecessary ceremony. Use distinct methods, following the convention used +Passing an empty tuple, `()`, for the common case of no application AAD is +needless ceremony. Use separate methods, following the convention used elsewhere: ```rust @@ -67,50 +104,142 @@ keyset.encrypt(42_i32).await?; keyset.encrypt_with_context(42_i32, "users/age").await?; ``` -The no-context method should select the same empty context currently -represented by `()`. This distinction should exist even if `KeysetCipher` -continues to use inherent methods rather than implementing a Vitamin C trait. +The method without context selects the same empty context that `()` +represents today. Make this split even if `KeysetCipher` keeps inherent +methods instead of implementing a Vitamin C trait. + +### Current behaviour -### Open design questions +`KeysetCipher::encrypt` is an inherent method, not a trait method: -- Can a context-bound adapter implement `AsyncCipher`, or should Vitamin C's - interface preserve the original context explicitly? -- How does that integration preserve batching across multiple documents? -- Should sealing and opening be separate trait capabilities, avoiding methods - that are present but always fail? -- Should decryption expose the corresponding `decrypt` / - `decrypt_with_context` distinction? +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let ciphertext = keyset.encrypt(42_i32, "users/age").await?; +``` -**Source:** [cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) -(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's batching -discussion). Vitamin C trait definitions are in `vitaminc-aead` 0.8.0, -`src/seal.rs`. +Vitamin C (`vitaminc-aead` 0.8.0) provides the `Cipher` trait and the +`AsyncCipher` trait. Stack implements the synchronous `Cipher` trait on +internal adapters after it fetches keys. `MintedKeys` seals leaves, and +`BoundKeys` opens them. Neither public cipher implements `AsyncCipher`. +Callers therefore cannot pass `KeysetCipher` to generic Vitamin C code that +needs that trait. -## 2. The Node / SealedValue boundary complicates opaque ciphertext storage +Each internal adapter implements only half of `Cipher`. `MintedKeys::open` +and `BoundKeys::seal` always return errors. -**Status:** Open. +### Why the mismatch exists -### Current behavior +Batching alone does not explain the separate API, because `AsyncCipher` +already batches through `seal_many` and `open_many`. The real reasons are +these: + +- `AsyncCipher`'s methods receive each leaf's derived AAD. ZeroKMS needs a + descriptor of the original structured context, and the derived leaf AAD + does not contain that context. +- Stack's layer for pending operations combines key requests across + documents, fields, and rows. The trait's batch interface works on one + document at a time. + +Any trait integration has to keep meeting these constraints. The current +separation is deliberate. It still leaves a gap for code that is generic +over Vitamin C's traits, and internal implementations with operations that +always fail. + +### Open questions + +- Can an adapter bound to a context implement `AsyncCipher`, or does + Vitamin C's interface need to carry the original context explicitly? +- How does that integration keep batching across several documents? +- Should sealing and opening be separate trait capabilities, so that no + method is present but always fails? +- Should decryption have the matching `decrypt` and `decrypt_with_context` + split? + +**Source:** +[cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) +(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's +discussion of batching). Vitamin C's trait definitions are in +`vitaminc-aead` 0.8.0, `src/seal.rs`. + +## 2. The Node and SealedValue types make opaque ciphertext storage harder + +**Status:** Open. Several decisions are recorded below. + +### Decisions so far + +- **Ordinary encryption produces one opaque ciphertext.** To encrypt a + value structurally, the caller chooses an explicit target. To encrypt a + composite plaintext as one ciphertext, the caller encodes it first with + `OpaqueBlob`. The + [implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends + adapting a native single sealed leaf through the existing transcoder. The + transcoder is the `target::transcode` protocol described in issue 13. This + avoids serialising a whole encrypted tree or encrypting again. +- **Collections, decided on 25 October 2026.** Direct `encrypt` takes a + single value. It does not lift itself over collections automatically. + Collections are still composed through `EncryptInto`, for example + `let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. + To encode and seal a whole collection as one ciphertext, use + `OpaqueBlob>`. The collection question had been reopened earlier. + This decision settles it. +- **Text encodings, decided on 25 October 2026.** Text conversions must use + the existing `Base64Text` and `Base85Text` wrappers from the EQL bindings. + The + [persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) + record the proposals and the limit on decoding Base85 in SQL. + +The follow-up of 25 October 2026 also set these directions: + +- Explore moving these general encoding types to a layer below Stack and + EQL, and supporting encoded field targets specific to ciphertext. +- Rust and PL/pgSQL must both be able to extract the stored ZeroKMS key tag + with simple code. +- A protocol for refined tags is being designed separately. The proposed + refinement replaces the ZeroKMS tag with a tag derived from it. The user + clarified that ZeroKMS checks policy and writes logs against the refined + tag, and returns the original tag. The decrypt path uses the original tag + to rebuild the leaf AAD. Record these two separate roles in the + implementation. + +### Open questions + +- Should byte buffers be spelled `Vec`, or use an explicit bytes + wrapper? +- What form should a simple round trip take that stores and loads opaque + ciphertext for scalar values, with direct conversion to bytes and back? + It could be a dedicated scalar result type, a wrapper, or conversion + helpers on the existing API. +- Should Base64 and Serde be supported conveniences? +- Does serialising a whole tree belong in Stack, or does it stay with the + caller? If it stays with the caller, say so explicitly, and keep it out + of the simple scalar path. + +The prototype's canonical leaf encoding is a useful starting point. It is +not a backward-compatibility obligation. + +### Current behaviour `KeysetCipher::encrypt` returns `StackCipherText`, an alias for Vitamin C's -`Node`. Encrypting an `i32` produces `Single(SealedValue)`, but -the return type also admits sequences, maps, empty containers, and passthrough -nodes. It does not retain the plaintext type as a type parameter. +`Node`. Encrypting an `i32` produces `Single(SealedValue)`. +The return type also allows sequences, maps, empty containers, and +passthrough nodes. It does not keep the plaintext type as a type parameter. | Capability | `StackCipherText` tree | `SealedValue` leaf | | --- | --- | --- | | Debug printing (`{:?}`) | Yes | Yes | -| Serde serialization/deserialization | No | Yes | -| Canonical byte encoding | None | `to_bytes()` / `from_bytes()` | -| Direct Base64 conversion | None | None; encode the canonical bytes separately | +| Serde serialisation and deserialisation | No | Yes | +| Canonical byte encoding | None | `to_bytes()` and `from_bytes()` | +| Direct Base64 conversion | None | None. Encode the canonical bytes separately | -The leaf's canonical bytes include its keyset id, key retrieval metadata, -and local ciphertext. That versioned byte format is the storage commitment; -the leaf's Serde representation is not the same compatibility commitment. +A leaf's canonical bytes contain its keyset ID, the metadata needed to +retrieve its key, and the local ciphertext. That versioned byte format is +the storage commitment. The leaf's Serde representation is not the same +compatibility commitment. -### Caller impact +### Effect on callers -Even the scalar storage path requires exposing the container representation: +Even storing a scalar makes the caller handle the container +representation: ```rust let ciphertext = keyset.encrypt(42_i32, ()).await?; @@ -125,58 +254,24 @@ let ciphertext = StackCipherText::Single(leaf); let value: i32 = cipher.decrypt(ciphertext, ()).await?; ``` -The caller must handle a shape mismatch that cannot occur for this input, -extract the leaf, and reconstruct the node to decrypt. There is no direct -`ciphertext.to_bytes()` or `serde_json::to_string(&ciphertext)` path. Base64 -adds a separate encoding step. Collections additionally require the caller -to choose a representation preserving the tree structure and node variants. - -Stored bytes are not sufficient by themselves to decrypt: the caller still -needs authorized ZeroKMS access and the same application context. The storage -API should make that requirement clear without exposing tree internals for -the simple scalar case. - -### Proposed direction and open decisions - -**Subsequent decision:** ordinary encryption should produce one opaque -ciphertext; choose structural encryption through an explicit target, or -encode a composite plaintext first with `OpaqueBlob`. The -[implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends -adapting a native single sealed leaf through the existing transcoder, -without serializing an entire encrypted tree or re-encrypting it. - -**Collection decision, 2026-10-25:** direct `encrypt` handles a single-value -input and does not automatically lift over collections. Retain collection -composition through `EncryptInto`, including -`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. -Use `OpaqueBlob>` to encode and seal a complete collection as one -ciphertext. This settles the previously reopened collection question. -The spelling for byte buffers (`Vec` versus an explicit bytes wrapper) -remains a separate detail to resolve. - -Provide a straightforward opaque ciphertext storage round trip for scalar -values, with direct byte conversion and reconstruction. Decide whether this -means a dedicated scalar result type, a wrapper, or conversion helpers on -the existing API. - -Also decide whether Base64 and Serde should be supported conveniences, and -whether whole-tree serialization belongs in Stack or remains caller-owned. -If it remains caller-owned, make that distinction explicit and keep it out -of the simple scalar path. The prototype's canonical leaf encoding is a -useful starting point, not a backward-compatibility obligation. - -**Follow-up, 2026-10-25:** text conversions must use the existing -`Base64Text` / `Base85Text` wrappers from EQL bindings. Explore moving these -general encoding types below Stack/EQL and supporting ciphertext-specific -encoded field targets. The persisted ZeroKMS key tag must be extractable -simply in Rust and PL/pgSQL. A refinable-tag protocol is being designed -separately. The user clarified that ZeroKMS checks policy/logs against the -refined tag and returns the original tag for reconstruction of leaf AAD; -record these distinct roles in the implementation. The -[persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) -record the proposals and the Base85/SQL decoding constraint. - -**Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) +The caller has to: + +1. handle a shape mismatch that cannot happen for this input; +2. extract the leaf; +3. rebuild the node before decrypting. + +There is no direct `ciphertext.to_bytes()` or +`serde_json::to_string(&ciphertext)` path. Base64 adds a separate encoding +step. For collections, the caller also has to choose a representation that +keeps the tree structure and the node variants. + +Stored bytes alone are not enough to decrypt. The caller also needs +authorised access to ZeroKMS and the same application context. The storage +API should make that clear without exposing tree internals in the simple +scalar case. + +**Sources:** +[ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) (`StackCipherText`, `SealedValue`) and [storage documentation and public re-exports](../../packages/stack-encrypt/src/lib.rs). Vitamin C's container definition is in `vitaminc-aead` 0.8.0, @@ -184,139 +279,124 @@ Vitamin C's container definition is in `vitaminc-aead` 0.8.0, ## 3. Plaintext-side encryption has different syntax and context requirements -**Status:** Open. - -### Current behavior +**Status:** Open. The receiver for ordinary `encrypt` was decided on 24 +October 2026. The differing requirements for an empty context are still +open. -The current Vitamin C `Encrypt` trait describes a plaintext through `visit`; -it does not provide an `encrypt` method. Importing Stack's re-export of that -trait does not enable `10_i32.encrypt(&keyset)`. Vitamin C 0.8.0 documents -this trait as replacing the older visitor-based trait that drove a cipher -directly. Its `ContextTag` wrapper has encryption methods, but those require -Vitamin C cipher traits and do not make the syntax available on bare integers -or accept Stack's public ciphers directly. +### Decision: the keyset is the receiver for ordinary encryption -Stack supplies its own blanket `EncryptInto` extension trait: - -```rust -use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; - -let ciphertext: StackCipherText = 10_i32 - .encrypt_into_with_context(&keyset, nonempty!("users/age")) - .await?; -``` - -The output type selects the `EncryptFrom` implementation. For -`StackCipherText`, that implementation requires an `Encrypt + Clone` source -and an `AeadContext`, which must be nonempty. - -Although `EncryptInto` also has `encrypt_into(&keyset)`, that method requires -the target's context to implement `Default`. `AeadContext` does not, so this -no-context form cannot produce `StackCipherText`. It is available for other -targets with default contexts, such as declarations carrying their own -contexts. - -The scalar derive's implicit context choice is also a discoverability concern: -`#[stash(plaintext = i32)]` generates `type Context = CallerContext`, even -for a record containing only a `StackCipherText` field (whose own context is -`AeadContext`). The generated operation converts between those types. The -reviewer found that hidden choice surprising; whether it needs an API change -or clearer diagnostics/documentation remains undecided. The `Default` bound -belongs to `EncryptInto::encrypt_into`, not to the `EncryptFrom` trait itself. - -### Caller impact and open decisions - -Plaintext-side encryption exists, but is not a symmetric spelling of -`keyset.encrypt(value, context)`: it uses different method names, selects an -output through a target trait, adds a `Clone` bound for this output, and -refuses the empty context accepted by the inherent method. - -Resolve the differing empty-context requirements alongside issue 1; -renaming alone would not make the two paths equivalent. This inconsistency -is separate from the receiver choice below. - -### Receiver choice: plaintext or keyset? - -**Decision (2026-10-24):** use only the keyset receiver for the ordinary -`encrypt` method: +**Decided on 24 October 2026.** Use only the keyset as the receiver of the +ordinary `encrypt` method: ```rust tenant_keyset.encrypt(value).await?; ``` -Do not add a plaintext-side `value.encrypt(&keyset)` counterpart. -**`EncryptInto` remains.** This decision concerns only the `encrypt` method, -not removal of the target-directed encryption API. Implementation is pending; -the current inherent method still requires its context argument (issue 1). +Do not add a plaintext-side counterpart, `value.encrypt(&keyset)`. -The two receiver forms express the same inputs and operation. Neither inherently provides -better type safety, ownership behavior, performance, or batching; those -depend on the signatures and implementation, not which argument is `self`. +**`EncryptInto` stays.** This decision covers only the `encrypt` method. It +does not remove the API that encrypts into a chosen target. Implementation +is pending. The current inherent method still requires its context argument +(issue 1). + +Both receiver forms take the same inputs and perform the same operation. +Neither gives better type safety, ownership behaviour, performance, or +batching by itself. Those depend on the signatures and the implementation, +not on which argument is `self`. | Consideration | Plaintext receiver | Keyset receiver | | --- | --- | --- | | Mental model | Transform this value using a capability | Ask this encryption capability to process a value | -| Discoverability | Requires importing an extension trait; methods appear on plaintext types | Encryption methods are found on the encryption handle | -| Fluent composition | Reads naturally after an expression producing plaintext | Reads naturally when selecting a keyset before an operation | -| Operational scope | Keyset and its authority appear as an argument | Keyset and its authority lead the call | -| API surface | Adds methods to primitives and application types | Keeps execution methods on library-owned handles | +| Discoverability | The caller imports an extension trait. Methods appear on plaintext types | Encryption methods are found on the encryption handle | +| Fluent composition | Reads naturally after an expression that produces plaintext | Reads naturally when a keyset is chosen before an operation | +| Operational scope | The keyset and its authority appear as an argument | The keyset and its authority come first in the call | +| API surface | Adds methods to primitives and application types | Keeps execution methods on handles the library owns | | Generic integration | Can work through a cipher trait | Can work through a cipher trait | -**Rationale:** Stack performs an asynchronous operation through a configured KMS -capability; putting that capability first makes its scope visible and gives -callers one place to discover operations. Plaintext traits can describe how -values are represented without also owning the execution API. - -Keeping both equivalent spellings adds documentation, completion, and naming -choices without adding capability. Familiarity with an earlier API is a -migration consideration, but does not by itself justify two permanent entry -points. +The reasons for the decision: -The target-directed `EncryptInto` surface can choose outputs other than -ordinary ciphertext and is retained explicitly. Its context inconsistency -remains open independently of this receiver decision. No decision about -decryption or batch method receivers is made here. +- Stack performs an asynchronous operation through a configured KMS + capability. Putting that capability first makes its scope visible. It + also gives callers one place to find operations. +- Plaintext traits can describe how values are represented without also + owning the execution API. +- Two equivalent spellings add documentation, completion entries, and + naming choices, but no capability. Familiarity with an earlier API matters + for migration, but does not by itself justify two permanent entry points. -**Sources:** [plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) -(`EncryptInto`, `EncryptFrom for StackCipherText`) and -[context types](../../packages/stack-encrypt/src/target/context.rs) -(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, -`src/plaintext/mod.rs`. +The `EncryptInto` API can choose outputs other than ordinary ciphertext. The +decision keeps it explicitly. Its context inconsistency is open, separately +from this decision. This decision does not cover the receivers for +decryption or batch methods. -## 4. The scalar derive cannot declare a fixed literal context +### Current behaviour -**Status:** Direction agreed; implementation pending. +Vitamin C's current `Encrypt` trait describes a plaintext through `visit`. +It has no `encrypt` method. Importing Stack's re-export of that trait does +not make `10_i32.encrypt(&keyset)` work. Vitamin C 0.8.0 documents this +trait as the replacement for an older trait. The older trait used a visitor +to drive a cipher directly. -The desired combination is a derived output for a bare `i32`, a fixed -context declared on that output, and `10_i32.encrypt_into(&keyset)`. -The current macro rejects `#[stash(plaintext = i32, context = "example")]`: -the literal `context` attribute is supported only with `struct = ..`. +Vitamin C's `ContextTag` wrapper has encryption methods, but those require +Vitamin C's cipher traits. The methods do not make the syntax available on +bare integers. They also do not accept Stack's public ciphers directly. -A supported derive-only workaround wraps the integer in a plaintext struct: +Stack supplies its own blanket extension trait, `EncryptInto`: ```rust -struct IntegerInput { - value: i32, -} - -#[derive(stack_encrypt::EncryptFrom)] -#[stash(struct = IntegerInput, context = "example")] -struct EncryptedInteger { - value: stack_encrypt::StackCipherText, -} +use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; -// With EncryptInto imported: -let encrypted: EncryptedInteger = IntegerInput { value: 10 } - .encrypt_into(&keyset) +let ciphertext: StackCipherText = 10_i32 + .encrypt_into_with_context(&keyset, nonempty!("users/age")) .await?; ``` -This form declares the field's context as the pair `("example", "value")`, -rendered as the ZeroKMS descriptor `example/value`, and uses defaultable -`DeclaredContext` for optional caller extensions. It is not a direct -conversion from a bare integer. A manual `EncryptFrom` implementation -can supply its own context; the ordinary literal attribute cannot express -that on a scalar derive today. +The output type selects the `EncryptFrom` implementation. For +`StackCipherText`, that implementation requires a source that implements +`Encrypt + Clone`, and an `AeadContext`. An `AeadContext` must not be +empty. + +`EncryptInto` also has `encrypt_into(&keyset)`. That method requires the +target's context to implement `Default`. `AeadContext` does not, so this +form without context cannot produce `StackCipherText`. The form works for +other targets whose contexts have defaults, such as declarations that +include their own contexts. + +The scalar derive also chooses a context without saying so. That choice is +hard to discover. `#[stash(plaintext = i32)]` generates +`type Context = CallerContext`, even for a record whose only field is a +`StackCipherText`, whose own context is `AeadContext`. The generated +operation converts between the two types. + +The reviewer found that hidden choice surprising. It is undecided whether +this needs an API change, or clearer diagnostics and documentation. The +`Default` bound belongs to `EncryptInto::encrypt_into`, not to the +`EncryptFrom` trait itself. + +### Effect on callers + +Plaintext-side encryption exists, but it is not a mirror image of +`keyset.encrypt(value, context)`. Compared with the keyset method, it: + +- uses different method names; +- selects an output through a target trait; +- adds a `Clone` bound for this output; +- refuses the empty context that the inherent method accepts. + +Resolve the differing requirements for an empty context together with +issue 1. Renaming alone would not make the two paths equivalent. This +inconsistency is separate from the receiver decision above. + +**Sources:** +[plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto`, `EncryptFrom for StackCipherText`) and +[context types](../../packages/stack-encrypt/src/target/context.rs) +(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, +`src/plaintext/mod.rs`. + +## 4. The scalar derive cannot declare a fixed literal context + +**Status:** Direction agreed. Implementation pending. ### Agreed direction @@ -332,88 +412,138 @@ struct EncryptedInteger { let encrypted: EncryptedInteger = 10_i32.encrypt_into(&keyset).await?; ``` -With a fixed context, the generated operation supplies that context and -exposes a defaultable caller-context type, enabling `encrypt_into`. Without -a fixed context, the ordinary scalar derive continues to use `CallerContext` -and requires `encrypt_into_with_context`. Preserve explicit `context_type` -and `context_field` configurations; their interaction with a fixed context -needs validation rules rather than a silent override. +With a fixed context, the generated operation supplies that context. It +exposes a caller-context type that has a default, so `encrypt_into` works. +Without a fixed context, the ordinary scalar derive keeps using +`CallerContext` and requires `encrypt_into_with_context`. + +Keep the explicit `context_type` and `context_field` configurations. Their +interaction with a fixed context needs validation rules, not a silent +override. -This addresses the scalar gap at the derive layer without requiring -`CallerContext` itself to implement `Default` or changing the `EncryptFrom` -trait. A bare scalar has no input field from which the derive can extract -context; runtime context can still be supplied by the caller. +This closes the scalar gap in the derive. It does not need `CallerContext` +itself to implement `Default`. It also leaves the `EncryptFrom` trait +unchanged. A bare scalar has no input field for the derive to take context +from. The caller can still supply context at run time. -Additional caller context is accepted. The existing mechanism to reuse is -`DeclaredContext::under(own)`, exposed on operation descriptions through -`Encryption::under(own)`. `DeclaredContext` implements `Default`; its empty -default preserves the declared context unchanged. A caller extension is -combined as `(own, caller)`, never substituted for the declared context. +### Combining caller context with the fixed context -Recommended combination semantics follow this existing mechanism: +Extra context from the caller is accepted. Reuse the existing mechanism, +`DeclaredContext::under(own)`. Operation descriptions expose it through +`Encryption::under(own)`. + +`DeclaredContext` implements `Default`, and its +empty default leaves the declared context unchanged. A caller extension is +combined as `(own, caller)`. It never replaces the declared context. + +The recommended rules follow this existing mechanism: | Caller input | Effective structured context, with static context `C` | | --- | --- | | No extension | `C` | | One extension `P` | `(C, P)` | -| Successive extensions `P`, then `Q` via `DeclaredContext::with` | `((C, P), Q)` | -| One extension containing the pair `(P, Q)` | `(C, (P, Q))` | +| Extensions `P`, then `Q`, through `DeclaredContext::with` | `((C, P), Q)` | +| One extension that contains the pair `(P, Q)` | `(C, (P, Q))` | -This is structured composition, not string concatenation. Order and grouping -matter, and decrypt must reproduce the same structure. `NonEmpty::with` -builds the corresponding pairs directly. The existing test +This is structured composition, not string concatenation. Order and +grouping matter, and decryption must rebuild the same structure. +`NonEmpty::with` builds the matching pairs directly. The existing test `under_gives_a_subtree_its_own_context_which_the_callers_extends` in -[target.rs](../../packages/stack-encrypt/tests/target.rs) verifies that an -operation's declared context plus a caller extension agrees with a directly -composed context. +[target.rs](../../packages/stack-encrypt/tests/target.rs) checks one case. +An operation's declared context plus a caller extension must agree with a +context composed directly. + +The derive implementation still has to specify how a static literal is +represented: for example, as text or as a parsed path label. Encryption and +decryption then produce the same descriptor and AEAD context. This choice is +separate from keeping the `EncryptInto` trait. + +### Current behaviour and workaround + +The goal is a derived output for a bare `i32`, a fixed context declared on +that output, and `10_i32.encrypt_into(&keyset)`. The current macro rejects +`#[stash(plaintext = i32, context = "example")]`. The literal `context` +attribute works only with `struct = ..`. + +A workaround that uses only the derive wraps the integer in a plaintext +struct: + +```rust +struct IntegerInput { + value: i32, +} -The exact representation of a static literal (notably text versus a parsed -path label) still needs to be specified when implementing the derive, so -the descriptor and AEAD context remain consistent across encryption and -decryption. This decision is separate from retaining the `EncryptInto` trait. +#[derive(stack_encrypt::EncryptFrom)] +#[stash(struct = IntegerInput, context = "example")] +struct EncryptedInteger { + value: stack_encrypt::StackCipherText, +} -**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +// With EncryptInto imported: +let encrypted: EncryptedInteger = IntegerInput { value: 10 } + .encrypt_into(&keyset) + .await?; +``` + +This form declares the field's context as the pair `("example", "value")`. +ZeroKMS receives it as the descriptor `example/value`. For optional caller +extensions, the form uses `DeclaredContext`, a type with a default. It is +not a direct conversion from a bare integer. + +A manual `EncryptFrom` implementation can supply its own context. The +ordinary literal attribute cannot do that on a scalar derive today. + +**Sources:** +[macro validation](../../packages/stack-encrypt-derive/src/shape.rs) (`a_container_context_requires_a_struct`) and [generated implementations](../../packages/stack-encrypt-derive/src/encrypt.rs) (`struct_derive`, `value_derive`). ## 5. EncryptFrom does not define encrypt_from -**Status:** Open. +**Status:** Open. No replacement design has been chosen. + +Review the trait names and method names, or where the convenience function +lives. Keep the distinction between declaring an operation and executing +it. -`EncryptFrom` defines the associated type `Context`, the operation-description -constructor `encryption()`, and `indexes()`. It does not define a function -named `encrypt_from`. +`EncryptFrom` defines the associated type `Context`, the constructor +for an operation description, `encryption()`, and `indexes()`. It does not +define a function named `encrypt_from`. -A function with that name does exist, but on the blanket extension trait -`EncryptInto`: `Target::encrypt_from(&value, &keyset, context)` delegates to -`keyset.encrypt_as`. Calling that function requires `EncryptInto` in scope -and a target implementing `EncryptFrom`. +A function with that name exists on the blanket extension trait +`EncryptInto`. `Target::encrypt_from(&value, &keyset, context)` delegates to +`keyset.encrypt_as`. To call it, the caller needs `EncryptInto` in scope and +a target that implements `EncryptFrom`. -This differs from the standard Rust naming relationship: `From` defines -`from`, while `Into` defines `into`. The names suggest that same -relationship here, but `EncryptFrom` declares an operation rather than -executing a conversion, and the target-side execution function belongs to -the source-side extension trait. That makes method discovery and the -required trait import surprising. +This breaks the standard Rust naming pattern. `From` defines `from`, and +`Into` defines `into`. The names here suggest the same pattern, but two +things differ: -Review the trait and method names, or where the convenience function lives, -while preserving the distinction between declaring an operation and -executing it. No replacement design has been selected. +- `EncryptFrom` declares an operation. It does not execute a conversion. +- The execution function on the target side belongs to the extension trait + on the source side. -**Source:** [target operation traits](../../packages/stack-encrypt/src/target/operations.rs) +As a result, finding the method, and knowing which trait to import, is +surprising. + +**Source:** +[target operation traits](../../packages/stack-encrypt/src/target/operations.rs) (`EncryptFrom`, `EncryptInto`, including `EncryptInto::encrypt_from`). -## 6. Plan builder encrypt_into reads like the execution method +## 6. The plan builder's encrypt_into reads like the execution method + +**Status:** Open. Revisit during the plan API discussion. -**Status:** Open; revisit during the plan API discussion. +`Plan::value::().encrypt_into::()` is an inherent builder method. It +selects an output target and requires `T: EncryptFrom`. It does not +implement `EncryptInto`. It does not execute encryption either. -`Plan::value::().encrypt_into::()` is an inherent builder method that -selects an output target requiring `T: EncryptFrom`. It does not implement -`EncryptInto` or execute encryption. Sharing the name with -`value.encrypt_into(&keyset).await?` caused confusion during this review. +Its name is the same as `value.encrypt_into(&keyset).await?`. The shared +name caused confusion during this review. -Consider renaming the builder method to `target`: +A proposal for later discussion is to rename the builder method to +`target`: ```rust Plan::value::() @@ -421,186 +551,228 @@ Plan::value::() .build() ``` -This names the selection being made and distinguishes it from executing a -conversion. The rename is a proposal for later discussion, not a decision -or an implemented API change. `EncryptInto::encrypt_into` remains unchanged -by this proposal. +The name `target` describes the choice being made, and separates it from +executing a conversion. The rename is not a decision. The API has not +changed. The proposal leaves `EncryptInto::encrypt_into` unchanged. -**Source:** [value plan builder](../../packages/stack-encrypt/src/plan/value.rs) +**Source:** +[value plan builder](../../packages/stack-encrypt/src/plan/value.rs) (`ValueStart::encrypt_into`). -## 7. Mixed encrypted and plaintext fields lack a direct output-derive spelling +## 7. The output derive cannot mix encrypted and plaintext fields directly -**Status:** Passthrough attribute proposal accepted in principle; implementation pending. +**Status:** The passthrough attribute is accepted in principle. +Implementation pending. -For a `User` with encrypted `name` and `email` and plaintext `id` and -`created_at`, the fields-plan API directly supports `.encrypt(pick(..))` -and `.passthrough(pick(..))`. It batches the encrypted fields and returns -`FieldValues`; callers can extract those values into a typed storage struct. - -The current `EncryptFrom` derive has no passthrough/copy attribute for source -fields. `#[stash(default)]` initializes an output from a default or expression; -it does not copy a field from the input. Merely keeping the original field -type does not select passthrough either: derived fields require target traits. +### Accepted proposal: a passthrough attribute The proposed `#[stash(passthrough)]` attribute on output fields was accepted -as sensible: it copies the matching source field unchanged, alongside fields -whose types select encryption. Plaintext passthrough fields remain unencrypted and -unauthenticated; making a field part of the encryption context is a separate -choice, not implied by copying it. - -This gap is specific to Stack's output-side `EncryptFrom` derive. Vitamin C -already has plaintext-side `#[derive(Encrypt, Decrypt)]` with -`#[aead(passthrough)]` on source fields. Stack's `keyset.encrypt(&user, aad)` -can consume that derived source, yielding a generic `StackCipherText` map -with sealed and passthrough nodes. It does not generate a named -`EncryptedUser` type or expose the output as typed Rust fields. Its map-entry -AAD derivation also differs from the per-field contexts of a Stack fields -plan; the two routes should not be presented as interchangeable storage -formats. There is currently no source-side derive that generates the typed -encrypted counterpart illustrated in this review. - -**Sources:** [plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) +as sensible. It copies the matching source field unchanged, next to fields +whose types select encryption. + +Passthrough fields stay unencrypted and unauthenticated. Making a field +part of the encryption context is a separate choice. Copying a field does +not imply it. + +### Current behaviour + +Take a `User` with encrypted `name` and `email`, and plaintext `id` and +`created_at`. The fields-plan API supports this directly with +`.encrypt(pick(..))` and `.passthrough(pick(..))`. It batches the encrypted +fields and returns `FieldValues`. Callers can extract those values into a +typed storage struct. + +The current `EncryptFrom` derive has no attribute to copy a source field +through. `#[stash(default)]` initialises an output from a default or an +expression. It does not copy a field from the input. Keeping the original +field type does not select passthrough either, because derived fields need +target traits. + +### How Vitamin C's derive differs + +This gap applies only to Stack's output-side `EncryptFrom` derive. Vitamin +C already has a plaintext-side `#[derive(Encrypt, Decrypt)]`, with +`#[aead(passthrough)]` on source fields. Stack's +`keyset.encrypt(&user, aad)` can take that derived source. The result is a +generic `StackCipherText` map with sealed and passthrough nodes. + +That route has three limits: + +- It does not generate a named `EncryptedUser` type. The output is not + exposed as typed Rust fields. +- It derives AAD for map entries differently from the per-field contexts of + a Stack fields plan. Do not present the two routes as interchangeable + storage formats. +- No source-side derive exists that generates the typed encrypted + counterpart shown in this review. + +**Sources:** +[plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) (`FieldsBuilder::passthrough`) and [derive field attributes](../../packages/stack-encrypt-derive/src/attrs.rs) (`FieldAttrs`). ## 8. Clarify whole-value encryption, layouts, and searchable outputs -**Status:** Open architectural discussion; recommendations below are not decisions. +**Status:** Open architectural discussion. The recommendations below are +not decisions. Three user workflows need clear support: -1. Serialize a complete value into bytes and encrypt those bytes as one +1. Serialise a complete value into bytes, and encrypt those bytes as one opaque ciphertext. -2. Produce a chosen layout, such as `EncryptedUser`, with independently - encrypted fields and explicit plaintext fields. -3. Produce a layout with additional encrypted forms or search terms, such - as EQL targets. - -The existing Vitamin C plaintext derive produces a structural ciphertext -tree, not a single encrypted serialization. It is a generic output layout -for workflow 2, not workflow 1. The distinction affects key count, -independent field access, visible structure, and authenticated boundaries. -Encoding the completed ciphertext tree into one byte string would not -change those properties into whole-value encryption. +2. Produce a chosen layout, such as `EncryptedUser`, with fields encrypted + independently and explicit plaintext fields. +3. Produce a layout with extra encrypted forms or search terms, such as + EQL targets. ### Recommended model -Distinguish the encryption granularity (one encoded value or selected -fields) from the output representation and its requested operations. -Workflow 3 should extend workflow 2 through field targets that request -ciphertext, terms, or both, rather than introducing a separate execution -system. The generic tree can remain a layout option without being the -mandatory scalar storage representation. - -- **Whole-value path:** the follow-up review favors an explicit named - wrapper, provisionally `OpaqueBlob`, which owns serialization and - deserialization through one library-selected codec. It produces one opaque - persistable ciphertext; decrypting into `OpaqueBlob` selects the matching - decode path. Today callers can serialize into `Vec`, which Vitamin C - treats as a single byte leaf, but still encounter the node/leaf storage gap. -- **Layout path:** select a target with `EncryptFrom`/`EncryptInto`, or a - plan, with an explicit passthrough choice. A source-side derive that - generates a target would be authoring convenience over the same system, - not another set of encryption semantics. -- **Searchable layout path:** compose core encryption/index operations - through targets. EQL adapters own their storage representation; Stack - owns execution. Write and query operations must share the relevant - context and normalization rules. - -Keep source representation, requested operations, output encoding, and -execution distinct internally, while giving each workflow a short public -path. An application should not need to learn visitor machinery or plans -to encrypt and persist a scalar. Arbitrary serialization must not become an -implicit replacement for the typed normalization used by indexes. - -Before selecting more trait names, write end-to-end examples for all three -workflows: encrypt, persist, load, decrypt, plus query for searchable targets. -Use those examples to evaluate the public surface and error messages. - -### Follow-up: opinionated whole-value wrapper +Separate the encryption granularity from the output representation and the +operations it requests. The granularity is either one encoded value or +selected fields. Workflow 3 should extend workflow 2 through field targets +that request ciphertext, search terms, or both. It should not add a separate +execution system. The generic tree can stay as one layout option without +being the required storage representation for scalars. + +- **Whole-value path.** The follow-up review favours an explicit named + wrapper, provisionally `OpaqueBlob`. The wrapper owns serialisation + and deserialisation through one codec that the library chooses. It + produces one opaque ciphertext that can be stored. Decrypting into + `OpaqueBlob` selects the matching decoding path. Today callers can + serialise into `Vec`. Vitamin C treats a `Vec` as a single byte + leaf. Callers still meet the storage gap between node and leaf (issue 2). +- **Layout path.** Select a target with `EncryptFrom` and `EncryptInto`, or + a plan, with an explicit passthrough choice. A source-side derive that + generates a target would make authoring easier on the same system. It + would not add another set of encryption semantics. +- **Searchable layout path.** Compose the core encryption and index + operations through targets. EQL adapters own their storage + representation, and Stack owns execution. Write operations and query + operations must share the relevant context and normalisation rules. + +Inside the library, keep these four things separate: source +representation, requested operations, output encoding, and execution. Give +each workflow a short public path. An application should not need to learn +visitor machinery or plans to encrypt and store a scalar. Arbitrary +serialisation must not silently replace the typed normalisation that +indexes use. + +Before choosing more trait names, write examples for all three workflows +from start to end: encrypt, store, load, and decrypt, plus query for +searchable targets. Use those examples to judge the public API and its +error messages. + +### Why Vitamin C's derive is not whole-value encryption + +The existing Vitamin C plaintext derive produces a structural ciphertext +tree, not one encrypted serialisation. It is a generic output layout for +workflow 2, not workflow 1. The difference affects four properties: the +number of keys, independent access to fields, the visible structure, and +the authenticated boundaries. Encoding the completed ciphertext tree +into one byte string would not give it those properties of whole-value +encryption. + +### Follow-up: a whole-value wrapper chosen by the library The first public example now uses -`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` rather than -manual JSON encoding. The wrapper name is provisional; the direction is to -make blob serialization an explicit library-owned behavior. The stored result -remains opaque ciphertext, and decryption returns an `OpaqueBlob` that -can be unwrapped. The codec and its authenticated versioning must be specified; -the wrapper does not silently change how unwrapped structs are encrypted. - -Constant-time serialization/deserialization is a desired property to -investigate, not a property established by choosing a binary codec or using -constant-time AEAD. The first revised example provisionally uses Serde traits, -whose arbitrary user implementations cannot carry that guarantee. Define -which lengths and shapes may be public and whether a stricter supported-type -profile is needed before promising secret-independent execution. The examples -document this unresolved boundary separately from the wrapper ergonomics. - -Investigate `serdect` as part of that work. It is already a Stack Encrypt -dependency; check its encoding/decoding guarantees and their scope, and -evaluate applicability to ciphertext serialization and the proposed blob -codec. No whole-object constant-time guarantee or codec choice is inferred -from its use. +`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` instead of +encoding JSON by hand. The wrapper name is provisional. The direction is to +make blob serialisation an explicit behaviour that the library owns. + +The stored result stays opaque ciphertext. Decryption returns an +`OpaqueBlob` that the caller can unwrap. The codec and its +authenticated versioning still need a specification. The wrapper does not +silently change how unwrapped structs are encrypted. + +### Constant-time serialisation is not yet established + +Constant-time serialisation and deserialisation is a property to +investigate. Choosing a binary codec, or using constant-time AEAD, does not +establish it. The first revised example uses Serde traits provisionally. +Arbitrary user implementations of those traits cannot give that guarantee. + +Before promising execution that does not depend on secrets, define which +lengths and shapes may be public. Also decide whether a stricter profile of +supported types is needed. The examples document this open question +separately from how easy the wrapper is to use. + +Investigate `serdect` as part of that work. Stack Encrypt already depends +on it. Check what its encoding and decoding guarantees are and what they +cover. Then judge whether it applies to serialising ciphertext and to the +proposed blob codec. Its current use does not imply a constant-time +guarantee for whole objects, or a codec choice. ## 9. Ownership of the value-encryption API across Stack and Vitamin C -**Status:** Open architectural discussion; moving the traits is proposed, -not authorized implementation work. +**Status:** Open architectural discussion. Moving the traits is proposed. +It is not authorised implementation work. + +### Recommendation -The review has repeatedly crossed from Stack's target declarations into -Vitamin C's plaintext visitor traits, cipher traits, contexts, derives, and -ciphertext tree. The user proposes moving the encryption traits into Stack -Encrypt and retaining general primitives such as protected values and -randomness in Vitamin C. +Stack should own the public model for encrypting values. That includes: -**Recommendation:** Stack should own the public value-encryption model, -including the plaintext traits/derives, target declarations, contexts, -ciphertext persistence, and execution semantics. Vitamin C can retain -reusable cryptographic and memory-protection primitives. Moving files alone -does not resolve ambiguous names (`Encrypt` currently describes a source), -duplicate entry points, or the missing whole-value/storage path. +- the plaintext traits and derives; +- target declarations; +- contexts; +- storing ciphertext; +- execution semantics. -One Stack-owned core layer may serve Rust and language guests without -pulling in KMS transport or HTTP. Applications should still have one -coherent `stack_encrypt` entry point; an internal dependency split need not -become another public API users must understand. +Vitamin C can keep reusable primitives for cryptography and for protecting +memory. Moving files alone does not fix three problems: + +- ambiguous names: for example, `Encrypt` currently describes a source; +- duplicate entry points; +- the missing path for whole values and storage. + +One core layer owned by Stack could serve both Rust and language guests, +without pulling in KMS transport or HTTP. Applications should still have one +consistent entry point, `stack_encrypt`. An internal split of dependencies +should not become another public API that users have to understand. + +### Where the proposal came from + +The review kept crossing from Stack's target declarations into Vitamin C's +plaintext visitor traits, cipher traits, contexts, derives, and ciphertext +tree. The user proposes moving the encryption traits into Stack Encrypt, +and keeping general primitives, such as protected values and randomness, in +Vitamin C. ### Migration constraints - Today `stack-encrypt` depends on both `vitaminc-aead` and - `vitaminc-encrypt`, while `vitaminc-encrypt` also depends on - `vitaminc-aead`. Moving the latter's traits into the top-level Stack crate - and making the existing backend depend on Stack would introduce a cycle. - Move the value-level backend adapters too, or separate a lower-level - core/primitive boundary. -- Context composition, type tags, ciphertext framing, index normalization, - and binding transport frames must agree across the new consumers. They - may change without preserving prototype data or providing migration. -- Retain one canonical trait/type identity. Update consumers directly; - compatibility re-exports are unnecessary. Copied independent trait - definitions would require duplicate implementations and adapters. -- Account for guest/FFI consumers and EQL target implementations, not just - Rust application imports. The no-HTTP/WASM path must remain usable. + `vitaminc-encrypt`, and `vitaminc-encrypt` also depends on + `vitaminc-aead`. Moving the traits of `vitaminc-aead` into the top-level + Stack crate, and making the existing backend depend on Stack, would create + a dependency cycle. Either move the backend adapters for values too, or + draw a separate boundary for a lower-level core of primitives. +- These must agree across the new consumers: context composition, type + tags, ciphertext framing, index normalisation, and the transport frames of + the bindings. They can change without keeping prototype data and without a + migration. +- Keep one canonical identity for each trait and type. Update consumers + directly. Re-exports for compatibility are not needed. Independent copies + of trait definitions would need duplicate implementations and adapters. +- Account for guest and FFI consumers and EQL target implementations, not + only Rust application imports. The path without HTTP, for WebAssembly, must + stay usable. **Sources:** [Stack dependencies](../../packages/stack-encrypt/Cargo.toml), -[guest ABI dependencies](../../packages/stack-guest-abi/Cargo.toml), and +[guest application binary interface (ABI) dependencies](../../packages/stack-guest-abi/Cargo.toml), +and [EQL encryption adapters](../../packages/eql/crates/eql-bindings/src/encryption.rs). -The installed `vitaminc-encrypt` 0.8.0 manifest confirms its dependency on +The installed `vitaminc-encrypt` 0.8.0 manifest confirms that it depends on `vitaminc-aead`. -## 10. Public encrypt_as duplicates encrypt_into +## 10. The public encrypt_as duplicates encrypt_into -**Status:** Decision agreed; implementation pending. +**Status:** Decision agreed. Implementation pending. -`KeysetCipher::encrypt_as` is currently an inherent method requiring the -output to implement `EncryptFrom`. It executes that target's operation -declaration. `EncryptInto::encrypt_into` and -`EncryptInto::encrypt_into_with_context` delegate to it, supplying either -the default context or the caller's context. +### Decision -**Decision:** remove `encrypt_as` from the public API. Keep `encrypt_into` -and `encrypt_into_with_context` as the public target-directed entry points: +Remove `encrypt_as` from the public API. Keep `encrypt_into` and +`encrypt_into_with_context` as the public entry points for encrypting into +a target: ```rust let encrypted: EncryptedInteger = value.encrypt_into(&keyset).await?; @@ -609,196 +781,256 @@ let encrypted: EncryptedInteger = value .await?; ``` -The operation-execution machinery may remain internal; removing the public -method does not remove target declarations, batching, or execution. Public -examples should use the retained entry points when illustrating the proposed -API. Earlier descriptions of `encrypt_as` in this log describe current code. +The machinery that executes operations can stay internal. Removing the +public method does not remove target declarations, batching, or execution. +Public examples of the proposed API should use the entry points that stay. +Earlier mentions of `encrypt_as` in this log describe the current code. -This is distinct from issue 3's decision about ordinary encryption: -`keyset.encrypt(value)` remains the sole ordinary `encrypt` spelling, while +This is separate from issue 3's decision about ordinary encryption. +`keyset.encrypt(value)` stays the only spelling of ordinary `encrypt`. `value.encrypt_into(&keyset)` selects a particular output type. The -`encrypt_from` naming/location question in issue 5 remains open. +question in issue 5 about the name and location of `encrypt_from` is still +open. -**Source:** [target operations](../../packages/stack-encrypt/src/target/operations.rs) +### Current behaviour + +`KeysetCipher::encrypt_as` is currently an inherent method. It requires the +output to implement `EncryptFrom`, and executes that target's operation +declaration. `EncryptInto::encrypt_into` and +`EncryptInto::encrypt_into_with_context` delegate to it. They supply either +the default context or the caller's context. + +**Source:** +[target operations](../../packages/stack-encrypt/src/target/operations.rs) (`KeysetCipher::encrypt_as`, `EncryptInto`). -## 11. Context-field storage does not yet extract context from an input struct +## 11. Context fields do not yet take context from an input struct -**Status:** Open. +**Status:** Open. The syntax is still open. The shorthand +`context(field = "id")` is recommended. + +### Current behaviour + +`#[stash(context_field)]` exists on derived output types: + +- On encryption, it stores the supplied context in that output field. +- On decryption, it uses the stored context. It can also check the stored + context against an expected value that the caller supplies separately. -`#[stash(context_field)]` exists on derived output types. On encryption it -stores the supplied context in that output field; on decryption it uses the -stored context, optionally checking an independently supplied expected value. It does not automatically select a field from a plaintext input struct. -The macro currently treats this field as the complete context and rejects -it alongside a literal `context`. Since `struct = User` requires a literal -context, that form cannot currently mark its `id` with `context_field`. -The public examples originally described passing the id explicitly without -explaining this distinction. +The macro treats this field as the complete context, and rejects it next to +a literal `context`. Because `struct = User` requires a literal context, +that form cannot mark its `id` with `context_field`. The public examples +first described passing the ID explicitly, without explaining this +difference. -### Clarified model: context source and context storage are independent +### Clarified model: where context comes from and where it is stored -Two similar scenarios must be distinguished: +Two similar scenarios need to be told apart: -1. The caller supplies context as a function argument; the encrypted output - stores it for later use. This is the existing scalar-output - `context_field` behavior. -2. The encryption declaration obtains context from one or more fields of - the plaintext. Whether those values are also stored is a separate choice. +1. The caller supplies context as a function argument, and the encrypted + output stores it for later use. This is what scalar-output + `context_field` does today. +2. The encryption declaration takes context from one or more fields of the + plaintext. Whether those values are also stored is a separate choice. -The second scenario can include storage too: read `User.id`, use it to bind -the encrypted fields, and carry it into `EncryptedUser.id`. The caller need -not pass the same value twice. Several fields, such as a tenant id and user -id, can supply a structured composite context. +The second scenario can include storage too. It reads `User.id`, uses it to +bind the encrypted fields, and copies it into `EncryptedUser.id`. The caller +does not pass the same value twice. Several fields, such as a tenant ID and +a user ID, can supply a structured composite context. -| Context source | Stored in the encrypted output? | Recovery requirement | +| Context source | Stored in the encrypted output | What decryption needs | | --- | --- | --- | -| Caller argument | Yes | Read the stored context; optionally check an independently supplied expected value | +| Caller argument | Yes | Read the stored context. Optionally check it against an expected value supplied separately | | Caller argument | No | Supply the same context again | -| Plaintext field(s) | Yes | Reconstruct context from the stored field(s); optionally check expected values | -| Plaintext field(s) | No | Supply equivalent context independently before decrypting | - -Store a source-derived context independently of the ciphertext that needs -it, so decrypt does not depend on first opening that ciphertext to discover -its own context. Binding ciphertext to a stored context authenticates that -relationship; it does not by itself establish that the entire record is at -the destination the application expected. - -The derive should express source selection and output storage as separate -concepts, even if a shorthand covers the common read-and-store case. Avoid -assuming that one `context_field` attribute should silently perform both. -Syntax remains open. Define field ordering/grouping for composite context, -literal prefixes, optional caller extensions, `from` mappings, and expected -context checks. These capabilities are proposed, not implemented. - -The recommended common-case shorthand is `context(field = "id")`, expanding -exactly to `encrypt(field = "id"), decrypt(field = "id")`. It selects the -same Rust field name on the two different inputs; it does not implicitly -copy or store that field. Keep the directional form for different names or -sources. Initially reject mixing shorthand and directional selectors to -avoid ambiguous precedence. The context examples now use this shorthand -for the plaintext-id-to-stored-id case. - -**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +| One or more plaintext fields | Yes | Rebuild the context from the stored fields. Optionally check expected values | +| One or more plaintext fields | No | Supply equivalent context separately before decrypting | + +Store a context taken from the source separately from the ciphertext that +needs it. Decryption then does not have to open that ciphertext first to +find its own context. Binding a ciphertext to a stored context +authenticates that relationship. It does not by itself prove that the whole +record is where the application expected it to be. + +### Recommended syntax + +The derive should express two separate concepts: selecting the source, and +storing in the output. A shorthand can cover the common case of reading and +storing. Do not assume that one `context_field` attribute should silently +do both. + +The syntax is still open. Define each of these: + +- the ordering and grouping of fields for a composite context; +- literal prefixes; +- optional caller extensions; +- `from` mappings; +- checks of expected context. + +These capabilities are proposed, not implemented. + +The recommended shorthand for the common case is `context(field = "id")`. +It expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. It +selects the same Rust field name on the two different inputs. It does not +copy or store that field implicitly. + +- Keep the directional form for different names or different sources. +- At first, reject mixing the shorthand with directional selectors, so + that precedence is never ambiguous. + +The context examples now use this shorthand for the case where a plaintext +ID becomes a stored ID. + +**Sources:** +[macro validation](../../packages/stack-encrypt-derive/src/shape.rs) (`Record::parse`, `context_type`) and [context-field output assembly](../../packages/stack-encrypt-derive/src/encrypt.rs) (`assignments`, `value_derive`). ## 12. Default is not the same as optional caller context -**Status:** Open; exposed by the directional-context examples. +**Status:** Open. The directional-context examples showed this problem. No +mechanism is chosen. -`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` make the -no-context form available whenever the relevant associated `Context` type -implements `Default`. That works for deliberately defaultable types such as -`DeclaredContext` and `ExpectedContext`, and currently excludes +`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` offer the form +without context whenever the associated `Context` type implements +`Default`. That works for types that are meant to have defaults, such as +`DeclaredContext` and `ExpectedContext`. It currently excludes `CallerContext` and `AeadContext`. -However, a required caller value can have a Rust default: if a declaration -exposes `Context = u64`, the no-context form supplies zero. The directional -proposal `encrypt(caller = u64)` must not accidentally mean "optional, with -zero as the default". +A required caller value can still have a Rust default. If a declaration +exposes `Context = u64`, the form without context supplies zero. The +directional proposal `encrypt(caller = u64)` must not accidentally mean +"optional, with zero as the default". + +Separate the application's type for the context value from the trait's +type or capability for caller input. Two options: -Separate the application's context value type from the trait's caller-input -type or capability. Options include a non-defaultable `RequiredContext` -wrapper that accepts ergonomic conversions from `T`, or a dedicated marker -controlling context-free entry points. No mechanism is decided yet. +- a `RequiredContext` wrapper with no default, accepting convenient + conversions from `T`; +- a dedicated marker that controls the entry points without context. -This matters equally to required context on decrypt. Source-derived context -can require no encrypt argument, while decrypt may require an external -value or accept an optional expectation of stored context. Do not assume the -two associated input types must match, or that either always equals the -complete logical context used by encryption. +Required context matters on decryption too. Context taken from the source +may need no argument for encryption. Decryption may need an external value, +or accept an optional expectation of the stored context. Do not assume that +the two associated input types must match. Do not assume that either one +always equals the complete logical context that encryption uses. -See the [context scenario examples](2026-10-24-stack-encrypt-context-examples.md) -for the proposed selectors and input requirements. +See the +[context scenario examples](2026-10-24-stack-encrypt-context-examples.md) +for the proposed selectors and what each one requires as input. -**Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) +**Source:** +[convenience methods](../../packages/stack-encrypt/src/target/operations.rs) (`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). -## 13. JSON layouts and encrypted-output adaptation - -**Status:** Open; initial assessment of the JSON tree implementation on -`next` at `b1049824b`. No replacement API selected. - -**Agreed follow-up:** investigate replacing the EQL JSON adapter's -serialization round trips with direct typed conversions. This does not yet -select an extended visitor protocol or authorize a particular replacement -API. - -`Layout` describes a target selected by a value carrying runtime options. -It associates a `Target: EncryptFrom` and an `Index`, and supplies an -`encryption(&self)` description plus the configured index for queries. -`EncryptFrom::encryption()` instead selects a description by output type. -For JSON, both routes can produce `JsonDocument`: selecting that type uses -default options, while `Json::new(options)` can specify ordered paths and -whether the root is sealed. A plan retains those options for its queries. - -This is broader than choosing a Rust record's field layout. JSON options -affect cryptographic structure, query capability, and information exposed -to the server. The same output type can represent different configurations; -the type alone does not identify them. The JSON tree belongs to the -searchable-layout workflow, rather than being an opaque serialized blob. - -The existing `target::transcode` protocol serves a different stage: a -`Reader` hands completed ciphertext, structural markers, metadata, and terms -to a destination `Visitor`. `Encryption::transcode()` uses it to adapt -encrypted output without an intermediate serialization tree. It does not -choose JSON indexing options or generate missing search terms. - -The new EQL adapter currently takes another route. `JsonTree` implements -`Layout`, delegates encryption to the -engine's `Json` layout, and wraps its completed `JsonDocument` with EQL -version and column metadata. Its envelope conversion uses a -`serde_json::Value` intermediate; a private helper also named `transcode` -converts query/extraction types through `to_value`/`from_value`. Neither is -the existing `target::transcode` protocol. That protocol currently has no -JSON-document reader or visitor event. The conversion processes encrypted -output and query metadata; it is not a plaintext serialization bypass. - -On decrypt, the EQL target validates its stored column against any supplied -expectation, reconstructs the engine document through the wire form, and -delegates opening to the engine. Its layout object is not required to -perform that conversion. - -### Questions to resolve - -- Keep encryption configuration distinct from encrypted-output adaptation. - Decide whether `Layout` is the clearest public name for a configured - target, especially alongside the earlier use of "layout" for output - records. -- Decide how runtime target options fit the retained `EncryptInto` API; - currently type-directed conversion selects the target's defaults, while - plans accept a configured layout via `encrypt_with_layout`. -- Evaluate a direct typed JSON-document adapter versus extending the - encrypted-output visitor. Avoid forcing the JSON protocol into generic - map/sequence events that lose its header, shared document key, projection, - and extraction semantics. A shared architectural boundary does not - necessarily require one universal visitor. -- Consider whether a general configured-target abstraction should require - an index: today's `Layout` requires one even though encryption-only - targets can also need options. -- Preserve the link between write configuration and query configuration; - a plan's reuse of the configured index is useful independently of naming. - -**Sources:** [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) -(`Layout`, `json_document`, `EncryptFrom for JsonDocument`), -[encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs), -[JSON options and document](../../packages/stack-encrypt/src/json/mod.rs), -and [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) -(`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). - -### Static Rust surface versus runtime configuration - -The intended public Rust experience is traits and derives. Treat plans as -the lower-level composition interface supporting those APIs and the dynamic -FFI interface, rather than requiring Rust callers to write a plan for JSON. -The derive already emits plans; its implementation can retain that shared -execution path without making the plan builder the primary user example. - -JSON already has a type-directed path. Given an initialized `cipher` and -its `keyset`, this uses existing methods and default JSON options: +## 13. JSON layouts and adapting encrypted output + +**Status:** Open. This is a first assessment of the JSON tree +implementation on the `next` branch at `b1049824b`. No replacement API is +chosen. + +**Agreed follow-up:** investigate replacing the serialisation round trips in +the EQL JSON adapter with direct typed conversions. This does not yet choose +an extended visitor protocol, or authorise a particular replacement API. + +### How JSON configuration and targets work today + +`Layout` describes a target that is selected by a value carrying run-time +options. It links a `Target: EncryptFrom` with an `Index`. It supplies an +`encryption(&self)` description, and the configured index for queries. +`EncryptFrom::encryption()`, by contrast, selects a description by +output type. + +For JSON, both routes can produce `JsonDocument`: + +- Selecting that type uses the default options. +- `Json::new(options)` can set ordered paths, and whether the root is + sealed. Under a sealed root, the root entry's ciphertext contains the + whole document, so it can be opened as a complete document. + +A plan keeps those options for its queries. + +This goes beyond choosing the field layout of a Rust record. JSON options +affect the cryptographic structure, what queries can do, and what +information is exposed to the server. The same output type can represent +different configurations, so the type alone does not identify the +configuration. The JSON tree belongs to the searchable-layout workflow +(issue 8). It is not an opaque serialised blob. + +### The existing transcode protocol does a different job + +The existing `target::transcode` protocol works at a later stage. A +`Reader` hands completed ciphertext, structural markers, metadata, and +search terms to a destination `Visitor`. `Encryption::transcode()` uses it +to adapt encrypted output without an intermediate serialisation tree. The +protocol does not choose JSON indexing options. It does not generate +missing search terms either. + +The new EQL adapter takes another route: + +- `JsonTree` implements `Layout`. It + delegates encryption to the engine's `Json` layout, and wraps the + completed `JsonDocument` with EQL version and column metadata. +- Its envelope conversion goes through a `serde_json::Value` intermediate. +- A private helper that is also named `transcode` converts query types and + extraction types. It calls `to_value`, then `from_value`. + +Neither conversion is the existing `target::transcode` protocol. That +protocol has no reader or visitor event for a JSON document today. The +conversion processes encrypted output and query metadata. It is not a way +to skip plaintext serialisation. + +On decryption, the EQL target checks its stored column against any expected +value supplied. It rebuilds the engine document through the wire form, and +delegates opening to the engine. Its layout object is not needed for that +conversion. + +The sources for this section are: + +- + [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) + (`Layout`, `json_document`, `EncryptFrom for JsonDocument`); +- + [encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs); +- [JSON options and document](../../packages/stack-encrypt/src/json/mod.rs); +- + [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) + (`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). + +### Open questions about JSON configuration + +- How should the API keep encryption configuration separate from adapting + encrypted output? +- Is `Layout` the clearest public name for a configured target, given that + this log earlier used "layout" for output records? +- How do run-time target options fit the `EncryptInto` API that stays? + Today type-directed conversion selects the target's defaults, and plans + accept a configured layout through `encrypt_with_layout`. +- Should JSON use a direct typed adapter for JSON documents, or an extended + visitor for encrypted output? Do not force the JSON protocol into generic + map and sequence events. Those events lose its header, shared document + key, projection, and extraction semantics. A shared architectural boundary + does not have to mean one universal visitor. +- Should a general abstraction for configured targets require an index? + Today's `Layout` requires one, but targets that only encrypt can also need + options. +- How does the design keep the link between write configuration and query + configuration? A plan's reuse of the configured index is useful whatever + the names are. + +### Static Rust API compared with run-time configuration + +The intended public Rust experience is traits and derives. Plans are the +lower-level composition interface. They support those APIs and the dynamic +FFI interface. Rust callers should not have to write a plan for JSON. + +The derive already emits plans. Its implementation can keep that shared +execution path without making the plan builder the main user example. + +JSON already has a type-directed path. Given an initialised `cipher` and its +`keyset`, this code uses existing methods and the default JSON options: ```rust use eql_bindings::v4::json_tree::JsonTreeDocument; @@ -817,61 +1049,73 @@ let restored: Value = encrypted.decrypt_into(&keyset, column).await?; assert_eq!(restored, profile); ``` -This produces searchable JSON with a sealed root and no ordered paths. -The explicit decrypt context checks the expected column against the stored -identifier. The snippet is source-checked, not compiled here. +This produces searchable JSON with a sealed root and no ordered paths. The +explicit decryption context checks the expected column against the stored +identifier. The snippet was checked against the source, not compiled. The earlier `SearchableEmail` example fixes its recipe in the target type: ciphertext plus an equality term. `EncryptFrom::encryption()` takes no configuration value. JSON's default target also fixes a recipe in its type. -What `Layout::encryption(&self)` adds is access to instance configuration: -one `JsonTreeDocument` output type can be produced with `$.age` ordered, -with `$.salary` ordered, or with different root settings, selected at -runtime. Its configured index also supplies the matching query behavior. - -Search metadata itself does not require `Layout`. A custom target could -fix ordered JSON paths in a manual `EncryptFrom` implementation using the -existing JSON operation. A derive could eventually generate such a fixed -policy. Runtime-configured targets need an additional configuration input; -today `encrypt_into_with_context` accepts context but no layout/options, -and custom JSON configurations use the lower-level plan or `keyset.run` -path. Keep those options separate from cryptographic context. - -The cryptographic distinction is separate from the API distinction: JSON -uses a specialized document operation with a shared document data key and -node/extraction semantics. It cannot be implemented merely by adding a -generic equality-term field to the ordinary ciphertext tree. That explains -the JSON operation, but does not by itself require a public `Layout` trait. - -### Naming obscures roles + +What `Layout::encryption(&self)` adds is access to configuration on an +instance. One `JsonTreeDocument` output type can be produced with `$.age` +ordered, with `$.salary` ordered, or with different root settings, chosen +at run time. Its configured index also supplies the matching query +behaviour. + +Search metadata itself does not require `Layout`: + +- A custom target could fix ordered JSON paths in a manual `EncryptFrom` + implementation, using the existing JSON operation. +- A derive could later generate such a fixed policy. +- Targets configured at run time need an extra configuration input. Today + `encrypt_into_with_context` accepts context, but no layout or options. + Custom JSON configurations use the lower-level plan, or the `keyset.run` + path. + +Keep those options separate from cryptographic context. + +The cryptographic difference is separate from the API difference. JSON uses +a specialised document operation, with a shared document data key and its +own node and extraction semantics. Adding a generic equality-term field to +the ordinary ciphertext tree cannot implement it. That explains why the +JSON operation exists. It does not by itself require a public `Layout` +trait. + +### Names hide the roles of the JSON types | Current type | Role | | --- | --- | -| `serde_json::Value` | Plaintext input and recovered plaintext | -| `stack_encrypt::json::JsonOptions` | Runtime configuration | -| `stack_encrypt::target::Json` | Configured index; also implements `Layout` | +| `serde_json::Value` | Plaintext input, and recovered plaintext | +| `stack_encrypt::json::JsonOptions` | Run-time configuration | +| `stack_encrypt::target::Json` | Configured index. Also implements `Layout` | | `stack_encrypt::json::JsonDocument` | Encrypted engine document, including search metadata | -| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target/layout | +| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target or layout | | `eql_bindings::v4::json_tree::JsonTreeDocument` | Encrypted EQL storage envelope | -Review names that state these roles, using `Encrypted` for ciphertext-bearing -documents, `Index` for index descriptions, and `Target` or `Config` for -configuration, with module paths distinguishing EQL from engine types. -No replacement names are selected. `Json`'s dual index/layout role should -be considered before simply renaming it. +Review names that state these roles: + +- `Encrypted` for documents that contain ciphertext; +- `Index` for index descriptions; +- `Target` or `Config` for configuration; +- module paths that separate EQL types from engine types. + +No replacement names are chosen. Consider `Json`'s double role as index and +layout before renaming it. ### Candidate: a fixed-policy wrapper around the EQL output -**Proposal, not implemented or agreed syntax.** Attach the fixed policy to -an output newtype, while the inner library type owns the database format. -The alias below only clarifies the existing type's role; it is not a -library rename. +This is a proposal. It is not implemented. Its syntax is not agreed. + +The fixed policy is attached to an output newtype. The inner library type +defines the database format. The alias below only makes the existing +type's role clearer. It does not rename anything in the library. The later [implementation plan](2026-10-24-stack-encrypt-api-plan.md) -recommends a general typed-options factory, `target(options = factory)`, -instead of the JSON-specific attribute below. Dynamic callers deserialize -options into the same concrete type; Rust and Go support ship together. -This sketch is retained to explain the earlier alternative. +recommends a general factory for typed options, `target(options = factory)`, +instead of the JSON-specific attribute below. Dynamic callers deserialise +options into the same concrete type. Rust and Go support ship together. This +log keeps the sketch to explain the earlier alternative. ```rust use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; @@ -905,272 +1149,476 @@ assert_eq!(restored, payroll); ``` The proposed `stash(transparent, json_tree(...))` selects the configured -JSON operation for the inner target and wraps its completed output. It must -not invoke the ordinary scalar/record derivation on the wrapper, nor invoke -the inner target's default encryption and then try to add ordering terms. -It inherits the EQL target's caller-context and decrypt behavior. Serde's -existing `transparent` attribute preserves the EQL envelope; it does not -provide database-driver trait implementations, which remain a separate -integration concern. The root stays sealed under this candidate's defaults. +JSON operation for the inner target, and wraps its completed output. It +must not: + +- run the ordinary scalar or record derivation on the wrapper; +- run the inner target's default encryption and then try to add ordering + terms. + +The wrapper inherits the EQL target's caller context and decryption +behaviour. Serde's existing `transparent` attribute keeps the EQL envelope. +It does not provide trait implementations for database drivers. Those stay +a separate integration concern. Under this candidate's defaults, the root +stays sealed. The generated declaration must also expose the configured index metadata -for future typed queries, so queries can reuse the policy instead of -repeating paths. The query-facing syntax remains undesigned. The core derive -must delegate configured output construction to a capability of the inner -target, rather than depending directly on the EQL crate. Whether to express -that capability through `Layout` or another trait remains open. +for future typed queries. Queries can then reuse the policy instead of +repeating the paths. The syntax for queries is not designed yet. The core +derive must delegate the construction of configured output to a capability +of the inner target, instead of depending directly on the EQL crate. It is +still open whether `Layout` or another trait expresses that capability. ### Alternative: configuration on a plaintext wrapper -The user's proposed `JsonPlaintext::new(value).with_ordering_path(path)` can -still yield the exact EQL wire format. Configuration source and output -representation are independent: an explicit `EqlEncryptedJsonTree` target -can accept that input and execute the same configured JSON operation. The -wrapper must be treated as an encryption request, not serialized as part of -the plaintext. The current API does not provide this wrapper. - -This is attractive when paths come from runtime configuration, but repeats -fixed policy at call sites and needs a way to reuse that policy for queries. -Prefer the output newtype for the fixed-policy example. If the runtime -wrapper is added, initially do not implement conversion from it into -fixed-policy targets: avoid silently overriding or merging conflicting -policies. Neither syntax permits callers to redefine EQL's required fields -or its cryptographic wire protocol. - -A fixed Rust type describes the policy for new writes; transparent Serde -deserialization alone does not prove a stored document was produced with -that policy. Ordered paths are not all recorded in the document header. -Policy changes and heterogeneous existing rows need explicit compatibility -and migration semantics before claiming stronger read-time guarantees. +The user proposed `JsonPlaintext::new(value).with_ordering_path(path)`. It +can still produce the exact EQL wire format. Where the configuration comes +from is independent of the output representation. + +An explicit +`EqlEncryptedJsonTree` target can accept that input and execute the same +configured JSON operation. The wrapper must be treated as an encryption +request, and not serialised as part of the plaintext. The current API does +not provide this wrapper. + +This wrapper is attractive when paths come from run-time configuration. +However, it repeats a fixed policy at each call site, and needs a way to +reuse that policy for queries. For the fixed-policy example, prefer the +output newtype. + +If the run-time wrapper is added, do not implement conversion from it into +fixed-policy targets at first. This avoids silently overriding or merging +conflicting policies. Neither syntax lets callers redefine EQL's required +fields or its cryptographic wire protocol. + +A fixed Rust type describes the policy for new writes. Transparent Serde +deserialisation alone does not prove that a stored document used that +policy. The document header does not record all ordered paths. Policy +changes, and existing rows written with different policies, need explicit +compatibility and migration rules. Only then can the API claim stronger +guarantees when it reads. ## 14. Decrypt convenience methods need explicit context variants -**Required direction:** provide a no-context method and a separate -`_with_context` method, mirroring the encrypt-side distinction. The user -prefers the name `decrypt_from`. +**Status:** Direction required by the user. The receiver, and the migration +of the competing method, are recommendations still to confirm. + +**Required direction:** provide a method without context and a separate +`_with_context` method, mirroring encryption. The user prefers the name +`decrypt_from`. + +**Recommendation:** use the pair directed at the destination in the +preferred public examples. `User::decrypt_from(encrypted, &keyset)` follows +Rust's `From` convention. `encrypted.decrypt_from(...)` would reverse it. + +These are not settled user decisions. Do not change the declaration trait +only because its convenience spelling changes. Handle trait discoverability +with issue 5, and the capability for optional context with issue 12. + +### Current behaviour + +These methods exist today on the blanket extension trait `DecryptFrom`: -Today `ciphertext.decrypt_into(cipher, context)` exists on the blanket -`DecryptFrom` extension trait. `Plaintext::decrypt_from(ciphertext, cipher)` -and `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)` also -exist on that trait. `DecryptInto

` separately declares the opening -operation. These method/trait names are easy to confuse. +- `ciphertext.decrypt_into(cipher, context)`; +- `Plaintext::decrypt_from(ciphertext, cipher)`; +- `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)`. -**Recommendation:** use the destination-directed pair in the preferred -public examples. `User::decrypt_from(encrypted, &keyset)` follows Rust's -From convention; `encrypted.decrypt_from(...)` would reverse it. The -receiver and migration of the competing method remain recommendations to -confirm, not settled user decisions. Do not change the declaration trait -merely because its convenience spelling changes. Address trait discoverability -with issue 5 and optional-context capability with issue 12. +`DecryptInto

` separately declares the opening operation. These method +and trait names are easy to confuse. ## 15. Coordinate the Go JSON work with the shared API redesign -The user identified [PR #1353](https://github.com/cipherstash/stack/pull/1353) -as parallel agent work. The PR is currently a Go JSON design document with -implementation tasks #1348 through #1352, tracked by #1347. Its dynamic -JSON/options path, public names, context behavior, and staged extraction -support must be reconciled with the configured-target and trait/context -contracts developed here. Preserve compatible catalog, transport, engine, -and testing work rather than designing or implementing it twice. - -The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records -the concrete reconciliation points after reading the PR write-up. This is -a local coordination item; no message or review was posted to that agent -or PR. - -## 16. Integrate the prior query design rather than reopening it - -The user supplied **Stack Encrypt query design** (2026-10-21), read in full -on 2026-10-25 from `/Users/dan/Downloads/Stack Encrypt query design.md`. -It proposes field-owned index selection, one answering index per operator, -typed field handles, a single capability declaration, and shared query -derivation/refusals. It is explicitly a discussion draft with open questions. - -The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses that -draft as the baseline and records concrete reconciliation points: our -output-side derives and configured options factories, JSON operators and -extraction, context inputs, static/dynamic portability, and validation -timing. Older compatibility requirements are superseded by the user's -prototype clarification. The source document was left unchanged. - -## 17. Use consistent definition-selection names for encrypt and query - -The user finds `.using(...)` for encryption and `.on(...)` for queries -confusing and asks to consider one name for both. Recommend `.using(...)`: -the selected definition or field supplies the operation's policy. Final -spelling is not yet accepted. Retain the operation-specific bounds: -encryption selects the output target's associated definition; queries -require a field or single-value target implementing `QueryTarget`. -This is a naming change, not a merger of those capabilities. - -The user also prefers `cipher.using(&field).query(op, input)` and explicitly -requires strict typestate. Supporting both orders is acceptable to them. -The plan recommends definition-first as canonical, with operator-first -converging to the same typed request. Missing definition/required context -must prevent awaiting on the static path; repeated selection must not -silently overwrite a request's definition. The current `.query(...).on(...)` -implementation is not claimed to already provide this API. - -## 18. Single-ciphertext representation needs consuming, cleanup-aware inputs - -**Decision, 2026-10-25:** use `OpaqueEncode` / `OpaqueDecode` for single-value -encoding/decoding, distinguishing one opaque ciphertext from target layouts. -Both direct `encrypt` and target-directed `encrypt_into` consume input and -are responsible for cleanup of owned plaintext artifacts, including the -context-taking variants. Explicit passthrough fields move to the output. -The plan records source cleanup capabilities, -immediate guarding before returning an unpolled request, cleanup of encoded -buffers/scratch on all ordinary exit paths, and boundaries for caller copies, -custom serialization, passthrough output, and Go host/guest ownership. -`Serialize` alone cannot establish cleanup of an `OpaqueBlob` source. -Consumption and zeroization are distinct; no completed audit is claimed. - -**Follow-up decision:** plaintext types must implement the existing -`Zeroize` trait; do not introduce our own cleanup trait for now. For the -proposed Serde blob, this means `Serialize + Zeroize` on input and -`DeserializeOwned + Zeroize` on owned decode. Library guards must actually -invoke cleanup; partially constructed Serde values need separate treatment. - -**Decrypt decision:** retain `keyset.decrypt` / `decrypt_with_context` for -ordinary opaque ciphertext and `T::decrypt_from` / -`decrypt_from_with_context` for declared targets. Return an ordinary owned -`T: Zeroize` on success, transferring cleanup responsibility to the caller. -Stack guards plaintext intermediates while it owns them. Explicit -drop-guarded results should be supported without becoming the default; -`Zeroize` alone does not imply cleanup on drop. This does not resolve the -separate partial-deserialization cleanup requirement. - -**Codec recommendation:** reuse Vitamin C frame machinery with a blob -mapping. The 0.8.0 transport Serde implementation already exists and guards -serialization buffers, and its frame writer wipes allocations on growth. -Do not reuse that mapping unchanged: option presence can be lost -(`Some(())` and `None` both encode as null), maps require text keys, and -successful `to_vec` returns an unguarded vector. See the +The user identified +[pull request #1353](https://github.com/cipherstash/stack/pull/1353) as +work by a parallel agent. The pull request is currently a design document +for Go JSON, with implementation tasks #1348 to #1352, tracked by #1347. + +These parts of it must be reconciled with the contracts for configured +targets, traits, and context developed here: + +- its dynamic path for JSON and options; +- its public names; +- its context behaviour; +- its staged support for extraction. + +Keep the compatible work on the catalog, the transport, the engine, and +testing. Do not design it twice, and do not implement it twice. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records the +specific points to reconcile, written after reading the pull request. This +is a local coordination item. No message or review was posted to that agent +or pull request. + +## 16. Integrate the prior query design instead of reopening it + +The user supplied **Stack Encrypt query design**, dated 21 October 2026. It +was read in full on 25 October 2026 from +`/Users/dan/Downloads/Stack Encrypt query design.md`. It is explicitly a +draft for discussion, with open questions. It proposes: + +- index selection owned by the field; +- one index that answers each operator; +- typed field handles; +- one declaration of capabilities; +- shared rules for deriving and refusing queries. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses +that draft as the baseline. The plan records specific points to reconcile: + +- this log's output-side derives and factories for configured options; +- JSON operators and extraction; +- context inputs; +- portability between the static and dynamic interfaces; +- when validation happens. + +The user's clarification about the prototype replaces older compatibility +requirements. The source document was not changed. + +## 17. Use the same name to select a definition for encryption and queries + +**Status:** `.using(...)` is recommended. The final spelling is not yet +accepted. + +The user finds it confusing that encryption uses `.using(...)` and queries +use `.on(...)`, and asked to consider one name for both. The recommendation +is `.using(...)`, because the selected definition or field supplies the +operation's policy. + +Keep the bounds specific to each operation. This is a change of name, not +a merger of the two capabilities: + +- Encryption selects the associated definition of the output target. +- Queries require a field, or a single-value target, that implements + `QueryTarget`. + +The user also prefers `cipher.using(&field).query(op, input)`. They +explicitly require strict typestate: the request's type records which steps +are done, so the compiler refuses an incomplete request. They accept +supporting both orders. The plan recommends putting the definition first as +the canonical order. The order that starts with the operator then produces +the same typed request. + +On the static path: + +- a missing definition, or missing required context, must prevent awaiting + the request; +- selecting a definition again must not silently replace the request's + definition. + +This log does not claim that the current `.query(...).on(...)` +implementation already provides this API. + +## 18. A single ciphertext needs inputs that are consumed and cleaned up + +**Status:** Decided on 25 October 2026, with the follow-up decisions below. +No completed audit is claimed. + +### Decision on encoding traits and consumed input + +**Decided on 25 October 2026.** Use `OpaqueEncode` and `OpaqueDecode` to +encode and decode single values. These names separate one opaque ciphertext +from target layouts. + +Both direct `encrypt` and target-directed `encrypt_into` consume their +input, including the variants that take context. Both are responsible for +cleaning up the owned plaintext data they create. Explicit passthrough +fields move to the output. + +The plan records: + +- what cleanup each source must support; +- guarding immediately, before returning a request that has not been + polled; +- cleaning up encoded buffers and scratch space on every ordinary exit + path; +- the limits for copies held by the caller, custom serialisation, + passthrough output, and ownership between the Go host and the guest. + +`Serialize` alone cannot ensure that an `OpaqueBlob` source is cleaned +up. Consuming a value and zeroising it are different things. + +### Follow-up decision: use the existing Zeroize trait + +Plaintext types must implement the existing `Zeroize` trait. Do not add a +cleanup trait of our own for now. For the proposed Serde blob, this means +`Serialize + Zeroize` on input, and `DeserializeOwned + Zeroize` on owned +decoding. Library guards must actually call the cleanup. Serde values that +are only partly built need separate handling. + +### Decryption decision + +Keep `keyset.decrypt` and `decrypt_with_context` for ordinary opaque +ciphertext. Keep `T::decrypt_from` and `decrypt_from_with_context` for +declared targets. On success, return an ordinary owned `T: Zeroize`. This +hands the cleanup responsibility to the caller. + +Stack guards intermediate plaintext values while Stack is their owner. Support +results with an explicit drop guard, but do not make them the default. +`Zeroize` alone does not mean the value is cleaned up when dropped. This +does not resolve the separate requirement to clean up after a partial +deserialisation. + +### Codec recommendation + +Reuse Vitamin C's frame machinery with a mapping for blobs. The 0.8.0 Serde +implementation for transport already exists and guards serialisation +buffers. Its frame writer wipes allocations when they grow. + +Do not reuse that mapping unchanged, for three reasons: + +- It can lose whether an option is present: `Some(())` and `None` both + encode as null. +- Maps require text keys. +- A successful `to_vec` returns a vector with no guard. + +The [codec assessment](2026-10-24-stack-encrypt-api-plan.md#blob-codec-assessment-vitamin-c-frames) -for the required work and Borsh fallback. Neither codec choice nor the -`Zeroize` bound establishes constant-time behavior or partial-decode cleanup. +lists the work needed, and Borsh as the fallback codec. Neither the codec +choice nor the `Zeroize` bound establishes constant-time behaviour, or +cleanup after partial decoding. -**Priority update:** first specify stable, versioned storage bytes; the -existing frame codec was ephemeral transport. Further zeroization and timing -implementation improvements may follow without changing stored format. -The upstream action is +### Priority update + +First specify stable, versioned storage bytes. The existing frame codec was +designed for short-lived transport. Later improvements to zeroisation and +timing can follow without changing the stored format. The action needed +upstream is [Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). ## 19. First-release completeness and remaining review decisions -**Accepted, 2026-10-25:** encoded ciphertext wrappers are first-class field -targets; final wrapper names/package placement come last. ORE does not -answer equality initially, even for integers. Preserve a future explicit -capability for equality-preserving encodings; never infer exact equality -from ordering support for text or other lossy encodings. Keep operator -provider overlap validation and avoid silently changing existing definitions. - -The JSON first version must be comprehensive across Rust and dynamic/Go, -including extraction, opening entries, children/counts and subtree walks. -Claude's earlier Go plan deferred walks; the new scope supersedes that -deferral. The [operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) -lists required operations, current evidence, proposed Rust/dynamic surfaces, -SQL use, missing-path/null semantics and implementation acceptance gates. +**Status:** The decisions below were accepted on 25 October 2026. The +detailed JSON operation contract is still under user review. + +### Encoded ciphertext and equality + +- Encoded ciphertext wrappers are first-class field targets. Their final + names and package placement are decided last. +- ORE does not answer equality at first, even for integers. +- Allow for a future explicit capability for encodings that preserve + equality. Never infer exact equality from ordering support for text or + other lossy encodings. +- Keep the validation that checks for overlapping operator providers. + Avoid silently changing existing definitions. + +### The JSON first version must be comprehensive + +The first JSON version must be comprehensive across Rust and the dynamic +interface for Go. It includes extraction, opening entries, listing and +counting children, and walks over subtrees. Claude's earlier Go plan +deferred walks. The new scope replaces that deferral. + +The +[operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) +lists: + +- the operations required; +- the current evidence; +- the proposed Rust and dynamic APIs; +- how SQL uses them; +- the rules for missing paths and null; +- the acceptance checks for implementation. + Names and packaging are explicitly the last design activity. -**Further accepted decisions:** custom Rust stages can remain explicitly -Rust-only with `NotPortable` for unsupported dynamic export; built-ins retain -Rust/Go parity. All key-independent checks precede key resolution/loading. -The detailed JSON operation contract is still under user review. - -## 20. Plaintext-derived keysets and optional provider policies - -The user requested routing a plaintext record to a keyset by ID/name from -one of its fields; assessment of Vitamin C's unused `AsyncCipher`; and -ZeroKMS lock context including principal JWT claim selectors. Keyset and -lock-context capabilities must remain optional for future AWS KMS/Vault -backends. See the [provider-capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) -for current source evidence, proposed static/dynamic declarations and -acceptance work. - -Current facts: Stack resolves keysets by ID/name but generation batches use -one preselected keyset. Stack deliberately orchestrates provider requests -outside Vitamin C's leaf walk and already implements synchronous `Cipher`. -Its provider payload layer supports lock context but Stack Encrypt sends -empty lock context. The existing data/index key abstractions still embed -ZeroKMS protocol types; future providers need a neutral base plus optional -capabilities, not no-op implementations of mandatory ZeroKMS features. - -Recommendations, not yet accepted: derive key routing as an explicit target -selector and resolve it in the client executor; partition batches by resolved -key/provider/principal requirements; carry lock context as a separate typed -provider policy on both generation and retrieval; reject unsupported required -features before I/O. Keep key routing, ciphertext context, search scope and -principal policy separate. Revisit cipher adapter cleanup during packaging -without discarding cross-record batching or native/WASM support. - -## 21. Packaging, naming and retirement of AsyncCipher - -The user favors removing Vitamin C `AsyncCipher` and moving application -encryption behavior into Stack Encrypt. The -[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes -Stack ownership of value traversal, declarations, target/index adapters and -execution; a small shared `stack-encrypt-format` crate for codecs and typed -encodings; a ZeroKMS-specific client crate; and reusable primitives remaining -in Vitamin C. It includes an acyclic dependency map and public name glossary. - -Stack does not use `AsyncCipher`, but Vitamin C's async helpers and tests do. -Retire that execution path with its consumers and move relevant coverage; -do not treat deletion of the trait alone as complete. Normalization/PRF/ORE -adapters also depend on the existing value model and need separation from -their primitives. Moving the frame implementation refines the earlier codec -ownership recommendation; coordinate stable storage work under Vitamin C -#519. Names and package moves remain proposals, not implementation changes. - -## 22. Customer-driven sequencing: Go JSON and match search first - -The customer needs Go JSON support and robust Bloom-filter handling with -tokenizers/token filters, including two-character input. Standalone Rust -release quality is secondary. Claude's JSON-tree PR stack headed by #1363 has now merged. Revise the -older Go plan in #1353 against it and the agreed API direction. Vitamin C -#520 and #526 implement the storage profiles; colleague review and release -are pending. Stack still needs to adopt that release. - -Freeze the public Go contract and stored binary/search semantics before the -customer release. Internal plans may change behind that interface. Adopt and -stabilize the Vitamin C codec in its current repository now; defer physical -crate moves, AsyncCipher retirement and final Rust derives. The earlier -prototype compatibility exemption does not permit breaking customer data -after this release. - -The supplied `Short-ngrams.md` explores short whole-word filters and variant -queries, which must be distinguished from arbitrary two-character substring -search. The current tokenizer permits length 2, but the current one-match -output and one/range-query-term shapes need expansion for named filters and -Boolean alternatives. Resolve the scheme's leakage, SQL representation and -performance before release; do not silently defer the customer requirement. - -### Merged implementation reconciliation, 2026-10-27 - -Reviewed `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` against the -previous `ebdb5de0a` baseline. #1372/#1373/#1378/#1402 deliver the Go JSON -catalog, configured targets, operators, request-bound opening, typed document -adapters and executable documentation. #1347–#1352 are closed. #1369/#1370 -integrate Go opaque storage; final Go releases still require replacing the -temporary Vitamin C pseudo-version pins. Rust dependencies remain on 0.8.0. - -#1379 adds cascade authentication and randomized stored/projection order. -Wildcard `->` returns an unspecified match, not the first document element. -Post-cascade data is the format to freeze; unchanged header version 1 does -not imply compatibility with the old prototype AAD. Read verification still -does not authenticate result completeness, absence or database row identity. - -#1397 implements deterministic, domain-separated short-word terms inside -the existing Bloom filter, enabled by EQL's pinned match profile. This avoids -a separate short-word index but is not the random-variant mechanism explored -earlier, and it does not by itself resolve the user's frequency/leakage -concern. Native Go index options still refuse configuration (#1112). The -uncommitted ngram research and Go delivery document are unchanged. - -The refreshed API plan and JSON operation inventory mark landed work as -implemented. Remaining items include the formal interoperability spec and -conformance corpus, parent-bound child opening, configurable Go match -pipelines, direct typed EQL conversion and later context-role separation. -The customer release does not wait for final Rust traits or crate moves. -This review inspected source and existing tests; runtime/database suites -were not rerun and no production code was changed. +### Further accepted decisions + +- Custom Rust stages can stay explicitly Rust-only, and use `NotPortable` + where dynamic export is not supported. Built-in stages keep parity between + Rust and Go. +- Every check that does not need a key runs before keys are resolved or + loaded. + +## 20. Keysets chosen from plaintext, and optional provider policies + +**Status:** The recommendations below are not yet accepted. + +### Request + +The user asked for three things: + +- route a plaintext record to a keyset, by ID or name, taken from one of its + fields; +- an assessment of Vitamin C's unused `AsyncCipher`; +- ZeroKMS lock context, including selectors for claims in the principal's + JWT (JSON Web Token). Lock context is a ZeroKMS feature that binds a data + key to claims about the authenticated identity, the principal, that makes + the request. + +Keyset and lock-context capabilities must stay optional, for future AWS KMS +and Vault backends. The +[provider capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) +gives the current evidence from the source, proposed static and dynamic +declarations, and the work needed for acceptance. + +### Current facts + +- Stack resolves keysets by ID or name, but each generation batch uses one + keyset chosen beforehand. +- Stack deliberately coordinates provider requests outside Vitamin C's walk + over leaves, and already implements the synchronous `Cipher`. +- Its layer for provider payloads supports lock context, but Stack Encrypt + sends an empty lock context. +- The existing abstractions for data keys and index keys still embed + ZeroKMS protocol types. Future providers need a neutral base plus optional + capabilities, not empty implementations of required ZeroKMS features. + +### Recommendations + +- Derive key routing as an explicit target selector, and resolve it in the + client executor. +- Split batches by the key, provider, and principal requirements they + resolve to. +- Carry lock context as a separate typed provider policy, on both key + generation and key retrieval. +- Reject unsupported required features before any I/O. +- Keep these four things separate: key routing, ciphertext context, search + scope, and principal policy. Search scope is the set of values that index + terms are derived under, such as a column and a tenant. A query can then + match every row in that scope. +- Revisit cleanup of the cipher adapters during packaging. Do not lose + batching across records, or native and WebAssembly support. + +## 21. Packaging, naming, and retiring AsyncCipher + +**Status:** The names and package moves are proposals. No implementation has +changed. + +The user favours removing Vitamin C's `AsyncCipher`, and moving encryption +behaviour for applications into Stack Encrypt. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes: + +- Stack owns value traversal, declarations, target adapters, index + adapters, and execution; +- a small shared `stack-encrypt-format` crate contains codecs and typed + encodings; +- a client crate specific to ZeroKMS; +- reusable primitives stay in Vitamin C. + +It includes a dependency map with no cycles, and a glossary of public names. + +Stack does not use `AsyncCipher`. Vitamin C's async helpers and tests do +use it. Retire that execution path together with its consumers, and move the +relevant test coverage. Deleting the trait alone is not complete. + +The adapters for normalisation, PRF (pseudorandom function), and ORE also +depend on the existing value model. They need to be separated from their +primitives. + +Moving the frame implementation refines the earlier recommendation about +who owns the codec. Coordinate the work on stable storage under +Vitamin C #519. + +## 22. Customer-driven order of work: Go JSON and match search first + +**Status:** Order of work set by the customer's needs. Merged work +reconciled on 27 October 2026. + +### The customer's needs come first + +The customer needs Go JSON support. They also need dependable Bloom-filter +handling with tokenizers and token filters, including input of two +characters. The quality of a standalone Rust release is secondary. + +- Claude's stack of JSON-tree pull requests, headed by #1363, has now + merged. Revise the older Go plan in #1353 against it, and against the + agreed API direction. +- Vitamin C #520 and #526 implement the storage profiles. Review by a + colleague, and the release, are pending. Stack still needs to adopt that + release. + +### What to freeze before the customer release + +Freeze the public Go contract, and the stored binary and search semantics, +before the customer release. Internal plans can change behind that +interface. + +Adopt and stabilise the Vitamin C codec in its current repository now. +Defer these: + +- moving crates physically; +- retiring `AsyncCipher`; +- the final Rust derives. + +The earlier exemption for prototype compatibility does not allow breaking +customer data after this release. + +### Short words in match search + +The supplied `Short-ngrams.md` explores Bloom filters for short whole words, +and queries over variants. These must be kept separate from search for any +two-character substring. + +The current tokenizer allows a length of 2. To support named filters and +Boolean alternatives, these need to grow: + +- the current output of one match; +- the current shapes of a single query term and a range query term. + +Resolve the scheme's leakage, its SQL representation, and its performance +before the release. Do not silently defer the customer's requirement. + +### Reconciliation with merged work, 27 October 2026 + +This review compared `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` +with the previous baseline, `ebdb5de0a`. + +#### What the merged Go work provides + +Pull requests #1372, #1373, #1378, and #1402 provide: + +- the Go JSON catalog; +- configured targets; +- operators; +- reads that Rust verifies against the original request. A prepared read + keeps its request, and Rust derives and checks the selectors and read keys + again from that request when it opens rows; +- adapters for typed documents; +- documentation that runs as code. + +Items #1347 to #1352 are closed. Pull requests #1369 and #1370 integrate +opaque storage in Go. Final Go releases still need the temporary Vitamin C +pseudo-version pins replaced. Rust dependencies stay on 0.8.0. + +#### Cascaded tags and random order + +Pull request #1379 adds cascaded tags: each container's tag covers the tags +of its children. It also randomises the order of stored entries and of +projections. A wildcard `->` returns an unspecified match, not the first +element of the document. + +The format to freeze is the data written after #1379. The header version is +still 1, but that does not mean the data is compatible with the old +prototype AAD. Verification on read still does not authenticate that a +result is complete, that something is absent, or which database row it +came from. + +#### Short-word terms + +Pull request #1397 implements short-word terms that are deterministic and +domain-separated, inside the existing Bloom filter. EQL's pinned match +profile turns them on. This avoids a separate index for short words. + +It is not the random-variant mechanism explored earlier. It does not by +itself resolve the user's concern about frequency and leakage. The native Go +index options still refuse configuration (#1112). The uncommitted research +on ngrams and the Go delivery document are unchanged. + +#### Remaining work + +The refreshed API plan and the JSON operation inventory mark landed work as +implemented. The remaining items include: + +- the formal interoperability specification and conformance corpus; +- opening a child bound to its parent; +- configurable match pipelines in Go; +- direct typed EQL conversion; +- later, separating the roles of context. + +The customer release does not wait for the final Rust traits or crate +moves. + +#### What this reconciliation checked + +This review inspected the source and the existing tests. It did not run the +run-time or database test suites again, and it changed no production code. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index c927b7d67..8f3ec73d0 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -1,21 +1,62 @@ # Context sources and storage: directional API examples -Date: 2026-10-24. Design exploration accompanying the -[API review log](2026-10-24-stack-encrypt-api-review.md) and +Date: 24 October 2026. This design exploration goes with the +[API review log](2026-10-24-stack-encrypt-api-review.md) and the [complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md). **Everything introduced here is proposed syntax, not an implemented API.** -The purpose is to compare context behavior across the meaningful source and -storage combinations before selecting attributes or changing traits. The -`Ciphertext` and `passthrough` proposals are inherited from the workflow draft. +None of the code below compiles against Stack Encrypt today. + +This document +compares what happens to encryption context in each useful combination of +context source and context storage. That comparison comes before anyone +chooses attributes or changes traits. The `Ciphertext` and `passthrough` +proposals come from the workflow draft. The workflow draft is the complete +public workflow examples document linked above. + +## Terms used in this document + +These terms appear throughout. Each one is also explained where it first +matters: + +- **Encryption context** is a value that each ciphertext is bound to. Stack + Encrypt gives it to the cipher as additional authenticated data (AAD). + AAD is not encrypted, but decryption fails unless the same value is + supplied again. The same context also scopes a field's search index terms + (see `packages/stack-encrypt/src/target/context.rs`). +- **Static prefix** is the fixed string in the declaration, such as + `"users"`. Together with the field name it forms a field's static base. +- **Logical row context**, written `R`, is the part of the context that + changes from record to record, such as a user id. +- **Effective context** is the complete context one field is encrypted + under: its static base combined with `R`. +- **Selector** is a declaration inside `context(...)` that says where to + get `R`, for one direction or for both. +- **Caller argument** is a context value the application passes to a + `*_with_context` method, rather than one read from a record. +- **Passthrough field** is an output field copied in cleartext from the + input, with `#[stash(passthrough)]`. A passthrough field is not encrypted + and not authenticated. +- **`from_context` field** is an output field that stores the computed `R` + in cleartext, so that decryption can read it back. +- **Caller extension** is an existing feature of the simple static form. + The caller adds an extra context value to the declared static context. +- **Keyset** is a set of encryption keys held in ZeroKMS, CipherStash's key + management service. `cipher.default_keyset()` chooses which keyset + encrypts. +- **The derive** means the `EncryptFrom` and `DecryptInto` derive macros in + `packages/stack-encrypt-derive`. ## Recommendation: directional sources, separate storage -Borrow Serde's nested directional attribute style: it supports separate -serialization and deserialization names within one `rename` attribute. -See [Serde's field attributes](https://serde.rs/field-attrs.html). +Name each direction, and say for each one where to get context. Store +the context with a separate field annotation. -For encryption, use direction names to say **where to obtain context**: +The nested direction style comes from Serde. Serde's `rename` attribute can +contain separate names for serialisation and deserialisation. See +[Serde's field attributes](https://serde.rs/field-attrs.html). + +For encryption, the direction names say **where to get context**: ```rust #[stash(context( @@ -25,107 +66,133 @@ For encryption, use direction names to say **where to obtain context**: ))] ``` -The two `field` selectors refer to different objects: the plaintext input -when encrypting, and the encrypted input when decrypting. They must yield the -same typed context value. Unlike a Serde rename, these are not independent -cryptographic names: changing the context value or its shape between the two -directions prevents decryption. +The two `field` selectors refer to different objects. On encrypt, the +selector reads the plaintext input. On decrypt, it reads the encrypted +input. Both must produce the same typed context value. + +Unlike the two names in a Serde rename, the two selectors are not +independent cryptographic names. If the context value or its shape differs +between the two directions, decryption fails. -For the common case where both directions use the same Rust field name, -the recommended shorthand is: +Both directions often use the same Rust field name. For that case, the +recommended shorthand is: ```rust #[stash(context(prefix = "users", field = "id"))] ``` -This expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. -It still reads two different objects: the plaintext input on encrypt and the -encrypted input on decrypt. It does not copy or store the field implicitly; -the output layout must make that value available, normally with passthrough. +This shorthand expands exactly to +`encrypt(field = "id"), decrypt(field = "id")`. It still reads two +different objects: the plaintext input on encrypt, and the encrypted input +on decrypt. The shorthand does not copy or store the field by itself. The +output layout must make that value available, normally with passthrough. Use the explicit directional form when the names or sources differ. -Use a separate field annotation to **store the computed context** when it is -not already carried by ordinary passthrough fields: +To **store the computed context**, use a separate field annotation. This is +needed only when ordinary passthrough fields do not already contain the +context value: ```rust #[stash(from_context)] scope: u64, ``` -`from_context` fills an output metadata field from the computed logical -context. It does not choose the context source. A passthrough source field -can already carry the context value, in which case no `from_context` field -is needed. No new source-side encryption methods or public `encrypt_as` -entry point are introduced. +A `from_context` field is an output metadata field filled from the +computed logical context. It does not choose where the context comes from. +A passthrough source field can already contain the context value. In that +case, no `from_context` field is needed. This proposal adds no new +source-side encryption methods and no public `encrypt_as` entry point. + +These names are better than bare `context(in)` and `context(out)`. Each of +those terms could mean any of three things: -This is preferable to bare `context(in)` / `context(out)`: either term could -mean an operation direction, an input/output object, or storing a value. -`encrypt`, `decrypt`, and `from_context` give those roles distinct names. +- an operation direction; +- an input object or an output object; +- storing a value. + +The names `encrypt`, `decrypt`, and `from_context` give those three roles +separate names. ## Proposed vocabulary and semantics +The table lists each proposed form inside `context(...)` and what it means. + | Syntax inside `context(...)` | Meaning | | --- | --- | | `prefix = "users"` | Static base for this layout's field contexts | | `field = "id"` | Shorthand for both `encrypt(field = "id")` and `decrypt(field = "id")` | -| `encrypt(caller = u64)` | Obtain the logical context from a required `u64` argument | -| `encrypt(field = "id")` | Obtain it from the plaintext input's `id` | -| `decrypt(caller = u64)` | Obtain it from a required `u64` argument on decrypt | -| `decrypt(field = "scope")` | Obtain it from the encrypted input's cleartext `scope` | +| `encrypt(caller = u64)` | Get the logical context from a required `u64` argument | +| `encrypt(field = "id")` | Get it from the plaintext input's `id` | +| `decrypt(caller = u64)` | Get it from a required `u64` argument on decrypt | +| `decrypt(field = "scope")` | Get it from the encrypted input's cleartext `scope` | | `encrypt(tuple(field = "tenant_id", field = "id"))` | Build one ordered, typed pair from plaintext fields | | `decrypt(tuple(field = "id", caller = u64))` | Build one pair from a stored field and the decrypt argument | -`tuple(...)` is a declarative selector, not a Rust expression or a function -that gets access to keys. It can nest. `field` names resolve at compile time; -the first design supports direct fields, not arbitrary paths or callbacks. -There is at most one caller argument selector per direction; use a tuple or -typed context object as that argument when it carries several values. +`tuple(...)` is a declarative selector. It is not a Rust expression or a +function that gets access to keys. A `tuple(...)` can contain another +`tuple(...)`. + +The derive resolves `field` names at compile time. The first design +supports direct fields only, not arbitrary paths or callbacks. Each +direction has at most one caller argument selector. When that argument +needs several values, make it a tuple or a typed context object. + +A declaration uses either the shared `field` shorthand, or both explicit +directional selectors. For the initial API, the derive rejects a mix of the +shorthand and directional selectors. Rejecting the mix avoids inventing +precedence rules, and keeps the expansion predictable. A record whose names +differ between directions uses the explicit form. -Use either the shared `field` shorthand or both explicit directional -selectors. For the initial API, reject mixing the shorthand with directional -selectors instead of inventing precedence rules. That keeps the expansion -predictable; a record with asymmetric names uses the explicit form. `context = "users"` -remains the existing simple spelling for a static prefix with optional caller -extensions; it is used in scenario 0. +remains the existing simple spelling for a static prefix with optional +caller extensions. Scenario 0 uses it. ### Composition rule -Call the logical row context `R`. For a field named `name`, its static base is -`B = ("users", "name")` and its effective context is `(B, R)`. With only a -static prefix and no extension it is just `B`, not `(B, ())`. +Call the logical row context `R`. A field named `name` has the static base +`B = ("users", "name")`. Its effective context is `(B, R)`. With only +a static prefix and no extension, the effective context is `B`, not +`(B, ())`. -The selector's result is **one structured value**. A pair `(tenant_id, id)` -produces `(("users", "name"), (tenant_id, id))`, not -`((("users", "name"), tenant_id), id)`. `from_context` stores `R`, not the -field-dependent effective context. Static prefixes and field identities are -reconstructed from the declaration. +The selector's result is **one structured value**. A pair +`(tenant_id, id)` produces `(("users", "name"), (tenant_id, id))`, not +`((("users", "name"), tenant_id), id)`. A `from_context` field stores `R`, +not the effective context. The effective context differs from field to +field. Static prefixes and field identities are rebuilt from the +declaration. -Names in this proposal do not change existing byte encodings. Mapping new -selectors onto the existing context machinery must preserve types, order, -grouping, and descriptor rendering. A new explicit selector is a new -declaration; it must not silently reinterpret existing stored data. +The names in this proposal do not change existing byte encodings. When new +selectors are mapped onto the existing context code, they must preserve +types, order, grouping, and descriptor rendering. A new explicit selector +is a new declaration. It must not silently reinterpret data that is already +stored. ### Storage and expected context -When all of `R` can be reconstructed from stored fields, the default decrypt -call uses those fields. The caller may additionally supply an expected `R` -through the existing `ExpectedContext` pattern: check equality first, then -use the stored value. That expected value is **not another extension**. +Sometimes stored fields contain all of `R`. Then the default decrypt call +rebuilds `R` from those fields. The caller can also supply an expected `R` +through the existing `ExpectedContext` pattern. Decryption then checks +that the two are equal, and uses the stored value. That expected value is +**not another extension**. -When a decrypt selector needs a caller argument, that argument is required -to construct `R`; no no-context convenience is available. It is not an -expected-value check on context already fully available from the record. +A decrypt selector can need a caller argument. That argument is then +required to build `R`, and no call without a context is available. It is +not a check of an expected value against context the record already +contains in full. -If a future caller needs both a required input and a separate expectation -on the assembled `R`, use a distinct checked-context input or a named API. -Do not overload one unlabelled argument to mean both. That combined check -API remains outside this proposal; scenario 7 makes the need visible. +A future caller could need both a required input and a separate check of +the assembled `R`. That caller would use a distinct checked-context input +or a named API. Do not make one unlabelled argument mean both. This +proposal does not include that combined-check API. Scenario 7 shows where +the need arises. ## Scenario matrix -| Scenario | Encrypt obtains context from | Context available in stored output | Decrypt obtains context from | -| --- | --- | --- | --- | +Each scenario combines one context source with one way of storing it. The +table shows where each scenario gets and stores the context. + +| Scenario | Encrypt gets context from | Context available in stored output | Decrypt gets context from | +| ---: | --- | --- | --- | | 0 | Static prefix; no row context | Declaration | Declaration | | 1 | Caller argument | Not stored | Caller argument | | 2 | Caller argument | Dedicated metadata field | Metadata field | @@ -137,21 +204,24 @@ API remains outside this proposal; scenario 7 makes the need visible. | 8 | Plaintext `(tenant_id, id)` | Neither component is stored in cleartext | Caller supplies the pair | An encrypted copy of a context field does not make that context available -before decrypt. Scenarios 3 and 8 retain encrypted copies to recover the -complete plaintext object; callers still need the context independently. +before decryption. Scenarios 3 and 8 keep encrypted copies so that +decryption can recover the complete plaintext object. Callers still need to +supply the context separately. ## Shared code for the examples -Each scenario defines an output type and an async function. Combine this -prelude, the scenario blocks, and the final `main` into one candidate program. -They use real file serialization through the same helper, so no example -relies on a context value being secretly carried inside an in-memory handle. -All use proposed APIs and therefore cannot yet be typechecked against Stack. +Each scenario defines an output type and an async function. Together, this +prelude, the scenario blocks, and the final `main` form one candidate +program. Every example stores its output in a real file and reads it back +through the same helper. So no example depends on a context value hidden +inside an in-memory handle. All the examples use proposed APIs, so they +cannot be typechecked against Stack yet. -Dependencies match the workflow draft: Stack Encrypt, Serde with derives, -serde_json, tempfile, and Tokio. ZeroKMS credentials are required once these -APIs exist. `scope = 7` in caller examples is external application context, -not a keyset selector or an authorization credential. +The dependencies match the workflow draft: Stack Encrypt, Serde with +derives, serde_json, tempfile, and Tokio. Once these APIs exist, the +examples need ZeroKMS credentials. In the caller examples, `scope = 7` is +external application context. It is not a keyset selector or an +authorisation credential. ```rust use serde::{de::DeserializeOwned, Deserialize, Serialize}; @@ -191,8 +261,9 @@ fn store_and_reload(value: &T) -> Result { ## 0. Static prefix only -This is the baseline. There is no dynamic row context to store or supply. -It retains the existing static-context form and proposed passthrough support. +This example shows the baseline: a static prefix and nothing else. There +is no row context to store or supply. The example keeps the existing +static-context form, and uses the proposed passthrough support. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -219,15 +290,17 @@ async fn example_0() -> Result<()> { } ``` -For this shorthand, caller extensions remain available as previously agreed: -encrypt and decrypt must receive the same extension. The remaining examples -use explicit directional selectors, so their argument requirements follow -those selectors; there is no additional implicit extension layered on top. +With this shorthand, caller extensions stay available, as previously +agreed. Encrypt and decrypt must receive the same extension. The remaining +examples use explicit directional selectors. Their argument requirements +come from those selectors alone. No implicit extension is added on top. ## 1. Caller context, not stored -`scope` is absent from the output. The caller must provide it again. A -plaintext `id` in this layout is unrelated to the chosen context source. +This example shows a context that the caller supplies and the output does +not store. `scope` is absent from the output, so the caller must provide it +again on decrypt. The plaintext `id` in this layout has nothing to do with +the chosen context source. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -262,13 +335,16 @@ async fn example_1() -> Result<()> { } ``` -Logical context: `7_u64`. Effective name context: `(("users", "name"), 7_u64)`. +Logical context: `7_u64`. Effective context for `name`: +`(("users", "name"), 7_u64)`. ## 2. Caller context, stored in an output metadata field -The source is still the function argument. `from_context` stores its value; -the decrypt selector explicitly chooses that stored field. `scope` is output -metadata, not a field to add to the recovered `User`. +This example shows a caller-supplied context that the output stores, so +decrypt needs no argument. The source is still the function argument. The +`from_context` field stores its value, and the decrypt selector explicitly +chooses that stored field. `scope` is output metadata. It is not a field to +add to the recovered `User`. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -305,15 +381,18 @@ async fn example_2() -> Result<()> { } ``` -Logical context remains `7_u64`. It is stored once, not appended a second time -on decrypt. This generalizes the existing caller-supplied-and-stored case to +The logical context is still `7_u64`. The output stores it once, and +decrypt does not append it a second time. This example extends the +existing case, where the caller supplies the context and it is stored, to the field-by-field layout. ## 3. Plaintext field as context, no cleartext context stored -The encrypt selector reads `User.id`; no argument is needed. The output's -`id` is encrypted, so it cannot supply the context required to open itself. -The reader obtains the id from its external record lookup. +This example shows a context read from a plaintext field that is then +encrypted, so the output has no cleartext copy. The encrypt selector reads +`User.id`, so encrypt needs no argument. The output's `id` is encrypted. An +encrypted `id` cannot supply the context needed to decrypt itself. The +reader gets the id from its own record lookup, outside the record. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -345,16 +424,19 @@ async fn example_3() -> Result<()> { } ``` -Both directions construct `R = 42_u64`. Supplying a different id fails to -open the encrypted fields. The id is not dropped or defaulted when rebuilding -`User`; its encrypted copy is opened using the externally supplied context. +Both directions build `R = 42_u64`. If the reader supplies a different id, +the encrypted fields fail to decrypt. Rebuilding `User` does not drop the +id or use a default. Decryption opens the encrypted copy of the id with +the context the reader supplied. ## 4. Plaintext field as context, stored as the corresponding output field -This is the natural ID-bound form of workflow example 2. The context selector -and the passthrough annotation have separate jobs: select `id`, and copy it. -It uses the common-field shorthand, equivalent to the explicit pair of -directional selectors introduced above. +This example shows a context read from a plaintext field and stored as a +passthrough copy of the same field. It is the obvious id-bound form of +workflow example 2. The context selector and the passthrough annotation +have separate jobs: the selector chooses `id`, and passthrough copies it. +The example uses the shared `field` shorthand. That shorthand is the same +as the explicit pair of directional selectors introduced above. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -392,17 +474,21 @@ async fn example_4() -> Result<()> { } ``` -The second decrypt call uses the `ExpectedContext` conversion. It checks -the stored id instead of constructing `(stored_id, expected_id)`. Changing -`id` alone breaks decryption; moving an entire consistent record is detected -only when the reader supplies the expected id (or another placement check). -`created_at` remains ordinary unauthenticated passthrough metadata. +The second decrypt call uses the `ExpectedContext` conversion. That +call checks the stored id. It does not build `(stored_id, expected_id)`. + +Changing `id` alone breaks decryption. Moving a whole consistent record is +detected only if the reader supplies the expected id, or checks placement +another way. `created_at` stays ordinary passthrough metadata. Passthrough +metadata is not authenticated. ## 5. Several plaintext fields as context, stored in separate fields -Add a tenant id to the input. The context is the ordered pair `(tenant_id, id)`. -The output renames `id` to `user_id` to demonstrate why direction-specific -field selectors are useful. No duplicate context metadata field is needed. +This example shows a context built from two plaintext fields, both stored +as passthrough fields. The input gains a tenant id, and the context is the +ordered pair `(tenant_id, id)`. The output renames `id` to `user_id`. The +rename shows why separate field selectors for each direction are useful. +No extra context metadata field is needed. ```rust #[derive(Debug, PartialEq)] @@ -455,17 +541,21 @@ async fn example_5() -> Result<()> { } ``` -Both directions construct `(7_u64, 42_u64)`. Rust field order does not choose -the context order: the selector does. A Serde wire rename also does not change -the selector; it names Rust fields, not serialized property names. +Both directions build `(7_u64, 42_u64)`. The selector sets the order of +the context, not the order of the Rust fields. A Serde rename of the +stored property also leaves the selector unchanged. Selectors name Rust +fields, not serialised property names. ## 6. Plaintext field plus caller argument, store the combined context -Here `R = (User.id, caller_scope)`. `binding` stores that complete pair. The -original id is also retained as ciphertext so `User` can be reconstructed -without adding a new rule that maps pieces of metadata into plaintext fields. -That deliberately redundant representation keeps this example focused on -context composition rather than projection conveniences. +This example shows a context built from a plaintext field and a caller +argument, with the whole pair stored. Here +`R = (User.id, caller_scope)`, and the `binding` field stores that +complete pair. The output also keeps the original id as ciphertext, so +that decryption can rebuild `User`. Without that copy, the derive would +need a new rule that maps pieces of metadata into plaintext fields. The +id is stored twice on purpose, so that this example stays about how the +context is put together, not about mapping metadata to fields. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -501,14 +591,17 @@ async fn example_6() -> Result<()> { } ``` -The encryption function accepts only the caller-supplied component (`7`), not -the pair the derive can construct itself. A checked decrypt would expect the -whole reconstructed pair `(42, 7)`, not just that caller component. +The encryption function accepts only the part the caller supplies (`7`). +It does not accept the whole pair, because the derive can build the pair +itself. A checked decrypt would expect the whole rebuilt pair `(42, 7)`, +not only the caller's part. ## 7. Plaintext field plus caller argument, store only the field component -Store `id` as a normal field, but keep the external scope out of the output. -Decryption combines the stored id with the scope supplied by the reader. +This example shows a context built from a plaintext field and a caller +argument, with only the field stored. The output stores `id` as a normal +passthrough field, and leaves the external scope out. Decryption combines +the stored id with the scope the reader supplies. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -543,17 +636,19 @@ async fn example_7() -> Result<()> { } ``` -This also constructs `(42_u64, 7_u64)` in both directions. It shows why -"stored context" cannot always be represented as a single yes/no flag: -some components may be stored and others supplied externally. A separate -expected-id check can be performed against `loaded.id` before opening when -the application also needs to assert the record's destination. +This example also builds `(42_u64, 7_u64)` in both directions. It shows +why one yes-or-no flag cannot always describe whether context is stored. +Some parts may be stored, and others supplied from outside the record. +The application can also need to check which record it has. If so, it can +compare `loaded.id` with an expected id before decrypting. ## 8. Several plaintext fields as context, keep neither component in cleartext -The encrypt side obtains `(tenant_id, id)` from the source. The reader supplies -the complete pair from outside the encrypted record; both source fields are -recovered from ciphertext after that context has been established. +This example shows a context built from two plaintext fields, with neither +stored in cleartext. The encrypt side gets `(tenant_id, id)` from the +source. The reader supplies the complete pair from outside the encrypted +record. Once that context is supplied, decryption recovers both source +fields from their ciphertexts. ```rust #[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] @@ -587,12 +682,14 @@ async fn example_8() -> Result<()> { } ``` -This is a new typed selector contract. It does not mean flattening the pair -into text, or treating separately chained extensions as interchangeable with -one tuple-valued argument. +This is a new contract for a typed selector. It does not mean flattening +the pair into text. It also does not make separately chained extensions +interchangeable with one argument whose value is a tuple. ## Candidate program entry point +This `main` runs all nine scenarios in order. + ```rust #[tokio::main] async fn main() -> Result<()> { @@ -611,10 +708,12 @@ async fn main() -> Result<()> { ## Consequences for the traits and derive -The logical context `R` is not always the value a caller passes. For example: +The logical context `R` is not always the value a caller passes. The table +compares `R` with the caller's input in the scenarios that have a row +context: | Scenario | Logical `R` | Caller input on encrypt | Caller input on decrypt | -| --- | --- | --- | --- | +| ---: | --- | --- | --- | | 1 | `u64` | Required `u64` | Required `u64` | | 2 | `u64` | Required `u64` | Optional expectation of `u64` | | 3 | `u64` | None | Required `u64` | @@ -623,101 +722,133 @@ The logical context `R` is not always the value a caller passes. For example: | 7 | `(u64, u64)` | Required `u64` component | Required `u64` component | | 8 | `(u64, u64)` | None | Required pair | -Today `EncryptFrom::Context` and `DecryptInto

::Context` already belong -to separate traits and can differ. They describe the input the caller needs -to supply, not necessarily the final context used by every operation. That -distinction should be explicit in documentation; `ContextInput` might be a -clearer name, but a rename is not decided here. +Today, `EncryptFrom::Context` already belongs to a different trait from +`DecryptInto

::Context`. The two types can differ. -No-caller-input selectors can expose a defaultable input, such as `()`, to -support `encrypt_into`. Stored-context decrypt uses the existing -`ExpectedContext` idea. Required caller selectors must remain required -even if their Rust type implements `Default`: a required `u64` must not -silently become zero merely because `u64: Default`. +Each `Context` type describes the input the caller needs to supply. That +input is not always the final context every operation uses. The documentation should state that difference. The name +`ContextInput` might be clearer, but this document does not decide on a +rename. -That exposes a real constraint in today's convenience methods, whose only +To support `encrypt_into`, a selector that needs no caller input can +expose an input that has a default, such as `()`. A decrypt that reads +stored context uses the existing `ExpectedContext` idea. A required +caller selector must stay required, even if its Rust type implements +`Default`. A required `u64` must not silently become zero only because +`u64: Default`. + +That rule exposes a real limit in today's convenience methods. Their only test is `Context: Default`. An implementation of this proposal would need -a non-defaultable `RequiredContext` input (with an ergonomic conversion -from `T`), or a separate trait capability controlling context-free calls. -The examples show bare caller values at call sites, but do not assume that -the associated input type can be bare `u64`. Choose that mechanism before -claiming the directional API enforces required inputs. +one of two things: + +- a `RequiredContext` input that has no default, with an easy + conversion from `T`; +- a separate trait capability that controls calls without a context. + +The examples pass bare caller values at call sites. They do not assume +that the associated input type can be a bare `u64`. Choose that mechanism +before claiming the directional API enforces required inputs. The current public `Encryption` constructors must also gain or expose the -core-owned source-selection machinery needed by the derive. Existing plan -context fields demonstrate source-dependent context, but are label-based, -carry the field, and use different composition rules. They are not already -an implementation of this proposal's general typed, optionally stored `R`. +source-selection code the derive needs, owned by the core crate. + +Plan context fields are the existing context fields of the plan builder, +in `packages/stack-encrypt/src/plan/build.rs`. They already show context +that depends on the source. But plan context fields are based on labels, +carry the field, and use different composition rules. So they do not +already implement this proposal's general `R`: a typed value that can be +stored. ## Rules the implementation should enforce +The implementation should enforce these rules: + - Selectors for both directions must produce the same context type and - ordered structure. Different Rust field names are fine; different values + ordered structure. Different Rust field names are fine. Different values or grouping are not a supported way to encrypt and decrypt one record. - Resolve field names and types at compile time. A decrypt selector cannot - read an encrypted field to obtain the context needed to decrypt it. + read an encrypted field to get the context needed to decrypt it. - A `from_context` field is cleartext output metadata with the logical - context's type. It cannot also be encrypted, defaulted, or independently - copied from an unrelated source. Ordinary passthrough fields do not - become context merely because their names are `id` or `tenant_id`. -- Extra metadata is not invented as a plaintext field during recovery. - Every original plaintext field must have a defined recovery path; no - silent omission or defaulting to compensate for a context-only source. -- Check supplied expectations before key retrieval where possible. A - context value read from storage remains untrusted until encrypted fields - authenticate under it. An expectation check alone is not authentication. -- If a selected source value is duplicated in stored context metadata and - in a separately recovered plaintext field, validate their agreement - before returning the reconstructed object. Do not return a different id + context's type. It cannot also be encrypted, set to a default, or copied + separately from an unrelated source. An ordinary passthrough field does + not become context only because its name is `id` or `tenant_id`. +- Recovery does not invent extra metadata as a plaintext field. Every + original plaintext field must have a defined way to be recovered. No + field is silently left out, or set to a default, because a source was + used only for context. +- Check supplied expectations before fetching keys, where possible. A + context value read from storage stays untrusted until the encrypted + fields authenticate under it. An expectation check alone is not + authentication. +- A selected source value can appear both in stored context metadata and + in a separately recovered plaintext field. If so, check that the two + agree before returning the rebuilt object. Never return an id different from the one the encrypted fields were bound to. -- Stored context should be required in the serialized schema; a missing - field must not silently fall back to a default context. Empty-context - policy must be explicit and retain the current nonempty target rule. -- For indexed targets, the selected context also scopes their terms unless - a future API explicitly separates index context from ciphertext context. - A row-specific id therefore changes search semantics; these examples - concern ciphertext layouts and do not silently promise cross-row queries. - -Required tests include storage round trips for every scenario, missing -required arguments, wrong expected values, altered stored context, tuple -reordering/regrouping, renamed source/output fields, and attempted circular -decrypt dependencies. Same-typed field swaps may evade a type check, so -round-trip and mismatch tests remain necessary even with typed selectors. +- Stored context should be required in the serialised schema. A missing + field must not silently fall back to a default context. The policy for + empty context must be explicit. That policy must keep the current rule + that a target's context is nonempty. +- An indexed target is a field that also produces search index terms. For + an indexed target, the selected context also scopes those terms, unless + a future API explicitly separates index context from ciphertext + context. So a row-specific id changes how search works. These + examples are about ciphertext layouts. They do not silently promise + queries that match across rows. + +The required tests are: + +- storage round trips for every scenario; +- missing required arguments; +- wrong expected values; +- altered stored context; +- reordered or regrouped tuples; +- renamed source or output fields; +- attempts to make decryption depend on its own output in a circle. + +Two fields of the same type can be swapped without failing a type check. +So round-trip and mismatch tests are still needed, even with typed +selectors. ## Alternatives worth comparing -1. **Field-level `context(encrypt, decrypt)` flags.** Concise for one id, - but references the output declaration while the encrypt source lives - elsewhere, leaves multi-field order implicit, and cannot readily describe - caller-plus-field composition. Prefer explicit container selectors first. -2. **One `context = expression` callback.** Flexible, but hides dependencies, - makes reverse reconstruction and cycle checks harder, and encourages - arbitrary application code in the declaration. Keep simple declarative - selectors as the initial surface. -3. **A typed context object with a dedicated derive.** Could reduce repetition - for large composite contexts and carry a stronger encoding contract. - Consider it after the scalar/tuple cases establish the desired semantics. - -The recommendation is the directional selector plus independent storage -model, not a commitment to every spelling shown here. In particular, -`from_context`, `prefix`, the tuple notation, and how required inputs are -represented remain review choices. +1. **Field-level `context(encrypt, decrypt)` flags.** These are concise for + one id. But they refer to the output declaration, while the encrypt + source is declared elsewhere. They leave the order of several fields + unstated, and cannot easily describe a context built from a caller + argument and a field. Prefer explicit selectors on the container first. +2. **One `context = expression` callback.** This is flexible, but it hides + dependencies. It makes rebuilding the context in reverse harder, and + makes cycles harder to detect. It also encourages arbitrary application + code in the declaration. Keep simple declarative selectors as the + initial API. +3. **A typed context object with a dedicated derive.** This could reduce + repetition for large composite contexts, and give a stronger encoding + contract. Consider it once the single-value and tuple cases have settled + the intended behaviour. + +The recommendation is the model of directional selectors with separate +storage. It is not a commitment to every spelling shown here. In +particular, these remain choices for review: `from_context`, `prefix`, the +tuple notation, and how required inputs are represented. ## Evidence and validation -Current behavior was checked in -[context types](../../packages/stack-encrypt/src/target/context.rs), -[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), -[derive validation](../../packages/stack-encrypt-derive/src/shape.rs), and -[plan context fields](../../packages/stack-encrypt/src/plan/build.rs). -The directional nesting is inspired by Serde, not provided by Serde. - -These are candidate API examples. Syntax/link checks do not establish macro -support, type correctness, or encryption behavior. No live encryption or -database tests are claimed, and no library implementation is changed. - -Validation completed: `rustfmt` parsed the combined nine-scenario candidate -program and the introductory fragments (wrapped as Rust items), using edition -2024. Local links, Markdown fences, and trailing whitespace were checked in -this document and both companion documents. No proposed macro was expanded -or typechecked. +The current behaviour was checked in these files: + +- [context types](../../packages/stack-encrypt/src/target/context.rs); +- [target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs); +- [derive validation](../../packages/stack-encrypt-derive/src/shape.rs); +- [plan context fields](../../packages/stack-encrypt/src/plan/build.rs). + +Serde inspired the directional nesting, but Serde does not provide it. + +These are candidate API examples. Checks of syntax and links do not show +that the macros support them, that the types are correct, or that +encryption works as described. This document claims no live encryption +or database tests, and changes no library code. + +The following validation was completed. `rustfmt` parsed the combined +candidate program for the nine scenarios, using edition 2024. It also +parsed the introductory fragments, wrapped as Rust items. Local links, +Markdown fences, and trailing whitespace were checked in this document and +both companion documents. No proposed macro was expanded or typechecked. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index ee700cabc..4dd2376ea 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -1,61 +1,140 @@ # Stack Encrypt: complete public API examples for review -Date: 2026-10-24. Companion to the -[API review log](2026-10-24-stack-encrypt-api-review.md). -The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) -explore directional source selection, storing context, and decrypt inputs -across nine scenarios. Their attributes are proposals, not implemented APIs. -The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) -covers the comprehensive first-release JSON workflows with Rust, dynamic -and SQL examples, including extraction and walks. - -The user clarified on 2026-10-25 that this API is an early prototype: -backward compatibility with its old APIs and encodings is not required. - -This document tests the proposed public API against three complete workflows: -encrypt, persist, load, and decrypt a user; the searchable workflow also -constructs and applies a query. It is an API design draft, not documentation -for a released version. The examples deliberately leave plans and operation -descriptions out of the application code. - -Each main example is a separate program, with all imports and types shown. -They use the same user fields and synthetic data. `created_at` is an `i64` -Unix timestamp in seconds so that date-library configuration does not distract -from the encryption API. The storage files live in a temporary directory and -are removed when the program exits. Files stand in for a database boundary; -the examples really encode, write, read, and decode their stored forms. - -## What is agreed, and what these examples propose - -**Ownership decision (2026-10-25):** `encrypt` and `encrypt_into` consume -their inputs, including context-taking variants. Their implementations must -guard owned plaintext and temporary buffers for cleanup on success, error, -and request drop; passthrough fields intentionally transfer to the output. -The single-ciphertext capabilities are named `OpaqueEncode` / `OpaqueDecode`. -Plaintext types must implement `Zeroize`; no new cleanup trait is planned. -The candidate programs below predate that bound: update their source derives -and any reuse of consumed input when making them executable. -`Serialize` alone does not establish the cleanup contract for `OpaqueBlob`. - -**Decrypt decision (2026-10-25):** retain direct `keyset.decrypt` and -`decrypt_with_context` for opaque ciphertext, alongside destination-directed -`T::decrypt_from` and `decrypt_from_with_context` for declared targets. -Successful calls return ordinary owned `T: Zeroize`; the caller then owns -cleanup responsibility. Stack guards intermediate plaintext. Support -explicit drop-guarded results without imposing a wrapper by default; -`Zeroize` alone does not wipe returned values on drop. These are accepted -API contracts, not a claim that the prototype already implements them. - -**Collection decision (2026-10-25):** direct `keyset.encrypt(value)` handles -one single-value input and produces `Ciphertext`; it does not automatically -encrypt collections element by element. Collection batching uses -`EncryptInto`, for example +Date: 24 October 2026. + +This document tests the proposed public Rust API of Stack Encrypt against +three complete workflows. The code in it is a proposal. The code does not +compile against today's crate. This document is not documentation for a +released version. +Reviewers need to decide whether the API these examples show is the one to +build. + +The questions to settle are listed in +"What to review across the three workflows", near the end. + +The three workflows are: + +1. Encrypt a whole user record as one encrypted value, store it, load it, + and decrypt it. +2. Encrypt only some fields of the record, store it, load it, and decrypt + it. +3. Do the same as workflow 2, but also store a search term for the email. + Then build a query and use it to find the record. + +A fourth program shows how the same search works today with the Encrypt +Query Language (EQL) types, using APIs that already exist. + +## Related documents + +This document is a companion to the +[API review log](2026-10-24-stack-encrypt-api-review.md). Two other +documents cover related ground: + +- The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) + work through nine scenarios. They cover where a context comes from, whether + the context is stored, and what a decrypt call takes as input. The + attributes in that document are proposals, not implemented APIs. +- The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) + covers all the JSON workflows planned for the first release, with Rust, + dynamic, and SQL examples, including extraction and walks. + +The user clarified on 25 October 2026 that this API is an early prototype. +The new API does not need to stay backward compatible with the old APIs or +their encodings. + +## Terms used in this document + +The Stack Encrypt crate is `packages/stack-encrypt`. This document uses +these terms from it: + +- **Keyset**: a set of keys in ZeroKMS, the CipherStash key service. + Every encrypted value is sealed under one keyset, and the sealed value + records which keyset that was. +- **Data key**: the key that encrypts one value. Stack Encrypt requests data + keys from ZeroKMS in batches. +- **Context**: extra input that is bound to a ciphertext when it is sealed, + such as `("users", "email")` or a record id. Decryption fails unless the + reader supplies the same context. Encryption uses the context as + additional authenticated data (AAD): data that encryption authenticates + but does not encrypt. Search terms are also derived from the context. +- **Leaf**: one encrypted value. "Sealing" a leaf means encrypting and + authenticating it. +- **Envelope**: the complete stored form of a ciphertext. It includes the + sealed bytes and the metadata needed to fetch the key again, so the caller + does not store anything separately. +- **Target**: a Rust type that a value is encrypted into or decrypted from. + The target decides which ciphertext and which search terms are produced. +- **Passthrough field**: a field copied unencrypted from the plaintext type + to the encrypted type, and back again on decrypt. +- **Plan**: the internal description of which fields a conversion encrypts, + indexes, or passes through. Today a caller can build one with + `Plan::fields()`. +- **Equality term**: a value stored next to a ciphertext that lets a query + find an exact match without decrypting. +- **Zeroize**: overwrite memory that held plaintext, so the plaintext does + not stay in memory after use. `Zeroize` is the Rust trait that does this. + +## Decisions already made, and what these examples propose + +The first parts of this section record decisions made on 25 October 2026. +The table after them lists every API surface in the examples and its status. + +### Ownership decision (25 October 2026) + +`encrypt` and `encrypt_into` take ownership of (consume) their inputs. This +includes the variants that take a context. Their implementations must guard +the owned plaintext and temporary buffers, so they are cleaned up on +success, on error, and when the request is dropped. Passthrough fields are +deliberately moved into the output. + +Other parts of this decision: + +- The capabilities for a single ciphertext are named `OpaqueEncode` and + `OpaqueDecode`. +- Plaintext types must implement `Zeroize`. No new cleanup trait is planned. +- `Serialize` alone does not establish the cleanup contract for + `OpaqueBlob`. + +The candidate programs below were written before the `Zeroize` requirement. +When making them executable, update their source derives, and remove any +reuse of input that has been consumed. + +### Decrypt decision (25 October 2026) + +Stack Encrypt keeps two ways to decrypt: + +- the direct `keyset.decrypt` and `decrypt_with_context`, for an opaque + ciphertext; +- the destination-directed `T::decrypt_from` and + `decrypt_from_with_context`, for a declared target. + +A successful call returns an ordinary owned `T: Zeroize`. The caller is then +responsible for cleaning it up. Stack Encrypt guards intermediate plaintext. + +The API must support results that are wiped when they are dropped, if the +caller asks for them, without forcing a wrapper on every caller. `Zeroize` +alone does not wipe a returned value when it is dropped. These are accepted +API contracts. This document does not claim that the prototype already +implements them. + +### Collection decision (25 October 2026) + +The direct `keyset.encrypt(value)` takes one single-value input and produces +one `Ciphertext`. It does not encrypt a collection element by element. + +To encrypt each element of a collection in one batch, use `EncryptInto`, +for example: `let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. -`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` instead produces -one ciphertext for the encoded collection. The latest recommendation -applies the same rule to byte vectors, with no wrapper or special case -needed to preserve prototype behavior. These calls require the proposed -opaque ciphertext and no-context target support. + +`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` produces one +ciphertext for the encoded collection instead. + +The latest recommendation applies the same rule to byte vectors. No wrapper +or special case is needed to keep the prototype's behaviour. These calls +need the proposed opaque ciphertext and support for targets that take no +context. + +### Status of each API surface | Surface | Status in this review | | --- | --- | @@ -68,66 +147,102 @@ opaque ciphertext and no-context target support. | `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | | `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | | `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | -| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64/Base85 storage compose with encrypt/decrypt targets; final names remain for the naming pass | -| `Ciphertext` as an encryption/decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | +| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64 or Base85 storage compose with encrypt and decrypt targets; final names remain for the naming pass | +| `Ciphertext` as an encryption or decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | | Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | | Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair, symmetric with direct encryption | | `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair on `DecryptFrom` | | `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | -The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or -passthrough APIs. They are complete candidate programs, not programs claimed -to compile against today's crate. The names and contracts above remain -subject to review where marked proposed or provisional. +The first three Rust programs use the proposed `OpaqueBlob`, `Ciphertext`, +and passthrough APIs, or some of them. Each is a complete candidate program. +None is claimed to compile against today's crate. The names and contracts in +the table can still change where they are marked proposed or provisional. -The proposed opaque ciphertext is intentionally untyped: the decrypt call's -expected type selects decoding. A typed wrapper remains an open alternative. -Its byte decoder parses a stored envelope; authentication happens on decrypt. -Context and leaf encodings must agree across the new Rust and dynamic -implementations; preserving the prototype's encoding is not required. +### Notes on the proposed types -The trait name `DecryptFrom` is somewhat surprising: it supplies the +The proposed opaque ciphertext deliberately carries no plaintext type. The +type that the decrypt call expects selects how the plaintext is decoded. A +typed wrapper is still an open alternative. + +The ciphertext's byte decoder +only parses a stored envelope. Authentication happens when it is decrypted. +The new Rust implementation and the dynamic implementation must agree on +the context and leaf encodings. They do not need to keep the prototype's +encoding. + +The trait name `DecryptFrom` may surprise readers. The trait supplies the convenience functions, while `#[derive(DecryptInto)]` describes how an -encrypted output recovers plaintext. These examples keep that existing -distinction visible rather than silently redesigning it. +encrypted output recovers its plaintext. These examples keep that existing +split visible, rather than redesigning it without saying so. + +### What the examples leave out, and why + +The examples deliberately keep plans and operation descriptions out of the +application code. The application code shows only the public calls. + +Each main example is a separate program, with all imports and types shown. +All of them use the same user fields and made-up data. `created_at` is an +`i64` Unix timestamp in seconds, so that configuring a date library does not +distract from the encryption API. + +Files stand in for a database. The programs do encode, write, read, and +decode their stored forms. The files are written to a temporary directory. +The directory is removed when the program exits. ## Setup shared by the programs -The proposed programs use `stack-encrypt` with its default `http` feature, -`serde` with `derive`, `serde_json`, `tempfile`, and `tokio` with `macros` and -`rt-multi-thread`. No published version currently supplies all proposed APIs, -so this draft does not give a misleading installable Cargo manifest. +The proposed programs use these crates: + +- `stack-encrypt`, with its default `http` feature; +- `serde`, with `derive`; +- `serde_json`; +- `tempfile`; +- `tokio`, with `macros` and `rt-multi-thread`. + +No published version of `stack-encrypt` supplies all the proposed APIs yet. +This draft therefore gives no Cargo manifest, because one would suggest the +programs can be installed and run. + +`StackCipher::new()` uses the configured ZeroKMS credentials. Those come +from a CLI login, or from these environment variables: `CS_CLIENT_ID`, +`CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. -`StackCipher::new()` uses the configured ZeroKMS credentials: a CLI login or -`CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. -The second client in each program represents a later reader using the same -authorized configuration. The default keyset is selected for brevity; an -already-selected tenant keyset can be used in its place. +The second client in each program stands for a later reader that uses the +same authorised configuration. The examples select the default keyset to +keep them short. A tenant keyset that the application has already selected +can be used in its place. -Decryption uses the client because the sealed value carries its keyset id. -It still requires access to the corresponding keys and the correct context. -The examples do not depend on in-memory fake keys surviving a reload. +Decryption uses the client, because the sealed value records its keyset id. +The reader still needs access to the matching keys and the correct context. +The examples do not rely on in-memory fake keys surviving a reload. ## 1. Encrypt the whole user through an explicit blob wrapper -**Intent:** store one encrypted document. All four fields, including the ID -and creation time, are inside one encrypted serialization. There are no -independently readable fields or search terms in the stored ciphertext. - -`OpaqueBlob::new(user)` explicitly requests that treatment. Stack owns the -serialization and matching deserialization, rather than requiring every -application to select and invoke a codec. The proposed wrapper owns the input; -`new` only wraps it, and serialization errors surface from the encryption -operation. The context identifies the application's document schema and binds -this document to an externally known record id. - -For this draft the wrapper accepts `Serialize + Zeroize` on write and -`DeserializeOwned + Zeroize` on read. This does -not claim that arbitrary Serde implementations can satisfy a constant-time -contract. The initial wrapper has one library-selected, versioned binary -codec, not a caller-selected codec parameter. Selecting and specifying that -codec remains implementation/design work; no particular binary format is -claimed to be implemented or constant-time here. +**What this example shows:** how to store one encrypted document. All four +fields, including the id and creation time, are inside one encrypted +serialization. The stored ciphertext contains no fields that can be read +separately, and no search terms. + +`OpaqueBlob::new(user)` explicitly asks for that treatment. Stack Encrypt +owns the serialization and the matching deserialization, so each +application does not have to choose and call a codec. The proposed wrapper +takes ownership of the input. `new` only wraps the value. The +encryption call returns any serialization error. + +The context names the application's document schema +(`"user-document-v1"`). The context also binds this document to a record id +that the application knows from outside the ciphertext. + +In this draft, the wrapper accepts `Serialize + Zeroize` on write and +`DeserializeOwned + Zeroize` on read. This does not claim that every Serde +implementation can meet a constant-time contract. + +The first version of the +wrapper has one versioned binary codec that the library chooses. The caller +cannot choose a codec. Choosing and specifying that codec is still design +and implementation work. This document does not claim that any binary format +is implemented, or that any is constant-time. ```rust use serde::{Deserialize, Serialize}; @@ -187,73 +302,115 @@ async fn main() -> Result<(), Box> { } ``` -**What the API must guarantee:** the serialized bytes are one encrypted leaf -under one data key. The envelope contains the retrieval metadata, so the -caller never needs to save an IV or key tag separately. A different context -must fail to open it. Stack owns the matching encode/decode pair and its -compatibility contract. A blob-format discriminator and version must be -authenticated, for example inside the encrypted payload, so decoding does -not rely on guessing a codec or on unauthenticated format metadata. This -can reuse the canonical outer ciphertext envelope; the blob payload is a -new, explicit format rather than silently reinterpreting previously encrypted -raw bytes. Application schema evolution remains explicit: a codec version -does not automatically migrate changes to `User`. - -**Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` -form. A matching no-context `reader.decrypt(loaded)` is accepted for symmetry; -it is not the signature of today's two-argument decrypt method. This document -uses explicit context in the full example so the saved format is unambiguous. - -**Meaning of the wrapper:** `OpaqueBlob` is the plaintext-side request -and the decoded value, not the stored ciphertext type. `Ciphertext` remains -the persisted envelope. `into_inner()` returns an ordinary `User`. The -wrapper does not by itself promise to zeroize arbitrary input/returned types; -the codec should keep temporary plaintext buffers in wipeable storage. -`#[derive(Encrypt)]` on `User` currently means a structural ciphertext tree -and is not a substitute for wrapping the value as one blob. +### What the API must guarantee for example 1 + +- The serialized bytes are one encrypted leaf under one data key. +- The envelope contains the metadata needed to fetch the key again. The + caller never needs to save an initialisation vector (IV) or a key tag + separately. +- A different context must fail to open the ciphertext. +- Stack Encrypt owns the matching encode and decode pair, and the contract + for keeping them compatible. +- The blob format's identifier and version must be authenticated, for + example by putting them inside the encrypted payload. Decoding then does + not depend on guessing a codec, or on format metadata that is not + authenticated. +- The blob can reuse the canonical outer ciphertext envelope. The blob + payload is a new, explicit format. It must not quietly reinterpret raw + bytes that were encrypted before. +- Changes to the application's schema still need explicit handling. A codec + version does not migrate changes to `User` automatically. + +### Encrypting without an application context + +Without an application context, use the form +`keyset.encrypt(OpaqueBlob::new(user))`. A matching no-context +`reader.decrypt(loaded)` is accepted, so that encrypt and decrypt match. It +is not the signature of today's decrypt method. Today's method takes two +arguments. +This document uses an explicit context in the full example so that the +saved format is unambiguous. + +### What the wrapper means + +`OpaqueBlob` is the request on the plaintext side, and the decoded +value. It is not the stored ciphertext type. `Ciphertext` is still the +stored envelope. `into_inner()` returns an ordinary `User`. + +The wrapper does not by itself promise to zeroize arbitrary input or +returned types. The codec should keep temporary plaintext buffers in memory +that can be wiped. + +`#[derive(Encrypt)]` on `User` currently means a structural tree of +ciphertexts, one for each field. That derive cannot replace wrapping the +value as one blob. ### Timing contract to investigate -The desired property is serialization/deserialization whose timing does not -depend on secret contents, under an explicit policy for lengths and shape. -Constant time does not mean identical elapsed time for differently sized -messages; what may vary must be declared public. The general requirement is -secret-independent control flow and memory access, including outside the -cryptographic primitive. See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). - -A library-owned binary codec gives us control over that implementation, but -does not establish the property by itself. Strings, collection lengths, -enum variants, validation errors, allocations, and user-defined Serde code -can all introduce value-dependent work. Supporting arbitrary `Serialize` / -`Deserialize` implementations cannot guarantee constant-time behavior for -the whole operation. - -Before claiming such a guarantee, define supported types, permitted public -shape/length information, and restrictions on custom implementations. A -stricter profile may need bounded shapes, fixed-width encodings, padding, -and controlled encoders/decoders, with implementation-level verification. -Padding ciphertext to a fixed size alone does not make the preceding -serialization constant-time. Authentication must complete before decoding -the plaintext blob. This is a design requirement to investigate, not a -guarantee made by these example programs or a constant-time audit result. - -**Investigation candidate: `serdect`.** It is already a dependency of -`stack-encrypt`. Review its supported encodings and precise constant-time -guarantees, including decoding/error behavior and length assumptions. Assess -where it can help with ciphertext/byte serialization and whether any of its -techniques apply to the `OpaqueBlob` codec. Do not assume guarantees for byte -encoding extend to serializing arbitrary objects through Serde. This is a -follow-up investigation, not a completed assessment or codec selection. +The goal is serialization whose timing does not depend on the secret +contents, in both directions. An explicit policy sets what may vary in +lengths and shape. + +Constant time does not mean that messages of different sizes take the same +elapsed time. The policy must declare which properties may vary, and treat +them as public. In general, neither control flow nor memory access may +depend on secrets. That applies outside the cryptographic primitive too. +See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). + +A binary codec that the library owns gives us control over the +implementation, but does not give the property by itself. Each of these can +add work that depends on the values: + +- strings; +- collection lengths; +- enum variants; +- validation errors; +- memory allocations; +- Serde code that users write. + +Supporting arbitrary `Serialize` and `Deserialize` implementations cannot +guarantee constant-time behaviour for the whole operation. + +Before claiming such a guarantee, define these things: + +- the supported types; +- which shape and length information may be public; +- the restrictions on custom implementations. + +A stricter profile may need bounded shapes, fixed-width encodings, padding, +and controlled encoders and decoders, with verification at the +implementation level. Padding the ciphertext to a fixed size does not by +itself make the serialization before it constant-time. Authentication must +finish before the plaintext blob is decoded. + +This section states a design requirement to investigate. These example +programs make no such guarantee. This section is not the result of a +constant-time audit. + +**Candidate to investigate: `serdect`.** `serdect` is already a dependency +of `stack-encrypt`. The investigation needs to: + +- review its supported encodings and its exact constant-time guarantees, + including its timing when decoding and on errors, and its assumptions + about lengths; +- assess where it can help serialize ciphertexts and bytes; +- assess whether any of its techniques apply to the `OpaqueBlob` codec. + +Do not assume that its guarantees for encoding bytes extend to serializing +arbitrary objects through Serde. This is a follow-up investigation. It is +not a finished assessment. It does not select a codec. ## 2. Encrypt selected fields into a chosen output layout -**Intent:** store an ordinary record with plaintext `id` and `created_at`, -and independently encrypted `name` and `email`. The output type states the -layout. The two encrypted fields need no search indexes. +**What this example shows:** how to store a record in which `id` and +`created_at` stay as plaintext. `name` and `email` are each encrypted +separately. The output type declares the layout. The two +encrypted fields need no search indexes. -The proposed ciphertext Serde implementation makes persistence compositional: -ordinary Serde derives on the output encode plaintext fields normally and -ciphertext fields as Base64 envelopes. There is no manual `Node` traversal. +The proposed Serde implementation for `Ciphertext` lets persistence be +built from ordinary parts. Ordinary Serde derives on the output type encode +the plaintext fields as usual, and encode the ciphertext fields as Base64 +envelopes. The application does not walk any `Node` values by hand. ```rust use serde::{Deserialize, Serialize}; @@ -311,72 +468,95 @@ async fn main() -> Result<(), Box> { } ``` -**Context and execution:** `name` uses the structured context -`("users", "name")`, and `email` uses `("users", "email")`. The keyset is -shared, but each encrypted field receives its own data key. One conversion -batches those key requests. The decrypt derive reconstructs `User`, opening -the ciphertext fields and copying the passthrough fields back. - -**Plaintext semantics:** copying `id` does not automatically bind the sealed -fields to it. Both plaintext fields remain unencrypted and unauthenticated. -That is the requested passthrough behavior, not a promise of whole-record -integrity. The example's name/email contexts distinguish columns, not rows. - -If the application wants row binding, it can use the existing -`user.encrypt_into_with_context(&keyset, expected_id)` entry point and the -matching `User::decrypt_from_with_context(loaded, &reader, expected_id)`. -The effective contexts then become `(("users", "name"), expected_id)` and +### Context and execution in example 2 + +`name` uses the structured context `("users", "name")`, and `email` uses +`("users", "email")`. The two fields share the keyset, but each encrypted +field gets its own data key. One conversion sends the key requests for both +fields in one batch. The decrypt derive rebuilds `User`: it opens the +ciphertext fields and copies the passthrough fields back. + +### What passthrough fields do and do not protect + +Copying `id` does not bind the sealed fields to it. Both plaintext fields +stay unencrypted and unauthenticated. That is the passthrough behaviour that +was asked for. It does not promise integrity for the whole record. The +example's `name` and `email` contexts tell columns apart, not rows. + +If the application wants each ciphertext bound to its row, it can use the +existing entry point `user.encrypt_into_with_context(&keyset, expected_id)` +and the matching +`User::decrypt_from_with_context(loaded, &reader, expected_id)`. The +contexts that take effect are then `(("users", "name"), expected_id)` and `(("users", "email"), expected_id)`. The reader supplies the expected id -from its record lookup. That is an explicit variation, not an automatic -effect of the passthrough attribute. - -**Existing `context_field` support:** the derive does have -`#[stash(context_field)]`, but its current behavior is to store the -caller-supplied context in an output field and use it again when decrypting. -It does not read an id from the plaintext `User`. It also supplies the complete -context: the macro rejects combining it with a literal `context = "users"`. -Because the current `struct = User` form requires that literal, the attribute -cannot currently be used on example 2's field-by-field declaration. - -**Separate source from storage:** obtaining context from plaintext fields -and carrying context into the encrypted output are independent choices. -An extended derive could read `User.id`, combine it with the declared field -contexts, and also copy it into `EncryptedUser.id`. That would enable row -binding without passing the id twice. The same model should accommodate -several source fields, such as `(tenant_id, id)`, with explicit ordering and -grouping. - -Alternatively, context can be passed as an argument and stored, passed and -not stored, or derived from plaintext fields without being stored. In either -non-stored case, the reader needs equivalent context from elsewhere before -decrypting. Source selection and storage should be separately expressible; -whether one annotation can conveniently request both remains a syntax choice. -Today's `context_field` only covers the caller-supplied-and-stored case. - -For the source-derived-and-stored case, a reader could use the stored id, -with an optional independently supplied expected id to reject a whole record -moved to another destination. Changing the stored id alone must fail -decryption. The values needed to reconstruct context must be available -without first opening the ciphertext that requires them. - -**Review point:** the declaration is concise, but requires an explicit -`EncryptedUser` type. A future source-side derive could generate that type; -it should generate this same layout and context behavior. Also decide whether -`User::decrypt_from(...)` is the desired public decrypt spelling before -adding a competing no-context `loaded.decrypt_into(...)` convenience. +from its record lookup. This is an explicit variation. The passthrough +attribute does not cause it. + +### Existing `context_field` support + +The derive already has `#[stash(context_field)]`. Its current behaviour is +to store the context the caller supplied in an output field, and to use it +again when decrypting. It does not read an id from the plaintext `User`. + +`context_field` also supplies the complete context, so the macro refuses to +combine it with a literal `context = "users"`. The current `struct = User` +form requires that literal. As a result, the attribute cannot be used today +in example 2's field-by-field declaration. + +### Where context comes from, compared with where it is stored + +Taking context from plaintext fields and carrying context into the encrypted +output are separate choices. An extended derive could read `User.id`, +combine it with the field contexts declared on the type, and also copy it +into `EncryptedUser.id`. That would bind each ciphertext to its row without +passing the id twice. The same model should handle several source fields, +such as `(tenant_id, id)`, with explicit ordering and grouping. + +There are other combinations. Context can be: + +- passed as an argument and stored; +- passed as an argument and not stored; +- taken from plaintext fields and not stored. + +In either case where the context is not stored, the reader needs the same +context from somewhere else before decrypting. The API should let callers +express the source and the storage separately. Whether one annotation can +ask for both is still a syntax choice. Today's `context_field` only covers +the case where the caller supplies the context and it is stored. + +Suppose the context is taken from plaintext fields and also stored. A +reader could then use the stored id. To reject a whole record that was moved +to another place, the reader could also supply an expected id from +elsewhere. Changing the stored id alone must make decryption fail. The values needed to rebuild the +context must be available without first opening the ciphertext that +requires them. + +### Review points for example 2 + +The declaration is short, but it requires an explicit `EncryptedUser` type. +A future derive on the source type could generate that type. If it does, it +should generate this same layout and the same context behaviour. + +Reviewers also need to decide whether `User::decrypt_from(...)` is the +public spelling for decryption that we want. That decision comes before +adding a competing no-context convenience, `loaded.decrypt_into(...)`. ## 3. Use the same layout model with a searchable email target -**Intent:** keep the same plaintext fields, encrypt the name normally, and -store the email's ciphertext plus an equality term. Search terms come from -the original typed plaintext through their declared operation, not from the -JSON encoding used to persist the output. +**What this example shows:** the same plaintext fields as example 2, with +the name encrypted as before. The email is stored as its ciphertext plus an +equality term. Search terms are made from the original typed plaintext, +through the operation the target declares. They are not made from the JSON +encoding used to store the output. -`SearchableEmail` is an application-defined scalar target. It uses the caller -context supplied by the outer record: `("users", "email")`. It does not -declare another fixed context or append another `email` component. The -`ciphertext` and `equality` member names are output layout, not additional -cryptographic context segments. +`SearchableEmail` is a scalar target that the application defines. It uses +the context that the outer record supplies: `("users", "email")`. It does +not declare a fixed context of its own. It does not add another `email` part +to the context. + +The member names `ciphertext` and `equality` only describe the output +layout. They do not add parts to the cryptographic +context. ```rust use serde::{Deserialize, Serialize}; @@ -465,56 +645,93 @@ async fn main() -> Result<(), Box> { } ``` -The filter is an in-memory scan over persisted encrypted records, not a -claim of database index use. It demonstrates the entire query contract: -derive the term, compare it with the stored term, and open the matched -ciphertext. An equality match is not authentication of the stored record; -the decrypt operation authenticates its encrypted fields. Passthrough -semantics are the same as in example 2. - -Collection conversion is already supported by Stack's blanket -`EncryptFrom> for Vec` implementation when `T: EncryptFrom` -and its context supports cloning. It collects every row's pending operations -before dispatch: these two users require four data keys (name and email for -each), issued through one batched `generate_keys` call. Equality terms use -the current local index-key backend. This count concerns encryption work, -not client initialization or authentication. - -One logical `generate_keys` call is not an unlimited single HTTP request: -the KMS client chunks batches according to its configured key-count limit. -This four-key example fits in one normal request; larger collections may -span several requests without changing the public call. - -The matching collection decrypt implementation batches retrieval across -rows, with one `retrieve_keys` call per keyset represented in the ciphertext. -An empty collection requires no data-key call. Collection order is preserved -by the conversion; no new whole-collection authentication is implied. - -This target-directed vector is a vector of independent output records. It -is distinct from encrypting a plaintext vector into a single -`StackCipherText` sequence tree, and from wrapping the vector in `OpaqueBlob` -to encode/encrypt the whole collection as one value. Batching is a property -of execution, not a change to the chosen layout. - -This example uses the common field context for all rows so one query term -can search them. Adding a row id to that context would also scope the term -to that row. Separating ciphertext row binding from column-wide search -context is a further design question, not something these examples silently -promise. Likewise, applications querying across keysets need the appropriate -term per keyset; no cross-keyset comparison is implied. +### What the query in example 3 shows + +The filter scans the stored encrypted records in memory. It does not claim +that a database index is used. The filter shows the whole query contract: + +1. Make the query term. +2. Compare it with the stored term. +3. Open the matching ciphertext. + +An equality match does not authenticate the stored record. The decrypt +operation authenticates the record's encrypted fields. Passthrough fields +work as they do in example 2. + +### How collection conversion batches key requests + +Stack Encrypt already supports collection conversion, through its blanket +implementation `EncryptFrom> for Vec` when `T: EncryptFrom` and +the target's context can be cloned. + +The conversion collects every row's +pending operations before sending any of them. The two users need four data +keys, one each for name and email in each row. All four are issued through +one batched `generate_keys` call. Equality terms use the current local +backend for index keys. + +This count is about the encryption work only. It +does not include setting up or authenticating the client. + +One logical `generate_keys` call is not always a single HTTP request. The +KMS (key management service) client splits a batch into chunks, according +to its configured limit on the number of keys. This example's four keys fit +in one normal request. A larger collection may need several requests, with +no change to the public call. + +The matching collection decrypt implementation also batches key retrieval +across rows. It makes one `retrieve_keys` call for each keyset found in the +ciphertexts. An empty collection needs no data-key call. The conversion +keeps the collection's order. It does not add any authentication of the +collection as a whole. + +### Three different ways to encrypt a vector + +This target-directed vector is a vector of separate output records. It is +different from both of these: + +- encrypting a plaintext vector into a single `StackCipherText` sequence + tree; +- wrapping the vector in `OpaqueBlob`, to encode and encrypt the whole + collection as one value. + +Batching is a property of how the work is run. It does not change the +layout that was chosen. + +### Limits of the search context + +This example uses the same field context for every row, so that one query +term can search all of them. Adding a row id to that context would also +limit the term to that row. How to bind each ciphertext to its row while +keeping a search context that covers the whole column is still an open +design question. These examples do not promise it. + +In the same way, an application that queries across keysets needs a +separate query term for each keyset. The examples do not imply any +comparison across keysets. ### EQL targets: a concrete adapter boundary -The example above is a generic searchable layout, **not an EQL payload**. -For EQL, use the adapter's types, metadata, serialization, and query target. -For example, today's `eql_bindings::v4::text::TextEq` already represents an -email ciphertext plus equality term; `TextEqQuery` represents the query. +The example above is a generic searchable layout. It is **not an EQL +payload**. For EQL, use the adapter's types, metadata, serialization, and +query target. Here, the adapter is the `eql-bindings` crate. The crate +converts between Stack Encrypt values and the stored EQL format. + +For example, today's `eql_bindings::v4::text::TextEq` already contains an email +ciphertext plus an equality term. `TextEqQuery` is the matching query type. + +The program below is a separate example for one field. It uses existing APIs +and avoids the proposed `Ciphertext` and passthrough attributes. It needs: + +- `eql-bindings`, with its `stack-encrypt` feature; +- `stack-encrypt`; +- `serde_json`; +- `tempfile`; +- `tokio`. -The following separate field-level program uses existing APIs and avoids -the proposed `Ciphertext` and passthrough attributes. It needs `eql-bindings` -with its `stack-encrypt` feature, `stack-encrypt`, `serde_json`, `tempfile`, -and `tokio`. It is marked `no_run` because it needs live credentials; it has -been checked against source signatures here, not compiled or executed. +The program is marked `no_run` because it needs live credentials. The program +has been checked against the source signatures. It has not been compiled or +run. ```rust use eql_bindings::{ @@ -554,56 +771,76 @@ async fn main() -> Result<(), Box> { } ``` -`TextEq` includes its EQL version, identifier, ciphertext, and equality term. -Its context is `NonEmpty`. The typed identifier comes from the -application's expected table/column on both writes and reads; decryption -checks it against the stored identifier before opening the ciphertext. -The `v4` namespace is intentional: this Rust producer profile targets EQL -v4, not the TypeScript EQL v3 integrations. - -There is a gap before we can simply replace `SearchableEmail` with `TextEq` -in the outer derive. That derive currently passes `CallerContext`; EQL's -target needs `NonEmpty` and carries its identifier in the output. -The context adaptation must be designed explicitly, including whether caller -extensions are representable. The EQL example demonstrates what that adapter -must preserve without inventing an implicit conversion or a nonexistent SQL -client API. PostgreSQL DDL, parameter binding, and index engagement are a -separate adapter example to add once that boundary is settled. +`TextEq` includes its EQL version, its identifier, the ciphertext, and the +equality term. Its context type is `NonEmpty`. The application +supplies the typed identifier for the expected table and column, on both +writes and reads. Decryption checks it against the stored identifier before +opening the ciphertext. + +The `v4` namespace is deliberate: this Rust producer profile targets EQL v4. +The TypeScript integrations use EQL v3. + +### Gap before `TextEq` can replace `SearchableEmail` + +There is a gap before `TextEq` can replace `SearchableEmail` in the +outer derive. That derive currently passes a `CallerContext`. The EQL target +needs a `NonEmpty`, and stores its identifier in the output. + +The conversion between the two contexts must be designed explicitly. The +design includes whether extensions that the caller adds to the context can +be represented. The EQL example shows what that adapter must keep. It does +not invent an implicit conversion, or an SQL client API that does not exist. + +PostgreSQL DDL (the statements that create tables and indexes), parameter +binding, and whether queries use the index belong in a separate adapter +example. That example is to be added once the boundary is settled. ## What to review across the three workflows +Each row of this table is one design question. The second column gives the +answer these examples chose. + | Question | Concrete choice made by these examples | | --- | --- | | What is one encrypted unit? | One `OpaqueBlob` encoding in example 1; one selected field in examples 2 and 3 | | How is the layout selected? | Ordinary encrypt returns an opaque value; `EncryptInto` selects a declared output | | How is persistence exposed? | Canonical bytes for one ciphertext; Serde composed over a typed layout | -| How is plaintext recovered? | Decrypt/decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | -| How do indexes fit? | Additional field targets under the same layout/execution model | +| How is plaintext recovered? | Decrypt and decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | +| How do indexes fit? | Additional field targets under the same layout and execution model | | Who chooses context? | The application for a blob; the layout plus optional caller extensions for fields; an explicit identifier for EQL | | Do examples need plans or visitors? | No; those remain implementation and advanced composition mechanisms | | Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | -The principal unresolved choices exposed by the examples are the opaque -ciphertext name and storage contract, the blob -codec and timing contract, EQL context adaptation, and optional source-side output -generation. None requires reintroducing public `encrypt_as`. +### Main open questions + +The examples leave these questions open: + +- What is the opaque ciphertext called, and what is its storage contract? +- What is the blob codec, and what is its timing contract? +- How is an EQL context adapted? +- Does a derive on the source type generate the output type? + +None of them requires bringing back a public `encrypt_as`. ## Source checks and validation limits -The draft was checked against the existing -[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), -[context composition](../../packages/stack-encrypt/src/target/context.rs), -[derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs), -[decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs), -[canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs), and -[EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). - -All four programs were parsed successfully by `rustfmt` using Rust edition -2024. All local links resolve; code fences and whitespace were checked. -These checks establish syntax and document integrity, not type correctness. - -The candidate APIs are not implemented by this document. The three proposed -programs cannot yet be typechecked against the repository. No live encryption, -database execution, or cryptographic compatibility tests are claimed. Once the surface is agreed -and implemented, promote these programs to executable examples and verify -storage round trips, matching queries, and context mismatches. +The draft was checked against the existing code: + +- [target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs); +- [context composition](../../packages/stack-encrypt/src/target/context.rs); +- [derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs); +- [decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs); +- [canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs); +- [EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). + +`rustfmt` parsed all four programs without errors, using Rust edition 2024. +All local links resolve. The code fences and whitespace were checked. These +checks confirm the syntax and the document's integrity, not that the types +are correct. + +This document does not implement the candidate APIs. The three proposed +programs cannot yet be type-checked against the repository. No live +encryption, database runs, or cryptographic compatibility tests are claimed. +Once the API surface is agreed and implemented, turn these programs into +executable examples. Then verify the storage round trips, the matching +queries, and that a wrong context fails. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index dc6d2e7e0..20cd7a865 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -1,117 +1,178 @@ # JSON tree: operation inventory and release contract -Updated: 2026-10-27 against merged `next` -`ffeed7adb3f7d8739cda53261046147852bffc82`. Companion to the +Updated 27 October 2026 against merged `next` +`ffeed7adb3f7d8739cda53261046147852bffc82`. This document accompanies the [API plan](2026-10-24-stack-encrypt-api-plan.md). -The Go JSON feature is implemented through #1372/#1373/#1378/#1402. This -inventory distinguishes current support from remaining requirements. The -public Rust derive/facade example below remains proposed, not executable -code for the current engine. The +## Summary + +The Go JSON feature is implemented, through #1372, #1373, #1378, and #1402. +This document lists every JSON operation the release needs, with what is +supported now and what work remains. Each operation has an ID from J01 to +J20, used in the table of operations below. Two operations still have +gaps: J14 and J17. The SQL types and functions come from Encrypt Query +Language (EQL). + +The Rust example in this document uses the proposed public derive and +facade. That example is not executable code for the current engine. The [refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) -and its engine/Go findings document the exact source evidence. +and its findings on the engine and Go give the exact source evidence. -## Current names and policy +## Terms used in this document -| Role | Current public Go / dynamic / SQL name | +| Term | Meaning | +| --- | --- | +| EQL | The SQL in `packages/eql` that stores and queries encrypted values | +| Entry | The encrypted record for one node of a document. A document is stored as one entry for each object, array, and scalar in it | +| Selector | A keyed pseudorandom value derived from a node's path, with every array position replaced by the wildcard. Every element of an array shares one selector | +| Token | A value derived from a node's concrete path. The database uses it to find that node and the entries beneath it | +| Needle | The JSON value that a containment query looks for | +| EqAt | Equality of a scalar at a path: `EqAt` in Go, `eq_at` in the dynamic engine | +| Containment projection | The set of selector and value pairs, one for each node, stored beside the entries. PostgreSQL's `jsonb @>` compares two of these sets to test containment | +| Subtree read | A read that returns a node and every entry beneath it, through `eql_v4.json_tree_subtree`, and opens them as one value | +| Cascaded tags | Each container's authentication tag covers the tags of its children, so a change anywhere below a node stops that node opening. Added by #1379 | +| Sealed root | The default. The root entry's ciphertext is the whole document, and the root token finds no other entry | +| A JSON document whose root is not sealed | The root is stored as an ordinary container entry (`sealed_root` false). Called "open root" in the code | +| Walk budget | The most entries a subtree read returns, counted across all its tokens, before the database falls back to the root | +| Root fallback | Past the walk budget, the database returns the sealed root alone. The client opens the whole document and picks out the requested values | +| Wildcard bound | How many elements each `[*]` in a read path expands to. One extra probe at the bound detects a longer array | +| Reads that Rust verifies against the original request | Prepared reads: Rust derives the request again when the rows are opened, and checks every row against it | +| Post-cascade profile | The storage and search format as it is after the cascaded tags change in #1379 | +| Facade | The proposed public Rust API of `stack-encrypt` | +| AAD | Additional authenticated data (AAD): data that an authentication tag covers but the ciphertext does not contain | + +## Current names + +| Role | Current public name in Go, the dynamic engine, and SQL | | --- | --- | | Configured target | `JsonSearch` | | Encrypted document | `eql.JSONSearch`, `public.eql_v4_json_search` | -| Containment / EqAt operand | `eql.JSONSearchQuery`, `eql_v4.query_json` | -| Ordering operand | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | +| Operand for containment and EqAt | `eql.JSONSearchQuery`, `eql_v4.query_json` | +| Operand for ordering | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | | Encrypted entry | `eql.JSONEntry`, `public.eql_v4_json_entry` | -| Prepared selector / subtree | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| Prepared selector and prepared subtree read | `encrypt.JSONSelection`, `encrypt.JSONRead` | | SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | -| Singular plaintext result | `JSONValue{Found, JSON}` | - -Handwritten Rust EQL adapters retain their `JsonTree*` names; the internal -engine `JsonDocument` is not the Go ciphertext type. `eql.JSON` is the -storage-only type. Do not use the old `query_json_tree` / `eql_v4_json_tree` -domains in current recipes. - -A field's configured target options are resolved into native `JsonOptions` -and reused for writes, queries and reads. The shipped profile uses column -identity and keyset selection. Separate row binding and search scope remain -accepted future API work, not supported EQL context inputs. Reject those -unsupported declarations; do not append row IDs to the existing shared -context. Queries do not gain row isolation merely because their returned -ciphertexts authenticate. - -## Required inventory and current support - -| ID | Operation | Current behavior / remaining work | -| --- | --- | --- | -| J01 | Encrypt object/array/scalar/null | Implemented; raw JSON supports every root, typed adapters support declared structs/maps/slices | -| J02 | Decrypt complete document | Implemented; raw JSON text or declared Go type | -| J03 | Batch write/read | Implemented; shared executor and provider chunking | -| J04 | Contains | Implemented `Contains`, `contains`, `@>` with `query_json`; generated needle guards apply | -| J05 | Contained by | Implemented `ContainedBy`, `contained_by`, `<@` | -| J06 | Scalar equality at path | Implemented `EqAt`, `eq_at`; value selectors, not ordering equality | -| J07 | Four ordering predicates | Implemented runtime paths and `At.` accessors; ordered number/string paths only | -| J08 | Inclusive range | Implemented `Between`, `between_at`, named lower/upper operands | -| J09 | Sort | Selector plus explicit SQL; wildcard sort needs defined reduction, not an assumed first array element | -| J10 | Path exists | Selector plus `json_tree_path_exists` | -| J11 | Extract one match | Selector plus `->` / path-query-first; wildcard choice is unspecified | -| J12 | Extract as encrypted text | `->>`; still encrypted entry JSON, not plaintext | -| J13 | Enumerate all matches | SQL path query exists; bounded wildcard subtree is the supported grouped plaintext route | -| J14 | Open extracted scalars | `JSONSelection.Open` handles one optional entry per document group. Multi-entry selector groups are refused; do not label them supported | -| J15 | Prepare positional/wildcard read | Implemented `Subtree`, `subtree`; token list plus retained trusted request | -| J16 | Fetch/open subtree | `json_tree_subtree(document, tokens, budget)`, `JSONRead.Open` / `OpenOne` | -| J17 | Immediate children | `Children` prepares a concrete token; SQL returns child entries. Parent-bound child-scalar opening remains unimplemented | -| J18 | Count children | `ChildCount` token and SQL nullable count; not authenticated completeness | -| J19 | Budget/fallback | Implemented, with original requested selection applied after root fallback | -| J20 | Stored-value comparisons | Existing document containment and entry ordering, within compatible column/keyset/profile | - -The comprehensive release requirement remains. Do not equate closed tracking -issues with implementation of a parent-bound child opener, or feed child -rows into an unrelated selector/subtree request. Complete that opener for -the J17 plaintext-child requirement; until then, fetch and open the parent's -subtree to inspect its children, documenting the different transfer/leakage -cost. SQL all-match enumeration is distinct from the supported bounded -subtree plaintext interface. +| Result of opening one value | `JSONValue{Found, JSON}` | + +The handwritten Rust EQL adapters keep their `JsonTree*` names. The +internal engine type `JsonDocument` is not the Go ciphertext type. +`eql.JSON` is the type for storage only. Do not use the old domains +`query_json_tree` and `eql_v4_json_tree` in current recipes. + +## Context and scope policy + +The SDK resolves the target options configured for a field into native +`JsonOptions`. The SDK uses the same options for writes, queries, and +reads. The released profile uses column identity and keyset selection. + +Separate row binding and search scope are accepted future API work. Row +binding and search scope are not supported as EQL context inputs today. +So: + +- reject a declaration that asks for them; +- do not add row IDs to the existing shared context. + +A query does not isolate rows only because the ciphertexts it returns +authenticate. + +## Required operations and current support + +| ID | Operation | Current support | Limits and remaining work | +| --- | --- | --- | --- | +| J01 | Encrypt an object, array, scalar, or null | Implemented. Raw JSON supports every kind of root. Typed adapters support declared structs, maps, and slices | None listed | +| J02 | Decrypt a complete document | Implemented, to raw JSON text or a declared Go type | None listed | +| J03 | Write and read in batches | Implemented, with a shared executor and chunking by the provider | None listed | +| J04 | Contains | Implemented: `Contains`, `contains`, and `@>` with `query_json` | Generated code checks the needle and refuses some shapes | +| J05 | Contained by | Implemented: `ContainedBy`, `contained_by`, and `<@` | None listed | +| J06 | Scalar equality at a path | Implemented: `EqAt`, `eq_at` | Compares value selectors, not ordering terms | +| J07 | The four ordering predicates | Implemented in runtime paths and in `At.` accessors | Ordered number and string paths only | +| J08 | Inclusive range | Implemented: `Between`, `between_at`, with named lower and upper operands | None listed | +| J09 | Sort | A selector and explicit SQL | Sorting at a wildcard path needs a defined reduction. Do not assume the first array element | +| J10 | Path exists | A selector and `json_tree_path_exists` | None listed | +| J11 | Extract one match | A selector and `->`, or the first result of a path query | At a wildcard path, which element is returned is unspecified | +| J12 | Extract as encrypted text | `->>` | The result is still encrypted entry JSON, not plaintext | +| J13 | Enumerate all matches | An SQL path query | The supported way to get grouped plaintext is a subtree read with a wildcard bound | +| J14 | Open extracted scalars | `JSONSelection.Open` opens one optional entry for each document group | Groups with more than one entry for a selector are refused. Do not label them supported | +| J15 | Prepare a read at a position or wildcard | Implemented: `Subtree`, `subtree`. Returns a token list, and keeps the trusted request | None listed | +| J16 | Fetch and open a subtree | `json_tree_subtree(document, tokens, budget)`, then `JSONRead.Open` or `OpenOne` | None listed | +| J17 | Immediate children | `Children` prepares a concrete token. SQL returns the child entries | Opening child scalars bound to their parent is not implemented | +| J18 | Count children | A `ChildCount` token. SQL returns a count that can be NULL | The count is not an authenticated proof of completeness | +| J19 | Walk budget and root fallback | Implemented. After root fallback, the client applies the selection originally requested | None listed | +| J20 | Comparisons between stored values | Existing document containment and entry ordering | Only within a compatible column, keyset, and profile | + +### Open requirements for J13 and J17 + +The release still needs every operation in the table. Closed tracking +issues do not mean an opener bound to the parent exists. + +For J17, build an opener for child scalars that is bound to their parent. +Do not pass child rows to an unrelated selector or subtree request. Until +the opener exists, fetch and open the parent's subtree to inspect its +children. Document that this transfers different data and leaks +differently. + +For J13, enumerating all matches in SQL is not the same as the supported +interface. The supported interface returns plaintext from a bounded +subtree read. ## Semantics that changed or are now implemented -- #1379 randomizes stored entries, object ordinals and containment projection - order. `->` at a wildcard selects an unspecified element. Use all matches - with an explicit `EXISTS`, `MIN` or `MAX` rule for ordering/sorting. -- Cascade authentication commits containers to their child ciphertext tags. - Nonempty standalone container entries fail `JsonNotALeaf`, also used for - alteration; empty container entries authenticate but fail `OpenedType`. - All ordinary containers need a subtree read. A requested sealed root can - still open as a complete document. -- The dynamic engine now rejects positional selector/ordering/EqAt paths. - Concrete `[n]` positions belong to subtree reads; wildcard selectors address - a path class. Independent EqAt predicates need not match one array element; - one containment needle preserves their correlation. -- The strict Rust parser now rejects duplicate keys and out-of-range integer - literals before key I/O. Typed Go encoding occurs first and follows - `encoding/json`, including invalid-UTF8 replacement. Raw JSON is checked - directly. Neither path promises byte-for-byte JSON text preservation. -- SQL NULL, missing selection, JSON null and empty containers are distinct. - `JSONValue.Found` carries selection presence. `OpenOne` rejects wildcards; - plural subtree reads return requested values in document order. -- Prepared reads retain the cipher, field definition, path and wildcard - bound. `Open(ctx, rows)` asks Rust to rederive and verify the request. - No serialized guest-local key handle is required. Rows cannot supply their - own expected path. Keep one group per database document. -- Cascade strengthens verification beneath a returned root. It does not - authenticate the whole result set, absence, freshness or the intended - database row. Whole results/probes can still be withheld. -- Wildcard bound defaults to 64 with boundary probes and an 8,192-token cap. - Walk budget independently defaults to 64; zero disables root fallback. - An open-root document cannot use sealed-root fallback. The budget is not - a universal response-size cap. -- Header version remains 1 despite the changed cascade/AAD semantics. Freeze - the post-cascade profile explicitly; the unchanged header is not evidence - of compatibility with older prototype ciphertexts. Randomized query - projection order requires canonical semantic comparisons in parity tests. +- **Random order.** #1379 randomises the stored order of entries, the + ordinals of object members, and the order of the containment + projection. `->` at a wildcard selects an unspecified element. For + ordering or sorting, use all matches with an explicit `EXISTS`, `MIN`, + or `MAX` rule. +- **Cascaded tags.** Each container's tag covers its children's + ciphertext tags. A container entry with children, opened on its own, + fails with `JsonNotALeaf`. The same error is returned for an altered + entry. An empty container entry authenticates, but fails with + `OpenedType`. Every ordinary container needs a subtree read. A sealed + root that the caller asks for can still open as a complete document. +- **No positional paths in queries.** The dynamic engine now rejects + positional paths for selectors, ordering, and EqAt. Concrete positions + such as `[n]` are for subtree reads. A wildcard selector addresses every + path that differs only in its array positions. Two separate EqAt + predicates can match different elements of one array. One containment + needle keeps them matched to the same element. +- **Strict checks on JSON input.** The Rust parser now rejects duplicate + keys and out-of-range integer literals before key I/O, that is, before + any request to the key provider. For typed Go values, Go encodes first, + with `encoding/json`. That package replaces invalid UTF-8. The parser + checks raw JSON directly. Neither path promises to keep the JSON text + byte for byte. +- **NULL, missing, null, and empty are different results.** SQL NULL, a + missing selection, JSON null, and an empty container are different. `JSONValue.Found` records + whether the selection was present. `OpenOne` rejects wildcards. Subtree + reads that return several values return the requested values in + document order. +- **Reads that Rust verifies against the original request.** A prepared + read keeps the cipher, the field definition, the path, and the wildcard + bound. `Open(ctx, rows)` asks Rust to derive the request again and + verify it. No serialised key handle local to the guest is needed. Rows + cannot supply their own expected path. Keep one group for each database + document. +- **What cascaded tags do not prove.** Cascaded tags make verification + stronger beneath the node at the top of a returned read. Cascaded tags + do not authenticate the whole result set, the absence of a value, freshness, + or that the row is the intended database row. The database can still + withhold whole results or probes. +- **Defaults for reads.** The wildcard bound defaults to 64, with probes + at the bound and a cap of 8,192 tokens. The walk budget is separate, and + also defaults to 64. A walk budget of 0 turns off root fallback. A JSON + document whose root is not sealed cannot fall back to the sealed root. + The walk budget does not limit the size of every response. +- **Header version.** The header version is still 1, although the cascade + changed what the AAD binds. Freeze the post-cascade profile explicitly. + The unchanged header does not show compatibility with older prototype + ciphertexts. The query projection order is now random, so parity tests + need to compare projections in a canonical form, by meaning. ## Current SQL recipes -The examples assume `profile public.eql_v4_json_search`. Parameters are the -SDK's generated query/selector/token values, never plaintext paths. Add the -application's authorization and tenant predicates as usual. +The examples assume a column declared as +`profile public.eql_v4_json_search`. Each parameter is a query, selector, +or token value that the SDK generates, never a plaintext path. Add the +application's usual authorisation and tenant predicates. ```sql -- Contains / contained by / EqAt operands. @@ -149,16 +210,18 @@ CROSS JOIN LATERAL eql_v4.json_tree_children(u.profile, $1::text) AS c(entry); SELECT id, eql_v4.json_tree_child_count(profile, $1::text) FROM users; ``` -Child count is zero for scalars/empty containers and NULL for a missing node -or the sealed root. Do not convert NULL to zero. The former token/gates and -`json_tree_subtree_at` recipes are removed. Current full Go programs and -parameter binding examples live in -[the JSON example](../../languages/golang/encrypt/example/json/main.go). +The child count is 0 for a scalar or an empty container. The count is +NULL for a missing node or for the sealed root. Do not convert NULL to 0. + +The earlier recipes with tokens and gates, and with +`json_tree_subtree_at`, are removed. +[The JSON example](../../languages/golang/encrypt/example/json/main.go) +contains complete current Go programs and examples of binding parameters. ## Current dynamic grammar and results -This is implemented data-plan syntax, without an illustrative versioned -outer envelope: +The example below shows the implemented syntax of a data plan. It leaves +out a versioned outer envelope: ```json { @@ -173,21 +236,34 @@ outer envelope: } ``` -Operations select the field through the ABI. Their named inputs include -`{"json"}`, `{"path","json"}`, `{"path","low","high"}`, `{"path"}` -and `{"path","wildcard_bound"}`. JSON values are UTF-8 JSON text. Query -results identify `encoding` (`eql`, `text`, `text_array`, or scalar `bytes`), -SQL domain and operands. Capabilities identify input shapes. Opening uses -`json_open` / `se_json_open` with the original `selector` or `subtree` -request and grouped encrypted rows. The general versioned definition/schema -package remains required work; it is not the current runtime grammar. +Operations select the field through the application binary interface +(ABI) between the host and the guest. The named inputs of operations +include: + +- `{"json"}` +- `{"path","json"}` +- `{"path","low","high"}` +- `{"path"}` +- `{"path","wildcard_bound"}` + +JSON values are UTF-8 JSON text. Query results state the `encoding` +(`eql`, `text`, `text_array`, or scalar `bytes`), the SQL domain, and the +operands. Capabilities state the shapes of input. -## Rust write/read example +To open, call `json_open` or `se_json_open` with the original `selector` +or `subtree` request and the grouped encrypted rows. -Proposed public code. `target(column = ...)` is illustrative declaration -syntax for an explicit EQL column identity; the exact attribute spelling -belongs to the forthcoming specification/naming pass. The options factory -is typed Rust, not JSON hidden inside an attribute. +A general versioned package of definitions and schemas is still required +work. That package is not the current runtime grammar. + +## Proposed Rust write and read example + +The code below is proposed public code, not executable code for the +current engine. +`target(column = ...)` illustrates syntax that declares an explicit EQL +column identity. The coming specification and naming pass decides the +exact attribute spelling. The options factory is typed Rust, not JSON +hidden inside an attribute. ```rust use eql_bindings::v4::json_tree::JsonTreeDocument; @@ -241,39 +317,69 @@ async fn main() -> Result<(), Box> { } ``` -This example explicitly declares column-wide scope and no row binding by -its fixed standalone definition; the normative schema must express that -choice rather than silently replacing missing required tenant scope. -For an output-derived `EncryptedUser` with `id: PrimaryKey` and -`tenant_id: ScopeKey`, select those fields through the context/scope -attributes described in the API plan. Obtain its JSON field handle from -`EncryptedUser::definition()?.profile()` and require `TenantId` on query -requests. Store/return both ID fields for decrypt. If caller input is used, -use the agreed `_with_context` variants with a typed input separating -binding and scope. No row ID is ever an index/query input implicitly. +### Scope and row binding in the example + +The example's fixed standalone definition explicitly declares scope across +the whole column, and no row binding. The normative schema must state that +choice. The schema must not silently replace a required tenant scope that is +missing. + +Take an `EncryptedUser` whose definition is derived on the encrypted +output type, with `id: PrimaryKey` and `tenant_id: ScopeKey`. +For that type: + +1. Select those fields through the context and scope attributes described + in the API plan. +2. Get its JSON field handle from + `EncryptedUser::definition()?.profile()`, and require `TenantId` on + query requests. +3. Store and return both ID fields for decryption. +4. If the caller supplies input, use the agreed `_with_context` variants, + with a typed input that keeps binding and scope separate. + +A row ID is never an index or query input implicitly. ## Remaining work and release gates -1. Capture the implemented wire grammar, errors, presence states and context - profile in the promised formal specification and conformance corpus. - General Rust definitions/caching and additional context roles can follow. -2. Close the explicit J14/J17 scope gaps. The supported all-match plaintext - route is subtree reads; a parent-bound child opener is separate engine - work. Do not add generic raw-entry decryption as a shortcut. -3. Replace the internal EQL serialize-to-Value/deserialize bridge with typed - conversion while preserving the frozen profile. This is an optimization, - not a missing Go JSON execution path. -4. Run the existing Rust/guest/Go/PostgreSQL suites on the release candidate, - mapping tests to J01–J20. Include typed/raw inputs, numeric/null cases, - both root policies, wildcard bounds, fallback, cascade tampering and known - omission limitations. Inspect real SQL parameter casts and index use. -5. Freeze post-cascade storage/search vectors and the matching EQL bundle. - Compare canonical projection semantics and cross-opening; ordinary - randomized bytes need not match. Replace temporary Go Vitamin C pins - before a final release. Opaque storage adoption is already done and is - a separate format from JSON-tree storage. - -This refresh is source and documentation review, not a fresh runtime test -run or a complete cryptographic audit. Final Rust example compilation, -package extraction and stronger zeroization/constant-time work remain later -API-plan activities; they must preserve the released Go/data contract. +1. Write down the implemented wire grammar, errors, presence states, + and context profile in the formal specification and the set of + conformance tests that were promised. General Rust definitions, + caching, and more context roles can come later. +2. Close the stated scope gaps in J14 and J17. Subtree reads are the + supported way to get plaintext for all matches. An opener for children + bound to their parent is separate engine work. Do not add generic + decryption of raw entries as a shortcut. +3. Replace the internal EQL bridge with typed conversion. The bridge + serialises to `Value` and deserialises again. Keep the frozen profile. + The change is an optimisation, not a missing execution path for Go + JSON. +4. Run the existing Rust, guest, Go, and PostgreSQL suites on the release + candidate, and map the tests to J01 to J20. Include: + - typed and raw inputs; + - numeric and null cases; + - both root policies (sealed and not sealed); + - wildcard bounds; + - root fallback; + - tampering that the cascaded tags detect; + - the known limits on detecting omitted results. + + Inspect the real SQL parameter casts and index use. +5. Freeze the post-cascade storage and search vectors and the matching EQL + bundle. Compare projections by their canonical meaning, and test + cross-opening (one implementation opening another's ciphertexts). + Ordinary random bytes do not need to match. Replace the temporary Go + pins of Vitamin C before a final release. Opaque storage is already + adopted. Opaque storage is a separate format from JSON tree storage. + +## What this refresh did not check + +This refresh is a review of source and documentation, not a new runtime +test run or a complete cryptographic audit. + +Later activities in the API plan include: + +- compiling the final Rust example; +- extracting packages; +- stronger zeroisation and constant-time work. + +These activities must keep the released Go and data contract. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 09c7bd248..2552b8621 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -1,81 +1,138 @@ # Stack Encrypt packaging and public names -Date: 2026-10-25. Recommendation for review, accompanying the +Dated 25 October 2026, with an implementation update on 27 October 2026. +This recommendation for review accompanies the [implementation plan](2026-10-24-stack-encrypt-api-plan.md). -These are proposed boundaries and names, not implemented APIs or accepted -renames. Existing companion examples retain their working names until this -proposal is accepted. The detailed JSON operation contract remains under -separate user review. - -**Implementation update, 2026-10-27:** Go JSON and opaque storage have -landed on `next` through #1369/#1370/#1372/#1373/#1378/#1402; #1397 adds -short whole-word match terms. The customer path is remaining configuration -and conformance work plus release readiness, including replacing temporary -Vitamin C Go module pins. Rust stays on crates.io Vitamin C 0.8.0. Neither -crate extraction nor final Rust facade completion is a customer release gate. - -Source baseline: the current documentation branch and inspected local Vitamin -C 0.8.0 sources. Stack's -[cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) -explain its existing batching boundary. The -[EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) establishes -the optional downstream engine dependency, and -[encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) -are the concrete shared-type extraction candidates. This is a source-level -design assessment, not an audit of every external Vitamin C consumer. + +## Status + +This document proposes boundaries and names. The proposals are not +implemented APIs or accepted renames. The existing companion examples keep their +working names until this proposal is accepted. The detailed JSON operation +contract is under separate user review. + +## Summary of recommendations + +- Stack Encrypt owns the application value model and its execution. + Vitamin C keeps the reusable cryptographic primitives. +- Add one new crate, `stack-encrypt-format`, and rename `stack-kms` to + `stack-zerokms`. Use modules, not crates, for the other boundaries. +- Retire Vitamin C's `AsyncCipher` trait. +- Adopt the public glossary for Rust. Keep the merged Go and SQL names for + JSON. + +## Update on 27 October 2026 + +Go JSON and opaque storage are merged into `next`, through #1369, #1370, +#1372, #1373, #1378, and #1402. #1397 adds match terms for short whole +words. + +What remains before customers can use them is configuration and +conformance work, and release readiness. That includes replacing the +temporary Go module pins of Vitamin C. Rust stays on Vitamin C 0.8.0 from +crates.io. A customer release does not wait for crate extraction or for the +final Rust facade (the proposed public Rust API). + +## Sources for this assessment + +This assessment is based on the current documentation branch and on the +local Vitamin C 0.8.0 sources that were inspected: + +- Stack's + [cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) + explain the boundary of its existing batching. +- The [EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) + shows that `eql-bindings`, the Rust crate for Encrypt Query Language + (EQL), sits downstream of the engine, with an optional dependency on + it. +- The + [encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) + are the concrete candidates for extracting shared types. + +This is a design assessment at the source level. It is not an audit of +every external consumer of Vitamin C. ## Ownership rule -Stack Encrypt owns the application value model and its execution: turning -owned Rust/dynamic values into ciphertext, structured targets and search -operands, and reconstructing plaintext. Vitamin C owns reusable cryptographic -and protected-memory primitives. A normal Stack Encrypt user should not need -to import Vitamin C encryption traits to implement an application type. +Stack Encrypt owns the application value model and its execution. That +means turning owned Rust values and dynamic values into ciphertext, +structured targets, and search operands, and reconstructing plaintext. +Vitamin C owns reusable primitives for cryptography and protected memory. +An ordinary Stack Encrypt user should not have to import Vitamin C +encryption traits to implement an application type. -Moving code should remove duplicate abstractions, not preserve both public -surfaces under new names. The user's prototype compatibility exemption -applies to this API; review other Vitamin C consumers before deleting their -dependencies upstream. +Moving code should remove duplicate abstractions. Moving code should not +keep both public surfaces under new names. The user's compatibility +exemption for prototypes applies to this API. Before deleting +dependencies upstream, review the other consumers of Vitamin C that use +them. ## Proposed packages | Package | Responsibility | | --- | --- | -| `stack-encrypt` | Public Rust API, value traversal/reconstruction, target declarations and definitions, context/scope roles, normalization and searchable-target adapters, plans, batching, JSON-tree execution, and dynamic definition validation/execution | -| `stack-encrypt-derive` | Derives generating the same declarations and definitions available by hand; re-export derives from `stack-encrypt` | -| `stack-encrypt-format` — new | Pure data representations/codecs shared by the engine and bindings: frame primitives and required scalar representations, stable blob profile, ciphertext envelope parsing/encoding, and validated Base64/Base85 text; no key requests or execution runtime | -| `stack-zerokms` — rename current `stack-kms` | ZeroKMS protocol client, authentication integration, native keysets, lock-context request types, and responses; it is not the neutral multi-provider abstraction | -| `eql-bindings` | EQL versioned storage/query types, SQL-compatible JSON serialization, catalog and Stack target adapters; stays downstream of the engine | -| Vitamin C crates | Protected memory, randomness, byte-level AEAD/key operations, HMAC/PRF and ORE algorithm primitives, and reusable canonical context encoding | - -Keep `stack-auth` and `stack-profile` separate: they have consumers outside -encryption. Keep `stack-guest-abi` for the host/guest calling convention. -Neither the ABI nor the Go host should own a second target-policy model. -Put the formal dynamic contract, schemas and conformance fixtures beside -Stack Encrypt; its implementation uses the same definition types and -validation as static Rust. - -Use modules for the remaining boundaries initially. In particular, do not -create separate execution, JSON-policy, query, context or provider-traits -crates merely to mirror each concept. - -The format crate has a concrete consumer justification: EQL's wire types -currently work without its optional `stack-encrypt` integration. They need -shared encoding types without an encryption client/runtime dependency. Make -Serde, schema and TypeScript support feature-scoped where practical; define -implementations alongside the owned types so downstream orphan rules do not -force duplicate wrappers. Re-export the ordinary format types through Stack -Encrypt, so users normally need only its dependency. - -The format crate is a single implementation with distinct profiles. Blob -storage, host/guest transport, and ciphertext envelopes have different -contracts and versioning requirements; sharing code must not silently make -the ephemeral transport Serde mapping the persisted blob format. This does -not introduce persistence for arbitrary internal `Node` trees: structured -outputs still use their explicit target layout and ordinary Serde composition. +| `stack-encrypt` | The public Rust API; traversing and reconstructing values; target declarations and definitions; context and scope roles; normalisation and adapters for searchable targets; plans; batching; JSON tree execution; and validating and executing dynamic definitions | +| `stack-encrypt-derive` | Derive macros that generate the same declarations and definitions a user can write by hand. `stack-encrypt` re-exports the derives | +| `stack-encrypt-format` (new) | Pure data representations and codecs shared by the engine and the bindings: frame primitives and the required scalar representations, the stable blob profile, parsing and encoding of the ciphertext envelope, and validated Base64 and Base85 text. No key requests and no execution runtime | +| `stack-zerokms` (the current `stack-kms`, renamed) | The ZeroKMS protocol client, authentication integration, native keysets, request types for lock context, and responses. Not the neutral abstraction over several providers | +| `eql-bindings` | EQL storage and query types by version, JSON serialisation compatible with SQL, the catalog, and Stack target adapters. Stays downstream of the engine | +| Vitamin C crates | Protected memory; randomness; AEAD and key operations on bytes; HMAC, PRF, and ORE algorithm primitives; and reusable canonical encoding of context | + +The abbreviations in the last row are: + +- AEAD: authenticated encryption with associated data; +- HMAC: hash-based message authentication code; +- PRF: pseudorandom function; +- ORE: order-revealing encryption. + +### Crates that stay as they are + +Keep `stack-auth` and `stack-profile` separate, because they have +consumers outside encryption. Keep `stack-guest-abi` for the calling +convention between the host and the guest. Neither the application binary +interface (ABI) nor the Go host should own a second model of target +policy. + +Put the formal dynamic contract, its schemas, and its conformance fixtures +beside Stack Encrypt. The implementation of the dynamic contract uses the +same definition types and validation as static Rust. + +### Modules, not more crates + +At first, use modules for the remaining boundaries. In particular, do not +create separate crates for execution, JSON policy, queries, context, or +provider traits only so that each concept has its own crate. + +### Why the format crate is needed + +The format crate has a concrete consumer. EQL's wire types now work +without EQL's optional `stack-encrypt` integration. Those wire types need +shared encoding types without depending on an encryption client or +runtime. + +- Put Serde, schema, and TypeScript support behind features, where + practical. +- Define the trait implementations beside the types the crate owns. + Otherwise Rust's orphan rule forces downstream crates to duplicate + wrappers. The orphan rule says an implementation must be in the crate + of the trait or of the type. +- Re-export the ordinary format types through Stack Encrypt, so users + normally need only that one dependency. + +The format crate is one implementation with separate profiles. Each of +these has its own contract and versioning requirements: blob storage, the +transport to and from the guest, and ciphertext envelopes. Sharing code must not +silently turn the Serde mapping for short-lived transport into the stored +blob format. + +The format crate does not add a way to store arbitrary internal `Node` +trees. Structured outputs still use their explicit target layout and +ordinary Serde composition. ## Dependency direction +In the diagram, an arrow points from a crate to a crate it depends on. + ```mermaid flowchart TD App[Application / Rust or guest adapter] --> EQL[eql-bindings] @@ -88,181 +145,259 @@ flowchart TD Engine --> Vitamin[Vitamin C primitives] ``` -The derive emits Stack paths but does not depend on the engine as a normal -runtime dependency. No Vitamin C primitive depends back on Stack Encrypt. -`stack-zerokms` does not depend on the engine. Provider traits belong to -`stack_encrypt::provider`; a Stack-owned adapter implements them using the -ZeroKMS client. This avoids an engine/client dependency cycle. +- The derive crate emits Stack paths. The derive crate does not depend on + the engine as a normal runtime dependency. +- No Vitamin C primitive depends back on Stack Encrypt. +- `stack-zerokms` does not depend on the engine. +- The provider traits belong in `stack_encrypt::provider`. An adapter + that Stack owns implements them with the ZeroKMS client. This avoids a + dependency cycle between the engine and the client. + +### Providers as optional features + +Make ZeroKMS an optional provider feature, separate from the HTTP feature. +Then a different provider does not bring its protocol into the core +contract. Keep support for injected providers, and for guest and native +builds. -Make ZeroKMS an optional provider feature independently of the HTTP feature, -so a different provider does not pull its protocol into the core contract. -Preserve injected providers and guest/native builds. Data-key support is the -base capability; index-key support, named-keyset resolution and lock context -remain explicit optional capabilities. Unsupported required capabilities -fail before key I/O. Future provider adapters can become separate downstream -crates when needed; do not design fictitious AWS/Vault keysets now. +Data key support is the base capability. Index key support, resolving +named keysets, and lock context stay explicit optional capabilities. A +required capability that is not supported fails before key I/O, that is, +before any request to the key provider. Future provider adapters can +become separate downstream crates when needed. Do not design invented AWS +or Vault keysets now. -Do not re-export EQL from Stack Encrypt: EQL's target integration already -depends on Stack Encrypt. Applications explicitly import their EQL types. +Do not re-export EQL from Stack Encrypt, because EQL's target integration +already depends on Stack Encrypt. Applications import their EQL types +explicitly. ## What moves out of Vitamin C -Move application-facing plaintext/ciphertext visitors, structural nodes, -value conversion traits and derives, tree walking, collection composition, -and application encryption helpers into Stack's value/target/plan layers. -Keep advanced extension points public only where target implementers need -them; they are not the normal root API. Internal nodes do not become a -second public opaque ciphertext type. - -Move the generic frame machinery and codecs needed by Stack into the format -crate. Stack owns the stored blob profile and its compatibility promise. -This refines the earlier recommendation to reuse Vitamin C's frame codec: -reuse its implementation, with a lossless storage mapping, while changing -its long-term ownership. Coordinate this handoff under -[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); retain -the issue's format specification and fixture work. No upstream issue or -repository has been changed by this proposal. - -Also move normalization and PRF/ORE adapters that understand Stack's logical -values, traversal, targets or query semantics. Keep independent cryptographic -algorithms below them. The current `vitaminc-normalize`, `vitaminc-prf` and -`vitaminc-ore` manifests depend on `vitaminc-aead`; normalization even uses -its physical scalar types. Moving only the encryption traits would therefore -leave a dependency back to the old value model. Separate primitive inputs -from application adapters as part of the move; do not add a reverse -Vitamin C dependency on Stack to make it compile. - -Retain canonical context byte construction in `vitaminc-context` where it -serves independent algorithms. Stack owns the semantic roles and selection -rules (`PrimaryKey`, `ScopeKey`, ciphertext context, query scope and caller -input capabilities), reusing that canonical encoding instead of inventing -an incompatible second one. Provider policy remains a separate role. +Move these parts into Stack's value, target, and plan layers: + +- the visitors for plaintext and ciphertext that applications use; +- structural nodes; +- the value conversion traits and their derives; +- tree walking; +- collection composition; +- the application encryption helpers. + +Keep advanced extension points public only where people who implement +targets need them. These extension points are not part of the normal API +at the crate root. Internal nodes do not become a second public type for +opaque ciphertext. + +### Frames and codecs + +Move the generic frame code and the codecs that Stack needs into the +format crate. Stack owns the stored blob profile and its compatibility +promise. + +This refines the earlier recommendation to reuse Vitamin C's frame codec. +Reuse its implementation, with a lossless mapping to storage, but change +who owns it in the long term. Coordinate this handover under +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). Keep +that issue's work on the format specification and the fixtures. This +proposal has not changed any upstream issue or repository. + +### Normalisation and index adapters + +Also move the normalisation, PRF, and ORE adapters that understand Stack's +logical values, traversal, targets, or query semantics. Keep the +independent cryptographic algorithms below them in the dependency graph. + +The current manifests of `vitaminc-normalize`, `vitaminc-prf`, and +`vitaminc-ore` depend on `vitaminc-aead`. Normalisation even uses the +physical scalar types of `vitaminc-aead`. So moving only the encryption +traits would leave a dependency back to the old value model. + +As part of the move, separate primitive inputs from application adapters. +Do not make Vitamin C depend on Stack to get the code to compile. + +### Context encoding + +Keep the construction of canonical context bytes in `vitaminc-context`, +where independent algorithms use it. Stack owns the semantic roles and +the selection rules: `PrimaryKey`, `ScopeKey`, ciphertext context, query +scope, and caller input capabilities. Stack reuses that canonical encoding +instead of inventing a second encoding that is not compatible. Provider +policy stays a separate role. ## Retire AsyncCipher -Recommend removing Vitamin C's `AsyncCipher` and its associated application -execution helpers as the Stack executor takes ownership. Do not introduce a -replacement public catch-all async cipher trait in Stack. +Recommendation: remove Vitamin C's `AsyncCipher` trait and its helpers for +executing application encryption, as the Stack executor takes over that +work. `AsyncCipher` is the Vitamin C trait that seals and opens leaves +asynchronously. Do not add a replacement public async cipher trait in +Stack that covers every case. + +Stack already plans and groups the remote key operations, and then runs +the cryptography locally and synchronously. Stack's current `MintedKeys` +and `BoundKeys` implementations use synchronous operations of the leaf +cipher. `AsyncCipher` models a different place to put the asynchronous +work. The asynchronous boundary that matters is the provider and the +public pending operation, with Stack controlling the batching. + +Stack does not use `AsyncCipher`. Vitamin C does use it: `aead/src/seal.rs`, +`context.rs`, `typed.rs`, `walk.rs`, and `normalize/src/terms.rs` expose or +use async helpers, and their tests implement the trait. To retire it: -Stack already plans and groups remote key operations, then performs local -synchronous cryptography. Its current `MintedKeys`/`BoundKeys` implementations -use synchronous leaf cipher operations; `AsyncCipher` models a different -place to put asynchronous work. The meaningful async boundary is the provider -and the public pending operation, with batching controlled by Stack. +1. Move the tests that cover batching, ordering, failures, and + reconstruction with the code they test. +2. Delete the retired trait and its helpers. +3. Audit external consumers before the upstream release. -It is unused by Stack, but not unused inside Vitamin C: `aead/src/seal.rs`, -`context.rs`, `typed.rs`, `walk.rs` and `normalize/src/terms.rs` expose or use -async helpers, and their tests implement it. Migrate the relevant batching, -ordering, failure and reconstruction coverage with the code, then delete -the retired trait/helper surface. Audit external consumers before the -upstream release. Separate internal leaf seal/open capabilities where useful -instead of retaining implementations that reject half of a combined trait. +Where useful, separate the internal capabilities for sealing leaves and +opening leaves. Do not keep implementations that reject half of a +combined trait. ## Recommended public glossary -| Name/path | Meaning | +| Name or path | Meaning | | --- | --- | -| `stack_encrypt::Cipher` | Provider-backed execution handle; replaces `StackCipher` | -| `KeysetCipher` | Execution handle bound to a resolved keyset, available for providers supporting keysets | -| `Ciphertext` | One opaque encrypted value with a specified storage envelope; replaces the public use of `StackCipherText = Node` | -| `OpaqueBlob` | Explicit owned whole-value serialization adapter | -| `EncryptInto`, `DecryptFrom` | Target-directed encryption and destination-directed decryption convenience traits, with context variants | -| `OpaqueEncode`, `OpaqueDecode` | Single-ciphertext representation traits | -| `target::{EncryptTarget, DecryptSource}` | Declaration traits; matching derives are re-exported at the crate root | -| `target::TargetDefinition` | Validated immutable definition shared by static and dynamic paths | -| `context::{PrimaryKey, ScopeKey}` | Eligibility wrappers; neither automatically selects a context or keyset | -| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes; owned in the format crate and re-exported here | -| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated encoded opaque-ciphertext representations and supported field targets; encoding validity alone is insufficient | -| `plan::Plan` | Explicit low-level composition interface; output selection is `.target::()` | +| `stack_encrypt::Cipher` | Execution handle that uses a provider for keys. Replaces `StackCipher` | +| `KeysetCipher` | Execution handle bound to a resolved keyset. Available for providers that support keysets | +| `Ciphertext` | One opaque encrypted value with a specified storage envelope. Replaces the public use of `StackCipherText = Node` | +| `OpaqueBlob` | Explicit adapter that serialises a whole owned value | +| `EncryptInto`, `DecryptFrom` | Convenience traits for encryption directed by the target and decryption directed by the destination, with variants that take context | +| `OpaqueEncode`, `OpaqueDecode` | Traits that represent a value as a single ciphertext | +| `target::{EncryptTarget, DecryptSource}` | Declaration traits. The matching derives are re-exported at the crate root | +| `target::TargetDefinition` | Validated, immutable definition shared by the static and dynamic paths | +| `context::{PrimaryKey, ScopeKey}` | Wrappers that mark a field as eligible. Neither selects a context or a keyset automatically | +| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes. Owned in the format crate and re-exported here | +| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated, encoded representations of opaque ciphertext, and supported field targets. Valid encoding alone is not enough | +| `plan::Plan` | Explicit low-level interface for composition. The output is selected with `.target::()` | | `provider` | Neutral provider contracts and capability bounds | -| `provider::zerokms` | ZeroKMS adapter and policy extension surface, including `LockContext` | -| `dynamic` | Versioned definition loading, capability discovery and execution through the common engine | - -Keep the already-agreed `.using(...)`, `definition()`, `encrypt`, -`encrypt_with_context`, `encrypt_into`, `encrypt_into_with_context`, -`decrypt_from` and `decrypt_from_with_context` spellings. Keep direct -`cipher.decrypt`/`decrypt_with_context`. Remove plaintext `.encrypt`, -redundant `encrypt_as`, and public `decrypt_into` as already agreed. - -Use `TargetDefinition` for the configured public contract and `Layout` only -for the advanced internal composition concept where that distinction is -needed. A layout does not name a plaintext document or a stored ciphertext. -Name low-level operations `seal`/`open` where they act on bytes; reserve the -application encrypt/decrypt vocabulary for the public value operations. - -Keep ciphertext representations in the format crate independent of traits; -Stack implements its own target traits for those types. Plain encoded text -is not itself an encryption target. Construction must validate both layers -for the ciphertext wrappers; avoid an unchecked public string constructor. +| `provider::zerokms` | The ZeroKMS adapter and the surface for its policy extension, including `LockContext` | +| `dynamic` | Loading versioned definitions, discovering capabilities, and executing through the common engine | + +### Spellings already agreed + +Keep these agreed spellings: + +- `.using(...)` +- `definition()` +- `encrypt` +- `encrypt_with_context` +- `encrypt_into` +- `encrypt_into_with_context` +- `decrypt_from` +- `decrypt_from_with_context` + +Keep the direct `cipher.decrypt` and `decrypt_with_context`. As already +agreed, remove `.encrypt` on plaintext, the redundant `encrypt_as`, and +the public `decrypt_into`. + +### Definition, layout, seal, and open + +Use `TargetDefinition` for the configured public contract. Use `Layout` +only for the advanced internal concept of composition, where that +distinction is needed. A layout does not name a plaintext document or a +stored ciphertext. + +Name low-level operations on bytes `seal` and `open`. Keep the words +"encrypt" and "decrypt" for the public operations on application values. + +### Ciphertext types in the format crate + +Keep the ciphertext representations in the format crate independent of +traits. Stack implements its own target traits for those types. Plain +encoded text is not itself an encryption target. Constructing a +ciphertext wrapper must validate both layers: the encoding and the +ciphertext. Avoid a public string constructor without checks. ## JSON names by role -Keep the existing EQL package and version modules. The merged public Go -surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery` and -`JSONEntry`, configured through `JsonSearch`; SQL uses `eql_v4_json_search`, -`query_json`, `query_json_leaf` and `eql_v4_json_entry`. Prepared Go objects -are `encrypt.JSONSelection` and `JSONRead`. - -The table below remains a proposed Rust facade glossary. Its `JsonTree*` -names describe handwritten Rust adapters, not the registered Go target. -Do not rename the working Go/SQL surface to this glossary as a customer -release prerequisite. Later Rust renames must preserve frozen dynamic IDs. - -| Working/current name | Recommended name | Role / home | -| --- | --- | --- | -| `JsonPlaintext` | `JsonPlaintext` | Owned plaintext adapter, `stack_encrypt::json` | -| `JsonOptions` | `JsonTreeOptions` | Validated tree-target policy, `stack_encrypt::json` | -| `JsonSelection` | `JsonSelection` | Proposed plaintext selection result, `stack_encrypt::json` | -| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document, `eql_bindings::v4` | -| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry, `eql_bindings::v4` | -| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result, `eql_bindings::v4` | -| `JsonTreeQuery` | `JsonContainmentQuery` | Containment/path-value query operand, `eql_bindings::v4` | -| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Ordering query operand, `eql_bindings::v4` | -| `JsonTreeSelector` | `JsonSelector` | Encoded path selector, `eql_bindings::v4` | -| `JsonSubtreeKey` | `JsonWalkRequest` | Client-held walk token and request binding; engine type with a typed EQL adapter | - -`JsonContainmentQuery` can still be the physical operand for a path equality -operation where the agreed SQL recipe uses containment; operation and wire -operand need not have the same name. `JsonWalkRequest` describes the actual -role better than "key": it carries sensitive capability/request state, -not the data-encryption key. Exact operation behavior remains governed by -the separate JSON contract review. - -Rename the internal target index `Json` to `JsonTreeIndex`. Keep the internal -engine `JsonDocument` out of the ordinary facade, with a role-qualified name -such as `EncryptedJsonTree` if adapter authors require public access. -`EncryptedJson` always denotes the EQL-compatible stored form, not any -arbitrary JSON serialization of encrypted nodes. An application's -`EncryptedUser` remains its own declared record layout. - -Dynamic target/operation identifiers are explicitly versioned contract -identifiers, not Rust type names serialized into JSON. Select their exact -spellings in the interoperability specification and use the same glossary -in Go. Rust renames alone must not accidentally redefine stored SQL formats -or dynamic discriminants. +Keep the existing EQL package and its version modules. The merged public +Go surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery`, +and `JSONEntry`, configured through `JsonSearch`. In SQL, the names are +`eql_v4_json_search`, `query_json`, `query_json_leaf`, and +`eql_v4_json_entry`. The prepared Go objects are `encrypt.JSONSelection` +and `JSONRead`. + +The table below is still a proposed glossary for the Rust facade. Its +`JsonTree*` names describe the handwritten Rust adapters, not the +registered Go target. A customer release does not need the working Go +and SQL surface renamed to match this glossary. Do not rename it for that +reason. Later Rust renames must keep the frozen dynamic IDs. + +| Working or current name | Recommended name | Role | Home | +| --- | --- | --- | --- | +| `JsonPlaintext` | `JsonPlaintext` | Adapter for owned plaintext | `stack_encrypt::json` | +| `JsonOptions` | `JsonTreeOptions` | Validated policy for the tree target | `stack_encrypt::json` | +| `JsonSelection` | `JsonSelection` | Proposed result of a plaintext selection | `stack_encrypt::json` | +| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document | `eql_bindings::v4` | +| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry | `eql_bindings::v4` | +| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result | `eql_bindings::v4` | +| `JsonTreeQuery` | `JsonContainmentQuery` | Query operand for containment and for a value at a path | `eql_bindings::v4` | +| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Query operand for ordering | `eql_bindings::v4` | +| `JsonTreeSelector` | `JsonSelector` | Encoded path selector | `eql_bindings::v4` | +| `JsonSubtreeKey` | `JsonWalkRequest` | Walk token held by the client, and its binding to the request | Engine type, with a typed EQL adapter | + +### Notes on the JSON names + +`JsonContainmentQuery` can still be the physical operand of an equality +at a path, where the agreed SQL recipe uses containment. An operation and +its wire operand do not need the same name. + +`JsonWalkRequest` describes the real role better than "key" does. The +type contains sensitive capability and request state, not the data +encryption key. The separate review of the JSON contract decides the +exact behaviour of each operation. + +Rename the internal target index `Json` to `JsonTreeIndex`. Keep the +internal engine type `JsonDocument` out of the ordinary facade. If adapter +authors need public access to it, give it a name that states its role, +such as `EncryptedJsonTree`. + +`EncryptedJson` always means the stored form +that is compatible with EQL. `EncryptedJson` never means some other JSON +serialisation of encrypted nodes. An application's `EncryptedUser` stays +its own declared record layout. + +The dynamic identifiers of targets and operations are explicitly +versioned contract identifiers. They are not Rust type names serialised +into JSON. Choose their exact spellings in the interoperability +specification, and use the same glossary in Go. A Rust rename on its own +must not redefine stored SQL formats or dynamic discriminants by accident. ## Implementation order and acceptance -1. Specify and validate the now-implemented Go JSON and opaque storage - contracts. Freeze post-cascade JSON/search semantics and exact fixtures. - Replace temporary Vitamin C Go module pins for a final release. -2. Complete outstanding Go match configuration and JSON read/conformance - gaps. Short whole-word matching is implemented in EQL; its leakage and - profile compatibility still require explicit release acceptance. -3. After delivery, establish the shared format crate and move the value model, - traversal and dependent normalization/index adapters into Stack. Preserve - frozen encodings and an acyclic graph; move tests with their code. -4. Complete neutral provider extraction and protocol-package renaming, then - retire Vitamin C's async application execution surface. Required Go-facing - policy and routing contracts must already have been handled before release. -5. Complete public Rust trait/derive names, imports and executable examples. - These changes must not rename the shipped Go API, reinterpret stored - profiles or require customer re-encryption/reindexing. - -Acceptance includes an acyclic Cargo graph; an EQL wire-only build without -the engine; engine builds without ZeroKMS and without HTTP; native/guest -builds; compiled public examples; the moved behavior tests; and shared -Rust/Go/SQL format/operation fixtures. Full timing and memory-cleanup -improvements remain follow-up work under the accepted format-first priority; -record actual guarantees honestly. No implementation changes, tests of the -proposed API, code moves, or upstream removals are claimed by this document. +1. Specify and validate the Go JSON and opaque storage contracts. Both + are now implemented. Freeze the post-cascade JSON and search semantics + and the exact fixtures. "Post-cascade" means as they are after #1379. + That change made each container's tag cover the tags of its children. + Replace the temporary Go module pins of Vitamin C for a final release. +2. Complete the outstanding Go match configuration, and close the gaps in + JSON reads and conformance. Short whole-word matching is implemented in + EQL. Its leakage and profile compatibility still need explicit + acceptance for the release. +3. After the customer release, set up the shared format crate. Move the + value model, traversal, and the normalisation and index adapters that + depend on it into Stack. Keep the frozen encodings and an acyclic + graph. Move tests with their code. +4. Complete the neutral provider extraction and the renaming of the + protocol package. Then retire Vitamin C's surface for asynchronous + application execution. The policy and routing contracts that Go needs + must already have been handled before the release. +5. Complete the public Rust trait and derive names, imports, and + executable examples. These changes must not rename the released Go API + or reinterpret stored profiles. The changes must not require customers to + re-encrypt or reindex. + +Acceptance includes: + +- an acyclic Cargo graph; +- an EQL build of the wire types only, without the engine; +- engine builds without ZeroKMS and without HTTP; +- native and guest builds; +- compiled public examples; +- the behaviour tests that moved; +- shared fixtures for formats and operations across Rust, Go, and SQL. + +Full improvements to timing and memory clean-up remain follow-up work, +under the accepted priority that puts the format first. Record only the +guarantees that are true. + +This document does not claim any implementation changes, tests of the +proposed API, code moves, or upstream removals. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index 0ce88185e..99202c076 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -1,77 +1,130 @@ -# Key selection, lock context and provider capabilities +# Key selection, lock context, and provider capabilities -Date: 2026-10-25. Assessment accompanying the -[API implementation plan](2026-10-24-stack-encrypt-api-plan.md). +Dated 25 October 2026, with an update on 27 October 2026. This assessment +accompanies the [API implementation plan](2026-10-24-stack-encrypt-api-plan.md). -Status: recommendations for review, not accepted public signatures or an -implementation. The user requested plaintext-derived ZeroKMS keyset -selection, review of Vitamin C `AsyncCipher`, and optional lock-context -support without requiring future AWS KMS/Vault backends to emulate it. -The JSON operation contract remains under user review independently. +## Status and requests -## Distinct roles +This document contains recommendations for review. The recommendations +are not accepted public signatures. Nothing in them is implemented. -| Role | Selects/binds | Example | +The user who requested this assessment asked for three things: + +- ZeroKMS keyset selection derived from plaintext; +- a review of the Vitamin C `AsyncCipher` trait; +- optional lock context support, without requiring future backends, such + as AWS Key Management Service (KMS) or Vault, to emulate it. + +The JSON operation contract is under separate user review. + +## Summary of recommendations + +1. Add a declaration that selects the keyset from a plaintext field. Never + fall back to the default keyset. +2. Do not implement `AsyncCipher` on `StackCipher`. The packaging + recommendation now proposes retiring the trait. +3. Add lock context as an optional, typed ZeroKMS policy input. +4. Split the provider boundary into a base capability for data keys and + optional capabilities for everything else. + +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| Keyset | A ZeroKMS key domain. Data keys are generated in a keyset, and each keyset has a deterministic index key | +| Lock context | A ZeroKMS policy that ZeroKMS applies when it generates or retrieves a data key, such as a claim of the authenticated JSON Web Token (JWT) | +| `DataKeySource` | The trait in `stack-kms` that generates fresh data keys and derives them again for stored ciphertexts | +| `IndexKeySource` | The trait in `stack-kms` that loads the deterministic index key of a keyset, used to derive search terms | +| `Pending` | The request object that `StackCipher` returns. Settling it (awaiting it) sends the batched ZeroKMS calls | +| Key I/O | Any request to the key provider to generate or retrieve a key | +| AAD | Additional authenticated data (AAD): data that an authentication tag covers but the ciphertext does not contain | +| `MaybeSend` | A bound that means `Send` on native targets and no bound on WebAssembly (WASM) | +| Vitamin C | CipherStash's library of cryptographic primitives | + +## Four separate roles + +| Role | What it selects or binds | Example | | --- | --- | --- | -| Key selection | Provider key domain used to encrypt and derive index terms | ZeroKMS keyset ID or name selected from `tenant_id` | -| Ciphertext context | Application binding verified when opening ciphertext | Column + tenant scope + row ID | -| Search scope | Domain in which index/query terms compare | Column + tenant ID | -| Provider policy | Conditions the provider applies to key generation/retrieval | ZeroKMS lock context selecting authenticated JWT `sub` | +| Key selection | The provider key domain used to encrypt and to derive index terms | A ZeroKMS keyset ID or name selected from `tenant_id` | +| Ciphertext context | The application binding verified when a ciphertext is opened | The column, the tenant scope, and the row ID | +| Search scope | The domain in which index and query terms compare | The column and the tenant ID | +| Provider policy | Conditions the provider applies when it generates or retrieves a key | ZeroKMS lock context that selects the `sub` claim of the authenticated JWT | -These are separate logical inputs, even where the protocol binds several -into its tag/AAD. A `ScopeKey` does not select a ZeroKMS keyset by itself; -a keyset selector does not insert its source field into AAD; lock context -is not an alias for either ciphertext context or search scope. Sharing a -source value requires explicit declaration, not inference from a name. +These four roles are separate logical inputs, even where the protocol binds +several of them into its tag or AAD: -## Current implementation evidence +- A `ScopeKey` does not select a ZeroKMS keyset by itself. +- A keyset selector does not add its source field to the AAD. +- Lock context is not another name for ciphertext context or for search + scope. -**Update, 2026-10-27:** #1379 makes open-root JSON verification load the -index keyset as well as the data key. `Pending` settlement and generic -execution entry points now require `DataKeySource + IndexKeySource`. -The future neutral provider split below must preserve that verification -requirement for JSON while allowing data-key-only capabilities where an -operation genuinely needs no index key. This change does not implement -plaintext-derived keyset routing or ZeroKMS lock-context plumbing. +For two roles to share a source value, the definition must declare it +explicitly. The engine must not infer it from a name. +## Current implementation + +### Update on 27 October 2026 + +#1379 makes verification of a JSON document whose root is not sealed load +the index keyset as well as the data key. Settling a `Pending`, and the +generic execution entry points, now require +`DataKeySource + IndexKeySource`. + +The neutral provider split proposed below must keep that verification +requirement for JSON. The split must still allow capabilities with data +keys only, for operations that need no index key. + +#1379 does not implement keyset routing derived from plaintext, or the code +that passes ZeroKMS lock context through. + +### What the code does now - [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts - `IdentifiedBy` by UUID or name, resolves the keyset and its index key, - and returns a bound `KeysetCipher`. The client has a bounded keyset cache - and name-resolution freshness policy. `.default_keyset()` is already - bound at initialization. Current initialization requires both - `DataKeySource` and `IndexKeySource`. -- A sealed leaf carries the resolved keyset UUID. Client-scoped decrypt - groups retrieval by the UUIDs in the leaves; a bound keyset can constrain - which keyset is permitted. Names are lookup conveniences, not persistent - key identities. See [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). + an `IdentifiedBy` universally unique identifier (UUID) or name. The method resolves the keyset and its + index key, and returns a `KeysetCipher` bound to that keyset. The client + has a keyset cache of limited size, and a policy for how fresh a name + resolution must be. `.default_keyset()` is already bound when the client + is initialised. Initialisation now requires both `DataKeySource` and + `IndexKeySource`. +- Each sealed leaf contains the UUID of the resolved keyset. Decryption + through the client groups key retrieval by the UUIDs in the leaves. A + bound keyset can restrict which keyset is allowed. Names are a + convenience for lookup, not persistent key identities. See the + architecture decision record (ADR) + [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). - [Pending dispatch](../../packages/stack-encrypt/src/target/pending.rs) - currently takes one selected keyset for generates and groups retrieves by - their stored keyset IDs. It constructs generate/retrieve payloads without - lock context and passes no unverified request context. -- [Key-source traits](../../packages/stack-kms/src/key_source.rs) and - [payloads](../../packages/stack-kms/src/payload.rs) already model ZeroKMS - lock-context lists, optional decryption policies and unverified context. - They are ZeroKMS-specific abstractions used for provider injection/testing, - not yet a neutral contract for arbitrary KMS backends. Returned key types - also reflect the existing ZeroKMS/recipher integration. -- [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) - accepts `.withLockContext({ identityClaim: ["sub"] })`. Its documented - behavior is server resolution of claim values from the authenticating - principal. Encrypt/decrypt operations pass that selector to protect-ffi; - audit metadata travels separately as `unverifiedContext`. -- The pinned `zerokms-protocol` 0.12.31 `Context` enum includes `Tag`, - `Value` and `IdentityClaim`. Its `RetrievedKey` currently exposes key - material only; `stack-kms::DataKey` contains IV and key, not the original - authentication tag returned by the proposed refinement flow. Supporting - that flow remains a protocol/client response integration requirement, - consistent with the user's earlier explanation of intended behavior. - -## 1. Plaintext-derived keyset selection - -Add a key-selection declaration to the existing target definition, parallel -to context/scope source selectors. A useful minimal portable form is an -explicit ID or name selector over a plaintext field: + now uses one selected keyset for every key it generates. Dispatch groups + key retrievals by their stored keyset IDs. Dispatch builds the payloads + for generating and retrieving keys without lock context, and passes no + unverified request context. +- The [key source traits](../../packages/stack-kms/src/key_source.rs) and + the [payloads](../../packages/stack-kms/src/payload.rs) already model + lists of ZeroKMS lock context, optional decryption policies, and + unverified context. These traits are abstractions specific to ZeroKMS, + used to inject a provider and for testing. They are not yet a neutral + contract for any KMS backend. The returned key types also follow the + existing integration with ZeroKMS and recipher. +- The [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) + accepts `.withLockContext({ identityClaim: ["sub"] })`. Its + documentation says the server resolves the claim values from the + principal that authenticated. Encrypt and decrypt operations pass that + selector to protect-ffi. Audit metadata is passed separately, as + `unverifiedContext`. +- The `Context` enum in the pinned `zerokms-protocol` 0.12.31 includes + `Tag`, `Value`, and `IdentityClaim`. Its `RetrievedKey` now exposes key + material only. `stack-kms::DataKey` contains the initialisation vector + (IV) and the key. `DataKey` does not contain the original + authentication tag that the proposed refinement flow returns. To support + that flow, the protocol and the client need to return that tag in their + responses. This matches the user's earlier explanation of the intended + behaviour. + +## 1. Keyset selection derived from plaintext + +Recommendation: add a key selection declaration to the existing target +definition, beside the source selectors for context and scope. A useful +minimal portable form is an explicit selector, by ID or by name, over a +plaintext field: ```rust // Proposed attribute fragment. tenant_id contains the actual keyset name. @@ -84,22 +137,30 @@ explicit ID or name selector over a plaintext field: #[stash(keyset(id(field = "keyset_id")))] ``` -Neither selector creates a keyset. Missing, unknown, unauthorized, disabled -or incorrectly typed keysets are errors; never fall back to the default. -A business tenant ID need not be a keyset name/UUID. Where they differ, -configure an explicit tenant-to-key mapping/resolver. Start with direct -selectors and a shared registered resolver contract; do not encode arbitrary -Rust callbacks in dynamic definitions. Pure mappings can return a selector; -network lookup belongs to the execution/provider layer, not the derive. +Neither selector creates a keyset. A missing, unknown, unauthorised, +disabled, or incorrectly typed keyset is an error. Never fall back to the +default keyset. + +A business tenant ID does not have to be a keyset name or UUID. Where the +two differ, configure an explicit mapping or resolver from tenant to key. + +Start with direct selectors and one shared contract for registered +resolvers. Do not encode arbitrary Rust callbacks in dynamic definitions. +A pure mapping can return a selector. A network lookup belongs in the +execution or provider layer, not in the derive. -The existing manually selected form already fits the agreed API: +### Selecting the keyset by hand and by declaration + +The existing form, where the caller selects the keyset by hand, already +fits the agreed API: ```rust let tenant = cipher.keyset(IdentifiedBy::Name(user.tenant_id.clone().into())).await?; let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; ``` -The proposed derived route removes that repetition: +The proposed route, derived from the declaration, removes that +repetition: ```rust // EncryptedUser's definition declares how to select the keyset. @@ -107,93 +168,139 @@ let encrypted: EncryptedUser = user.encrypt_into(&cipher).await?; let encrypted: Vec = users.encrypt_into(&cipher).await?; ``` -`&cipher` is the client/provider executor which can resolve keysets; it is -not an unscoped request that silently encrypts under the default. Extend the -conversion executor capability to accept either a resolved keyset or a -client plus a declared selection recipe. The target's representation and -encryption operation description remain unchanged. Direct -`tenant_keyset.encrypt(value)` retains its accepted meaning. - -Execution order: - -1. Acquire guarded source ownership at request construction as already - required. Validate target/options, input shape, selectors, role/capability - requirements and every key-independent value check before provider I/O. -2. Evaluate the key-selection recipe from the guarded plaintext. Resolve - selectors through the configured provider; pin each resolved identity - for the duration of the request, including index derivation. -3. Partition a batch by resolved key identity, provider/authentication - session and compatible request-level options. Preserve original result - ordering; many tenants do not imply one remote call for the whole batch. -4. Use the selected keyset's index material for its searchable fields and - mint data keys in the selected keyset. Persist the resolved identity, - not a mutable name, in the ciphertext's provider metadata. - -Define conflict behavior: a preselected keyset cannot silently be overridden -by a target's route. If both are provided, validate that the resolved route -matches the bound keyset, or reject the combination explicitly. No implicit -precedence or fallback. Errors requiring name resolution can only occur -after that resolution; the accepted preflight rule covers key-independent -checks, not provider facts unavailable locally. - -For queries there is no plaintext user to inspect. The same target policy -must expose a caller-supplied tenant routing input or use an explicitly -selected keyset. A typed input may deliberately supply both key routing and -search scope; declaration determines both uses. Do not derive a keyset from -the email search value. Searching across multiple keysets is explicit -fan-out with a term per keyset, not one universally comparable query term. - -Decrypt uses the stable key identity stored with the ciphertext, as today. -Do not re-resolve a historical keyset name and substitute its current target. -An expected tenant/keyset restriction can constrain retrieval, with any -historical key mapping supplied explicitly. Merely trusting a tenant field -returned alongside a ciphertext does not authenticate application tenancy; -row binding or trusted expected context supplies that separate check. -Likewise, choosing a keyset from caller-controlled plaintext is routing, -not permission to use every tenant keyset accessible to a service account. +`&cipher` is the executor for the client and provider. That executor can +resolve keysets. `&cipher` is not a request without a scope that silently +encrypts under the default keyset. + +Extend the capability of the conversion executor so that it accepts +either a resolved keyset, or a client with a declared selection recipe. +The target's representation, and the description of its encryption +operation, do not change. A direct `tenant_keyset.encrypt(value)` keeps +its accepted meaning. + +### Execution order + +1. When the request is built, take guarded ownership of the source, as + already required. Before any provider I/O, validate: + - the target and its options; + - the input shape; + - the selectors; + - the role and capability requirements; + - every check on values that does not need a key. +2. Evaluate the key selection recipe on the guarded plaintext. Resolve the + selectors through the configured provider. Keep each resolved identity + fixed for the whole request, including index derivation. +3. Split a batch into groups by resolved key identity, by provider and + authentication session, and by compatible options at the request level. + Return results in the original order. A batch with many tenants does + not mean one remote call for the whole batch. +4. Use the index material of the selected keyset for its searchable + fields, and generate data keys in the selected keyset. Store the + resolved identity in the ciphertext's provider metadata, not a name + that can change. + +### A keyset selected both beforehand and by the target + +Define the behaviour for this conflict. A target's route must not silently +override a keyset selected beforehand. If both are given, either check +that the resolved route matches the bound keyset, or reject the +combination explicitly. Neither takes precedence implicitly, and there is +no fallback. + +Errors that need name resolution can only happen after that resolution. +The accepted preflight rule covers checks that need no key. The rule does +not cover facts about the provider that are not available locally. + +### Queries + +A query has no plaintext user record to inspect. So the same target +policy must either expose a tenant routing input that the caller supplies, +or use a keyset selected explicitly. One typed input can deliberately +supply both the key route and the search scope. The declaration decides +both uses. + +Do not derive a keyset from the email address being searched for. To +search across several keysets, fan out explicitly, with one query term for +each keyset. No single query term compares across all keysets. + +### Decryption + +Decryption uses the stable key identity stored with the ciphertext, as it +does today. Do not resolve a historical keyset name again and use what it +points to now. An expected tenant or keyset can restrict key retrieval, +with any historical key mapping supplied explicitly. + +Trusting a tenant field returned beside a ciphertext does not authenticate +the application's tenancy. Row binding, or a trusted expected context, +makes that separate check. In the same way, choosing a keyset from +plaintext that the caller controls is routing. That choice does not give +permission to use every tenant keyset a service account can access. ## 2. Vitamin C AsyncCipher -**Packaging follow-up:** the user favors removal. The +Recommendation: do not implement `AsyncCipher` on `StackCipher`. +`AsyncCipher` is the Vitamin C trait that seals and opens the leaves of +one document asynchronously. + +### Follow-up from packaging + +The user favours removing the trait. + +The [packaging recommendation](2026-10-25-stack-encrypt-packaging.md#retire-asynccipher) -now recommends retiring this trait and its application helpers as execution -moves into Stack, rather than adding the optional adapter considered below. -Vitamin C still has helper/test consumers to migrate; no removal has been -implemented. - -The absence of `AsyncCipher` is intentional in the current design, not an -unexplained omission. [cipher.rs](../../packages/stack-encrypt/src/cipher.rs) -documents the reason, and implements the synchronous Vitamin C `Cipher` -through `MintedKeys` (seal) and `BoundKeys` (open). - -Vitamin C 0.8.0 `AsyncCipher` takes leaf plaintext and derived AAD, with -`seal_many`/`open_many` operating on the leaves of one document. Stack's -executor retains the original provider descriptor and combines requests -across fields/documents/records before calling the leaf cipher. Leaf AAD -alone is not a reversible representation of those execution inputs. -Future routed keys and lock policies add more execution metadata rather -than removing the mismatch. The trait also requires `Sync` and `Send` -futures; Stack's provider traits use `MaybeSend` for native/WASM support. - -Recommendation: retain Stack's batch/provider orchestration and Vitamin C's -leaf cryptography. Do not implement `AsyncCipher` on `StackCipher` merely to -claim trait symmetry, encode provider policy secretly into AAD, or lose -cross-record batching. An explicitly configured adapter could implement the -trait for a restricted use case with provider descriptor/key route/policy -already supplied, but it is not required for the public redesign and must -advertise its limitations. A general refactor would need a richer request -contract and evidence that batching/metadata/WASM behavior remain intact. - -There is a smaller cleanup opportunity: each synchronous adapter currently -implements an unused half of `Cipher` by returning an error. Consider -separate seal/open capabilities or an internal adapter improvement during -the final dependency/packaging pass. Do not conflate that with requiring the -public Stack client to implement the async trait. +now recommends retiring this trait and its application helpers as +execution moves into Stack. That replaces the optional adapter considered +below. Vitamin C still has helpers and tests that use the trait. Those +helpers and tests need to be migrated. Nothing has been removed yet. + +### Why Stack leaves out AsyncCipher + +Stack leaves out `AsyncCipher` on purpose. The current design does not +omit it by accident. +[cipher.rs](../../packages/stack-encrypt/src/cipher.rs) documents the +reason. That file implements the synchronous Vitamin C `Cipher` trait +through two types: `MintedKeys` seals, and `BoundKeys` opens. + +In Vitamin C 0.8.0, `AsyncCipher` takes the plaintext of a leaf and its +derived AAD. Its `seal_many` and `open_many` work on the leaves of one +document. The trait does not match Stack in three ways: + +- Stack's executor keeps the original provider descriptor. The executor + combines requests across fields, documents, and records before it calls + the leaf cipher. The AAD of a leaf alone cannot be turned back into + those execution inputs. +- Future routed keys and lock policies add more execution metadata. They + do not remove the mismatch. +- The trait requires `Sync` and `Send` futures. To support both native + and WASM builds, Stack's provider traits use `MaybeSend`. + +### Original recommendation on AsyncCipher + +Keep Stack's orchestration of batches and providers, and Vitamin C's leaf +cryptography. Do not implement `AsyncCipher` on `StackCipher` only so that +the traits match. Do not encode provider policy secretly into AAD. Do not +lose batching across records. + +An adapter configured explicitly could implement the trait for a +restricted use, with the provider descriptor, key route, and policy +already supplied. The public redesign does not need that adapter. If +built, the adapter must state its limits. A general refactor would need a +richer request contract. It would also need evidence that batching, +metadata, and WASM behaviour still work. + +### A smaller clean-up + +Each synchronous adapter now implements the half of `Cipher` that it does +not use, by returning an error. During the final pass on dependencies and +packaging, consider separate capabilities for sealing and opening, or an +improvement to the internal adapter. Keep this clean-up separate from the +question of whether the public Stack client implements the async trait. ## 3. Lock context as an optional provider policy -Use a dedicated, typed ZeroKMS policy input, not the ordinary -`encrypt_with_context` argument and not a `SearchScope`. Proposed spelling: +Recommendation: pass lock context as its own typed ZeroKMS policy input. +Do not pass it as the ordinary `encrypt_with_context` argument, or as a +`SearchScope`. Proposed spelling: ```rust use stack_encrypt::providers::zerokms::{LockContext, LockContextExt}; @@ -207,95 +314,132 @@ let user = User::decrypt_from(encrypted, &tenant_keyset) .await?; ``` -This is illustrative module/method syntax pending the naming pass. The -operation builder retains owned plaintext guards while policy is attached. -The extension is present for a provider with the required capability; a -runtime-erased provider checks the equivalent capability before key I/O. +The module and method names illustrate the idea until the naming pass. +The operation builder keeps its owned plaintext guards while the policy is +attached. The extension is present for a provider with the required +capability. A provider whose type is erased at runtime checks the same +capability before key I/O. + Ordinary encryption without this optional policy works on providers that -do not support it. A request/definition requiring it must fail on those -providers, never discard it or substitute literal AAD. - -The input selects a claim **name**. ZeroKMS verifies the authenticating -principal and resolves its claim value; do not decode an unverified JWT in -the application and turn its `sub` into a caller assertion. Authentication -belongs to the configured provider/session. Use explicit, immutable -request/session binding when identities differ concurrently; avoid mutating -a global client's identity between operations. - -Support explicit literal tag/value items through the same provider policy, -while distinguishing those caller assertions from principal-derived claims. -Missing claims or provider policy failures are errors with no retry without -the lock. Forward policy on generate and retrieve. A required policy can -live in the target definition so callers cannot accidentally omit it; an -operation may supply an undeclared optional policy, but cannot replace or -weaken a definition-required one. Define composition/conflict rules rather -than last-call-wins behavior. Policy selectors may need to be available on -reads; define their storage or required caller input without serializing -JWTs, credentials or claim values resolved by the provider. - -Keep audit metadata (`UnverifiedContext`) separate: it is not trusted -identity, ciphertext context or a policy assertion. Preserve distinctions -between lock context and the protocol's separate decryption-policy feature; -do not silently reinterpret one as the other. - -Index/query derivation may use an already-loaded deterministic index key -without generating/retrieving a data key. Therefore lock context on those -data-key endpoints does not by itself restrict who can derive search terms -or view query results. If search authorization is required, define that -capability separately; do not infer that the TypeScript query method's -acceptance of `withLockContext` proves the new index path is protected. -Do not implicitly make search scope principal-specific to emulate a lock. - -Batching must preserve per-item lock inputs and separate incompatible -principal/provider sessions. If a provider supports per-payload policy, -retain it in each payload; if an option is request-wide, partition by it. -Any future data-key cache must respect policy/principal checking semantics; -a cache hit must not silently replace a required provider authorization or -audit event. This is a design requirement, not a claim of an existing cache -vulnerability. Do not broaden the current keyset/index-key cache into an -unqualified data-key authorization cache. - -## Provider-neutral base and progressive capabilities - -The public value/target model needs extensions, not replacement. The larger -refactor is the provider boundary, which currently imports ZeroKMS payloads, -UUID keyset IDs, policies, errors and returned key types directly. - -Recommended capability split (working names): +do not support it. On those providers, a request or definition that +requires lock context must fail. The request must never discard the +policy, or substitute literal AAD. + +### Claims come from the provider + +The input selects the **name** of a claim. ZeroKMS verifies the principal +that authenticated, and resolves the value of its claim. Do not decode an +unverified JWT in the application and turn its `sub` into an assertion by +the caller. Authentication belongs to the configured provider or session. + +When different identities are active at the same time, bind each request +or session to its identity explicitly and immutably. Do not change the +identity of a global client between operations. + +### Rules for lock context + +- Support explicit literal tag and value items through the same provider + policy. Keep these assertions by the caller distinct from claims derived + from the principal. +- A missing claim, or a failure of the provider policy, is an error. Do + not retry without the lock. +- Send the policy on both key generation and key retrieval. +- A required policy can be part of the target definition, so callers + cannot leave it out by mistake. An operation can supply an optional + policy that the definition does not declare. An operation cannot replace + or weaken a policy that the definition requires. +- Define rules for combining policies and for conflicts between them. Do + not let the last call win. +- Reads may need the policy selectors. Define how the selectors are + stored, or which input the caller must supply. Do not serialise JWTs, + credentials, or claim values that the provider resolved. +- Keep audit metadata (`UnverifiedContext`) separate. Audit metadata is + not a trusted identity, ciphertext context, or a policy assertion. +- Keep lock context distinct from the protocol's separate decryption + policy feature. Do not silently treat one as the other. + +### Lock context does not protect search + +Deriving index and query terms can use a deterministic index key that is +already loaded, without generating or retrieving a data key. So lock +context on the data key endpoints does not by itself restrict who can +derive search terms or see query results. + +If search needs authorisation, define that capability separately. The +TypeScript query method accepts `withLockContext`. That does not prove +that the new index path is protected. Do not imitate a lock by +making search scope specific to a principal implicitly. + +### Batching and caching with lock context + +Batching must keep the lock inputs of each item. Batching must also keep +incompatible principal sessions and provider sessions apart. If a provider +supports a policy on each payload, keep the policy in each payload. If an +option applies to the whole request, split the batch by that option. + +Any future cache of data keys must respect the semantics of checking +policy and principal. A cache hit must not silently replace a provider +authorisation or audit event that is required. This is a design +requirement. It does not claim that an existing cache has a +vulnerability. Do not widen the current cache of keysets and index keys +into a data key cache that authorises without those checks. + +## A neutral provider base with optional capabilities + +The public model of values and targets needs extending, not replacing. The +larger refactor is at the provider boundary. That boundary now imports +ZeroKMS payloads, UUID keyset IDs, policies, errors, and returned key +types directly. + +The recommended split of capabilities, with working names: | Capability | Required for | | --- | --- | -| Data-key generation/recovery and protected key material | Ordinary opaque and structural encryption/decryption | -| Index-key support | Searchable targets and query derivation | -| Keyset resolution by ID/name | ZeroKMS-style keyset selectors; not every provider | -| Lock-context support, including claim selectors | ZeroKMS policy extension; not every provider | - -Define a resolved provider key reference and versioned opaque recovery -metadata. Do not force every provider to supply a ZeroKMS UUID, IV/tag or -recipher response. A caller selects/configures a provider; ciphertext -metadata identifies a registered provider/key format, never arbitrary -network endpoints or credentials to execute. Keep ZeroKMS's key tag simple -to extract within its documented envelope profile for Rust/SQL consumers. -Other providers' metadata need not invent such a tag. Stable blob payload -encoding is independent of these provider-specific envelope profiles. - -Future AWS KMS/Vault integrations implement the minimal capabilities they -offer. They need not implement lock context, named keysets or searchable -index-key management merely to encrypt an opaque value. The application -can configure its own tenant-to-provider-key mapping without representing -it as a native ZeroKMS keyset. A searchable target requires an explicitly -supported index-key strategy; the engine must not assume every data-key -backend supplies deterministic search keys automatically. - -Refined-tag support also needs distinct stored/request tag and returned -original authentication-tag input. Preserve the user's intended ZeroKMS -flow when designing neutral responses; do not require that every provider -implements refinable tags or that a generic KMS returns that concept. +| Data key generation and recovery, and protected key material | Ordinary opaque and structural encryption and decryption | +| Index key support | Searchable targets and query derivation | +| Keyset resolution by ID or name | ZeroKMS-style keyset selectors. Not every provider offers it | +| Lock context support, including claim selectors | The ZeroKMS policy extension. Not every provider offers it | + +### Key references and metadata + +Define a reference to a resolved provider key, and versioned opaque +metadata for recovering the key. Do not force every provider to supply a +ZeroKMS UUID, an IV with its tag, or a recipher response. + +A caller selects and configures a provider. Ciphertext metadata identifies +a registered provider and key format. The metadata never names arbitrary +network endpoints or credentials to execute. + +Keep ZeroKMS's key tag simple for Rust and SQL consumers to extract, +within its documented envelope profile. Other providers' metadata does not +need to invent such a tag. The stable encoding of blob payloads does not +depend on these envelope profiles of each provider. + +### Future providers + +Future integrations with AWS KMS or Vault implement the minimal +capabilities they offer. To encrypt an opaque value, they do not need lock +context, named keysets, or management of index keys for search. The +application can configure its own mapping from tenant to provider key, +without representing it as a native ZeroKMS keyset. + +A searchable target requires an index key strategy that is explicitly +supported. The engine must not assume that every data key backend supplies +deterministic search keys automatically. + +### Refined tags + +Support for refined tags also needs two distinct inputs: the tag stored +and sent in the request, and the original authentication tag returned. +When designing neutral responses, keep the ZeroKMS flow that the user +intended. Do not require every provider to implement refinable tags, or a +generic KMS to return that concept. ## Dynamic interoperability -Provider capabilities join target capabilities in discovery and definition -validation. Example provider-specific extension (proposed logical schema): +Discovery and definition validation cover provider capabilities as well as +target capabilities. This example shows a proposed logical schema for an +extension specific to one provider: ```json { @@ -311,41 +455,62 @@ validation. Example provider-specific extension (proposed logical schema): } ``` -The field's declared type must admit the chosen selector; missing fields -are definition/input errors before provider resolution. Query routing -declares an equivalent caller input because it has no source record. -Providers and extensions are registered, versioned capabilities. Unsupported -required key selection/policy is an error during loading or binding to an -execution provider, before key I/O. Do not accept arbitrary extension maps -and ignore unfamiliar entries. Auth credentials stay in provider setup, not -in this declaration or persisted policy metadata. - -Go generation uses named selectors/policies and the same validated dynamic -definition; provider-specific methods/capabilities are explicit. The -interoperability specification must include mixed-keyset batches, policy -inputs, capability refusals, key identity persistence and the distinctions -between authenticated claims and caller-supplied audit metadata. +Rules for the dynamic definition: + +- The field's declared type must allow the chosen selector. A missing field + is a definition or input error, before the provider resolves anything. +- Query routing has no source record, so it declares an equivalent input + that the caller supplies. +- Providers and extensions are registered, versioned capabilities. +- Unsupported required key selection or policy is an error when the + definition is loaded or bound to an execution provider, before key I/O. +- Do not accept arbitrary extension maps, and do not ignore unfamiliar + entries. +- Authentication credentials stay in the provider setup. They are not in + this declaration or in stored policy metadata. + +Go code generation uses named selectors and policies, and the same +validated dynamic definition. Methods and capabilities specific to a +provider are explicit. The interoperability specification must cover: + +- batches with several keysets; +- policy inputs; +- refusals because a capability is missing; +- how key identities are stored; +- the difference between authenticated claims and audit metadata that the + caller supplies. ## Implementation and acceptance work -1. Specify neutral key references/recovery metadata and optional capability - traits without altering the accepted encrypt/decrypt conversion roles. -2. Add definition selectors and client-scoped routed execution. Cover ID - versus name, explicit mappings, conflict errors and no default fallback. -3. Extend request planning/grouping to carry resolved route and provider - policy. Preserve per-record ordering and guard lifetimes across async - resolution/cancellation. -4. Wire ZeroKMS lock context through both generate/retrieve, including - literal and principal-claim forms, and test required-policy enforcement. -5. Verify mixed-tenant batches, matching write/query keys, historical ID - decrypt, wrong expected keysets, missing claims, unsupported-provider - failures before I/O, and concurrent principal isolation. A minimal fake - provider without keysets/locks/index keys must still encrypt opaque values. -6. Resolve refined-tag response support with the ZeroKMS client/protocol. - Add integration fixtures without assuming the pinned API already returns - the original tag. -7. Defer the `AsyncCipher` adapter/trait cleanup decision to the packaging - pass; preserve the documented batching and WASM constraints meanwhile. - -No implementation changes, protocol guarantees for future providers, or -completed security audit are claimed by this assessment. +1. Specify neutral key references, recovery metadata, and optional + capability traits. Do not change the accepted conversion roles for + encryption and decryption. +2. Add selectors to definitions, and routed execution through the client. + Cover selection by ID compared with by name, explicit mappings, + conflict errors, and no fallback to the default. +3. Extend request planning and grouping to include the resolved route and + the provider policy. Across asynchronous resolution and cancellation, + keep the order of records. Keep the lifetimes of guards too. +4. Pass ZeroKMS lock context through both key generation and key + retrieval, in both the literal form and the principal claim form. Test + that a required policy is enforced. +5. Test these cases: + - batches with several tenants; + - matching keys for writes and queries; + - decryption by historical ID; + - wrong expected keysets; + - missing claims; + - failures for unsupported providers, before I/O; + - isolation between principals active at the same time. + + A minimal fake provider without keysets, locks, or index keys must + still encrypt opaque values. +6. Settle support for refined tags in responses with the ZeroKMS client + and protocol. Add integration fixtures without assuming the pinned API + already returns the original tag. +7. Leave the decision on the `AsyncCipher` adapter and the trait clean-up + to the packaging pass. Until then, keep the documented constraints on + batching and WASM. + +This assessment does not claim any implementation changes, any protocol +guarantees for future providers, or a completed security audit. From ab5fea0e9ad48ff6470eb9a36395f7b0ae730c07 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 11:57:00 +1100 Subject: [PATCH 06/42] docs: reconcile the Stack Encrypt plans with next on 7 November 2026 Check every plan item against the work merged into next since 27 October 2026, and record each one as done, partly done, or open. Record three decisions from 7 November 2026: the stored format written by Go v0.1.0-rc.3 must stay compatible, context and AAD have one definition, and parameters that take a context are named `context`. Replace "profile" with plain words except for conformance profiles and Vitamin C's storage profile. --- .../2026-10-24-stack-encrypt-api-plan.md | 569 +++++++++++++++--- .../2026-10-24-stack-encrypt-api-review.md | 282 ++++++--- ...26-10-24-stack-encrypt-context-examples.md | 46 +- ...026-10-24-stack-encrypt-public-examples.md | 87 +-- ...026-10-25-stack-encrypt-json-operations.md | 69 ++- .../2026-10-25-stack-encrypt-packaging.md | 57 +- ...-25-stack-encrypt-provider-capabilities.md | 34 +- 7 files changed, 904 insertions(+), 240 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 5188a4bbf..2131c7167 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1,7 +1,8 @@ # Stack Encrypt API implementation plan -First written on 24 October 2026. Checked against the merged code on 27 -October 2026. +First written on 24 October 2026. Checked against the merged code on 7 +November 2026, at `next` commit `e4990e49b`. The previous check was on 27 +October 2026, at `ffeed7adb`. Status: proposed implementation direction. This plan develops the [API review log](2026-10-24-stack-encrypt-api-review.md), the [public @@ -16,15 +17,16 @@ show proposed APIs, not working code in the current release. ## Summary -- The Go JSON foundations and Go opaque storage are merged. The customer - work that remains is conformance to the contract, closing the remaining - gaps in reads and configuration, and release readiness. See [Status on 27 - October 2026](#status-on-27-october-2026). +- The Go JSON foundations and Go opaque storage are merged. Since 27 + October 2026, three more things have merged. They are the owned Rust API for single values and collections, a conformance corpus for Rust and Go, and two new JSON queries. See [Status on 7 November + 2026](#status-on-7-november-2026). +- The customer work that remains is listed in [Remaining customer + work](#remaining-customer-work). It includes opening a child bound to its + parent, configurable Go match options, the gaps in the specification, and + release readiness. - The Go customer release comes first. The final Rust facade and the crate moves come after it, and do not hold it up. -- Until the customer release, nothing needs to stay compatible with the - prototype API or its stored data. After it, the Go interface and the - stored data must stay compatible. +- On 7 November 2026, the user decided that the stored format must not break from now on. The Go customer has started building on Go `v0.1.0-rc.3`. The Rust and Go code and APIs can still change in breaking ways. See [Stored format must stay compatible from 7 November 2026](#stored-format-must-stay-compatible-from-7-november-2026). - The [decision checkpoint](#decision-checkpoint-on-25-october-2026) lists the settled directions. The [remaining questions](#remaining-questions) table lists what is still open. @@ -33,14 +35,16 @@ show proposed APIs, not working code in the current release. | Term | Meaning | | --- | --- | -| AAD (additional authenticated data) | Data that encryption binds to a ciphertext without encrypting it. Decryption fails unless the same AAD is supplied again | +| AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | | AEAD (authenticated encryption with associated data) | The cipher that seals each value and authenticates its AAD | | Blob | A value that `OpaqueBlob` encodes as a whole and seals as one ciphertext | | Bloom filter | The bit array that the match index stores. Each term sets some bits, so a query can test whether a value may contain the term | | Cascaded tags | Each JSON container's authentication tag covers the tags of its children. #1379 added them | -| Column and keyset profile | The released configuration. It uses the column identity and the keyset selection, and accepts no row binding or search scope input | +| Column and keyset configuration | The released configuration. A JSON document's context is its column, and its keys come from the selected keyset. It accepts no row binding or search scope input | | Configured target | A target together with typed options, such as a JSON target with a list of ordered paths | -| Context | Values bound into a ciphertext, such as a column name or a row ID. Decryption fails unless the same context is supplied again | +| Conformance corpus | The shared test cases and requirement lists in `conformance/stack-encrypt`. Native Rust and the Go guest both run them | +| Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Each language runs the same cases. The files are in `conformance/stack-encrypt/v1/` | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | | Dynamic interface | The interface that takes declarations and requests as data instead of Rust types. Go uses it through the guest | | EQL (Encrypt Query Language) | The SQL types and functions in `packages/eql` that store and query encrypted values in PostgreSQL | | Facade | The proposed public Rust API of `stack-encrypt`: its traits, derives, and convenience methods | @@ -48,36 +52,138 @@ show proposed APIs, not working code in the current release. | Guest | The Rust engine compiled to WebAssembly, which the Go SDK runs | | Key I/O | Any request to ZeroKMS to resolve, load, or derive keys | | Keyset | A set of keys in ZeroKMS, the CipherStash key management service (KMS) | +| OPE (order-preserving encryption) | Encryption whose ciphertexts keep the order of the plaintexts | | Open and seal | To seal a value is to encrypt and authenticate it. To open it is to authenticate and decrypt it | | ORE (order-revealing encryption) | Index terms that reveal the order of the plaintexts, for range queries and sorting | -| OPE (order-preserving encryption) | Encryption whose ciphertexts keep the order of the plaintexts | +| Owned API | The new Rust conversion API, which consumes its plaintext input: `keyset.encrypt`, `EncryptInto`, and `DecryptFrom`, over `EncryptTarget` and `DecryptSource` declarations. #1404 added it | | Passthrough field | A field copied unencrypted from the plaintext type to the encrypted output | | Plan | The lower-level description of which fields a conversion encrypts, indexes, or passes through. Derives and dynamic declarations compile to plans | -| Post-cascade profile | The JSON storage and search format as it is after #1379 added cascaded tags and randomised order. This is the format to freeze | +| Post-cascade format | The JSON storage and search format after #1379 added cascaded tags and randomised order. Go rc.3 writes it, so it must stay compatible | | Pseudo-version | A Go module version that names a commit instead of a released tag | | Reads that Rust verifies against the original request | Go prepared reads. When the rows are opened, Rust derives the request again and checks every row against it | +| Refined tag | A proposed tag that Stack Encrypt would derive from the ZeroKMS key tag with an HMAC, and store in place of the original tag. See [Typed encodings and extractable ZeroKMS tags](#typed-encodings-and-extractable-zerokms-tags) | | Row binding | Context that ties a ciphertext to one row, such as the row ID, so that it opens only for that row. Also called ciphertext binding | | Search scope | The part of the context that also goes into index keys and search terms, such as a tenant ID. One query matches every row in the scope | -| Target | A type or dynamic name that says how a value is encrypted and stored, such as `JsonSearch` or `TextEq` | +| Storage profile | Vitamin C's versioned byte format for an opaque value that is stored. Vitamin C #520 added it | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [Stored format must stay compatible from 7 November 2026](#stored-format-must-stay-compatible-from-7-november-2026) | +| Structural API | The earlier Rust API over borrowed input and the `EncryptFrom` and `DecryptInto` declarations. Its convenience methods are now in `stack_encrypt::target::legacy` | | Target definition | The resolved, immutable form of a target's declaration: options, prepared indexes, context rules, and operation metadata | +| Target | A type or dynamic name that says how a value is encrypted and stored, such as `JsonSearch` or `TextEq` | | Typestate | A builder whose type changes with each call, so the compiler refuses to run an incomplete request | | Vitamin C | CipherStash's cryptography library, `vitaminc`. Its codec encodes values as binary frames | -## Status on 27 October 2026 +## Context and AAD + +The context is the data that a ciphertext is tied to, such as a column +name or a row ID. The field's declaration supplies part of the context, +such as `users/age`. The caller can supply the rest, such as a row ID. + +Stack Encrypt uses the context in three ways: + +1. **To derive the data key.** Stack Encrypt sends the context to ZeroKMS + as a text descriptor, such as `users/age`. ZeroKMS needs the same + descriptor to derive the key again. +2. **To build the AAD of each ciphertext.** The AAD is data that + encryption checks without encrypting it. Stack Encrypt builds it from + the context and the place of the value in its record. The AAD also + includes the format version, the keyset ID, and the key tag. +3. **To derive the search terms** for the field. + +A different context gives a different key and a different AAD, so +decryption fails. + +A parameter that takes the caller's context is named `context`. The name +`aad` is kept for a value that is an AAD, such as the derived AAD of one +leaf. Some public methods do not follow this rule yet. See [Name context +parameters `context`](#name-context-parameters-context). + +Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` +describes the descriptor and the AAD. The module documentation of +`packages/stack-encrypt/src/target/context.rs` describes the search terms. + +## Status on 7 November 2026 + +On 7 November 2026, this plan was checked against merged `next` at commit +`e4990e49b`. Each pull request description was read. Where the description +did not settle a status, the code on `next` was read too. No test suite was +run for this check. + +### Merged between 27 October and 7 November 2026 + +| Pull request | What it delivers | Plan items it affects | +| --- | --- | --- | +| #1404 | The owned API for one value, with the `Ciphertext` type and explicit context inputs. Renames the structural keyset methods to `encrypt_tree` and `decrypt_tree` | Requirements; decrypt interface; public traits; work areas A, B, and C | +| #1491 | Repeated Rust context extensions append in order, as in Go. Adds context conformance cases | Work area B; the dynamic specification | +| #1501 | Owned `Vec` collections, encrypted as one batch of key requests. Moves the Rust crates to Vitamin C 0.9 | Requirements; work areas C and D; cleanup | +| #1460, #1461, #1462, #1469 | Conformance profiles for scalars, declarations, runtime values, stored reads, value kinds, and query operators, run by native Rust and the Go guest | Implementation step 1; the dynamic specification | +| #1492 | The `eql-json-v1` conformance profile, run in two required CI jobs: a fast job without a database, and a job against PostgreSQL | Implementation steps 1 and 3; the JSON contract | +| #1493 | The `lifecycle-v1` conformance profile for ownership release, cancellation, cleanup, and key I/O | Cleanup acceptance work | +| #1444, #1445, #1446, #1451, #1452 | Two new JSON queries in the engine, EQL, and Go. #1444 documents a limit of native `jsonb` functions | The JSON contract | +| #1500 | Go documentation and a changelog for `v0.1.0-rc.3`, which was tagged on 2 November 2026 | Release readiness | +| #1403 | Fixes from the review of the release candidate. Bumps the crate versions that `stack-encrypt` 0.3.0 needs | Release readiness | + +### Remaining customer work + +The conformance corpus is now merged, so it is no longer the first item. +The customer work that remains: + +1. **Finish the specification around the corpus.** The conformance profiles are marked experimental. That means they test parts of the contract, but do not certify a whole SDK. The contract is not frozen. JSON Schema files, + protocol version identifiers, a release compatibility matrix, and fresh + cross-language exchange of JSON values are not done. Python and + TypeScript runners are not done. +2. **Open a child bound to its parent** (J17 in the JSON contract). No + merged pull request implements it. +3. **Configurable Go match options** (#1112), and the tokeniser and filter + pipeline. No merged pull request implements them. +4. **Freeze the release candidate.** Freeze the post-cascade JSON vectors + and the matching EQL bundle. The new conformance jobs run in CI, but + they do not freeze the stored format. +5. **Replace the temporary Go pins of Vitamin C** before a final Go + release. Go `v0.1.0-rc.3` is a prerelease, so it can keep them. + +### Differences between merged code and this plan + +These differences are recorded here. This plan's decisions are not changed +to match them. + +- **Byte vectors.** This plan recommends that `Vec` encrypts byte by byte into `Vec`. It recommends `OpaqueBlob>` for a whole vector. Pull request #1501 does the opposite. In #1501, `Vec` and `[u8; N]` encrypt into one `Ciphertext`. In #1501, `u8` is not an `OpaqueEncode` type. The reason given in #1501 is that Vitamin C has no single value for `u8`. Vitamin C could not read an element that only Stack could encode. + The user needs to decide whether to accept this. +- **The structural API stays beside the owned API.** This plan says to + remove replaced APIs directly. On `next`, `EncryptFrom`, `DecryptInto`, + the derives with those names, and the public `encrypt_as` remain. Their + borrowed convenience methods moved to `stack_encrypt::target::legacy`. + Issue #1488 tracks their removal. +- **Context composition.** #1491 makes repeated extensions append. That matches this plan's order: the base, then each extension. The dynamic + grammar and Go `Extend` accept only scalar extension parts, and refuse a + grouped part. Go accepts an empty text extension when the base is + nonempty, and Rust's `NonEmpty` refuses it. The context contract records + both differences as part of the supported API. +- **Compatibility of stored data.** Pull request #1491 changes which context repeated Rust extensions produce. The #1491 changelog explains how to read old records. Data written by Go `v0.1.0-rc.3` is not affected. The [stored-format section](#pull-request-1491-and-data-written-by-go-rc3) gives the evidence. +- **Vitamin C 0.9 and the order schemes.** Pull request #1501 moves to Vitamin C 0.9, `ore-rs` 0.11.0, and `clww-ore` 0.7.0. The #1501 description says that neither order-scheme release changes a stored order term. This check did not test that claim against rc.3 data. + +### Work in progress in open pull requests + +These pull requests target `next` and are not merged. Nothing in them is +counted as done in this plan. + +- #1503 adds `OpaqueBlob`. The wrapper encodes a value with Vitamin C's storage profile, and seals it as one `Ciphertext`. +- #1504 adds the `EncryptTarget` and `DecryptSource` derives, the + `#[stash(passthrough)]` attribute, and the context inputs + `ContextExtension` and `ExpectedContext`. It turns the public examples + for mixed and searchable records into tests. ### Merged Go work -On 27 October 2026, this plan was checked against merged `next` at commit -`ffeed7adb3f7d8739cda53261046147852bffc82`. This Go work is merged: +The check on 27 October 2026, against `next` at commit `ffeed7adb`, found +this Go work merged: - the Go JSON catalog, configured targets, queries, reads that Rust verifies against the original request, typed documents, and executable documentation (#1372, #1373, #1378, and #1402); - Go adoption of opaque storage (#1369 and #1370). -The active customer work is now conformance to the contract, closing the -remaining gaps in reads and configuration, and release readiness. That work -does not implement those foundations again. See the refreshed [Go JSON +The remaining customer work does not implement those foundations again. +See [Remaining customer work](#remaining-customer-work), the refreshed [Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) and the [JSON operations inventory](2026-10-25-stack-encrypt-json-operations.md). @@ -87,7 +193,7 @@ inventory](2026-10-25-stack-encrypt-json-operations.md). Pull request #1379 added cascaded tags, and randomised the order of stored entries and query projections. A wildcard `->` now selects an unspecified element, so old SQL recipes that relied on the first element in document -order are invalid. Freeze the post-cascade profile. Compare query semantics +order are invalid. Freeze the post-cascade format. Compare query semantics in tests, not the randomised byte order. Pull request #1397 implements short whole-word terms in the shared Bloom @@ -100,9 +206,9 @@ does not accept a stronger privacy claim for those terms. Go's Vitamin C storage modules are temporarily pinned to pseudo-versions at commit `06568ba93bd9`. The guard for a final release requires each pseudo-version to be replaced with a released module tag. Prereleases can -keep pseudo-versions. Rust remains on Vitamin C 0.8.0 from crates.io. -Schedule the crate moves and the final Rust facade after the customer -release. +keep pseudo-versions, and Go `v0.1.0-rc.3` still uses them. #1501 moved +the Rust workspaces from Vitamin C 0.8.0 to 0.9 from crates.io. Schedule +the crate moves and the final Rust facade after the customer release. ### Decisions made on 25 October 2026 @@ -151,9 +257,27 @@ recommendation](2026-10-25-stack-encrypt-packaging.md) defines: - coordinated migration work. The recommendation is a proposal for review. Nobody has accepted the -renames. None of the proposal is implemented. +renames. #1404 implements some of the names in its glossary that were +already agreed: `Ciphertext`, `OpaqueEncode`, `OpaqueDecode`, +`EncryptTarget`, and `DecryptSource`. No crate moves or renames are +implemented. + +### Stored format must stay compatible from 7 November 2026 + +**Decision of 7 November 2026.** The user decided: "The stored data must not break as of now. The Go customer has already started implementation with rc.3. We can make breaking changes to the code/API but not the stored format." + +The stored format is what is written to the database. It includes ciphertexts with their envelopes and headers. It includes index and search terms. It includes JSON tree entries. + +The baseline is the Go module `github.com/cipherstash/stack/languages/golang` at tag `languages/golang/v0.1.0-rc.3`. That tag is commit `114e08448`, built on 2 November 2026 from `next` at `929451b49`. The baseline also includes the EQL v4 SQL bundle, `eql-v4-install.sql`, attached to that release. The rc.3 changelog, from #1500, describes the release. + +This decision replaces the decision of 25 October 2026 for stored data. For the code and the APIs, the earlier decision still applies: Rust and Go APIs can change without compatibility aliases or deprecation periods. Any change that would alter the stored format must do one of two things: + +- keep data written by rc.3 readable and searchable; or +- add a new, versioned format beside the rc.3 format, and keep reading the rc.3 format. + +This plan does not choose between the two for any item. The open items are listed in [Work that the stored-format decision constrains](#work-that-the-stored-format-decision-constrains). -### No compatibility with the prototype +#### History: the decision of 25 October 2026 On 25 October 2026, the user clarified the compatibility scope. This API has existed only as an early prototype. Do not preserve its APIs, stored @@ -168,9 +292,28 @@ merged Go implementation does still keep a reader for the legacy JSON encoding of opaque values. Record that reader as existing behaviour, not as a new requirement to build machinery for migrating prototype data. -This exemption ends for the customer profiles when they are released. After -that, later refactors of the Rust API and packages must keep the Go -interface, and keep stored data and search compatible. +The 25 October decision said that this exemption would end when the customer's configurations were released. The decision of 7 November 2026 ends it for stored data now, before a final release. The Go interface is not frozen by that decision. + +#### Work that the stored-format decision constrains + +Each item below would change the stored format if it were built as first planned. Each must now keep rc.3 data readable and searchable, or add a new versioned format beside it. Which option to take for each is an open question. + +| Planned work | How it touches the stored format | +| --- | --- | +| Freezing the post-cascade format | rc.3 already writes it. The frozen format must be the rc.3 one, or a new version beside it | +| The refined-tag proposal | It changes the tag stored in the envelope, and what the AAD binds | +| Separating row binding and search scope | It changes the AAD and the context that index terms are derived under | +| The `OpaqueBlob` codec, including the Borsh fallback | Go rc.3 already writes opaque structs with Vitamin C's storage profile. A Rust codec must read those bytes | +| Encoded ciphertext field targets (Base64 and Base85) | They change how a ciphertext is written as text in a column | +| The `SealedValue` envelope layout and the JSON header version | The key-tag offsets and the header version 1 are what rc.3 wrote. The plan earlier said the offsets were not a commitment | +| Byte vectors | Whether Go rc.3 writes a byte string as one value, and so whether the byte-vector decision affects stored data, was not checked | +| Configurable match options and the tokeniser pipeline | New options must not change the terms of columns written with rc.3 defaults | + +#### Pull request #1491 and data written by Go rc.3 + +Pull request #1491 does not change data that Go rc.3 can write. It changes only the Rust typed chain's `extend` method in `plan/chain.rs`. In `dynamic/record.rs`, the path that the Go guest uses, #1491 only adds a test. Go's `Extend` already appended parts in order. + +Go rc.3 was built from `929451b49`. That commit is earlier than the merge of #1491. The data that #1491 affects is data written by repeated Rust `.extend` calls before #1491. That data is Rust-only. ## Requirements established in the review @@ -185,8 +328,7 @@ interface, and keep stored data and search compatible. - Use `EncryptInto` to batch a collection, including `Vec` into `Vec` without caller context. This division was agreed on 25 October 2026. The current recommendation for byte vectors follows the same - collection rule, and uses `OpaqueBlob` for a whole vector. Compatibility - with the prototype gives no reason to add an exception for raw bytes. + collection rule, and uses `OpaqueBlob` for a whole vector. Compatibility with the prototype API gives no reason to add an exception for raw bytes. - Rust traits and derives are the main static interface. Plans support those declarations, and their lowering to the dynamic interface and FFI. - Design static and dynamic support against the same contract for the engine @@ -206,12 +348,26 @@ interface, and keep stored data and search compatible. checks. Deliberate passthrough fields move to the output instead of being erased. - Keep EQL's contract between the engine and the SQL coherent, and keep the - shared batching and execution behaviour. Prototype representations need - not be preserved. + shared batching and execution behaviour. From 7 November 2026, the stored format of Go rc.3 must be preserved. Other prototype representations need not be. - Investigate the serialisation round trips in the EQL JSON adapter, and replace them with direct typed conversion. Keep the engine and the SQL in agreement. +### Status of these requirements on 7 November 2026 + +| Requirement | Status | What remains | +| --- | --- | --- | +| Keyset-directed `encrypt` and `encrypt_with_context`, with `EncryptInto` kept | Partly done (#1404) | Remove the public `encrypt_as`, which is still on `next` | +| One value gives one opaque ciphertext, and `encrypt` takes no collection | Partly done (#1404) | `OpaqueBlob`. #1503 is in progress | +| `EncryptInto` batches a collection without caller context | Done for `Vec` (#1501) | Byte vectors differ from this plan. See [Differences between merged code and this plan](#differences-between-merged-code-and-this-plan) | +| Rust traits and derives are the main static interface | Partly done (#1404) | The derives with the new names. #1504 is in progress | +| Static and dynamic support against one contract, with engine-level interoperability tests | Partly done (#1460, #1461, #1462, #1469, #1491, #1492) | The items in [Remaining customer work](#remaining-customer-work), item 1 | +| Decrypt with and without context, with no invented default | Done for one value (#1404) | Record targets. #1504 is in progress | +| `OpaqueEncode` and `OpaqueDecode`, and consuming `encrypt` and `encrypt_into` | Done for one value and for `Vec` (#1404, #1501) | Records. #1504 is in progress | +| `Zeroize` bound and guarded cleanup | Partly done (#1404, #1493, #1501) | See [Cleanup in target-directed encryption and at the Go boundary](#cleanup-in-target-directed-encryption-and-at-the-go-boundary) | +| Keep EQL's engine and SQL contract coherent | Done for the merged JSON queries. #1492 runs them against PostgreSQL in CI | None listed | +| Direct typed conversion in the EQL JSON adapter | Still open | No merged pull request changes it | + ## Recommended decision: general configured targets Use a general contract: a target plus typed options. Do not add a public @@ -454,13 +610,22 @@ Settle the names of declaration traits and convenience traits together with review issue 5. The public examples use the destination-directed pair. Remove the competing ciphertext method when implementing this interface. +Status on 7 November 2026: partly done. #1404 implements `keyset.decrypt`, +`decrypt_with_context`, `T::decrypt_from`, and `decrypt_from_with_context` +for one `Ciphertext`. #1501 adds them for `Vec`. The owned +`DecryptFrom` has no `decrypt_into` method. The structural +`decrypt_into` still exists in `target::legacy`. The opt-in adapter for +results that are wiped on drop is still open. + ## Recommended public traits and context input capabilities Status: the user accepted this direction on 24 October 2026. Separate the conversion methods from the `EncryptTarget` and `DecryptSource` declarations. Use explicit capabilities for caller context instead of -`Default`. Implementation, complete signatures, and updates to call sites -remain pending. +`Default`. #1404 implements this direction for opaque values, and #1501 +for collections of them. Records, complete signatures, and updates to call +sites remain pending. The status of each point is listed at the end of this +section. This section settles the direction of work areas A and B, later in this plan. Earlier examples keep their historical derive names until they are @@ -498,7 +663,7 @@ always fails. Plaintext visitor traits remain a separate representation layer that Stack owns. Moving or renaming them must keep a single canonical trait implementation, and a consistent wire contract across the new consumers. -Compatibility with the prototype is not needed. Configured-target options +Compatibility with the prototype API is not needed. The stored format must stay compatible. Configured-target options are another associated capability, not context. Declaration methods continue to return operation descriptions that the core @@ -527,7 +692,7 @@ capabilities: | `ExpectedContext` | Allowed | Accepted `C` | Leaving it out rebuilds the stored context, and supplying it also checks that context | `OptionalContext` does not append a synthetic empty prefix, and leaving out -an extension does not append `()`. `ExpectedContext` never extends the AAD. +an extension does not append `()`. `ExpectedContext` never extends the context, so it never changes the AAD. The table specifies behaviour, and the names can change without changing it. A requirement for nonempty input belongs to the accepted argument type or to the validated resolved context, not to whether the input is optional. @@ -588,6 +753,20 @@ The implementation must prove these points: examples](2026-10-24-stack-encrypt-context-examples.md) run through Rust and through dynamic plans. +Status of these points on 7 November 2026: + +| Point | Status | +| --- | --- | +| Leaving out required input fails to compile, even when `C: Default` | Done. #1404 adds `RequiredContext`, `AllowsNoContext`, and `AcceptsContext`. #1501 adds compile-fail tests, including a collection of `u64` | +| A target that accepts no input rejects context | Done. #1501 adds a compile-fail test for `NoCallerContext` | +| An optional extension and an expected context keep their meanings | Still open. `ContextExtension` and `ExpectedContext` as context inputs are in #1504, which is in progress | +| All nine directional scenarios run through Rust and dynamic plans | Still open. #1491 tests a base with ordered extensions and stored-context expectations, not the nine scenarios | + +#1404 implements the traits in the tables above, the associated type +`ContextInput`, and `ScalarContext`, the owned context of an opaque value. +`OptionalContext` gives an opaque value an empty context when the caller +leaves it out, through both the keyset and the target-directed API. + Keep support for owned sources, and keep batching. This separation of traits must not add a blanket `Clone` bound. The final signatures must show the support for borrowed sources and for owned sources. They must not hide the @@ -628,8 +807,12 @@ State the remaining requirements explicitly: - Opening by selector accepts at most one entry for each document group. - Use subtree reads to retrieve plaintext for all matches. - General inputs for search scope and row binding remain unsupported for EQL - targets. The existing column and keyset profile can be released without - claiming them. + targets. The existing column and keyset configuration can be released without claiming them. + +Two JSON queries were added after this section was first written. Pull requests #1445 and #1451 test several conditions on one array element. Pull requests #1446 and #1452 turn a jsonpath predicate into one query parameter. Neither query uses +an index. The [JSON operations +inventory](2026-10-25-stack-encrypt-json-operations.md) lists them as J21 +and J22. The JSON contract now uses randomised projection order and ciphertexts bound by cascaded tags. Semantic parity and cross-opening are required. Comparing @@ -754,8 +937,7 @@ the output, are initialised and shared with convenient `encrypt_into` calls. initialise a client or keyset. 8. **Apply the user's prototype scope.** These parts of the older document are not constraints here: decoding legacy match options, deprecation - requirements, release-preservation requirements, and frozen Go wire - formats. Keep the new Rust, Go, and EQL consumers aligned. The appendix's + requirements, release-preservation requirements, and frozen Go wire formats. The decision of 7 November 2026 now freezes the stored format, though not the Go API. Keep the new Rust, Go, and EQL consumers aligned. The appendix's `cipher.encrypt(plan, value)` and plaintext derive are illustrative older proposals. They do not replace our agreed entry points. @@ -995,7 +1177,8 @@ Accepted on 25 October 2026: outputs and layouts compose those operations. - The lifecycle and cleanup scope below are part of the accepted design. - The bound for plaintext cleanup is `Zeroize`. The implementation is not - finished. + finished. #1404 implements it for opaque values, and #1501 for + collections of them. ### Owned input is the direct API contract @@ -1090,7 +1273,7 @@ partial decoding. #### Stable storage format first Decided on 25 October 2026, as the priority: first define a stable, -versioned binary storage profile. The codec was first designed for +versioned binary storage format. The codec was first designed for short-lived transport, and stored blobs need a stronger compatibility commitment. Improvements to zeroisation and constant-time implementation can follow, provided they keep the meaning of the stored format. @@ -1103,7 +1286,7 @@ C #519](https://github.com/cipherstash/vitaminc/issues/519), created for this review, tracks this action. The issue explicitly separates storage stability from the transport-only reasons in Vitamin C #332. -#### Storage profile implemented in Vitamin C and Go +#### Vitamin C storage profile implemented in Vitamin C and Go By 27 October 2026, Vitamin C's dedicated storage profile, and its Go reader and writer, addressed the format work from #519. That work is in Vitamin @@ -1111,7 +1294,7 @@ C #520 and #526, carried onto upstream main by #544. Stack #1369 and #1370 adopt the Go codec: generated opaque structs call `vcffi.MarshalStorage` and `UnmarshalStorage`. -The profile is versioned (`VCS` plus a version). It supports explicit +The storage profile is versioned (`VCS` plus a version). It supports explicit optional values and typed map keys, and it is separate from transport. Do not report these properties of the transport mapping as defects of the new storage profile: @@ -1133,8 +1316,9 @@ cross-reading tests and the normative corpus matter. #### Remaining Rust and release work -Rust workspaces still use Vitamin C 0.8.0 from crates.io. So this work does -not implement the final proposed Rust `OpaqueBlob` facade. When that facade +On 27 October 2026, the Rust workspaces used Vitamin C 0.8.0 from +crates.io. #1501 moved them to 0.9. Neither change implements the final +proposed Rust `OpaqueBlob` facade. #1503 adds `OpaqueBlob` over the storage profile. #1503 is in progress. When that facade is built, adopt the dedicated storage entry points, instead of reusing the transport Serde codec for storage. Go's temporary pseudo-version dependency must move to released module tags for a final Go release. A new evaluation @@ -1143,8 +1327,7 @@ of serialisers is not on the critical path for customers. Stable bytes do not establish constant-time behaviour or complete cleanup. Keep the remaining work on guarded output, temporary buffers, partial decoding, and values that the caller owns, without changing the stored -format. The JSON tree keeps its own node and envelope encoding. Freeze its -post-cascade profile separately from opaque blobs. +format. The JSON tree keeps its own node and envelope encoding. Freeze its post-cascade format separately from opaque blobs. ### Cleanup in target-directed encryption and at the Go boundary @@ -1169,6 +1352,28 @@ Acceptance work: - cover nested source and blob cases; - cover owned inputs that are not `Clone`. +Status of this acceptance work on 7 November 2026: + +| Acceptance item | Status | +| --- | --- | +| Source cleanup and drop on successful and failing requests | Done for opaque values and collections. #1404 wipes the owned source before it returns the pending request. #1501 adds drop witnesses, which are test values that record whether they were wiped | +| Cancellation, and requests dropped before they are polled | Done. #1493 adds lifecycle tests, and #1501 adds them for collections | +| Nested source and blob cases | Partly done. #1501 tests a nested `Vec>` as one batch. Blob cases wait for #1503 | +| Owned inputs that are not `Clone` | Done for collections (#1501) | + +Other cleanup work that merged: + +- #1501 makes the frame codec write into a Vitamin C `Buffer`. A `Buffer` wipes each allocation it outgrows. +- #1501 makes the frame readers, and the Go guest's frame inputs, take + protected bytes. +- #1501 wipes the plaintexts decrypted so far when a later element of a + collection fails. +- #1493 records a known gap: plaintext that stays in guest memory after a + Go decrypt. + +Cleanup after a partial Serde deserialisation, and the opt-in adapter for +results that are wiped on drop, are still open. + Where necessary, inspect optimised code for wiping and copying behaviour. Ordinary tests alone cannot prove that no copy remains. Do not infer a current vulnerability, or a completed guarantee, from this design @@ -1187,7 +1392,7 @@ of opaque storage are now merged. - operator input and result encodings; - grouped reads; - errors; - - the column and keyset profile. + - the column and keyset configuration. 2. **Close the actual remaining customer gaps.** The gaps are opening a child bound to its parent, and complex ordered-path declarations if needed. The gaps also include configurable Go match options (#1112), and @@ -1203,7 +1408,7 @@ of opaque storage are now merged. 4. **Release the dependencies and the customer interface.** For a final release, replace Vitamin C's temporary Go pseudo-version pins. The existing guard allows them in prereleases. Opaque storage is already - integrated. Keep it separate from the JSON-tree storage profile. Build + integrated. Keep it separate from the JSON tree storage format. Build production guests and the matching SQL. 5. **Complete the Rust convenience APIs and packaging.** Implement public traits, derives, the definition lifecycle, and crate moves within the @@ -1211,10 +1416,20 @@ of opaque storage are now merged. that keeps the format. Use one execution engine throughout. The final Rust facade and package -layout do not gate the existing Go JSON profile. Do not reimplement +layout do not gate the existing Go JSON support. Do not reimplement configured JSON options, or reads that Rust verifies against the original request: both are already merged. +Status of each step on 7 November 2026: + +| Step | Status | What remains | +| --- | --- | --- | +| 1. Specify the implemented contract | Partly done. The conformance corpus covers declarations, runtime values, reads, context, scalars, value kinds, query operators, JSON operations, and lifecycle (#1460, #1461, #1462, #1469, #1491, #1492, #1493) | The specification itself. See [Required deliverable: dynamic interoperability specification](#required-deliverable-dynamic-interoperability-specification) | +| 2. Close the remaining customer gaps | Still open | Opening a child bound to its parent, and #1112. The new element and predicate queries (#1445, #1446, #1451, #1452) were not on this list | +| 3. Validate and freeze the release candidate | Partly done. #1492 runs the JSON suites against PostgreSQL in CI, and records the SQL bundle hash. #1403 fixes findings from the review of the release candidate | Freezing the post-cascade vectors and the EQL bundle | +| 4. Release the dependencies and the customer interface | Still open. Go `v0.1.0-rc.3` is a prerelease | Replacing the Go pseudo-version pins | +| 5. Complete the Rust convenience APIs and packaging | Partly done (#1404, #1501). #1503 and #1504 are in progress | Records, blobs, the definition lifecycle, and crate moves | + ### Required deliverable: dynamic interoperability specification The user requires a formal specification of the dynamic interoperability @@ -1224,6 +1439,34 @@ examples remain proposals until they are reconciled into that specification. Adding this deliverable does not claim that the specification has been written. +Status on 7 November 2026: partly done. The conformance corpus in +`conformance/stack-encrypt` contains the executable fixtures, and contract +documents for each conformance profile. It covers these parts of the required coverage +below: + +- the current declaration grammar, with accepted and rejected declarations + (#1460); +- runtime values, missing compared with null, and structured diagnostics + (#1461); +- validation stages, and which failures come before key I/O (#1462); +- context ordering and grouping, descriptors, AAD, and index context, for + the current shared context (#1491); +- JSON operations, the predicate grammar, and SQL oracles (#1492). + +These parts are not done: + +- **The format.** No JSON Schema files exist. The declaration grammar + contract explains that a schema of shapes alone would not enforce the + semantic rules. +- **Versions.** The conformance profiles are marked experimental: they do not certify a whole SDK. The contract is not frozen. The `v1` in the corpus is a corpus format version, not a + protocol version. +- **Roles.** The grammar for search scope, primary keys, and scope keys + does not exist. The context contract says that this split is separate + work. +- **Other runners and exchange.** Python and TypeScript runners, fresh + cross-language exchange of JSON values, and a release compatibility + matrix are not done. + Recommended format: a versioned specification package that contains: - [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) @@ -1276,8 +1519,7 @@ Required coverage: extraction. - **Versioning.** Explicit protocol and schema identifiers, handling of supported versions and capabilities, and rejection of unsupported - features. Define future evolution without adding compatibility obligations - for the prototype. + features. Define future evolution without adding compatibility obligations for the prototype API. The rc.3 stored format is the first version that later versions must keep reading. Conformance fixtures include accepted and rejected definitions and requests, expected error codes and paths, normalised definitions, and deterministic @@ -1302,19 +1544,19 @@ The configured-target work above covers review issue 13 in depth. The full API redesign also needs the work in this table. Each row names a concrete deliverable. Recommendations do not silently settle the open decisions. -| Work area | Review issues | Required deliverable | -| --- | --- | --- | -| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One map of signatures, imports, and receivers; operation-trait design; and migration of dependencies | -| Context contract | 1, 3, 4, 11, 12, 14 | Context rules for scalars and records, explicit optionality, and a matching dynamic representation | -| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde and Base64 behaviour, and a defined boundary with structural trees | -| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | -| Mixed records and collections | 7, 8, 11 | Derive support for passthrough, renaming, and context fields, and retained collection batching | -| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion, and Go coverage | -| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | -| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348 to #1352 with this contract, and reuse compatible implementation work | -| Prior query design | 16 | Reuse the supplied field-owned query model, and reconcile handles, configuration ownership, JSON capabilities, and portability | -| Consistent definition selector | 17 | Choose one method name for encrypt and query. Recommended: `using`, with bounds specific to each operation | -| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode` and `OpaqueDecode`, consuming `encrypt` and `encrypt_into`, guarded lifetimes for sources and buffers, and an explicit cleanup scope | +| Work area | Review issues | Required deliverable | Status on 7 November 2026 | +| --- | --- | --- | --- | +| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One map of signatures, imports, and receivers; operation-trait design; and migration of dependencies | Partly done (#1404, #1501). The structural API and `encrypt_as` remain | +| Context contract | 1, 3, 4, 11, 12, 14 | Context rules for scalars and records, explicit optionality, and a matching dynamic representation | Partly done (#1404, #1491). Directional selectors and records remain | +| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde and Base64 behaviour, and a defined boundary with structural trees | Partly done (#1404). Typed text encodings and encoded field targets remain | +| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | Still open. #1503 is in progress | +| Mixed records and collections | 7, 8, 11 | Derive support for passthrough, renaming, and context fields, and retained collection batching | Collections done (#1501). Record derives are in #1504, which is in progress | +| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion, and Go coverage | Still open for Rust. Go coverage gained two JSON queries (#1451, #1452) | +| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | Partly done. Format fixtures are in the conformance corpus. The public examples become tests in #1504, which is in progress | +| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348 to #1352 with this contract, and reuse compatible implementation work | Unchanged since 27 October 2026. #1353 is still open | +| Prior query design | 16 | Reuse the supplied field-owned query model, and reconcile handles, configuration ownership, JSON capabilities, and portability | Unchanged | +| Consistent definition selector | 17 | Choose one method name for encrypt and query. Recommended: `using`, with bounds specific to each operation | Still open | +| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode` and `OpaqueDecode`, consuming `encrypt` and `encrypt_into`, guarded lifetimes for sources and buffers, and an explicit cleanup scope | Partly done (#1404, #1493, #1501). Blob cleanup and partial decoding remain | ### A. Consolidate the public Rust method and trait structure @@ -1325,11 +1567,12 @@ sections below organise the full scope of the review, not the critical path for customers. Keep the agreed ordinary keyset receiver and the `EncryptInto` conversion -forms. Use the accepted destination-directed decrypt pair. Write out all +forms. Use the accepted destination-directed decrypt pair. + +Write out all signatures, imports, bounds, and return types before renaming traits. Keep the ordinary keyset `decrypt` and `decrypt_with_context` for opaque -ciphertexts. Remove redundant prototype conveniences without compatibility -shims. +ciphertexts. Remove redundant prototype conveniences without compatibility shims. The APIs can break. The stored format must not. Separate plaintext representation, target operation declarations, and execution conveniences explicitly. Resolve two inconsistencies: @@ -1355,6 +1598,20 @@ imports, including from generic callers. The method inventory has no unexplained duplicate execution path. Compile-fail tests cover the trait capabilities that are not supported. +Status on 7 November 2026: partly done. + +- **Done.** #1404 separates the declarations (`EncryptTarget`, + `DecryptSource`) from the conversion methods (`EncryptInto`, + `DecryptFrom`) for opaque values. #1501 extends them to `Vec`. Scalar + calls compile with the crate-root imports, and compile-fail tests cover + unsupported inputs and context. +- **Not done.** Blob, typed record, searchable record, and JSON calls do + not yet compile through the owned API. Pull requests #1503 and #1504 are in progress. +- **Duplicate path.** The method inventory still has a duplicate path. The + structural `EncryptFrom` and `DecryptInto`, `encrypt_as`, and the helpers + in `target::legacy` remain beside the owned API. Issue #1488 tracks their + removal. + ### B. Turn the context scenarios into a normative contract The nine scenarios in the [context @@ -1635,6 +1892,26 @@ dynamic declarations. Required input is enforced, altered or wrong context is refused, and existing context encodings agree across all new implementations. +Status on 7 November 2026: partly done. + +- **Composition order.** #1491 fixes it for the current shared context. A + field's base comes first, then each extension, appended in order. A + grouped extension stays one part. An empty extension adds nothing. + Before #1491, repeated Rust extensions replaced earlier ones. +- **Evidence.** #1491's conformance cases check the descriptor, AAD, and + index context bytes in native Rust. They check the plan that Go sends, + and cross-opening between the two languages. Other cases check + stored-context expectations, altered context, and refusals before key + I/O. +- **Required input.** #1404 enforces it for opaque values without using + `Default`. #1504 adds optional extensions and expected context as inputs. #1504 is in progress. +- **Empty context.** #1404 resolves the mismatch for the new `Ciphertext` + target. The target accepts an empty context through both APIs. EQL targets + still require a nonempty context. +- **Still open.** The nine directional scenarios, source-field selection + on records, the dynamic form of grouped extensions, and the split between + ciphertext binding and search scope. + ### C. Finish the ciphertext and whole-value storage contracts Accepted direction: a single-value plaintext encoding produces one opaque @@ -1678,6 +1955,17 @@ Keep generic structural encryption and searchable JSON explicit. Specify whether persistence of structural trees is supported, and in what stable format. Coordinate any protocol changes with their consumers. +Status on 7 November 2026: partly done by #1404. + +- **`Ciphertext`.** The new type stores exactly the existing `SealedValue` + v1 envelope. It has `to_bytes` and `from_bytes`. Parsing does not authenticate. Human-readable Serde writes padded standard Base64, and + binary Serde writes bytes. `Debug` uses Vitamin C's `OpaqueDebug`, so it + does not print the ciphertext. +- **Single values only.** `OpaqueEncode` is sealed. A type qualifies only if Vitamin C encrypts it as a single value. So `encrypt` does not compile for a struct, a map, or a `Vec`. +- **Trees.** Structural trees use `encrypt_tree` and `decrypt_tree`, and + their persistence is still unspecified. Nullable values also use the + tree API, because `Option` is not an `OpaqueEncode` type. + #### Typed encodings and extractable ZeroKMS tags Requirements added on 25 October 2026: @@ -1772,6 +2060,15 @@ Protocol details remain deferred. This is the intended behaviour, not a claim that the current client or engine already implements it. Add integration coverage for refinement and decryption when that work is merged. +Status on 7 November 2026: partly done. + +- **Done.** #1404 adds `Ciphertext::key_tag`. The accessor reads the + ZeroKMS key tag from the envelope without decrypting, and its + documentation says the tag is untrusted until the ciphertext opens. +- **Still open.** `to_base64` and `to_base85` with the typed text wrappers. + The encoded ciphertext field targets. The SQL and PL/pgSQL extraction + helper, and its Rust, Go, and SQL fixtures. The refined-tag flow. + #### Single values for encrypt, collections for EncryptInto Decided on 25 October 2026: direct `encrypt` handles one single-value input, @@ -1807,14 +2104,32 @@ Acceptance checks: Existing collection composition is kept. Opaque `Ciphertext`, and its target support without context, are new work. -The current prototype treats `Vec` as one byte value, but that is not a -compatibility constraint. The latest recommendation is uniform collection +The current prototype treats `Vec` as one byte value. On 25 October 2026, that was not a compatibility constraint. After the decision of 7 November 2026, it is a constraint if Go rc.3 stores byte strings that way. This was not checked. + +The latest recommendation is uniform collection behaviour. `Vec` to `Vec` encrypts individual bytes, while `OpaqueBlob>` encodes the complete vector as one value. Do not add a conversion from `Vec` to `Ciphertext`, or a bytes wrapper, only to keep -prototype behaviour. A distinct raw-byte capability would need its own use -case, and the blob codec need not reproduce the prototype's raw-byte -encoding. +prototype behaviour. + +A distinct raw-byte capability would need its own use +case. The blob codec need not reproduce the prototype's raw-byte encoding. A byte string that Go rc.3 stores must still open, since 7 November 2026. + +Status on 7 November 2026: partly done, with one difference from this +plan. + +| Acceptance check | Status | +| --- | --- | +| Reject direct `encrypt(Vec)` before execution | Done. `Vec` is not an `OpaqueEncode` type, and a compile-fail test in #1501 checks it | +| Support `Vec` to `Vec` without context | Done (#1501). Encrypting makes one `generate_keys` call, and decrypting makes one `retrieve_keys` call for each keyset | +| Keep target-directed opening and batching of collections | Done (#1501) | +| Recover the blob as a whole collection | Still open. #1503 is in progress | + +The difference is in byte vectors. Pull request #1501 makes `Vec` and `[u8; N]` encrypt into one `Ciphertext`. In #1501, `u8` is not an `OpaqueEncode` type. So `Vec` never encrypts byte by byte. That is the opposite of the recommendation above. + +The reason given in #1501 is that Vitamin C has no single value for `u8`. + +Pull request #1501 also leaves the order and length of a `Vec` unauthenticated. ADR-0010 decides this for stored rows. #### Blob codec, format, and timing requirements @@ -1840,8 +2155,7 @@ The constant-time question needs a concrete conclusion. Assess: - which custom implementations are allowed. If arbitrary Serde types remain supported, do not promise constant-time -serialisation from end to end. If that property is required, select a -restricted profile separately. Constant-time behaviour cannot be inferred +serialisation from end to end. If that property is required, select a restricted set of supported types separately. Constant-time behaviour cannot be inferred from the wrapper or its encryption primitive. Completion criteria for work area C: scalar and blob examples persist, and @@ -1849,6 +2163,10 @@ reload through a new cipher instance. The examples reject invalid encodings and verify fixtures for the selected format. The supported timing contract and the codec version are documented. +Status on 7 November 2026: partly done. The scalar half is merged (#1404). +The `record-scalars-v1` profile (#1460) checks that frozen Rust scalar +ciphertexts open in Go, and that fresh Go ciphertexts open in Rust. The blob half, its codec version, and its timing contract are in #1503. #1503 is in progress. + ### D. Complete record derives and preserve collection composition Implement explicit passthrough and renaming of source fields, independently @@ -1869,6 +2187,13 @@ Dynamic and Go declarations produce equivalent semantics. A plaintext-side derive that generates output types is a deferred convenience, not a prerequisite for these workflows. +Status on 7 November 2026: partly done. + +- **Collections.** #1501 composes an owned `Vec` into one logical batch. + Lifecycle tests check that an empty collection makes no provider call. +- **Records.** No record derive for the owned API is merged. Pull request #1504 adds one, with passthrough, renaming, nested targets, and stored context. Pull request #1504 is in progress. +- **Configured nested targets.** #1504 defers them. The JSON plaintext type is `serde_json::Value`. That type does not implement `Zeroize`. + ### E. Finish the configured policy's lifetime and query interface Use the query-design baseline and reconciliation in [Prior query @@ -1892,6 +2217,10 @@ of mandatory hand-written plans. Keep direct typed EQL conversion, and support in the catalog and guest, as deliverables of the configured-target sequence. +Status on 7 November 2026: still open. No merged pull request implements +typed factories, a definition lifecycle, or typed EQL conversion in Rust. +The catalog gained two JSON query roles in #1445 and #1446. + ### F. Make naming and coordinated integration part of completion Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for: @@ -1907,10 +2236,42 @@ and distinguish it from executing `EncryptInto`. New spellings and crate moves still need review. Remove replaced APIs directly, and update the EQL, Go, and guest consumers -together. There is no requirement to read prototype ciphertext or keep old -imports. Specify the selected encoding and cryptographic semantics, and set +together. There is no requirement to keep old imports. From 7 November 2026, ciphertext written by Go rc.3 must stay readable and searchable. Specify the selected encoding and cryptographic semantics, and set up fixtures for that contract from now on. +#### Name context parameters `context` + +Decided on 7 November 2026: a parameter that takes the caller's context is +named `context`. The name `aad` is kept only for a value that is an AAD, +such as the AAD that Vitamin C derives for one leaf. The private functions +`seal_leaf`, `open_leaf`, and `leaf_aad`, and those in `json/derive.rs`, +already follow the rule. + +On `next` at `e4990e49b`, these public methods in +`packages/stack-encrypt/src/cipher.rs` take a context through a parameter +named `aad`: + +| Method | Line | +| --- | --- | +| `KeysetCipher::encrypt_tree` | 1021 | +| `KeysetCipher::decrypt_tree` | 1036 | +| `StackCipher::decrypt` | 1105 | +| `PendingStackCipherText::plan`, whose bound is `IntoAad` | 1485 | +| `PendingStackCipherText::seal` | 1520 | +| `PendingStackCipherText::into_pending` | 1554 | + +Rename each of these parameters to `context`. This is a breaking change to +the Rust API only. The decision of 7 November 2026 allows it. The change +does not change the stored format. + +Vitamin C's `IntoAad` trait also takes a context. Renaming it is a change +in Vitamin C. Record it there as a follow-up. It is not done in this +repository. + +The Go SDK and the dynamic interface in `packages/stack-encrypt/src/dynamic` +have no public field or parameter named `aad`. I checked with `git grep` +on `next`. + Completion criteria for work area F: - all public examples are executable tests; @@ -1926,6 +2287,16 @@ storage in the [implementation sequence](#configured-target-implementation-sequence-and-completion-criteria). Those gates do not depend on the facade. +Status on 7 November 2026: partly done. + +- **Cross-opening.** The conformance corpus checks cross-opening between + native Rust and the Go guest for scalars, records, context, and JSON + (#1460, #1491, #1492). +- **Batching.** #1493 and #1501 check batching. +- **Guest builds.** CI builds the guests for the conformance jobs. +- **Not done.** The public examples are not yet tests. Pull request #1504 makes the record examples into tests. Pull request #1504 is in progress. The plan builder method is not renamed + to `target`. The replaced APIs are not removed. + ## Recommended order for the remaining design decisions ### Decision checkpoint on 25 October 2026 @@ -1955,7 +2326,7 @@ These directions are settled: named-type declarations, and explicit scope eligibility; - a versioned interoperability specification with shared conformance coverage; -- no backward compatibility with the prototype. +- no backward compatibility with the prototype API. On 7 November 2026, the user added that the stored format of Go rc.3 must not break. Do not reopen these choices only because older examples or descriptions of work areas still use provisional wording. @@ -1967,19 +2338,18 @@ from the obligations to specify and implement: | Remaining question | Decision required | | --- | --- | -| JSON contract completion | The main Go interface is implemented. Close the gaps in opening a child bound to its parent, and in formal conformance. Use the current semantics of cascaded tags and random order. Do not reopen operator design, or rename released Go APIs for cosmetic consistency | +| JSON contract completion | The main Go interface is implemented, and #1492 enrols it in conformance. Close the gap in opening a child bound to its parent, and keep the current semantics | +| Byte vectors | #1501 encrypts `Vec` as one value, unlike this plan. Accept that behaviour, or change the code | | Key selection and optional provider policy | Review keyset selection derived from plaintext, routed execution scoped to a client, and ZeroKMS lock context, as optional provider capabilities. In the provider-capabilities companion, assess the neutral boundary between data keys and index keys, and the deliberate non-use of Vitamin C's `AsyncCipher` | -| Customer storage profiles | Go opaque storage is integrated. For the final release, replace the temporary Go dependency pins, and freeze the post-cascade JSON and search profiles. Rust blob conveniences and crate moves can follow | +| Customer storage formats | Go opaque storage is integrated. For the final release, replace the Go pins, and freeze the post-cascade format | | Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenisers and filters remain, including #1112. Reconcile the earlier privacy concern explicitly. Merging support does not prove that leakage is unchanged | | Packaging and names (review in progress) | Review the packaging companion's shared format crate, value execution owned by Stack, a Vitamin C boundary of primitives only, retirement of `AsyncCipher`, the ZeroKMS client rename, and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again | ### Next steps -Next, complete the specification of the supported contract, and close the -remaining customer gaps in configuration and reads. Then validate the -release candidate, and resolve the pins for the final release. The original -end-to-end JSON implementation is already done. Packaging remains deferred. -Keep the initial ORE equality policy unchanged. +Next, close the remaining customer gaps in configuration and reads. Finish the specification around the merged conformance corpus. Then freeze the release candidate, and resolve the pins for the final release. The Rust conveniences for blobs and the crate moves can follow after the customer storage formats are final. + +The original end-to-end JSON implementation is already done. The first conformance corpus is also done. Packaging remains deferred. Keep the initial ORE equality policy unchanged. Specification and implementation work still includes: @@ -1990,6 +2360,11 @@ Specification and implementation work still includes: - Rust, Go, and SQL fixtures; - the flow of the refined tag and the original tag. +On 7 November 2026, the conformance corpus covers the current context +grouping, the current dynamic grammar, and Rust and Go fixtures. It does +not cover scope, version identifiers, stable error codes as a frozen list, +or the refined tag. + Record candidate choices in the formal specification for review. Do not treat every byte offset or adapter signature as a new architectural decision. The initial recommendation for persistence remains ordinary Serde @@ -2003,6 +2378,15 @@ Cleanup implementation still needs: - checks for cancellation and drop; - an explicit adapter for drop-guarded results. +Status on 7 November 2026: + +| Cleanup item | Status | +| --- | --- | +| Guarded encoder sinks | Partly done. #1501 makes the frame codec write into a `Buffer`. A `Buffer` wipes each allocation it outgrows | +| Handling of partial decoding | Partly done. #1493 and #1501 wipe the values decoded so far when a later value fails. A partly built Serde value is still open | +| Checks for cancellation and drop | Done (#1493, #1501) | +| An explicit adapter for drop-guarded results | Still open | + Under the user's latest sequencing decision, implementation improvements can follow the stable format. Keep the stored contract, and document the actual guarantees, instead of claiming that the eventual goals are already met. @@ -2014,8 +2398,7 @@ signatures and proof. That is implementation design within the recorded ownership model. It is not a reason to reopen whether writes and queries share a policy. -Passthrough implementation, transcoder adapters, collection composition, -moving traits, documentation updates, and acceptance tests remain work -items. These items are not reasons to describe the agreed API direction as -undecided. The prototype needs no backward-compatibility phase. The customer -release creates a new commitment for Go and for persisted data. +Passthrough implementation, transcoder adapters, moving traits, +documentation updates, and acceptance tests remain work items. Collection +composition for the owned API is done (#1501). These items are not reasons to describe the agreed API direction as +undecided. The prototype API needs no backward-compatibility phase. The stored format has needed one since 7 November 2026, because the Go customer builds on rc.3. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index ef9bc092c..8c07554f7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -7,9 +7,18 @@ opaque ciphertext and store the result. This document is a running log of the issues found. Each numbered section records one issue: what the code does now, what changes were proposed, and which decisions are made or still open. The proposed APIs are illustrations, -not implemented code. All the issues below are still unimplemented. Add a -numbered section for each new issue, and keep observations, proposed -changes, and open decisions apart. +not implemented code, unless a status line says otherwise. Add a numbered +section for each new issue, and keep observations, proposed changes, and +open decisions apart. + +On 7 November 2026, this log was checked against `next` at commit +`e4990e49b`. Some issues are now partly done. The owned API for one value and for collections is merged. A conformance corpus for Rust and Go is merged too. No issue is completely done. + +The [summary of +status](#summary-of-status) gives the status of each issue and the pull +requests behind it. "Current behaviour" sections describe the code when +each issue was written. Where a code block in one of them is now wrong, a +note follows the block. Three companion documents go with this log: @@ -23,17 +32,40 @@ Three companion documents go with this log: requirements, the recommended decisions, the order of work, and when the work is complete. -## No backward compatibility with the prototype +## The stored format must stay compatible from 7 November 2026 + +On 7 November 2026, the user decided: "The stored data must not break as of +now. The Go customer has already started implementation with rc.3. We can +make breaking changes to the code/API but not the stored format." + +The stored format is what is written to the database. It includes +ciphertexts with their envelopes and headers. It includes index and search +terms. It includes JSON tree entries. + +The baseline is what the Go module at +tag `languages/golang/v0.1.0-rc.3` writes, with the EQL v4 SQL bundle +attached to that release. The tag is commit `114e08448`, built on 2 +November 2026 from `next` at `929451b49`. + +From now on, a change that would alter the stored format must keep data +written by rc.3 readable and searchable. Otherwise it must add a new, +versioned format beside the rc.3 format. The Rust and Go code and APIs can +still change in breaking ways. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#work-that-the-stored-format-decision-constrains) +lists the planned work that this decision constrains. + +### History: no backward compatibility with the prototype On 25 October 2026 the user clarified that this API has never shipped beyond an early prototype. Backward compatibility, and migration of prototype data, -are not requirements. This replaces any earlier suggestion in this log to +were not requirements. That replaced any earlier suggestion in this log to keep old encodings or public imports. The new Rust, Go, guest, and EQL -(Encrypt Query Language) implementations must agree with each other. They do -not need to keep the prototype's representations. +(Encrypt Query Language) implementations must agree with each other. -Issue 22 limits this exemption: once the customer release ships, customer -data must not be broken. +The decision of 7 November 2026 replaces this one for stored data. The +earlier decision still applies to the code and the APIs. Issue 22 had said +the exemption would end when the customer release shipped. It now ends for +stored data before that release. ## Terms used in this log @@ -42,17 +74,19 @@ where they first appear. | Term | Meaning | | --- | --- | -| Vitamin C | CipherStash's Rust cryptography library, published as the `vitaminc-*` crates. Stack Encrypt uses version 0.8.0 | +| Vitamin C | CipherStash's Rust cryptography library, published as the `vitaminc-*` crates. Stack Encrypt used version 0.8.0 when this log started. #1501 moved it to 0.9 | | ZeroKMS | CipherStash's key management service, which issues and retrieves data keys | | KMS | Key management service | | EQL | Encrypt Query Language, the SQL and storage format for searchable encrypted data in PostgreSQL | | AEAD | Authenticated encryption with associated data | -| AAD | Additional authenticated data: data that the ciphertext is bound to, but that is not encrypted | -| Context | The application data a caller binds a ciphertext to, such as a column name. Stack Encrypt derives each leaf's AAD from it, and sends ZeroKMS a text descriptor of it, such as `users/age` | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | | Leaf | One sealed value, `SealedValue`, in Vitamin C's ciphertext tree | | Node | One element of Vitamin C's ciphertext tree, `Node`: a single leaf, a sequence, a map, an empty container, or a passthrough value | | Passthrough | A field that is copied into the output in plaintext, without encryption | -| Target | An output type that declares how a plaintext is encrypted into it, through `EncryptFrom` | +| Target | An output type that declares how a plaintext is encrypted into it, through `EncryptFrom`, or `EncryptTarget` in the owned API | +| Owned API | The Rust API added by #1404 that consumes its plaintext input: `keyset.encrypt`, `EncryptInto`, and `DecryptFrom`, over `EncryptTarget` and `DecryptSource` | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](#the-stored-format-must-stay-compatible-from-7-november-2026) | | Plan | The lower-level builder, `Plan`, that composes encryption operations. The derives generate plans, and the dynamic interface uses them | | Dynamic interface | The interface driven by data instead of Rust types, used through FFI (foreign function interface) and by the Go SDK | | Guest | The WebAssembly build of the Rust engine that the Go SDK runs | @@ -60,42 +94,53 @@ where they first appear. ## Summary of status -| Issue | Status on the last update | +| Issue | Status on 7 November 2026 | | --- | --- | -| 1. Cipher trait mismatch | Open. The user asked for separate `encrypt` and `encrypt_with_context` methods | -| 2. Opaque ciphertext storage | Open. Decisions recorded for ordinary encryption, collections, and text encodings | -| 3. Plaintext-side encryption | Open. Receiver decided on 24 October 2026 | -| 4. Fixed context on the scalar derive | Direction agreed. Implementation pending | -| 5. `EncryptFrom` has no `encrypt_from` | Open | -| 6. Plan builder `encrypt_into` | Open. Revisit during the plan API discussion | -| 7. Mixed encrypted and plaintext fields | Passthrough attribute accepted in principle. Implementation pending | -| 8. Whole-value encryption and layouts | Open architectural discussion | -| 9. Ownership across Stack and Vitamin C | Open architectural discussion. Moving the traits is proposed, not authorised | -| 10. Public `encrypt_as` | Decision agreed. Implementation pending | -| 11. Context from an input struct | Open | -| 12. `Default` compared with optional context | Open | -| 13. JSON layouts and output adaptation | Open. One follow-up agreed | -| 14. Decrypt context variants | Direction required by the user. Receiver still to confirm | -| 15. Go JSON work in pull request #1353 | Coordination item | -| 16. Prior query design | Adopted as the baseline | -| 17. One name for definition selection | Recommendation. Final spelling not accepted | -| 18. Consuming inputs with cleanup | Decisions made on 25 October 2026 | -| 19. First-release completeness | Decisions accepted on 25 October 2026. JSON contract under user review | -| 20. Keysets chosen from plaintext | Recommendations, not accepted | -| 21. Packaging and `AsyncCipher` | Proposals | -| 22. Customer-driven order of work | Order of work set by the customer's needs. Merged work reconciled on 27 October 2026 | +| 1. Cipher trait mismatch | Partly done. #1404 adds `encrypt`, `encrypt_with_context`, `decrypt`, and `decrypt_with_context` on the keyset. How Stack's ciphers relate to Vitamin C's traits is still open | +| 2. Opaque ciphertext storage | Partly done. #1404 adds `Ciphertext`; text encodings and `OpaqueBlob` (#1503, in progress) are still open | +| 3. Plaintext-side encryption | Partly done. #1404 implements the keyset receiver, and an empty context for `Ciphertext` through both APIs. The structural API keeps its different rules | +| 4. Fixed context on the scalar derive | Still open. #1491 makes the composition rules in this issue true for repeated extensions | +| 5. `EncryptFrom` has no `encrypt_from` | Partly done. #1404 adds `EncryptTarget` and `DecryptSource`; the structural traits remain | +| 6. Plan builder `encrypt_into` | Still open | +| 7. Mixed encrypted and plaintext fields | Still open. In progress in #1504 | +| 8. Whole-value encryption and layouts | Still open. In progress in #1503 and #1504 | +| 9. Ownership across Stack and Vitamin C | Still open. #1501 moves the Rust crates to Vitamin C 0.9 | +| 10. Public `encrypt_as` | Still open. `encrypt_as` is still public | +| 11. Context from an input struct | Still open | +| 12. `Default` compared with optional context | Partly done. #1404 adds explicit context inputs for the owned API. The structural API still uses `Default` | +| 13. JSON layouts and output adaptation | Still open | +| 14. Decrypt context variants | Partly done. #1404 implements both pairs for one value, and #1501 for collections. Records are in progress in #1504 | +| 15. Go JSON work in pull request #1353 | Coordination item. Unchanged | +| 16. Prior query design | Adopted as the baseline. Unchanged | +| 17. One name for definition selection | Still open | +| 18. Consuming inputs with cleanup | Partly done (#1404, #1493, #1501). Blob cleanup, partial Serde decoding, and drop-guarded results are still open | +| 19. First-release completeness | Partly done (#1445, #1446, #1451, #1452, #1492). Opening a child bound to its parent is still open | +| 20. Keysets chosen from plaintext | Still open | +| 21. Packaging and `AsyncCipher` | Still open | +| 22. Customer-driven order of work | Partly done. The conformance corpus is merged. Merged work was reconciled on 7 November 2026 | ## 1. Cipher trait mismatch and the required context argument -**Status:** Open. +**Status:** Partly done. #1404 adds `encrypt` and `encrypt_with_context`, and +the matching `decrypt` pair, on the keyset. The open questions about +Vitamin C's cipher traits remain. The user asked for two separate methods, so that the common case needs no empty context argument. Questions remain about how Stack's ciphers relate to Vitamin C's cipher traits. +### Decision of 7 November 2026: name context parameters `context` + +A parameter that takes the caller's context is named `context`. The name `aad` is kept only for a value that is an AAD. + +Several public methods, such as `encrypt_tree`, still name the context `aad`. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#name-context-parameters-context) +lists them as work to do. The rename breaks the Rust API only. It does not +change the stored format. + ### Requested API direction -Passing an empty tuple, `()`, for the common case of no application AAD is +Passing an empty tuple, `()`, for the common case of no application context is needless ceremony. Use separate methods, following the convention used elsewhere: @@ -117,6 +162,11 @@ let ciphertext = keyset.encrypt(42_i32, ()).await?; let ciphertext = keyset.encrypt(42_i32, "users/age").await?; ``` +Note added on 7 November 2026: this block is no longer current. #1404 +renamed the two-argument method to `encrypt_tree`. `keyset.encrypt(value)` +now takes one argument and returns a `Ciphertext`. The second argument of +`encrypt_tree` is named `aad` in the code, but it takes a context. + Vitamin C (`vitaminc-aead` 0.8.0) provides the `Cipher` trait and the `AsyncCipher` trait. Stack implements the synchronous `Cipher` trait on internal adapters after it fetches keys. `MintedKeys` seals leaves, and @@ -163,7 +213,7 @@ discussion of batching). Vitamin C's trait definitions are in ## 2. The Node and SealedValue types make opaque ciphertext storage harder -**Status:** Open. Several decisions are recorded below. +**Status:** Partly done. Several decisions are recorded below. Pull request #1404 adds `Ciphertext`. The type has `to_bytes`, `from_bytes`, `key_tag`, and Serde support as Base64 or bytes. The text encodings and `OpaqueBlob` are still open. ### Decisions so far @@ -214,8 +264,10 @@ The follow-up of 25 October 2026 also set these directions: caller? If it stays with the caller, say so explicitly, and keep it out of the simple scalar path. -The prototype's canonical leaf encoding is a useful starting point. It is -not a backward-compatibility obligation. +The prototype's canonical leaf encoding is a useful starting point. On 25 +October 2026 it was not a backward-compatibility obligation. Since 7 +November 2026 it is, because Go rc.3 writes that envelope. #1404's +`Ciphertext` uses the same envelope unchanged. ### Current behaviour @@ -234,7 +286,8 @@ passthrough nodes. It does not keep the plaintext type as a type parameter. A leaf's canonical bytes contain its keyset ID, the metadata needed to retrieve its key, and the local ciphertext. That versioned byte format is the storage commitment. The leaf's Serde representation is not the same -compatibility commitment. +compatibility commitment. Since 7 November 2026, data written by Go rc.3 +must stay readable. ### Effect on callers @@ -254,6 +307,12 @@ let ciphertext = StackCipherText::Single(leaf); let value: i32 = cipher.decrypt(ciphertext, ()).await?; ``` +Note added on 7 November 2026: this block is no longer current. With #1404, +`keyset.encrypt(42_i32, ())` does not compile. The tree form is `keyset.encrypt_tree(42_i32, ())`. + +For one value, `keyset.encrypt(42_i32)` returns a `Ciphertext`. A `Ciphertext` has `to_bytes` and `from_bytes`. `keyset.decrypt` opens it. `StackCipher::decrypt` still takes a tree and a +context, as in the last line. + The caller has to: 1. handle a shape mismatch that cannot happen for this input; @@ -279,9 +338,9 @@ Vitamin C's container definition is in `vitaminc-aead` 0.8.0, ## 3. Plaintext-side encryption has different syntax and context requirements -**Status:** Open. The receiver for ordinary `encrypt` was decided on 24 -October 2026. The differing requirements for an empty context are still -open. +**Status:** Partly done. The receiver for ordinary `encrypt` was decided on 24 October 2026. Pull request #1404 implements it. Pull request #1404 also gives `Ciphertext` +the same empty context through the keyset and through `EncryptInto`. The +structural API keeps its different requirements. ### Decision: the keyset is the receiver for ordinary encryption @@ -351,6 +410,12 @@ let ciphertext: StackCipherText = 10_i32 .await?; ``` +Note added on 7 November 2026: this block no longer compiles as written. This was checked by reading the source, not by compiling. + +Since #1404, the crate-root `EncryptInto` is the owned trait. `StackCipherText` is not an owned target on `next`. The structural form needs +`stack_encrypt::target::legacy::EncryptInto`. The owned equivalent is +`let ciphertext: Ciphertext = 10_i32.encrypt_into_with_context(&keyset, "users/age").await?;`. + The output type selects the `EncryptFrom` implementation. For `StackCipherText`, that implementation requires a source that implements `Encrypt + Clone`, and an `AeadContext`. An `AeadContext` must not be @@ -396,7 +461,8 @@ inconsistency is separate from the receiver decision above. ## 4. The scalar derive cannot declare a fixed literal context -**Status:** Direction agreed. Implementation pending. +**Status:** Direction agreed. Still open. #1491 makes repeated Rust +extensions append, so the composition table below now matches the code. ### Agreed direction @@ -455,7 +521,7 @@ context composed directly. The derive implementation still has to specify how a static literal is represented: for example, as text or as a parsed path label. Encryption and -decryption then produce the same descriptor and AEAD context. This choice is +decryption then produce the same descriptor and AAD. This choice is separate from keeping the `EncryptInto` trait. ### Current behaviour and workaround @@ -485,6 +551,10 @@ let encrypted: EncryptedInteger = IntegerInput { value: 10 } .await?; ``` +Note added on 7 November 2026: since #1404, "EncryptInto" here must be +`stack_encrypt::target::legacy::EncryptInto`. The crate-root `EncryptInto` is the owned trait. The `EncryptFrom` derive does not implement it. +This was checked by reading the source, not by compiling. + This form declares the field's context as the pair `("example", "value")`. ZeroKMS receives it as the descriptor `example/value`. For optional caller extensions, the form uses `DeclaredContext`, a type with a default. It is @@ -501,7 +571,10 @@ ordinary literal attribute cannot do that on a scalar derive today. ## 5. EncryptFrom does not define encrypt_from -**Status:** Open. No replacement design has been chosen. +**Status:** Partly done. #1404 adds the declaration traits `EncryptTarget` +and `DecryptSource`, executed by the owned `EncryptInto` and `DecryptFrom`. +The structural `EncryptFrom` and `DecryptInto` remain. #1504 is in +progress. Review the trait names and method names, or where the convenience function lives. Keep the distinction between declaring an operation and executing @@ -533,7 +606,7 @@ surprising. ## 6. The plan builder's encrypt_into reads like the execution method -**Status:** Open. Revisit during the plan API discussion. +**Status:** Still open. Revisit during the plan API discussion. `Plan::value::().encrypt_into::()` is an inherent builder method. It selects an output target and requires `T: EncryptFrom`. It does not @@ -561,8 +634,7 @@ changed. The proposal leaves `EncryptInto::encrypt_into` unchanged. ## 7. The output derive cannot mix encrypted and plaintext fields directly -**Status:** The passthrough attribute is accepted in principle. -Implementation pending. +**Status:** The passthrough attribute is accepted in principle. Still open. Pull request #1504 adds it to the owned derives. That pull request is in progress. ### Accepted proposal: a passthrough attribute @@ -593,7 +665,9 @@ target traits. This gap applies only to Stack's output-side `EncryptFrom` derive. Vitamin C already has a plaintext-side `#[derive(Encrypt, Decrypt)]`, with `#[aead(passthrough)]` on source fields. Stack's -`keyset.encrypt(&user, aad)` can take that derived source. The result is a +`keyset.encrypt(&user, aad)` can take that derived source. Since #1404, +that method is `keyset.encrypt_tree(&user, aad)`, and its `aad` parameter +takes a context. The result is a generic `StackCipherText` map with sealed and passthrough nodes. That route has three limits: @@ -615,7 +689,8 @@ That route has three limits: ## 8. Clarify whole-value encryption, layouts, and searchable outputs **Status:** Open architectural discussion. The recommendations below are -not decisions. +not decisions. #1503 is in progress for workflow 1, and #1504 for workflows +2 and 3. Three user workflows need clear support: @@ -693,7 +768,7 @@ establish it. The first revised example uses Serde traits provisionally. Arbitrary user implementations of those traits cannot give that guarantee. Before promising execution that does not depend on secrets, define which -lengths and shapes may be public. Also decide whether a stricter profile of +lengths and shapes may be public. Also decide whether a stricter set of supported types is needed. The examples document this open question separately from how easy the wrapper is to use. @@ -706,7 +781,9 @@ guarantee for whole objects, or a codec choice. ## 9. Ownership of the value-encryption API across Stack and Vitamin C **Status:** Open architectural discussion. Moving the traits is proposed. -It is not authorised implementation work. +It is not authorised implementation work. #1501 moves the Rust crates to +Vitamin C 0.9. `OpaqueEncode` now follows Vitamin C's `Scalar` marker for +which types are single values. ### Recommendation @@ -748,8 +825,9 @@ Vitamin C. draw a separate boundary for a lower-level core of primitives. - These must agree across the new consumers: context composition, type tags, ciphertext framing, index normalisation, and the transport frames of - the bindings. They can change without keeping prototype data and without a - migration. + the bindings. On 25 October 2026 they could change without keeping + prototype data and without a migration. Since 7 November 2026, stored + data written by Go rc.3 must stay readable and searchable. - Keep one canonical identity for each trait and type. Update consumers directly. Re-exports for compatibility are not needed. Independent copies of trait definitions would need duplicate implementations and adapters. @@ -766,7 +844,8 @@ The installed `vitaminc-encrypt` 0.8.0 manifest confirms that it depends on ## 10. The public encrypt_as duplicates encrypt_into -**Status:** Decision agreed. Implementation pending. +**Status:** Decision agreed. Still open: `encrypt_as` is still public on +`next`. ### Decision @@ -806,8 +885,8 @@ the default context or the caller's context. ## 11. Context fields do not yet take context from an input struct -**Status:** Open. The syntax is still open. The shorthand -`context(field = "id")` is recommended. +**Status:** Still open. The syntax is still open. The shorthand +`context(field = "id")` is recommended. No merged pull request adds it. ### Current behaviour @@ -890,8 +969,9 @@ ID becomes a stored ID. ## 12. Default is not the same as optional caller context -**Status:** Open. The directional-context examples showed this problem. No -mechanism is chosen. +**Status:** Partly done. The directional-context examples showed this +problem. Pull request #1404 implements the first option below for the owned API. It adds `RequiredContext`, with the capabilities `AllowsNoContext` and `AcceptsContext`. #1501 adds compile-fail tests for a `u64` context. The +structural API still uses `Default`. `EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` offer the form without context whenever the associated `Context` type implements @@ -1200,8 +1280,10 @@ guarantees when it reads. ## 14. Decrypt convenience methods need explicit context variants -**Status:** Direction required by the user. The receiver, and the migration -of the competing method, are recommendations still to confirm. +**Status:** Partly done. #1404 implements `T::decrypt_from` and +`decrypt_from_with_context`, and the keyset pair, for one value. #1501 +adds them for collections. The structural `decrypt_into` remains in +`target::legacy`. **Required direction:** provide a method without context and a separate `_with_context` method, mirroring encryption. The user prefers the name @@ -1223,6 +1305,10 @@ These methods exist today on the blanket extension trait `DecryptFrom`: - `Plaintext::decrypt_from(ciphertext, cipher)`; - `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)`. +Since #1404, these three methods are in `target::legacy::DecryptFrom`. The +crate-root `DecryptFrom` is the owned trait, with `decrypt_from` and +`decrypt_from_with_context` and no `decrypt_into`. + `DecryptInto

` separately declares the opening operation. These method and trait names are easy to confuse. @@ -1271,13 +1357,14 @@ that draft as the baseline. The plan records specific points to reconcile: - portability between the static and dynamic interfaces; - when validation happens. -The user's clarification about the prototype replaces older compatibility -requirements. The source document was not changed. +The user's clarification about the prototype replaced older compatibility +requirements. The decision of 7 November 2026 adds one back: the stored +format of Go rc.3 must not break. The source document was not changed. ## 17. Use the same name to select a definition for encryption and queries -**Status:** `.using(...)` is recommended. The final spelling is not yet -accepted. +**Status:** Still open. `.using(...)` is recommended. The final spelling is +not yet accepted. The user finds it confusing that encryption uses `.using(...)` and queries use `.on(...)`, and asked to consider one name for both. The recommendation @@ -1311,7 +1398,8 @@ implementation already provides this API. ## 18. A single ciphertext needs inputs that are consumed and cleaned up **Status:** Decided on 25 October 2026, with the follow-up decisions below. -No completed audit is claimed. +Partly done: #1404 and #1501 guard and wipe owned sources, and #1493 adds +lifecycle conformance. No completed audit is claimed. ### Decision on encoding traits and consumed input @@ -1388,7 +1476,9 @@ upstream is ## 19. First-release completeness and remaining review decisions **Status:** The decisions below were accepted on 25 October 2026. The -detailed JSON operation contract is still under user review. +detailed JSON operation contract is still under user review. Partly done: +#1492 enrols the JSON operations in conformance. Opening a child bound to +its parent is still open. ### Encoded ciphertext and equality @@ -1431,7 +1521,7 @@ Names and packaging are explicitly the last design activity. ## 20. Keysets chosen from plaintext, and optional provider policies -**Status:** The recommendations below are not yet accepted. +**Status:** Still open. The recommendations below are not yet accepted. ### Request @@ -1481,8 +1571,7 @@ declarations, and the work needed for acceptance. ## 21. Packaging, naming, and retiring AsyncCipher -**Status:** The names and package moves are proposals. No implementation has -changed. +**Status:** Still open. The names and package moves are proposals. Pull request #1404 uses some agreed names. No package has moved. The user favours removing Vitamin C's `AsyncCipher`, and moving encryption behaviour for applications into Stack Encrypt. The @@ -1512,7 +1601,7 @@ Vitamin C #519. ## 22. Customer-driven order of work: Go JSON and match search first **Status:** Order of work set by the customer's needs. Merged work -reconciled on 27 October 2026. +reconciled on 7 November 2026. The conformance corpus is merged. ### The customer's needs come first @@ -1523,7 +1612,7 @@ characters. The quality of a standalone Rust release is secondary. - Claude's stack of JSON-tree pull requests, headed by #1363, has now merged. Revise the older Go plan in #1353 against it, and against the agreed API direction. -- Vitamin C #520 and #526 implement the storage profiles. Review by a +- Vitamin C #520 and #526 implement the storage profile: Vitamin C's versioned byte format for an opaque value that is stored. Review by a colleague, and the release, are pending. Stack still needs to adopt that release. @@ -1541,7 +1630,8 @@ Defer these: - the final Rust derives. The earlier exemption for prototype compatibility does not allow breaking -customer data after this release. +customer data. Since 7 November 2026, that applies from now, not only +after the final release, because the customer builds on Go rc.3. ### Short words in match search @@ -1596,8 +1686,7 @@ came from. #### Short-word terms Pull request #1397 implements short-word terms that are deterministic and -domain-separated, inside the existing Bloom filter. EQL's pinned match -profile turns them on. This avoids a separate index for short words. +domain-separated, inside the existing Bloom filter. EQL's pinned match configuration turns them on. This avoids a separate index for short words. It is not the random-variant mechanism explored earlier. It does not by itself resolve the user's concern about frequency and leakage. The native Go @@ -1618,7 +1707,42 @@ implemented. The remaining items include: The customer release does not wait for the final Rust traits or crate moves. -#### What this reconciliation checked +### Reconciliation with merged work, 7 November 2026 + +This review compared `next` at `e4990e49b` with the previous baseline, +`ffeed7adb`. It read each pull request description, and the code where the +description was not enough. It ran no test suite. + +#### What merged + +- #1404 adds the owned API for one value. #1501 extends it to `Vec`, and + moves the Rust crates to Vitamin C 0.9. +- #1491 makes repeated Rust context extensions append in order, as Go + already did. +- #1460, #1461, #1462, #1469, #1491, #1492, and #1493 add a conformance + corpus that native Rust and the Go guest both run. +- #1444, #1445, #1446, #1451, and #1452 add element-scoped and jsonpath + predicate queries over JSON documents, in the engine, EQL, and Go. +- #1500 documents Go `v0.1.0-rc.3`. That version was tagged on 2 November 2026. +- #1403 fixes findings on the release candidate, and bumps the crate + versions that `stack-encrypt` 0.3.0 needs. + +#### Remaining work on 7 November 2026 + +- the specification around the conformance corpus: versions, schemas, + and the Python and TypeScript runners; +- opening a child bound to its parent; +- configurable match pipelines in Go; +- direct typed EQL conversion; +- freezing the release candidate's stored format; +- later, separating the roles of context. + +#### Differences from the plan + +#1501 encrypts `Vec` as one value. The plan recommends encrypting it +byte by byte. The API plan records this as a question for the user. + +#### What the reconciliation of 27 October 2026 checked This review inspected the source and the existing tests. It did not run the run-time or database test suites again, and it changed no production code. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 8f3ec73d0..45310cc9d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -7,6 +7,34 @@ Date: 24 October 2026. This design exploration goes with the **Everything introduced here is proposed syntax, not an implemented API.** None of the code below compiles against Stack Encrypt today. +## Implementation status on 7 November 2026 + +This document was checked against `next` at commit `e4990e49b`. The +directional selectors are not implemented. These parts of the model are +now in the code: + +- **Required input.** Pull request #1404 adds `RequiredContext`, + `NoCallerContext`, and `OptionalContext` for the owned API. A required + `u64` stays required, even though `u64` implements `Default`. +- **Composition.** Pull request #1491 makes repeated Rust extensions append + in order. A grouped extension stays one part. This matches the + composition rule below. +- **Conformance.** Pull request #1491 adds shared Rust and Go cases for a + base with extensions, for stored-context expectations, and for altered + context. The nine scenarios below are not among those cases. + +Still open, or in progress: + +- `ContextExtension` and `ExpectedContext` as caller inputs. Pull request + #1504 adds them, and is in progress. +- `context(...)`, `from_context`, `prefix`, and `tuple(...)`. No pull + request adds them. + +From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not +break. So a new selector must produce the same context bytes as rc.3 for +data that rc.3 can write. See the [API +plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). + This document compares what happens to encryption context in each useful combination of context source and context storage. That comparison comes before anyone @@ -19,11 +47,12 @@ public workflow examples document linked above. These terms appear throughout. Each one is also explained where it first matters: -- **Encryption context** is a value that each ciphertext is bound to. Stack - Encrypt gives it to the cipher as additional authenticated data (AAD). - AAD is not encrypted, but decryption fails unless the same value is - supplied again. The same context also scopes a field's search index terms - (see `packages/stack-encrypt/src/target/context.rs`). +- **Encryption context**, or context, is data that a ciphertext is tied + to, such as a column name or a row ID. Decryption needs the same context. + The context also scopes a field's search terms. See [Context and + AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad). +- **AAD** means additional authenticated data. Encryption checks it for one ciphertext, but does not encrypt it. Stack Encrypt builds it from + the context. - **Static prefix** is the fixed string in the declaration, such as `"users"`. Together with the field name it forms a field's static base. - **Logical row context**, written `R`, is the part of the context that @@ -737,8 +766,11 @@ caller selector must stay required, even if its Rust type implements `Default`. A required `u64` must not silently become zero only because `u64: Default`. -That rule exposes a real limit in today's convenience methods. Their only -test is `Context: Default`. An implementation of this proposal would need +That rule exposed a real limit in the convenience methods of 24 October +2026. Their only test was `Context: Default`. Since #1404, the owned +methods use the explicit capabilities `AllowsNoContext` and +`AcceptsContext` instead. The structural methods in `target::legacy` still +use `Default`. An implementation of this proposal would need one of two things: - a `RequiredContext` input that has no default, with an easy diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index 4dd2376ea..36bcbe97f 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -1,6 +1,7 @@ # Stack Encrypt: complete public API examples for review -Date: 24 October 2026. +Date: 24 October 2026. Implementation status checked on 7 November 2026, +against `next` at commit `e4990e49b`. This document tests the proposed public Rust API of Stack Encrypt against three complete workflows. The code in it is a proposal. The code does not @@ -39,8 +40,14 @@ documents cover related ground: dynamic, and SQL examples, including extraction and walks. The user clarified on 25 October 2026 that this API is an early prototype. -The new API does not need to stay backward compatible with the old APIs or -their encodings. +The new API does not need to stay backward compatible with the old APIs. + +On 7 November 2026, the user decided that the stored format must not break +from now on, because the Go customer builds on Go `v0.1.0-rc.3`. The stored +format is what is written to the database, such as ciphertext envelopes and +search terms. The APIs can still change. This replaces the decision of 25 +October 2026 for stored data. See the [API +plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). ## Terms used in this document @@ -52,11 +59,12 @@ these terms from it: records which keyset that was. - **Data key**: the key that encrypts one value. Stack Encrypt requests data keys from ZeroKMS in batches. -- **Context**: extra input that is bound to a ciphertext when it is sealed, - such as `("users", "email")` or a record id. Decryption fails unless the - reader supplies the same context. Encryption uses the context as - additional authenticated data (AAD): data that encryption authenticates - but does not encrypt. Search terms are also derived from the context. +- **Context**: data that a ciphertext is tied to, such as + `("users", "email")` or a record id. Decryption needs the same context. + Search terms are also derived from it. See [Context and + AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad). +- **AAD**: additional authenticated data. Encryption checks it for one ciphertext, but does not encrypt it. Stack Encrypt builds it from the + context. - **Leaf**: one encrypted value. "Sealing" a leaf means encrypting and authenticating it. - **Envelope**: the complete stored form of a ciphertext. It includes the @@ -130,29 +138,32 @@ for example: ciphertext for the encoded collection instead. The latest recommendation applies the same rule to byte vectors. No wrapper -or special case is needed to keep the prototype's behaviour. These calls +or special case is needed to keep the prototype's behaviour. Pull request +#1501 differs: it encrypts `Vec` and `[u8; N]` as one value. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#differences-between-merged-code-and-this-plan) +records this as a question for the user. These calls need the proposed opaque ciphertext and support for targets that take no context. ### Status of each API surface -| Surface | Status in this review | -| --- | --- | -| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | -| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | -| No public `encrypt_as` in these examples | Agreed direction; current implementation still exposes it | -| `#[stash(passthrough)]` on output fields | Accepted in principle; needs implementation on both encrypt and decrypt derives | -| A fixed context on a scalar output derive | Agreed direction; not required by these particular examples | -| `OpaqueBlob` | Agreed wrapper direction; provisional name. Requests whole-value serialization and one encrypted leaf, using a library-owned codec | -| `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | -| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | -| `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | -| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64 or Base85 storage compose with encrypt and decrypt targets; final names remain for the naming pass | -| `Ciphertext` as an encryption or decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | -| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | -| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair, symmetric with direct encryption | -| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair on `DecryptFrom` | -| `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | +| Surface | Status in this review | Implementation on 7 November 2026 | +| --- | --- | --- | +| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction | Done for one value (#1404) | +| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | Done for one value and for `Vec` (#1404, #1501) | +| No public `encrypt_as` in these examples | Agreed direction | Still open. `encrypt_as` is still public | +| `#[stash(passthrough)]` on output fields | Accepted in principle | Still open. In progress in #1504 | +| A fixed context on a scalar output derive | Agreed direction; not required by these examples | Still open | +| `OpaqueBlob` | Agreed wrapper direction; provisional name | Still open. In progress in #1503 | +| `Ciphertext` | Proposed name for an opaque single-value ciphertext | Done (#1404) | +| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the canonical sealed-leaf encoding | Done (#1404). The envelope is unchanged | +| `Ciphertext: Serialize + Deserialize` | Proposed: Base64 for human-readable formats, bytes for binary formats | Done (#1404) | +| Encoded ciphertext field targets | Accepted; final names remain for the naming pass | Still open | +| `Ciphertext` as an encryption or decryption target | Proposed single-value target | Done (#1404). `Vec` is one value, unlike the recommendation (#1501) | +| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path | Still open. In progress in #1503 | +| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair | Done for one value (#1404) | +| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair | Done for one value and `Vec` (#1404, #1501). Records are in #1504 | +| `SearchableEmail` in example 3 | Application-defined target | Still open. #1504 turns example 3 into a test | The first three Rust programs use the proposed `OpaqueBlob`, `Ciphertext`, and passthrough APIs, or some of them. Each is a complete candidate program. @@ -168,8 +179,7 @@ typed wrapper is still an open alternative. The ciphertext's byte decoder only parses a stored envelope. Authentication happens when it is decrypted. The new Rust implementation and the dynamic implementation must agree on -the context and leaf encodings. They do not need to keep the prototype's -encoding. +the context and leaf encodings. On 24 October 2026 they did not need to keep the prototype's encoding. Since 7 November 2026, they must keep reading the encodings that Go rc.3 writes. The trait name `DecryptFrom` may surprise readers. The trait supplies the convenience functions, while `#[derive(DecryptInto)]` describes how an @@ -325,9 +335,9 @@ async fn main() -> Result<(), Box> { Without an application context, use the form `keyset.encrypt(OpaqueBlob::new(user))`. A matching no-context -`reader.decrypt(loaded)` is accepted, so that encrypt and decrypt match. It -is not the signature of today's decrypt method. Today's method takes two -arguments. +`reader.decrypt(loaded)` is accepted, so that encrypt and decrypt match. +Since #1404, `keyset.decrypt(ciphertext)` takes one argument for an opaque +value. `OpaqueBlob` itself is not merged. This document uses an explicit context in the full example so that the saved format is unambiguous. @@ -377,7 +387,7 @@ Before claiming such a guarantee, define these things: - which shape and length information may be public; - the restrictions on custom implementations. -A stricter profile may need bounded shapes, fixed-width encodings, padding, +A stricter set of supported types may need bounded shapes, fixed-width encodings, padding, and controlled encoders and decoders, with verification at the implementation level. Padding the ciphertext to a fixed size does not by itself make the serialization before it constant-time. Authentication must @@ -662,7 +672,9 @@ work as they do in example 2. Stack Encrypt already supports collection conversion, through its blanket implementation `EncryptFrom> for Vec` when `T: EncryptFrom` and -the target's context can be cloned. +the target's context can be cloned. Since #1501, the owned API has the same +batching for `Vec` into `Vec`. The batching counts below apply to +both. The conversion collects every row's pending operations before sending any of them. The two users need four data @@ -771,13 +783,20 @@ async fn main() -> Result<(), Box> { } ``` +Note added on 7 November 2026: this program no longer uses existing APIs +as written. Since #1404, the crate-root `EncryptInto` and `DecryptFrom` are +the owned traits. `TextEq` implements only the structural declarations. So +the program needs `stack_encrypt::target::legacy::{DecryptFrom, +EncryptInto}`, or `keyset.encrypt_as`, as `eql-bindings` uses. This was +checked by reading the source, not by compiling. + `TextEq` includes its EQL version, its identifier, the ciphertext, and the equality term. Its context type is `NonEmpty`. The application supplies the typed identifier for the expected table and column, on both writes and reads. Decryption checks it against the stored identifier before opening the ciphertext. -The `v4` namespace is deliberate: this Rust producer profile targets EQL v4. +The `v4` namespace is deliberate: this Rust producer targets EQL v4. The TypeScript integrations use EQL v3. ### Gap before `TextEq` can replace `SearchableEmail` diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 20cd7a865..415ca7079 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -1,7 +1,7 @@ # JSON tree: operation inventory and release contract -Updated 27 October 2026 against merged `next` -`ffeed7adb3f7d8739cda53261046147852bffc82`. This document accompanies the +Updated 7 November 2026 against merged `next` at commit `e4990e49b`. The +previous update was on 27 October 2026, at `ffeed7adb`. This document accompanies the [API plan](2026-10-24-stack-encrypt-api-plan.md). ## Summary @@ -13,6 +13,17 @@ J20, used in the table of operations below. Two operations still have gaps: J14 and J17. The SQL types and functions come from Encrypt Query Language (EQL). +Since 27 October 2026, two operations have been added as J21 and J22. +J21 tests several conditions on one array element. J22 sends a jsonpath +predicate as one query parameter. Both are in the engine, EQL, and Go. The +`eql-json-v1` conformance profile, from #1492, enrols the JSON operations +in Rust, Go, and PostgreSQL tests. + +From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not +break. The stored format is what is written to the database, such as JSON +tree entries and their search terms. See the [API +plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). + The Rust example in this document uses the proposed public derive and facade. That example is not executable code for the current engine. The [refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) @@ -37,9 +48,11 @@ and its findings on the engine and Go give the exact source evidence. | Root fallback | Past the walk budget, the database returns the sealed root alone. The client opens the whole document and picks out the requested values | | Wildcard bound | How many elements each `[*]` in a read path expands to. One extra probe at the bound detects a longer array | | Reads that Rust verifies against the original request | Prepared reads: Rust derives the request again when the rows are opened, and checks every row against it | -| Post-cascade profile | The storage and search format as it is after the cascaded tags change in #1379 | +| Post-cascade format | The JSON storage and search format after #1379 added cascaded tags. Go rc.3 writes it, so it must stay compatible | +| Conformance profile | A named, versioned set of test cases for one area, such as `eql-json-v1`. Each language runs the same cases. The files are in `conformance/stack-encrypt/v1/` | | Facade | The proposed public Rust API of `stack-encrypt` | -| AAD | Additional authenticated data (AAD): data that an authentication tag covers but the ciphertext does not contain | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | ## Current names @@ -53,6 +66,8 @@ and its findings on the engine and Go give the exact source evidence. | Prepared selector and prepared subtree read | `encrypt.JSONSelection`, `encrypt.JSONRead` | | SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | | Result of opening one value | `JSONValue{Found, JSON}` | +| Operand for several conditions on one array element | `eql.JSONSearchElementsQuery`, `eql_v4.query_json_elements` | +| Operand for a compiled jsonpath predicate | `eql.JSONSearchPredicateQuery`, `eql_v4.query_json_predicate` | The handwritten Rust EQL adapters keep their `JsonTree*` names. The internal engine type `JsonDocument` is not the Go ciphertext type. @@ -63,7 +78,7 @@ internal engine type `JsonDocument` is not the Go ciphertext type. The SDK resolves the target options configured for a field into native `JsonOptions`. The SDK uses the same options for writes, queries, and -reads. The released profile uses column identity and keyset selection. +reads. The released configuration takes a document's context from its column, and its keys from the selected keyset. Separate row binding and search scope are accepted future API work. Row binding and search scope are not supported as EQL context inputs today. @@ -84,7 +99,7 @@ authenticate. | J03 | Write and read in batches | Implemented, with a shared executor and chunking by the provider | None listed | | J04 | Contains | Implemented: `Contains`, `contains`, and `@>` with `query_json` | Generated code checks the needle and refuses some shapes | | J05 | Contained by | Implemented: `ContainedBy`, `contained_by`, and `<@` | None listed | -| J06 | Scalar equality at a path | Implemented: `EqAt`, `eq_at` | Compares value selectors, not ordering terms | +| J06 | Scalar equality at a path | Implemented: `EqAt`, `eq_at` | Compares value selectors, not ordering terms. J21 keeps several tests on one array element | | J07 | The four ordering predicates | Implemented in runtime paths and in `At.` accessors | Ordered number and string paths only | | J08 | Inclusive range | Implemented: `Between`, `between_at`, with named lower and upper operands | None listed | | J09 | Sort | A selector and explicit SQL | Sorting at a wildcard path needs a defined reduction. Do not assume the first array element | @@ -98,7 +113,15 @@ authenticate. | J17 | Immediate children | `Children` prepares a concrete token. SQL returns the child entries | Opening child scalars bound to their parent is not implemented | | J18 | Count children | A `ChildCount` token. SQL returns a count that can be NULL | The count is not an authenticated proof of completeness | | J19 | Walk budget and root fallback | Implemented. After root fallback, the client applies the selection originally requested | None listed | -| J20 | Comparisons between stored values | Existing document containment and entry ordering | Only within a compatible column, keyset, and profile | +| J20 | Comparisons between stored values | Existing document containment and entry ordering | Only within a compatible column, keyset, and storage format | +| J21 | Several conditions on one array element | Implemented in the engine and EQL (#1445), and in Go as `Elements..Match` (#1451) | Needs ordered paths. Uses no index | +| J22 | A jsonpath predicate as one parameter | Implemented in the engine and EQL as `match` and `@@` (#1446), and in Go as `Match` (#1452) | Supports a subset of jsonpath. Uses no index | + +Status on 7 November 2026: no merged pull request changes J14 or J17. The +`eql-json-v1` profile enrols tests for J01 to J08, J10, J11, J15, J16, and +J19, with J21 and J22. Its requirement IDs are not the J numbers. I made this mapping from that conformance profile's contract. I did not check it test by test. + +In J21, equality compares folded ordering terms, not exact values. In J22, the EQL reference documents five cases where the answer differs from `jsonb_path_match`. ### Open requirements for J13 and J17 @@ -133,7 +156,12 @@ subtree read. such as `[n]` are for subtree reads. A wildcard selector addresses every path that differs only in its array positions. Two separate EqAt predicates can match different elements of one array. One containment - needle keeps them matched to the same element. + needle keeps them matched to the same element. So does J21. +- **Native `jsonb` functions give wrong answers.** The encrypted column is + a PostgreSQL domain over `jsonb`. So PostgreSQL's own `jsonb` functions + accept it, and run over the stored envelope, not the document. For + example, `jsonb_path_match(col, ...)` returns false for every row, with + no error. #1444 documents this and does not block the functions. - **Strict checks on JSON input.** The Rust parser now rejects duplicate keys and out-of-range integer literals before key I/O, that is, before any request to the key provider. For typed Go values, Go encodes first, @@ -162,7 +190,8 @@ subtree read. document whose root is not sealed cannot fall back to the sealed root. The walk budget does not limit the size of every response. - **Header version.** The header version is still 1, although the cascade - changed what the AAD binds. Freeze the post-cascade profile explicitly. + changed what the AAD binds. Freeze the post-cascade format explicitly. Go rc.3 writes this format. Since 7 November 2026, a later change must + keep it readable and searchable, or add a new header version beside it. The unchanged header does not show compatibility with older prototype ciphertexts. The query projection order is now random, so parity tests need to compare projections in a canonical form, by meaning. @@ -246,6 +275,12 @@ include: - `{"path"}` - `{"path","wildcard_bound"}` +Since #1445 and #1446, two more operations take these inputs: + +- `element_match` takes `{"path", "wildcard_bound", "predicates"}`. The + `wildcard_bound` is optional. +- `match` takes `{"predicate", "vars"}`, as `jsonb_path_match` does. + JSON values are UTF-8 JSON text. Query results state the `encoding` (`eql`, `text`, `text_array`, or scalar `bytes`), the SQL domain, and the operands. Capabilities state the shapes of input. @@ -341,16 +376,14 @@ A row ID is never an index or query input implicitly. ## Remaining work and release gates -1. Write down the implemented wire grammar, errors, presence states, - and context profile in the formal specification and the set of - conformance tests that were promised. General Rust definitions, +1. Write down the implemented wire grammar, errors, and presence states in the formal specification and the promised conformance tests. Write down the context rules of the released configuration there too. General Rust definitions, caching, and more context roles can come later. 2. Close the stated scope gaps in J14 and J17. Subtree reads are the supported way to get plaintext for all matches. An opener for children bound to their parent is separate engine work. Do not add generic decryption of raw entries as a shortcut. 3. Replace the internal EQL bridge with typed conversion. The bridge - serialises to `Value` and deserialises again. Keep the frozen profile. + serialises to `Value` and deserialises again. Keep the stored format. The change is an optimisation, not a missing execution path for Go JSON. 4. Run the existing Rust, guest, Go, and PostgreSQL suites on the release @@ -371,6 +404,16 @@ A row ID is never an index or query input implicitly. pins of Vitamin C before a final release. Opaque storage is already adopted. Opaque storage is a separate format from JSON tree storage. +### Status of this work on 7 November 2026 + +| Item | Status | +| --- | --- | +| 1. Specification and conformance | Partly done. #1492 adds the `eql-json-v1` conformance profile. That profile is experimental: it does not certify a whole SDK | +| 2. J14 and J17 | Still open | +| 3. Typed EQL conversion | Still open | +| 4. Run the suites on the release candidate | Partly done. #1492 runs them in CI, including a job against PostgreSQL | +| 5. Freeze the stored vectors and the EQL bundle | Still open. Go rc.3 already writes this format, so the frozen format must keep it readable | + ## What this refresh did not check This refresh is a review of source and documentation, not a new runtime diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 2552b8621..0438e1519 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -1,6 +1,7 @@ # Stack Encrypt packaging and public names -Dated 25 October 2026, with an implementation update on 27 October 2026. +Dated 25 October 2026, with implementation updates on 27 October 2026 and +7 November 2026. The last check was against `next` at commit `e4990e49b`. This recommendation for review accompanies the [implementation plan](2026-10-24-stack-encrypt-api-plan.md). @@ -21,6 +22,43 @@ contract is under separate user review. - Adopt the public glossary for Rust. Keep the merged Go and SQL names for JSON. +## Terms used in this document + +| Term | Meaning | +| --- | --- | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026) | + +## Update on 7 November 2026 + +Pull request #1404 implements some names from the glossary below. They are +`Ciphertext`, `EncryptInto`, `DecryptFrom`, `OpaqueEncode`, `OpaqueDecode`, +and the traits `target::EncryptTarget` and `target::DecryptSource`. Pull request #1504 adds the derives with those names. Pull request #1503 adds `OpaqueBlob`. Both are in progress. + +These parts of the proposal are not implemented: + +- the `Cipher` rename, `TargetDefinition`, `PrimaryKey`, and `ScopeKey`; +- the `encoding` module and the encoded ciphertext wrappers; +- `.target::()` on the plan builder, and the `provider` module; +- the `stack-encrypt-format` crate, and the rename of `stack-kms`; +- the retirement of `AsyncCipher`. + +`encrypt_as` is still public. The structural `decrypt_into` is still +public, in `target::legacy`. `StackCipherText` remains for the tree API. The tree methods are `encrypt_tree` and `decrypt_tree`. + +Other changes that affect packaging: + +- Pull request #1501 moves the Rust crates to Vitamin C 0.9. It also + re-exports `Buffer`, `ProtectedRef`, and `AsProtectedRef` from the crate + root. +- Pull request #1403 bumps `stack-kms` to 0.2.0 on crates.io. So the + proposed rename to `stack-zerokms` would now rename a published crate. + +From 7 November 2026, the stored format must not break. The user's +exemption for prototypes below now applies to the APIs only. Moving codecs +and adapters between crates must keep the bytes that Go rc.3 writes. + ## Update on 27 October 2026 Go JSON and opaque storage are merged into `next`, through #1369, #1370, @@ -29,8 +67,8 @@ words. What remains before customers can use them is configuration and conformance work, and release readiness. That includes replacing the -temporary Go module pins of Vitamin C. Rust stays on Vitamin C 0.8.0 from -crates.io. A customer release does not wait for crate extraction or for the +temporary Go module pins of Vitamin C. Rust stayed on Vitamin C 0.8.0 +from crates.io until #1501 moved it to 0.9. A customer release does not wait for crate extraction or for the final Rust facade (the proposed public Rust API). ## Sources for this assessment @@ -63,7 +101,8 @@ encryption traits to implement an application type. Moving code should remove duplicate abstractions. Moving code should not keep both public surfaces under new names. The user's compatibility -exemption for prototypes applies to this API. Before deleting +exemption for prototypes applies to this API, but not to the stored format +since 7 November 2026. Before deleting dependencies upstream, review the other consumers of Vitamin C that use them. @@ -73,7 +112,7 @@ them. | --- | --- | | `stack-encrypt` | The public Rust API; traversing and reconstructing values; target declarations and definitions; context and scope roles; normalisation and adapters for searchable targets; plans; batching; JSON tree execution; and validating and executing dynamic definitions | | `stack-encrypt-derive` | Derive macros that generate the same declarations and definitions a user can write by hand. `stack-encrypt` re-exports the derives | -| `stack-encrypt-format` (new) | Pure data representations and codecs shared by the engine and the bindings: frame primitives and the required scalar representations, the stable blob profile, parsing and encoding of the ciphertext envelope, and validated Base64 and Base85 text. No key requests and no execution runtime | +| `stack-encrypt-format` (new) | Pure data representations and codecs shared by the engine and the bindings: frame primitives and the required scalar representations, the stable format of stored blobs, parsing and encoding of the ciphertext envelope, and validated Base64 and Base85 text. No key requests and no execution runtime | | `stack-zerokms` (the current `stack-kms`, renamed) | The ZeroKMS protocol client, authentication integration, native keysets, request types for lock context, and responses. Not the neutral abstraction over several providers | | `eql-bindings` | EQL storage and query types by version, JSON serialisation compatible with SQL, the catalog, and Stack target adapters. Stays downstream of the engine | | Vitamin C crates | Protected memory; randomness; AEAD and key operations on bytes; HMAC, PRF, and ORE algorithm primitives; and reusable canonical encoding of context | @@ -119,7 +158,7 @@ runtime. - Re-export the ordinary format types through Stack Encrypt, so users normally need only that one dependency. -The format crate is one implementation with separate profiles. Each of +The format crate is one implementation with separate formats. Each of these has its own contract and versioning requirements: blob storage, the transport to and from the guest, and ciphertext envelopes. Sharing code must not silently turn the Serde mapping for short-lived transport into the stored @@ -190,7 +229,7 @@ opaque ciphertext. ### Frames and codecs Move the generic frame code and the codecs that Stack needs into the -format crate. Stack owns the stored blob profile and its compatibility +format crate. Stack owns the stored blob format and its compatibility promise. This refines the earlier recommendation to reuse Vitamin C's frame codec. @@ -370,7 +409,7 @@ must not redefine stored SQL formats or dynamic discriminants by accident. Replace the temporary Go module pins of Vitamin C for a final release. 2. Complete the outstanding Go match configuration, and close the gaps in JSON reads and conformance. Short whole-word matching is implemented in - EQL. Its leakage and profile compatibility still need explicit + EQL. Its leakage and its compatibility with the stored format still need explicit acceptance for the release. 3. After the customer release, set up the shared format crate. Move the value model, traversal, and the normalisation and index adapters that @@ -382,7 +421,7 @@ must not redefine stored SQL formats or dynamic discriminants by accident. must already have been handled before the release. 5. Complete the public Rust trait and derive names, imports, and executable examples. These changes must not rename the released Go API - or reinterpret stored profiles. The changes must not require customers to + or reinterpret stored data. The changes must not require customers to re-encrypt or reindex. Acceptance includes: diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index 99202c076..6f490033d 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -1,12 +1,22 @@ # Key selection, lock context, and provider capabilities -Dated 25 October 2026, with an update on 27 October 2026. This assessment +Dated 25 October 2026, with updates on 27 October 2026 and 7 November +2026. The last check was against `next` at commit `e4990e49b`. This assessment accompanies the [API implementation plan](2026-10-24-stack-encrypt-api-plan.md). ## Status and requests This document contains recommendations for review. The recommendations -are not accepted public signatures. Nothing in them is implemented. +are not accepted public signatures. Nothing in them is implemented. On 7 +November 2026, no merged pull request implemented keyset routing from +plaintext, lock context, or the neutral provider split. + +From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not +break. The stored format is what is written to the database, such as +ciphertext envelopes and search terms. The refined-tag work in item 6 +below changes the stored tag, so it must keep rc.3 data readable. See the +[API +plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). The user who requested this assessment asked for three things: @@ -37,7 +47,8 @@ The JSON operation contract is under separate user review. | `IndexKeySource` | The trait in `stack-kms` that loads the deterministic index key of a keyset, used to derive search terms | | `Pending` | The request object that `StackCipher` returns. Settling it (awaiting it) sends the batched ZeroKMS calls | | Key I/O | Any request to the key provider to generate or retrieve a key | -| AAD | Additional authenticated data (AAD): data that an authentication tag covers but the ciphertext does not contain | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | +| AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | | `MaybeSend` | A bound that means `Send` on native targets and no bound on WebAssembly (WASM) | | Vitamin C | CipherStash's library of cryptographic primitives | @@ -63,6 +74,19 @@ explicitly. The engine must not infer it from a name. ## Current implementation +### Update on 7 November 2026 + +- Pull request #1501 batches an owned `Vec` collection. Encrypting makes + one `generate_keys` call. Decrypting makes one `retrieve_keys` call for + each keyset. A ciphertext from another keyset is refused before any key + is retrieved. +- Pull request #1493 adds lifecycle conformance, including bounded + concurrency when the KMS client splits a batch into chunks. +- Pull request #1501 moves the Rust crates to Vitamin C 0.9. The section + on `AsyncCipher` below describes 0.8.0, and was not checked against 0.9. +- Pull request #1403 bumps `stack-kms` to 0.2.0, and `stack-auth` to + 0.44.0. Neither bump adds lock context or routing. + ### Update on 27 October 2026 #1379 makes verification of a JSON document whose root is not sealed load @@ -411,9 +435,9 @@ a registered provider and key format. The metadata never names arbitrary network endpoints or credentials to execute. Keep ZeroKMS's key tag simple for Rust and SQL consumers to extract, -within its documented envelope profile. Other providers' metadata does not +within its documented envelope format. Other providers' metadata does not need to invent such a tag. The stable encoding of blob payloads does not -depend on these envelope profiles of each provider. +depend on the envelope format of each provider. ### Future providers From 936567a3908a9993a3095c82e4a44a4b1c1c4a6d Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:12:21 +1100 Subject: [PATCH 07/42] docs: lead the Stack Encrypt API plan with its target state Restructure the API plan into three parts: the target state, shown as Rust and Go code; the remaining work to reach it, in customer order; and the open questions. Cut the history and long assessments, which the review log keeps. Update links in the other plans that pointed to removed sections. --- .../2026-10-24-stack-encrypt-api-plan.md | 2869 ++++------------- .../2026-10-24-stack-encrypt-api-review.md | 16 +- ...26-10-24-stack-encrypt-context-examples.md | 2 +- ...026-10-24-stack-encrypt-public-examples.md | 4 +- ...026-10-25-stack-encrypt-json-operations.md | 2 +- .../2026-10-25-stack-encrypt-packaging.md | 2 +- ...-25-stack-encrypt-provider-capabilities.md | 2 +- 7 files changed, 593 insertions(+), 2304 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 2131c7167..d96c1d3e7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1,2404 +1,691 @@ -# Stack Encrypt API implementation plan - -First written on 24 October 2026. Checked against the merged code on 7 -November 2026, at `next` commit `e4990e49b`. The previous check was on 27 -October 2026, at `ffeed7adb`. - -Status: proposed implementation direction. This plan develops the [API -review log](2026-10-24-stack-encrypt-api-review.md), the [public -examples](2026-10-24-stack-encrypt-public-examples.md), and the [context -examples](2026-10-24-stack-encrypt-context-examples.md) into implementation -choices, and sets the criteria for finishing each one. - -The plan marks which statements are the user's requirements and which are -recommendations that nobody has accepted yet. "The user" is the person who -asked for the review and who decides its open questions. The code examples -show proposed APIs, not working code in the current release. - -## Summary - -- The Go JSON foundations and Go opaque storage are merged. Since 27 - October 2026, three more things have merged. They are the owned Rust API for single values and collections, a conformance corpus for Rust and Go, and two new JSON queries. See [Status on 7 November - 2026](#status-on-7-november-2026). -- The customer work that remains is listed in [Remaining customer - work](#remaining-customer-work). It includes opening a child bound to its - parent, configurable Go match options, the gaps in the specification, and - release readiness. -- The Go customer release comes first. The final Rust facade and the crate - moves come after it, and do not hold it up. -- On 7 November 2026, the user decided that the stored format must not break from now on. The Go customer has started building on Go `v0.1.0-rc.3`. The Rust and Go code and APIs can still change in breaking ways. See [Stored format must stay compatible from 7 November 2026](#stored-format-must-stay-compatible-from-7-november-2026). -- The [decision checkpoint](#decision-checkpoint-on-25-october-2026) lists - the settled directions. The [remaining questions](#remaining-questions) - table lists what is still open. - -## Terms used in this plan +# Stack Encrypt API plan -| Term | Meaning | -| --- | --- | -| AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | -| AEAD (authenticated encryption with associated data) | The cipher that seals each value and authenticates its AAD | -| Blob | A value that `OpaqueBlob` encodes as a whole and seals as one ciphertext | -| Bloom filter | The bit array that the match index stores. Each term sets some bits, so a query can test whether a value may contain the term | -| Cascaded tags | Each JSON container's authentication tag covers the tags of its children. #1379 added them | -| Column and keyset configuration | The released configuration. A JSON document's context is its column, and its keys come from the selected keyset. It accepts no row binding or search scope input | -| Configured target | A target together with typed options, such as a JSON target with a list of ordered paths | -| Conformance corpus | The shared test cases and requirement lists in `conformance/stack-encrypt`. Native Rust and the Go guest both run them | -| Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Each language runs the same cases. The files are in `conformance/stack-encrypt/v1/` | -| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | -| Dynamic interface | The interface that takes declarations and requests as data instead of Rust types. Go uses it through the guest | -| EQL (Encrypt Query Language) | The SQL types and functions in `packages/eql` that store and query encrypted values in PostgreSQL | -| Facade | The proposed public Rust API of `stack-encrypt`: its traits, derives, and convenience methods | -| FFI (foreign function interface) | The boundary through which another language calls the Rust engine | -| Guest | The Rust engine compiled to WebAssembly, which the Go SDK runs | -| Key I/O | Any request to ZeroKMS to resolve, load, or derive keys | -| Keyset | A set of keys in ZeroKMS, the CipherStash key management service (KMS) | -| OPE (order-preserving encryption) | Encryption whose ciphertexts keep the order of the plaintexts | -| Open and seal | To seal a value is to encrypt and authenticate it. To open it is to authenticate and decrypt it | -| ORE (order-revealing encryption) | Index terms that reveal the order of the plaintexts, for range queries and sorting | -| Owned API | The new Rust conversion API, which consumes its plaintext input: `keyset.encrypt`, `EncryptInto`, and `DecryptFrom`, over `EncryptTarget` and `DecryptSource` declarations. #1404 added it | -| Passthrough field | A field copied unencrypted from the plaintext type to the encrypted output | -| Plan | The lower-level description of which fields a conversion encrypts, indexes, or passes through. Derives and dynamic declarations compile to plans | -| Post-cascade format | The JSON storage and search format after #1379 added cascaded tags and randomised order. Go rc.3 writes it, so it must stay compatible | -| Pseudo-version | A Go module version that names a commit instead of a released tag | -| Reads that Rust verifies against the original request | Go prepared reads. When the rows are opened, Rust derives the request again and checks every row against it | -| Refined tag | A proposed tag that Stack Encrypt would derive from the ZeroKMS key tag with an HMAC, and store in place of the original tag. See [Typed encodings and extractable ZeroKMS tags](#typed-encodings-and-extractable-zerokms-tags) | -| Row binding | Context that ties a ciphertext to one row, such as the row ID, so that it opens only for that row. Also called ciphertext binding | -| Search scope | The part of the context that also goes into index keys and search terms, such as a tenant ID. One query matches every row in the scope | -| Storage profile | Vitamin C's versioned byte format for an opaque value that is stored. Vitamin C #520 added it | -| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [Stored format must stay compatible from 7 November 2026](#stored-format-must-stay-compatible-from-7-november-2026) | -| Structural API | The earlier Rust API over borrowed input and the `EncryptFrom` and `DecryptInto` declarations. Its convenience methods are now in `stack_encrypt::target::legacy` | -| Target definition | The resolved, immutable form of a target's declaration: options, prepared indexes, context rules, and operation metadata | -| Target | A type or dynamic name that says how a value is encrypted and stored, such as `JsonSearch` or `TextEq` | -| Typestate | A builder whose type changes with each call, so the compiler refuses to run an incomplete request | -| Vitamin C | CipherStash's cryptography library, `vitaminc`. Its codec encodes values as binary frames | - -## Context and AAD +Date: 7 November 2026. Checked against `next` at commit `e4990e49b`. -The context is the data that a ciphertext is tied to, such as a column -name or a row ID. The field's declaration supplies part of the context, -such as `users/age`. The caller can supply the rest, such as a row ID. - -Stack Encrypt uses the context in three ways: - -1. **To derive the data key.** Stack Encrypt sends the context to ZeroKMS - as a text descriptor, such as `users/age`. ZeroKMS needs the same - descriptor to derive the key again. -2. **To build the AAD of each ciphertext.** The AAD is data that - encryption checks without encrypting it. Stack Encrypt builds it from - the context and the place of the value in its record. The AAD also - includes the format version, the keyset ID, and the key tag. -3. **To derive the search terms** for the field. +Stack Encrypt encrypts each value under its own data key from ZeroKMS, +CipherStash's key service. It also makes the search terms that let +PostgreSQL query the encrypted value. The Rust crate `stack-encrypt` is the +engine. The Go SDK runs the same engine through the dynamic interface. -A different context gives a different key and a different AAD, so -decryption fails. +This plan is for the engineers who build Stack Encrypt, and for the lead +who approves its design. It has three parts: -A parameter that takes the caller's context is named `context`. The name -`aad` is kept for a value that is an AAD, such as the derived AAD of one -leaf. Some public methods do not follow this rule yet. See [Name context -parameters `context`](#name-context-parameters-context). +1. [Target state](#target-state): the API we are building, shown as code. +2. [Remaining work](#remaining-work): what is left to build to reach it. +3. [Open questions and decisions to be + made](#open-questions-and-decisions-to-be-made). -Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` -describes the descriptor and the AAD. The module documentation of -`packages/stack-encrypt/src/target/context.rs` describes the search terms. +The history of each decision is in the [review +log](2026-10-24-stack-encrypt-api-review.md). -## Status on 7 November 2026 +## Terms -On 7 November 2026, this plan was checked against merged `next` at commit -`e4990e49b`. Each pull request description was read. Where the description -did not settle a status, the code on `next` was read too. No test suite was -run for this check. +| Term | Meaning | +| --- | --- | +| AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | +| Ciphertext | One encrypted value, as the Rust `Ciphertext` type. It is stored as bytes, or as Base64 text | +| Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Rust and Go run the same cases. The files are in `conformance/stack-encrypt/v1/` | +| Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | +| Dynamic interface | The engine's interface for languages other than Rust. It takes a declaration and values as data, not as Rust types | +| EQL (Encrypt Query Language) | The PostgreSQL types and functions in `packages/eql` that store encrypted values and answer queries on them | +| Guest | The Rust engine compiled to WebAssembly. The Go SDK runs it inside the Go process | +| Keyset | A set of keys in ZeroKMS. Every data key belongs to one keyset. A tenant can have its own keyset | +| Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | +| Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | +| Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | +| Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | +| Target | The encrypted type that a value is stored as, such as `Ciphertext`, an EQL type like `TextEq`, or a record type | -### Merged between 27 October and 7 November 2026 +## Target state -| Pull request | What it delivers | Plan items it affects | -| --- | --- | --- | -| #1404 | The owned API for one value, with the `Ciphertext` type and explicit context inputs. Renames the structural keyset methods to `encrypt_tree` and `decrypt_tree` | Requirements; decrypt interface; public traits; work areas A, B, and C | -| #1491 | Repeated Rust context extensions append in order, as in Go. Adds context conformance cases | Work area B; the dynamic specification | -| #1501 | Owned `Vec` collections, encrypted as one batch of key requests. Moves the Rust crates to Vitamin C 0.9 | Requirements; work areas C and D; cleanup | -| #1460, #1461, #1462, #1469 | Conformance profiles for scalars, declarations, runtime values, stored reads, value kinds, and query operators, run by native Rust and the Go guest | Implementation step 1; the dynamic specification | -| #1492 | The `eql-json-v1` conformance profile, run in two required CI jobs: a fast job without a database, and a job against PostgreSQL | Implementation steps 1 and 3; the JSON contract | -| #1493 | The `lifecycle-v1` conformance profile for ownership release, cancellation, cleanup, and key I/O | Cleanup acceptance work | -| #1444, #1445, #1446, #1451, #1452 | Two new JSON queries in the engine, EQL, and Go. #1444 documents a limit of native `jsonb` functions | The JSON contract | -| #1500 | Go documentation and a changelog for `v0.1.0-rc.3`, which was tagged on 2 November 2026 | Release readiness | -| #1403 | Fixes from the review of the release candidate. Bumps the crate versions that `stack-encrypt` 0.3.0 needs | Release readiness | - -### Remaining customer work - -The conformance corpus is now merged, so it is no longer the first item. -The customer work that remains: - -1. **Finish the specification around the corpus.** The conformance profiles are marked experimental. That means they test parts of the contract, but do not certify a whole SDK. The contract is not frozen. JSON Schema files, - protocol version identifiers, a release compatibility matrix, and fresh - cross-language exchange of JSON values are not done. Python and - TypeScript runners are not done. -2. **Open a child bound to its parent** (J17 in the JSON contract). No - merged pull request implements it. -3. **Configurable Go match options** (#1112), and the tokeniser and filter - pipeline. No merged pull request implements them. -4. **Freeze the release candidate.** Freeze the post-cascade JSON vectors - and the matching EQL bundle. The new conformance jobs run in CI, but - they do not freeze the stored format. -5. **Replace the temporary Go pins of Vitamin C** before a final Go - release. Go `v0.1.0-rc.3` is a prerelease, so it can keep them. - -### Differences between merged code and this plan - -These differences are recorded here. This plan's decisions are not changed -to match them. - -- **Byte vectors.** This plan recommends that `Vec` encrypts byte by byte into `Vec`. It recommends `OpaqueBlob>` for a whole vector. Pull request #1501 does the opposite. In #1501, `Vec` and `[u8; N]` encrypt into one `Ciphertext`. In #1501, `u8` is not an `OpaqueEncode` type. The reason given in #1501 is that Vitamin C has no single value for `u8`. Vitamin C could not read an element that only Stack could encode. - The user needs to decide whether to accept this. -- **The structural API stays beside the owned API.** This plan says to - remove replaced APIs directly. On `next`, `EncryptFrom`, `DecryptInto`, - the derives with those names, and the public `encrypt_as` remain. Their - borrowed convenience methods moved to `stack_encrypt::target::legacy`. - Issue #1488 tracks their removal. -- **Context composition.** #1491 makes repeated extensions append. That matches this plan's order: the base, then each extension. The dynamic - grammar and Go `Extend` accept only scalar extension parts, and refuse a - grouped part. Go accepts an empty text extension when the base is - nonempty, and Rust's `NonEmpty` refuses it. The context contract records - both differences as part of the supported API. -- **Compatibility of stored data.** Pull request #1491 changes which context repeated Rust extensions produce. The #1491 changelog explains how to read old records. Data written by Go `v0.1.0-rc.3` is not affected. The [stored-format section](#pull-request-1491-and-data-written-by-go-rc3) gives the evidence. -- **Vitamin C 0.9 and the order schemes.** Pull request #1501 moves to Vitamin C 0.9, `ore-rs` 0.11.0, and `clww-ore` 0.7.0. The #1501 description says that neither order-scheme release changes a stored order term. This check did not test that claim against rc.3 data. - -### Work in progress in open pull requests - -These pull requests target `next` and are not merged. Nothing in them is -counted as done in this plan. - -- #1503 adds `OpaqueBlob`. The wrapper encodes a value with Vitamin C's storage profile, and seals it as one `Ciphertext`. -- #1504 adds the `EncryptTarget` and `DecryptSource` derives, the - `#[stash(passthrough)]` attribute, and the context inputs - `ContextExtension` and `ExpectedContext`. It turns the public examples - for mixed and searchable records into tests. - -### Merged Go work - -The check on 27 October 2026, against `next` at commit `ffeed7adb`, found -this Go work merged: - -- the Go JSON catalog, configured targets, queries, reads that Rust verifies - against the original request, typed documents, and executable - documentation (#1372, #1373, #1378, and #1402); -- Go adoption of opaque storage (#1369 and #1370). - -The remaining customer work does not implement those foundations again. -See [Remaining customer work](#remaining-customer-work), the refreshed [Go JSON -plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) -and the [JSON operations -inventory](2026-10-25-stack-encrypt-json-operations.md). - -### JSON storage and match changes in #1379 and #1397 - -Pull request #1379 added cascaded tags, and randomised the order of stored -entries and query projections. A wildcard `->` now selects an unspecified -element, so old SQL recipes that relied on the first element in document -order are invalid. Freeze the post-cascade format. Compare query semantics -in tests, not the randomised byte order. - -Pull request #1397 implements short whole-word terms in the shared Bloom -filter. Configurable native Go match options and broader tokeniser and -filter work remain. This paragraph records implementation status only. It -does not accept a stronger privacy claim for those terms. - -### Temporary Vitamin C pins - -Go's Vitamin C storage modules are temporarily pinned to pseudo-versions at -commit `06568ba93bd9`. The guard for a final release requires each -pseudo-version to be replaced with a released module tag. Prereleases can -keep pseudo-versions, and Go `v0.1.0-rc.3` still uses them. #1501 moved -the Rust workspaces from Vitamin C 0.8.0 to 0.9 from crates.io. Schedule -the crate moves and the final Rust facade after the customer release. - -### Decisions made on 25 October 2026 - -The latest decisions about what to release: - -- Encoded ciphertext field targets are accepted. These are fields whose - stored output is the ciphertext as text, such as Base64. -- The comprehensive JSON contract for the first release, with Rust and - dynamic examples, is in [JSON - operations](2026-10-25-stack-encrypt-json-operations.md). The contract - includes extraction and walks. -- ORE does not provide equality at first. Keep the option of a future - explicit capability for encodings that preserve equality. -- Make stable blob storage bytes the priority, through [Vitamin - C #519](https://github.com/cipherstash/vitaminc/issues/519). Improvements - to zeroisation and timing can follow without changing the stored format. -- The review of names and packages is now in progress. - -Where these decisions conflict with earlier sequencing recommendations in -this plan, these decisions apply. - -### Follow-up decisions - -- Custom Rust query stages can be explicitly Rust-only. They export to the - dynamic interface as `NotPortable` until they are registered. Built-in - capabilities keep parity between Rust and Go. -- Every validation that does not need keys runs before keys are resolved or - loaded. -- The user is still reviewing the JSON operation contract. Its detailed API - proposals are not approved as a set. -- The [provider - capabilities](2026-10-25-stack-encrypt-provider-capabilities.md) companion - assesses the new questions about keyset routing, lock context, and - provider capabilities. - -### Packaging review - -The user favours retiring Vitamin C's `AsyncCipher`, and moving application -encryption behaviour into Stack. The [packaging and naming -recommendation](2026-10-25-stack-encrypt-packaging.md) defines: - -- crate ownership; -- a dependency graph with no cycles; -- the proposed shared format crate; -- role-based public names; -- coordinated migration work. +This section shows what a developer writes when every item in this plan is +built. A comment marks each piece of code that does not compile on `next` +today. [Remaining work](#remaining-work) says what each of those needs. -The recommendation is a proposal for review. Nobody has accepted the -renames. #1404 implements some of the names in its glossary that were -already agreed: `Ciphertext`, `OpaqueEncode`, `OpaqueDecode`, -`EncryptTarget`, and `DecryptSource`. No crate moves or renames are -implemented. +### Rust -### Stored format must stay compatible from 7 November 2026 +These examples use the `stack-encrypt` crate directly. -**Decision of 7 November 2026.** The user decided: "The stored data must not break as of now. The Go customer has already started implementation with rc.3. We can make breaking changes to the code/API but not the stored format." +#### Connect and choose a keyset -The stored format is what is written to the database. It includes ciphertexts with their envelopes and headers. It includes index and search terms. It includes JSON tree entries. +A client reads its credentials from the environment, or from the profile +that `npx stash auth login` writes. Every encryption uses one keyset. A +program can use the client's default keyset, or select another by name or +ID. -The baseline is the Go module `github.com/cipherstash/stack/languages/golang` at tag `languages/golang/v0.1.0-rc.3`. That tag is commit `114e08448`, built on 2 November 2026 from `next` at `929451b49`. The baseline also includes the EQL v4 SQL bundle, `eql-v4-install.sql`, attached to that release. The rc.3 changelog, from #1500, describes the release. +```rust +use stack_encrypt::{kms::IdentifiedBy, StackCipher}; -This decision replaces the decision of 25 October 2026 for stored data. For the code and the APIs, the earlier decision still applies: Rust and Go APIs can change without compatibility aliases or deprecation periods. Any change that would alter the stored format must do one of two things: +let cipher = StackCipher::new().await?; +let keyset = cipher.default_keyset(); +let tenant = cipher + .keyset(IdentifiedBy::Name("acme".to_string().into())) + .await?; +``` -- keep data written by rc.3 readable and searchable; or -- add a new, versioned format beside the rc.3 format, and keep reading the rc.3 format. +#### Encrypt and decrypt one value -This plan does not choose between the two for any item. The open items are listed in [Work that the stored-format decision constrains](#work-that-the-stored-format-decision-constrains). +`encrypt` takes ownership of the value and wipes it. It returns one +`Ciphertext`. A context ties the ciphertext to where it belongs, so it +cannot be decrypted anywhere else. -#### History: the decision of 25 October 2026 +```rust +use stack_encrypt::Ciphertext; -On 25 October 2026, the user clarified the compatibility scope. This API has -existed only as an early prototype. Do not preserve its APIs, stored -ciphertexts, or encodings for backward compatibility. No legacy decoders, -compatibility aliases, deprecation period, or data migration is required. -Choose the coherent new design, and update its Rust, Go, guest, and EQL -consumers together. +let secret: Ciphertext = keyset.encrypt("secret".to_owned()).await?; +let plaintext: String = keyset.decrypt(secret).await?; -Engine output must still agree with the SQL that consumes it. That is -current interoperability, not a requirement to support a legacy format. The -merged Go implementation does still keep a reader for the legacy JSON -encoding of opaque values. Record that reader as existing behaviour, not as -a new requirement to build machinery for migrating prototype data. +let card: Ciphertext = keyset + .encrypt_with_context("4111 1111 1111 1111".to_owned(), "users/42/card") + .await?; -The 25 October decision said that this exemption would end when the customer's configurations were released. The decision of 7 November 2026 ends it for stored data now, before a final release. The Go interface is not frozen by that decision. +// Store the ciphertext as bytes, or as Base64 text through Serde. +let bytes: Vec = card.to_bytes(); +let card = Ciphertext::from_bytes(&bytes)?; +let number: String = keyset.decrypt_with_context(card, "users/42/card").await?; +``` -#### Work that the stored-format decision constrains +`encrypt` accepts only single values: text, numbers, booleans, dates, +times, and decimals. A struct or a `Vec` does not compile. Use a record, a +collection, or an opaque blob instead. -Each item below would change the stored format if it were built as first planned. Each must now keep rc.3 data readable and searchable, or add a new versioned format beside it. Which option to take for each is an open question. +#### Encrypt a record -| Planned work | How it touches the stored format | -| --- | --- | -| Freezing the post-cascade format | rc.3 already writes it. The frozen format must be the rc.3 one, or a new version beside it | -| The refined-tag proposal | It changes the tag stored in the envelope, and what the AAD binds | -| Separating row binding and search scope | It changes the AAD and the context that index terms are derived under | -| The `OpaqueBlob` codec, including the Borsh fallback | Go rc.3 already writes opaque structs with Vitamin C's storage profile. A Rust codec must read those bytes | -| Encoded ciphertext field targets (Base64 and Base85) | They change how a ciphertext is written as text in a column | -| The `SealedValue` envelope layout and the JSON header version | The key-tag offsets and the header version 1 are what rc.3 wrote. The plan earlier said the offsets were not a commitment | -| Byte vectors | Whether Go rc.3 writes a byte string as one value, and so whether the byte-vector decision affects stored data, was not checked | -| Configurable match options and the tokeniser pipeline | New options must not change the terms of columns written with rc.3 defaults | - -#### Pull request #1491 and data written by Go rc.3 - -Pull request #1491 does not change data that Go rc.3 can write. It changes only the Rust typed chain's `extend` method in `plan/chain.rs`. In `dynamic/record.rs`, the path that the Go guest uses, #1491 only adds a test. Go's `Extend` already appended parts in order. - -Go rc.3 was built from `929451b49`. That commit is earlier than the merge of #1491. The data that #1491 affects is data written by repeated Rust `.extend` calls before #1491. That data is Rust-only. - -## Requirements established in the review - -- The keyset directs ordinary encryption through `encrypt` and - `encrypt_with_context`. Keep `EncryptInto` as a separate conversion - interface directed by the target. Remove the redundant public - `encrypt_as`. -- Encrypting a single value produces an opaque ciphertext. `OpaqueBlob` - explicitly encodes a composite value before sealing it as one value. - Direct `encrypt` does not automatically apply itself to each item of a - collection. -- Use `EncryptInto` to batch a collection, including `Vec` into - `Vec` without caller context. This division was agreed on 25 - October 2026. The current recommendation for byte vectors follows the same - collection rule, and uses `OpaqueBlob` for a whole vector. Compatibility with the prototype API gives no reason to add an exception for raw bytes. -- Rust traits and derives are the main static interface. Plans support those - declarations, and their lowering to the dynamic interface and FFI. -- Design static and dynamic support against the same contract for the engine - and definitions. Deliver the Go customer interface first. Polished public - Rust traits and derives can follow. Interoperability tests at the level of - the Rust engine are still required, so this order does not introduce a - second Go encryption implementation. -- Provide decrypt entry points with and without explicit context. The - variant without context must not invent a default value for a required - application input. -- `OpaqueEncode` and `OpaqueDecode` name the capabilities that represent a - value as a single ciphertext. `encrypt` and `encrypt_into`, including - their context-taking variants, consume their input. Both operations take - responsibility for cleaning up owned plaintext and plaintext artefacts. -- Ownership alone does not implement zeroisation. Require `Zeroize` on - plaintext types, and enforce cleanup through guarded storage and lifecycle - checks. Deliberate passthrough fields move to the output instead of being - erased. -- Keep EQL's contract between the engine and the SQL coherent, and keep the - shared batching and execution behaviour. From 7 November 2026, the stored format of Go rc.3 must be preserved. Other prototype representations need not be. -- Investigate the serialisation round trips in the EQL JSON adapter, and - replace them with direct typed conversion. Keep the engine and the SQL in - agreement. - -### Status of these requirements on 7 November 2026 - -| Requirement | Status | What remains | -| --- | --- | --- | -| Keyset-directed `encrypt` and `encrypt_with_context`, with `EncryptInto` kept | Partly done (#1404) | Remove the public `encrypt_as`, which is still on `next` | -| One value gives one opaque ciphertext, and `encrypt` takes no collection | Partly done (#1404) | `OpaqueBlob`. #1503 is in progress | -| `EncryptInto` batches a collection without caller context | Done for `Vec` (#1501) | Byte vectors differ from this plan. See [Differences between merged code and this plan](#differences-between-merged-code-and-this-plan) | -| Rust traits and derives are the main static interface | Partly done (#1404) | The derives with the new names. #1504 is in progress | -| Static and dynamic support against one contract, with engine-level interoperability tests | Partly done (#1460, #1461, #1462, #1469, #1491, #1492) | The items in [Remaining customer work](#remaining-customer-work), item 1 | -| Decrypt with and without context, with no invented default | Done for one value (#1404) | Record targets. #1504 is in progress | -| `OpaqueEncode` and `OpaqueDecode`, and consuming `encrypt` and `encrypt_into` | Done for one value and for `Vec` (#1404, #1501) | Records. #1504 is in progress | -| `Zeroize` bound and guarded cleanup | Partly done (#1404, #1493, #1501) | See [Cleanup in target-directed encryption and at the Go boundary](#cleanup-in-target-directed-encryption-and-at-the-go-boundary) | -| Keep EQL's engine and SQL contract coherent | Done for the merged JSON queries. #1492 runs them against PostgreSQL in CI | None listed | -| Direct typed conversion in the EQL JSON adapter | Still open | No merged pull request changes it | - -## Recommended decision: general configured targets - -Use a general contract: a target plus typed options. Do not add a public -`json_tree(...)` derive construct. Rust constructs concrete options -directly. The dynamic interface deserialises its options into that same -type. - -JSON supplies the options type, the validation, and the implementation -behind the general mechanism. Sharing a contract does not require shared -input syntax, or static Rust configuration that passes through -deserialisation. - -Three approaches were considered: - -| Approach | Benefit | Limitation | Recommendation | -| --- | --- | --- | --- | -| Typed Rust options factory | The compiler checks the options type, and normal Rust APIs and tools work | The function itself is not portable, but its concrete output has a data form | Preferred static interface | -| Named policy type that holds options | Reusable and easy to discover | Adds a provider trait and a marker type where a function is enough | Optional later convenience | -| Target plus options as data | Can be written in Go declarations and dynamic data | Needs deserialisation and semantic validation | Dynamic interface, producing the same typed options | - -The first recommendation put option field literals in Rust attributes, and -this recommendation replaces it. The first recommendation rejected a Rust -factory because its code cannot cross FFI. The rejection was too -restrictive: the common boundary is the factory's typed result, not the code -that builds it. This plan introduces no arbitrary encryption callback. The -factory only supplies configuration. - -The target owns: - -- its source capabilities, canonical output, and options and defaults; -- validation, the encryption description, and index and query capabilities; -- how its output is opened. - -Context remains a separate input. Configuration does not supply arbitrary -execution callbacks. Configuration also does not let a caller redefine EQL's -wire fields. - -Implement configuration as a target capability, with the working name -`ConfigurableTarget`. The capability lowers into the existing operation -descriptions. Its associated options type has a data representation. Rust -factories return that type directly. Registered dynamic targets deserialise -into it. - -Validate and normalise the options once, when a reusable declaration is -built. Both paths run the target's own semantic validator before key I/O. -Use validated deserialisation, for example a wire helper plus `TryFrom`. Do -not use a derived deserialiser that bypasses the checked builders. If the -public type promises validated invariants, keep any raw options fields -private. - -Compile-time guarantees cover factory resolution and signatures, the -association between target and options, and typed builder arguments. They do -not mean that a string is a valid JSON path, or that the factory runs at -compile time. Building a JSON path can currently fail. - -Start with a fallible factory contract, -`fn() -> Result`, with errors converted to the engine's -configuration error path. Propagate errors without panics or `unwrap`. A -later macro for literal paths can add stronger static validation. - -Resolve the factory once for each reusable configured declaration, and reuse -its normalised options for writes and queries. A factory can read mutable -external state, so its type alone cannot prove a fixed policy. Document that -providers of a fixed policy must be stable. Do not evaluate the provider -again when generating a query. Configuration that is meant to vary belongs -in a configured-target instance that the caller constructs explicitly. - -Unlike today's `Layout`, the general capability must not require exactly one -index: a target can have none, or several. Adapt the existing JSON layout -internally during the migration. Do not expose two competing concepts of a -configured target. - -For Go, the target catalog resolves registered targets. Custom Rust targets -remain possible without registration. The dynamic interface can address them -only once the guest build includes their adapter and descriptor. - -## Candidate static syntax - -`target(options = payroll_options)` names a Rust factory for the inner -target of a transparent newtype. The inner Rust type selects the output -representation and the associated options type. The spelling is a proposal. -The revised recommendation is typed factories for Rust, and typed -deserialisation for dynamic callers. +A record type says how each field of a plaintext struct is stored. The +derive seals each encrypted field under its own context, such as +`("users", "name")`. All the fields in a record share one batch of key +requests. ```rust -use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; -use stack_encrypt::json::JsonOptions; -use stack_encrypt::sem::TermError; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::target::{DecryptSource, EncryptTarget}; +use stack_encrypt::{Ciphertext, DecryptFrom, EncryptInto}; +use zeroize::Zeroize; + +#[derive(Zeroize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} -fn payroll_options() -> Result { - JsonOptions::new().ordering(["$.salary"]) +// A searchable email: its ciphertext and an equality term. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, } -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[serde(transparent)] -#[stash( - plaintext = serde_json::Value, - transparent, - target(options = payroll_options) -)] -struct EncryptedPayroll(EqlEncryptedJsonTree); -``` +// Not on `next` yet: the record derive and `passthrough`. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} -The factory body uses today's `JsonOptions` builder. The derive support is -proposed. The derive calls the factory, and type-checks its result against -the inner target's associated options type. The derive does not depend on -the meaning of JSON paths, or on JSON-specific option names. It does not -serialise or deserialise the factory's result. +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &cipher).await?; +``` -The configured target constructs its output, and the derive wraps it. Serde -transparency keeps that output's wire envelope. SQL driver traits are a -separate integration requirement. The same factory attribute should work on -a field of a derived output record, under the existing rules for selecting -context. +To tie every field to one row, the caller supplies the row ID as a context +extension, and supplies it again to decrypt. The extension also changes the +search terms, so a query must supply the same row ID. How to bind a +ciphertext to its row while its search terms cover the whole column is an +[open question](#open-questions-and-decisions-to-be-made). ```rust -let encrypted: EncryptedPayroll = payroll - .encrypt_into_with_context(&keyset, column.clone()) +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, user_id) .await?; - -let restored = serde_json::Value::decrypt_from_with_context( - encrypted, &keyset, column, -).await?; +let user = User::decrypt_from_with_context(encrypted, &keyset, user_id).await?; ``` -A fixed policy is a recipe for writes and queries. Transparent -deserialisation does not prove that an existing row was created with that -policy. In particular, the JSON header does not store the full policy of -ordered paths. +#### Store a field as an EQL type -## Candidate dynamic representation and Go requirements +A field can use an EQL type from the `eql-bindings` crate as its target. +The field is then stored as one EQL column, such as +`public.eql_v4_text_eq`. -Pull request #1372 implements this: string targets and configured object -targets share the native options validator. Use the merged `JsonSearch` -identifier: +```rust +use eql_bindings::v4::{integer::IntegerOrd, text::TextEq}; -```json -{ - "payroll": { - "context": ["employees", "payroll"], - "target": { - "name": "JsonSearch", - "options": { "ordered": ["$.salary"] } - } - } +#[derive(Zeroize)] +struct Member { + id: u64, + email: String, + age: i32, } -``` -Keep `"target": "TextEq"` as shorthand for that target's default options. -The object form and the JSON target options are now implemented. Reject -unknown options, incompatible options, unsupported target names, and invalid -paths during declaration validation. Do not fall back to defaults silently. -Specify one consistent JSON wire format and context encoding across the -Rust, dynamic, Go, and database paths. - -Go's generator and policy API must represent these same options, emit the -same declaration, and get the EQL value from the guest. Go must not assemble -the EQL envelope or reproduce JSON indexing. Discovery must expose the -options contract and the capabilities. Extend the existing target catalog -and resolver instead of installing a parallel JSON registry. - -The dynamic registry calls the concrete target's options deserialiser. It -does not look up or call the Rust application's factory by name. A Go caller -supplies equivalent options as data. If a Rust policy must be exported, -serialise its resolved options, not the function's identifier. This contract -does not require every Rust-only target to depend on Serde: only targets -registered with the dynamic interface need data conversion. - -These are now implemented: - -- the catalog and guest; -- the JSON value kind; -- the strict JSON text parser; -- the generated Go target options. - -Pull request #1373 adds operation input shapes and operand encodings. Pull -request #1378 adds typed Go document adapters. Remaining work includes: - -- the formal versioned specification; -- a general lifecycle and caching for definitions; -- direct typed EQL conversion; -- separating the roles within context. - -The current grammar for data plans is still a map of fields. The runtime -does not yet accept the illustrative versioned schema later in this plan. - -Writes and queries must reuse the normalised configured target in Rust and -Go. An ordering query on an undeclared path must fail before key I/O. - -Containment, path equality, ordering, and extraction need capability -metadata that can express their different shapes of operand and result. The -metadata cannot assume a single scalar query counterpart. Decryption must -use the stored cryptographic metadata and context. It must not require the -caller to remember the write's ordered paths. - -## Accepted decrypt interface and return ownership - -Decided on 25 October 2026: keep ordinary keyset-directed decryption for one -opaque ciphertext. Keep destination-directed `DecryptFrom` for declared -targets and layouts. Both paths have variants with and without context: - -```rust -let value: i32 = keyset.decrypt(encrypted).await?; -let value: i32 = keyset.decrypt_with_context(encrypted, context).await?; - -let plaintext = User::decrypt_from(encrypted, &keyset).await?; -let plaintext = User::decrypt_from_with_context(encrypted, &keyset, context) - .await?; +// Not on `next` yet: EQL types as fields of a record. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[stash(struct = Member, context = "members")] +struct EncryptedMember { + #[stash(passthrough)] + id: u64, + email: TextEq, + age: IntegerOrd, +} ``` -These are separate call examples. They do not reuse one consumed ciphertext -in sequence. Direct decrypt mirrors direct encrypt, and `DecryptFrom` -mirrors the target-directed `EncryptInto`. The existing input capabilities -still decide whether a caller can leave out context. Do not add a competing -public `decrypt_into` method that takes the ciphertext as its receiver. - -Successful decryption returns an ordinary owned `T: Zeroize`. Stack guards -the recovered bytes and intermediate plaintext for as long as they belong to -Stack. A successful return transfers responsibility for the value to the -caller. `Zeroize` does not automatically wipe that returned value on drop. - -Support results with a drop guard when the caller explicitly asks for one, -without requiring a wrapper around every decrypted application struct. The -exact opt-in adapter is implementation design within this decision. Guarding -the completed value does not solve cleanup after a partial deserialisation. -That cleanup remains required work. - -`User::decrypt_from(...)` reads as constructing a `User` from a ciphertext. -Calling `encrypted.decrypt_from(...)` would conflict with the usual From and -Into receiver convention, so keep the agreed destination receiver. The -destination-directed pair already exists on the blanket `DecryptFrom` -extension trait. - -Today's ciphertext method is `encrypted.decrypt_into(cipher, context)`, also -on `DecryptFrom`. That method has no separate overload without context. The -current declaration trait `DecryptInto

` is a different layer, so do not -rename it by accident while settling how the methods are called. - -Settle the names of declaration traits and convenience traits together with -review issue 5. The public examples use the destination-directed pair. -Remove the competing ciphertext method when implementing this interface. - -Status on 7 November 2026: partly done. #1404 implements `keyset.decrypt`, -`decrypt_with_context`, `T::decrypt_from`, and `decrypt_from_with_context` -for one `Ciphertext`. #1501 adds them for `Vec`. The owned -`DecryptFrom` has no `decrypt_into` method. The structural -`decrypt_into` still exists in `target::legacy`. The opt-in adapter for -results that are wiped on drop is still open. - -## Recommended public traits and context input capabilities - -Status: the user accepted this direction on 24 October 2026. Separate the -conversion methods from the `EncryptTarget` and `DecryptSource` -declarations. Use explicit capabilities for caller context instead of -`Default`. #1404 implements this direction for opaque values, and #1501 -for collections of them. Records, complete signatures, and updates to call -sites remain pending. The status of each point is listed at the end of this -section. - -This section settles the direction of work areas A and B, later in this -plan. Earlier examples keep their historical derive names until they are -migrated. - -Keep two user-facing conversion traits, both provided by blanket -implementations in the library: - -| Trait | Public methods | Receiver | -| --- | --- | --- | -| `EncryptInto` | `encrypt_into`, `encrypt_into_with_context` | Plaintext value; output inferred or selected | -| `DecryptFrom` | `decrypt_from`, `decrypt_from_with_context` | Plaintext destination type; ciphertext supplied as an argument | - -These methods run declarations through the common engine and return the -existing batched awaitable value. Target implementers do not override -execution. The ordinary `keyset.encrypt` and `encrypt_with_context` remain -inherent methods, and do not gain an `encrypt` synonym on the plaintext. -Their accepted decrypt counterparts are `keyset.decrypt` and -`decrypt_with_context`. - -Give the declaration traits names that state their role. From and Into names -would promise conversion methods that these traits do not define: +#### Query -| Proposed declaration | Implemented on | Describes | -| --- | --- | --- | -| `target::EncryptTarget

` | Encrypted output or term target | How to produce this target from `P` | -| `target::DecryptSource

` | Recoverable encrypted output | How this stored value yields `P` | - -The corresponding derives should use the same names. This replaces today's -declaration traits `EncryptFrom

` and `DecryptInto

`. It does not remove -the retained `EncryptInto` extension trait. A target that only produces -terms implements only `EncryptTarget`, so it has no decrypt method that -always fails. - -Plaintext visitor traits remain a separate representation layer that Stack -owns. Moving or renaming them must keep a single canonical trait -implementation, and a consistent wire contract across the new consumers. -Compatibility with the prototype API is not needed. The stored format must stay compatible. Configured-target options -are another associated capability, not context. - -Declaration methods continue to return operation descriptions that the core -owns. Encryption declarations do not receive the plaintext and the cipher -together. Decrypt declarations inspect the stored output and select how to -open it, without any key I/O. Plans, derives, and dynamic resolution reuse -those descriptions. Keep the declaration module public for custom targets, -and keep the normal method imports and derives available from the Stack -entry point. - -Use `ContextInput` as the name of the associated type on each declaration -trait. `ContextInput` describes only what the caller must or can supply. It -is not the complete cryptographic context. The complete context can also -include a prefix, and fields selected from the plaintext or the stored -output. Encrypt and decrypt declare this type independently. - -Recommended: input wrappers that the library owns, each with explicit -capabilities: - -| Proposed input | Argument left out | Argument supplied | Meaning | -| --- | --- | --- | --- | -| `NoCallerContext` | Allowed | Not accepted | The declaration derives all context and takes no caller input | -| `RequiredContext` | Not available | Required `C` | Supplies the context, or the missing component that the declaration selects | -| `OptionalContext` | Allowed | Accepted `C` | For an ordinary ciphertext, leaving it out means empty context, and a supplied context is used directly | -| `ContextExtension` | Allowed | Accepted `C` | Leaving it out keeps the declared base, and supplying it extends the base | -| `ExpectedContext` | Allowed | Accepted `C` | Leaving it out rebuilds the stored context, and supplying it also checks that context | - -`OptionalContext` does not append a synthetic empty prefix, and leaving out -an extension does not append `()`. `ExpectedContext` never extends the context, so it never changes the AAD. -The table specifies behaviour, and the names can change without changing it. -A requirement for nonempty input belongs to the accepted argument type or to -the validated resolved context, not to whether the input is optional. - -Search targets still require a nonempty effective context. Ordinary scalar -ciphertext should accept the same empty context through both the keyset API -and the target-directed API. - -Methods without context require an explicit, sealed `AllowsNoContext` -capability. `_with_context` methods require a matching `AcceptsContext` -capability with its accepted argument type. Neither capability is inferred -from `C: Default`. So `RequiredContext` stays required, even though -`u64` defaults to zero. - -The helper methods accept ordinary argument values, or convenient checked -conversions. Users do not wrap values in `RequiredContext` or -`ExpectedContext` by hand at every call. Conversion errors propagate before -key I/O. - -The wrappers that the core owns record presence and intent. Target -declarations select the sources and compose the context. For composite -required input, `C` can be a tuple or a named struct. It does not have to be -the full logical context. Keep the existing grouping rules and context -encodings. - -Examples where encrypt and decrypt have different requirements: - -| Declaration | Encrypt input | Decrypt input | -| --- | --- | --- | -| Caller row ID stored in output | `RequiredContext` | `ExpectedContext` | -| Source row ID also stored in output | `NoCallerContext` | `ExpectedContext` | -| Source row ID not recoverable before decrypt | `NoCallerContext` | `RequiredContext` | -| Static field base with optional unstored extension | `ContextExtension` | `ContextExtension` | - -In the last row, decrypting a ciphertext that has an extension requires the -same extension. That the method without context is available does not -promise that it opens every ciphertext of that Rust type. Where a -declaration explicitly allows it, a derive can expose an optional extension -instead of `NoCallerContext`. Expected-context checks do not authenticate -stored data themselves. Successful cryptographic opening is still necessary. - -The dynamic plan must encode the same requirements, source selectors, and -storage rules. Rust uses trait bounds to enforce which methods are -available, with or without context. The engine enforces the presence and -validity of dynamic input before key I/O. Omitted input, an explicit zero, -and an empty supplied value are three different things. Go generation can -expose convenient signatures from that metadata, but the guest stays -authoritative for runtime requests. - -The implementation must prove these points: - -- a call that leaves out required input fails to compile, even when - `C: Default`; -- a target that accepts only no input rejects context; -- an optional extension and an expected context keep their different - meanings; -- all nine directional scenarios in the [context - examples](2026-10-24-stack-encrypt-context-examples.md) run through Rust - and through dynamic plans. - -Status of these points on 7 November 2026: - -| Point | Status | -| --- | --- | -| Leaving out required input fails to compile, even when `C: Default` | Done. #1404 adds `RequiredContext`, `AllowsNoContext`, and `AcceptsContext`. #1501 adds compile-fail tests, including a collection of `u64` | -| A target that accepts no input rejects context | Done. #1501 adds a compile-fail test for `NoCallerContext` | -| An optional extension and an expected context keep their meanings | Still open. `ContextExtension` and `ExpectedContext` as context inputs are in #1504, which is in progress | -| All nine directional scenarios run through Rust and dynamic plans | Still open. #1491 tests a base with ordered extensions and stored-context expectations, not the nine scenarios | - -#1404 implements the traits in the tables above, the associated type -`ContextInput`, and `ScalarContext`, the owned context of an opaque value. -`OptionalContext` gives an opaque value an empty context when the caller -leaves it out, through both the keyset and the target-directed API. - -Keep support for owned sources, and keep batching. This separation of traits -must not add a blanket `Clone` bound. The final signatures must show the -support for borrowed sources and for owned sources. They must not hide the -existing difference behind the rename. - -## Go JSON implementation and PR #1353 - -The [Go JSON -plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) -has been updated against merged `next`. Tracking issue #1347 and steps #1348 -to #1352 are closed. #1372, #1373, and #1378 implement: - -- the catalog; -- configured JSON targets; -- dynamic execution; -- operands specific to each operation; -- opening that Rust verifies against the original request; -- the generated typed Go front end. - -Pull request #1402 supplies current documentation and the executable -database example. - -The released target identifier is `JsonSearch`. Go uses `eql.JSONSearch`, -`JSONSearchQuery`, `JSONSearchLeafQuery`, and `JSONEntry`. Do not impose the -proposed `EncryptedJson` glossary as a rewrite before release. Keep the -distinctions between roles, and freeze the actual customer names before -later Rust renames. - -Prepared Go reads keep their cipher and use `Open(ctx, rows)`. The engine -derives the trusted selectors and read keys again from the original request. -No new protocol for persistent guest handles is needed to implement that -verification. - -State the remaining requirements explicitly: - -- SQL for children and child counts exists, but there is no opener for - plaintext children that is bound to their parent. -- Opening by selector accepts at most one entry for each document group. -- Use subtree reads to retrieve plaintext for all matches. -- General inputs for search scope and row binding remain unsupported for EQL - targets. The existing column and keyset configuration can be released without claiming them. - -Two JSON queries were added after this section was first written. Pull requests #1445 and #1451 test several conditions on one array element. Pull requests #1446 and #1452 turn a jsonpath predicate into one query parameter. Neither query uses -an index. The [JSON operations -inventory](2026-10-25-stack-encrypt-json-operations.md) lists them as J21 -and J22. - -The JSON contract now uses randomised projection order and ciphertexts bound -by cascaded tags. Semantic parity and cross-opening are required. Comparing -raw term bytes is meaningful only with controlled test randomness. Go typed -JSON adapters follow `encoding/json`, while raw JSON goes through the -engine's strict parser. Do not claim that arbitrary Go encoders preserve -every input byte. - -## Prior query design: baseline and reconciliation - -### No ORE equality in the first API - -Decided on 25 October 2026: ORE does not support equality in the first API, -for any plaintext type. Equality uses the field's declared index that -supports equality. Parameterise capability selection by the index, by the -encoding or domain of the plaintext, and by the options. A future explicit -exact or injective integer encoding can then opt in to equality, without -changing the query syntax or the operator's wire identity. - -Never infer `Eq` only from support for ordering. Text and other lossy -encodings cannot establish plaintext equality. A future opt-in must keep the -rejection of overlaps, with one selected provider for each operator. It must -not silently invalidate existing `(Equality, Ore)` definitions. - -This policy does not disable JSON scalar equality. JSON scalar equality uses -value selectors, not ORE or OPE terms. - -### The query design draft of 21 October 2026 - -On 25 October 2026, the whole user-supplied **Stack Encrypt query design** -draft was read. The draft is dated 21 October 2026. It is 654 lines long, at -`/Users/dan/Downloads/Stack Encrypt query design.md`. The source document -was not changed. - -The document is explicitly a discussion draft, based on an earlier snapshot -of `next`. Its historical claims about code and releases were not checked -again as part of this reading. - -Use the draft as the baseline for query design. It is not a settled -specification, or a reason to redesign queries from scratch. - -### Existing proposal to carry forward - -- The caller selects an operator and a typed field or target handle. The - field supplies its own configured index. Remove `DefaultIndex` and the - caller's `.via`. -- Each field has at most one index that supports each operator. Refuse - overlaps at declaration validation, instead of choosing an index by - priority. -- Declare operators, index capabilities, and admitted source types once. - Derive or verify typed lookup, dynamic lookup, and discovery from that - declaration. Avoid another capability table in EQL. -- Share query derivation and refusals through the same semantic functions. - In particular, tokenise match input once, and keep the tokens needed for - probabilistic verification in the query object. -- Keep index and source types in typed field handles. Unsupported operators - are compile errors on the typed path, and validation errors on the dynamic - path. -- Model the analyser's handling of letter case once, in its stages, instead - of stating it in both the match configuration and the operator. - -The draft already proposes three designs to reuse: - -- the query call `cipher.query(op::Matches, text).on(&field).await?`; -- typed `FieldRef` handles; -- a loaded configuration and analyser that the dynamic plan owns (`Loaded` - and `Arc`). - -These designs do not yet settle how our typed factory options, derived from -the output, are initialised and shared with convenient `encrypt_into` calls. - -### Reconciliation with this review - -1. **Generate handles from the same output-target declaration.** The query - appendix proposes an `EncryptionPlan` derive on the plaintext side, and - generated encrypted counterparts. Our agreed starting point is the - output-side `EncryptTarget` and `DecryptSource` derives. Recommended: - emit typed handles from that declaration, without restating indexes - separately on the plaintext. A convenience derive on the source side can - later generate the same target. Hand-written plans remain lower-level - composition, not required usage. -2. **Settle who owns the results of typed factories.** A generic `Match` - can fix a policy identity in its type. An arbitrary factory can still - compute different values on different calls. Carry a resolved - configuration instance through writes and field handles, and settle the - API for its owner and its initialisation. The draft's ownership in the - dynamic plan is useful precedent. That precedent does not prove that all - static factory results are compile-time constants. -3. **Separate portable options from execution types.** JSON's Rust factory - and its dynamic deserialiser can construct the same `JsonOptions`. To - allow generic and custom analyser stages, the draft's `Match` and - `Match` deliberately have different execution types. Keep shared - validated options and semantic implementations for the portable subset. - Do not force all typed analysers through a serialisable enum of stages. -4. **Accepted portability scope.** Release built-in advertised capabilities - in Rust and Go together. Downstream custom Rust stages can stay - explicitly typed-only, with `NotPortable`, until they are registered. - This qualifies the draft's blanket "lands in Rust first" rule. This - qualification must not become permission to defer the Go JSON capability - under this plan. -5. **Include JSON capabilities before fixing lookup representations.** The - draft's dynamic enum of 16 shapes covers equality, match, and scalar - ordering, but not the new JSON target. For containment, `AtPath`, and - extraction, extend the operator identity, operand shapes, result shapes, - capability validation, and overlap rules. The draft's interim validated - index list, using one capability table, is a workable starting point. Its - specific closed enum is not a decision. -6. **Use the agreed model for context and scope.** Replace the assumption - that context is only a label with separate ciphertext binding and search - scope. Row binding does not flow into search terms. `PrimaryKey` is - eligible for context only. `ScopeKey` is eligible for both context and - scope. Apply the same declared scope to index generation and to queries. - Keep the requirement to return or supply row context when decrypting - selected results. -7. **Accepted validation timing.** All checks that do not need keys come - before keys are resolved or loaded, including checks on query input. This - replaces the draft's possible resolution of a named keyset before those - checks. Reuse the same validator for Go's `check_query` and for - execution, instead of copying it or the tokenisation. Provider facts, and - cryptographic checks that need keys, remain later stages. This rule for - validating operations does not forbid explicit earlier calls that - initialise a client or keyset. -8. **Apply the user's prototype scope.** These parts of the older document - are not constraints here: decoding legacy match options, deprecation - requirements, release-preservation requirements, and frozen Go wire formats. The decision of 7 November 2026 now freezes the stored format, though not the Go API. Keep the new Rust, Go, and EQL consumers aligned. The appendix's - `cipher.encrypt(plan, value)` and plaintext derive are illustrative older - proposals. They do not replace our agreed entry points. - -ORE equality is now decided, as described in [No ORE equality in the first -API](#no-ore-equality-in-the-first-api). Integration work remains on: - -- how typed field handles are constructed and spelled; -- dynamic `Index` compared with a shared boundary of semantic - functions; -- the feasibility of checking match options at compile time; -- Go's `check_query`; -- updates to the architecture decision records (ADRs). - -Type-safe ordering of stages does not by itself validate runtime numeric -values or string paths. Stronger claims about const checks need proof in the -implementation. Treat these items as specific integration decisions, not as -an open-ended query API task. Coordinate their Go interface with PR #1353. - -## Recommended owner: a reusable typed target definition - -Status: recommendation for review. Use an immutable target definition to -own: - -- the resolved options; -- prepared indexes and analysers; -- the rules for selecting context; -- the operation metadata. - -The output derive generates the definition's typed field accessors from the -same declaration that generates encryption. The definition contains no -plaintext, row values, credentials, or selected keyset. A field handle -borrows or shares that definition, and keeps its plaintext and index types. -It is not an independent name plus a newly constructed default index. - -### One selector name for encryption and queries - -On 25 October 2026, the user said that encryption's `using` and the query's -`on` are confusing, and asked for one name to be considered. Recommended: -`.using(...)` for both. In each case it selects the definition or field -whose declared policy governs the operation. This spelling is proposed, not -yet accepted. - -Encryption still accepts the requested target's associated definition type. -A query still requires `QueryTarget` on a selected field or -single-value definition. Sharing one name does not weaken either bound, or -make a whole multi-field record a query target. The older `on` examples in -the supplied query draft show its earlier spelling. - -Proposed usage, assuming the previously described `EncryptedUser` target: +A query encrypts the search value with the same keyset and the same field +declaration as the stored values. `definition()` returns the record's +declaration, built once. It has one method for each field. That method +returns the field's handle. The query value binds to SQL as a parameter. ```rust -let users = EncryptedUser::definition()?; +use stack_encrypt::op; -let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +// Not on `next` yet: `definition()`, field handles, and `using`. +// Today a query names a plan written by hand: +// `cipher.query(op::Eq, value).on(&plan)`. +let members = EncryptedMember::definition()?; -let query = cipher - .using(&users.email()) +let by_email = cipher + .using(&members.email()) .query(op::Eq, "alice@example.com") - .keyset(&keyset) .await?; +let older = cipher.using(&members.age()).query(op::Gt, &30).await?; +let adults = cipher + .using(&members.age()) + .query(op::Between, &(18, 65)) + .await?; +``` + +```sql +SELECT id, email, age FROM members WHERE email = $1::eql_v4.query_text_eq; +SELECT id, email, age FROM members WHERE age > $1::eql_v4.query_integer_ord; ``` -`definition()` gets the shared resolved definition of that mapping from -source to target. Plain `encrypt_into` gets the same definition internally, -and does not call the options factory separately. `.email()` is generated -from the declaration of the output field, not from a second schema that -someone maintains by hand. The method name `definition` is proposed. It -names a resolved declaration, not a builder that users must reconstruct -before every operation. +A match query (`op::Matches`) works in the same way on a field with a match +index. Its answer can include rows that do not match. So the caller +decrypts the rows, and checks each one. -### Call order and strict typestate +#### Encrypt a JSON document -On 25 October 2026, the user required strict typestate for the fluent API. -The user prefers the reading `cipher.using(&field).query(op, input)`, and is -open to supporting both call orders. Recommended: document the -definition-first form as the canonical query form, and support the -operator-first form through the same request states: +A JSON field is stored as one encrypted document in a +`public.eql_v4_json_search` column. Typed Rust options say which paths can +be ordered. A query can test containment, compare the value at a path, +match one element of an array, or run a jsonpath predicate. ```rust -let a = cipher.using(&users.email()).query(op::Eq, email).await?; -let b = cipher.query(op::Eq, email).using(&users.email()).await?; -``` +use eql_bindings::v4::json_tree::JsonTreeDocument; +use stack_encrypt::json::{JsonOptions, JsonPlaintext}; -Both forms are proposed. The current implementation is query-first, followed -by `.on(...)`. This recommendation changes that interface. +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].at"]) +} -| Starting state | Method | Resulting state | -| --- | --- | --- | -| Cipher | `using(&definition)` | Definition bound, no query selected | -| Definition bound | `query(op, input)` | Query with its definition, operator, and input known | -| Cipher | `query(op, input)` | Query waiting for a definition | -| Query waiting for a definition | `using(&definition)` | The same query state as the definition-first path | - -The transition that brings the definition together with the operator and -input has the `QueryTarget` bound. A handle bound in the -definition-first order does not need to fix the future operator in advance. -Combining the two determines the index. It also determines the context -requirements. - -Only the executable state implements `IntoFuture`. A bound definition alone, -a query missing its definition, and a query missing required context cannot -be awaited. - -A typed context-input transition completes required context. Cases with -optional input or no input start in a valid state with context left out. On -the static path, completing context must not be represented only as an -unchecked `Option` plus a runtime error for missing input. - -Do not expose another `using` transition after a query's definition has been -selected, or another transition on the request that selects a query. -Supporting these two orders does not promise every permutation of keyset, -context, and operation selection. Specify those transitions separately, -starting with context supplied after its required type is known. - -Share the final query request, validation, and execution implementation, -instead of duplicating them for each call order. Errors that depend on -input, such as an invalid path, still need runtime validation. Typestate -checks completeness and capability, not the meaning of arbitrary values. - -The same rules apply to encryption requests. A fixed-policy `encrypt_into` -has an implicit definition, so a caller can await it without `using`. -Selecting an explicit compatible definition is a single transition before -execution. This does not add a new ordinary `encrypt` receiver, or require -callers with a fixed policy to bind definitions by hand. - -Completion tests must include: - -- both query orders, with identical semantics; -- compile-fail cases for unsupported operators and inputs, awaiting too - early, missing required context, and selecting a definition twice. - -Borrowed definitions and shared handles must stay valid for the lifetime of -the request. - -### Fixed declarations - -Initialise the fixed definition lazily, in a way that can fail, and keep -each successful result for reuse. Allow explicit initialisation at startup -through `EncryptedUser::definition()?`. The shorthand write initialises the -definition on demand, and returns any error through its awaitable value. -Factory errors must not panic or perform key I/O. - -Do not cache initialisation failures permanently. Coordinate concurrent -first use so that successful initialisation happens once. Handle reentrant -and recursive initialisation in the implementation, instead of allowing a -deadlock. - -The identity of a definition is the complete concrete declaration, including -the mapping from source to target and any generic policy parameters. It is -not only the ciphertext's output type. Different instantiations must not -share a cache slot by accident. This is an implementation requirement. It -does not claim that a static inside a generic Rust function is automatically -one per monomorphisation. - -Fixed factories must describe stable application policy. They are not the -interface for reloading environment settings, or for options that differ for -each tenant. The cache makes writes and queries in one process share the -resolved policy. It does not prove that another process, or an older stored -row, uses that policy. Per-call context, source values, and keyset choice -remain separate inputs. - -### Runtime declarations - -Load or create a separate definition instance for runtime options. A -configured instance is not stored in the fixed-declaration cache. Use the -same type of definition and field handles, so that queries and writes share -that instance. - -Proposed Rust spelling for a configurable library target: +// Not on `next` yet: a JSON target with typed options. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("customers", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); -```rust -let payroll = EqlEncryptedJsonTree::configure(options)?; +let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?; +let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; -let encrypted: EqlEncryptedJsonTree = value - .encrypt_into_with_context(&keyset, column) - .using(&payroll) +let profiles = EncryptedProfile::definition()?; +let needle = cipher + .using(&profiles.document()) + .query(op::Contains, &json!({ "name": "Ada" })) .await?; ``` -Here `using` takes a resolved typed target definition, not a hand-built -plan. This requires a lazy conversion request: select the definition before -building the operation or calling a default factory. Awaiting the request -lowers it to the existing `Pending` and batch executor, and the shared -engine does not change. The definition's output and source types constrain -the conversion. Queries use the same definition, or its field handles, and -supply the required context and keyset for that call. - -A fixed-policy wrapper such as `EncryptedPayroll` does not allow its -declared options to be overridden at runtime. Use the configurable -underlying target for the runtime case, so that nothing silently overrides a -fixed policy. The exact configuration APIs for runtime definitions with -several fields are still to be specified. They must select typed options -without introducing another execution mechanism. - -Go's loaded dynamic plan owns its resolved definitions, and generated field -accessors reach those definitions. Go has no Rust factory to call. The guest -validates and builds the configuration once for each loaded declaration, and -then writes and queries use the loaded instance. A different runtime policy -is a different instance. It does not change an instance that operations in -progress are using. - -### Implementation proof and boundary - -Prove the cache and handle design early with: - -- two fixed mappings; -- several generic instantiations; -- concurrent first use; -- an initialisation failure followed by a retry; -- separate runtime instances. - -Prove that a query and its write use the same options without calling the -factory twice. Keep the source lifetimes, the support for owned sources, -batch lowering, and the builds without HTTP and for WebAssembly. A query on -a field without the operator must remain a compile error, with dynamic -equivalents checked at declaration or query validation. - -Prefer this approach, with generated definitions, over requiring callers to -write a second plan or putting configuration state in each ciphertext. Plans -are still the lower-level implementation of the definition. Portable -serialisation of options and discovery of index capabilities remain part of -its dynamic contract. The definition object does not itself need to -serialise arbitrary Rust closures or custom analyser code. - -## Single-ciphertext representation and plaintext ownership - -Accepted on 25 October 2026: - -- Use `OpaqueEncode` and `OpaqueDecode` to represent a value as a single - ciphertext. They replace the working names `EncodeValue` and - `DecodeValue`. -- Direct `encrypt` and target-directed `encrypt_into`, including their - context-taking variants, consume their values. Both operations take - responsibility for cleaning up owned plaintext. -- These are representation capabilities, and the engine still performs the - cryptography. `EncryptTarget` and `DecryptSource` describe how encrypted - outputs and layouts compose those operations. -- The lifecycle and cleanup scope below are part of the accepted design. -- The bound for plaintext cleanup is `Zeroize`. The implementation is not - finished. #1404 implements it for opaque values, and #1501 for - collections of them. - -### Owned input is the direct API contract - -`keyset.encrypt(value)` and its context-taking counterpart take an owned -single-value input. There is no implicit clone and no `Clone` bound. - -For example, passing a `String` transfers it. A borrowed `&str` does not -transfer the storage it refers to. So it must not receive the same cleanup -promise. Do not add a silent fallback to borrowed input in the consuming -operation. - -Decided on 25 October 2026: require `Zeroize` for plaintext types, and use a -guard that the library owns. For now, use that existing ecosystem trait, -instead of defining a new cleanup trait or another adapter for protected -ownership. `Zeroize` supplies the wiping operation, and the library must -arrange to call it on every supported exit path. A bound alone does not -arrange cleanup on drop. - -Acquire protected ownership synchronously when creating the encryption -request, before returning its future or builder. The body of an `async fn` -does not run until the future is polled. If the first guard were inside that -body, a request that is never polled would drop its source in the ordinary -way. A synchronous constructor that returns the typed future can set up the -guard first. Every later builder transition moves that guarded ownership, -instead of making a new unprotected copy. - -The representation trait can encode through a temporary internal borrow of -the guarded value, into a sensitive sink that the library owns. Public -consumption does not require giving an encoder unrestricted ownership of the -source, or an ordinary growable `Vec` for scratch work. Wipe and release -the source as soon as encoding no longer needs it. Keep the encoded -plaintext guarded until sealing completes. Keep the boundary of a -representation as a single leaf. - -### Cleanup scope and implementation obligations - -- Cover owned source storage, encoded buffers, codec scratch space, and any - plaintext copies that the library creates. Use established wiping - primitives that the compiler cannot remove, or protected storage. Do not - rely on `clear()` or an ordinary drop alone. -- Cover success, errors in encoding, configuration, KMS, and AEAD, partly - prepared batches, cancelled requests, and a request dropped before it is - polled. Use drop guards for ordinary unwinding too. A process abort or a - leaked future does not run destructors, so neither is covered. -- Avoid reallocations that leave uncleared buffers behind. Sensitive sinks - that grow must have a cleanup strategy for their old allocations. Prevent - plaintext from escaping in diagnostics, `Debug` output, caches, and - instrumentation. -- On decrypt, protect the recovered bytes and partial decode state until - cleanup, or until ownership passes successfully to the caller. - Successfully returned plaintext is kept on purpose. After the return, its - lifetime is the caller's responsibility, unless the return type itself is - protected. -- A Rust move does not prove that all physical copies were erased. In - particular, `Copy` inputs leave the caller's originals usable, and - compiler temporaries may exist. The API promise that can be enforced - covers supported owned storage and the artefacts that the library manages. - It does not cover copies that the caller keeps, or every remnant in - registers and on the stack. Check stronger claims with evidence from the - compiled code. - -The [zeroize documentation](https://docs.rs/zeroize/latest/zeroize/) -describes guards based on drop, wiping that the compiler cannot remove, and -the limits involving moves, stack copies, and reallocations. The source -currently uses `Protected>` at the AEAD boundary. That does not -establish cleanup of every caller value and intermediate from end to end. -This discussion is a design requirement, not a completed audit of -zeroisation in the source or the assembly. - -### Consequences for blobs, layouts, and Go - -`OpaqueBlob` cannot promise to clean up the source from `T: Serialize` -alone. Require `T: Serialize + Zeroize` for the proposed blob input that -uses Serde. Owned blob decoding requires `T: DeserializeOwned + Zeroize`. -Ordinary application structs can derive `Zeroize`. - -A generic Serde implementation can allocate or keep hidden copies, so strong -claims also need a supported contract for the codec and implementation. -Wrapping the final byte buffer does not make arbitrary user serialisation -code compliant. - -In particular, `T: Zeroize` only supplies cleanup once a `T` exists. Suppose -a derived `Deserialize` allocates a `String` field and then fails on a later -field. The fields built so far drop in the ordinary way, without calling -`T::zeroize`. Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix -a struct that was never constructed. Keep this as a concrete requirement for -decoder design and acceptance, and do not claim that the new bound solves -partial decoding. - -### Blob codec assessment: Vitamin C frames - -#### Stable storage format first - -Decided on 25 October 2026, as the priority: first define a stable, -versioned binary storage format. The codec was first designed for -short-lived transport, and stored blobs need a stronger compatibility -commitment. Improvements to zeroisation and constant-time implementation can -follow, provided they keep the meaning of the stored format. - -Do not defer decisions that affect stored bytes, such as option presence, -type mapping, and versions. Do not claim timing or cleanup properties that -are not implemented. The `Zeroize` input bound and the ownership direction -remain. [Vitamin -C #519](https://github.com/cipherstash/vitaminc/issues/519), created for -this review, tracks this action. The issue explicitly separates storage -stability from the transport-only reasons in Vitamin C #332. - -#### Vitamin C storage profile implemented in Vitamin C and Go - -By 27 October 2026, Vitamin C's dedicated storage profile, and its Go reader -and writer, addressed the format work from #519. That work is in Vitamin -C #520 and #526, carried onto upstream main by #544. Stack #1369 and #1370 -adopt the Go codec: generated opaque structs call `vcffi.MarshalStorage` and -`UnmarshalStorage`. - -The storage profile is versioned (`VCS` plus a version). It supports explicit -optional values and typed map keys, and it is separate from transport. Do -not report these properties of the transport mapping as defects of the new -storage profile: - -- the ambiguity of `Some(())`; -- the rules that allow only text map keys. - -Go generated shapes use `vc` tags and `GeneratedVersion2`. A reader rejects -unsupported future blob versions with a distinct error, and still falls back -to the old JSON encoding. Go storage mapping has these documented -qualifications: - -- nil containers become empty; -- supported nonfinite floats round-trip; -- generation refuses unsupported opaque types. - -Go pointers and Rust `Option` do not always write identical bytes, so -cross-reading tests and the normative corpus matter. - -#### Remaining Rust and release work - -On 27 October 2026, the Rust workspaces used Vitamin C 0.8.0 from -crates.io. #1501 moved them to 0.9. Neither change implements the final -proposed Rust `OpaqueBlob` facade. #1503 adds `OpaqueBlob` over the storage profile. #1503 is in progress. When that facade -is built, adopt the dedicated storage entry points, instead of reusing the -transport Serde codec for storage. Go's temporary pseudo-version dependency -must move to released module tags for a final Go release. A new evaluation -of serialisers is not on the critical path for customers. - -Stable bytes do not establish constant-time behaviour or complete cleanup. -Keep the remaining work on guarded output, temporary buffers, partial -decoding, and values that the caller owns, without changing the stored -format. The JSON tree keeps its own node and envelope encoding. Freeze its post-cascade format separately from opaque blobs. - -### Cleanup in target-directed encryption and at the Go boundary - -The same consuming contract applies to target-directed `encrypt_into`. For a -structural target, declared passthrough fields move into the output on -purpose. Wipe encrypted source fields and temporary copies, not plaintext -fields that the caller explicitly chose to keep. Where possible, the fan-out -to ciphertext and index operations should read a guarded source, instead of -needing a plaintext clone for each operation. - -As part of the Go implementation, specify cleanup for dynamic transport -between host and guest, and for JSON parsing buffers. The Rust guest cannot -erase Go caller objects that it never owned. Copies that the host manages -need cleanup by the host, and ordinary Go values do not gain Rust move -semantics. Document that boundary, instead of claiming that guest cleanup -erases every copy in the calling process. - -Acceptance work: - -- observe source cleanup and drop on successful and failing requests; -- test cancellation, and requests dropped before they are polled; -- cover nested source and blob cases; -- cover owned inputs that are not `Clone`. - -Status of this acceptance work on 7 November 2026: - -| Acceptance item | Status | -| --- | --- | -| Source cleanup and drop on successful and failing requests | Done for opaque values and collections. #1404 wipes the owned source before it returns the pending request. #1501 adds drop witnesses, which are test values that record whether they were wiped | -| Cancellation, and requests dropped before they are polled | Done. #1493 adds lifecycle tests, and #1501 adds them for collections | -| Nested source and blob cases | Partly done. #1501 tests a nested `Vec>` as one batch. Blob cases wait for #1503 | -| Owned inputs that are not `Clone` | Done for collections (#1501) | - -Other cleanup work that merged: - -- #1501 makes the frame codec write into a Vitamin C `Buffer`. A `Buffer` wipes each allocation it outgrows. -- #1501 makes the frame readers, and the Go guest's frame inputs, take - protected bytes. -- #1501 wipes the plaintexts decrypted so far when a later element of a - collection fails. -- #1493 records a known gap: plaintext that stays in guest memory after a - Go decrypt. - -Cleanup after a partial Serde deserialisation, and the opt-in adapter for -results that are wiped on drop, are still open. - -Where necessary, inspect optimised code for wiping and copying behaviour. -Ordinary tests alone cannot prove that no copy remains. Do not infer a -current vulnerability, or a completed guarantee, from this design -discussion. - -## Configured-target implementation sequence and completion criteria - -Commitments to customers replace the earlier sequence, in which the Rust and -Go front ends moved in step. The foundational Go JSON work and the adoption -of opaque storage are now merged. - -1. **Specify the implemented contract.** Keep the proposed schema envelopes - distinct from the current field-map grammar. Record these in the formal - interoperability specification and conformance fixtures: - - target options; - - operator input and result encodings; - - grouped reads; - - errors; - - the column and keyset configuration. -2. **Close the actual remaining customer gaps.** The gaps are opening a - child bound to its parent, and complex ordered-path declarations if - needed. The gaps also include configurable Go match options (#1112), and - the tokeniser and filter pipeline. #1397 already supplies short - whole-word matching through EQL. Confirm the leakage accepted for it, and - freeze its index semantics, instead of planning it as unimplemented. -3. **Validate and freeze the release candidate.** Run the existing suites - for the engine, for Go with the guest, and for PostgreSQL against the - selected bundle. Keep numeric and null behaviour, tamper checks on - cascaded tags, wildcard semantics, fallback semantics, known omission - limitations, and post-cascade stored vectors. Tests that are already - merged do not prove that this review ran a release build again. -4. **Release the dependencies and the customer interface.** For a final - release, replace Vitamin C's temporary Go pseudo-version pins. The - existing guard allows them in prereleases. Opaque storage is already - integrated. Keep it separate from the JSON tree storage format. Build - production guests and the matching SQL. -5. **Complete the Rust convenience APIs and packaging.** Implement public - traits, derives, the definition lifecycle, and crate moves within the - frozen Go and data contract. Direct EQL conversion is an optimisation - that keeps the format. - -Use one execution engine throughout. The final Rust facade and package -layout do not gate the existing Go JSON support. Do not reimplement -configured JSON options, or reads that Rust verifies against the original -request: both are already merged. - -Status of each step on 7 November 2026: - -| Step | Status | What remains | -| --- | --- | --- | -| 1. Specify the implemented contract | Partly done. The conformance corpus covers declarations, runtime values, reads, context, scalars, value kinds, query operators, JSON operations, and lifecycle (#1460, #1461, #1462, #1469, #1491, #1492, #1493) | The specification itself. See [Required deliverable: dynamic interoperability specification](#required-deliverable-dynamic-interoperability-specification) | -| 2. Close the remaining customer gaps | Still open | Opening a child bound to its parent, and #1112. The new element and predicate queries (#1445, #1446, #1451, #1452) were not on this list | -| 3. Validate and freeze the release candidate | Partly done. #1492 runs the JSON suites against PostgreSQL in CI, and records the SQL bundle hash. #1403 fixes findings from the review of the release candidate | Freezing the post-cascade vectors and the EQL bundle | -| 4. Release the dependencies and the customer interface | Still open. Go `v0.1.0-rc.3` is a prerelease | Replacing the Go pseudo-version pins | -| 5. Complete the Rust convenience APIs and packaging | Partly done (#1404, #1501). #1503 and #1504 are in progress | Records, blobs, the definition lifecycle, and crate moves | - -### Required deliverable: dynamic interoperability specification - -The user requires a formal specification of the dynamic interoperability -layer, developed during implementation. Do not write it afterwards as -documentation, after the Rust API has diverged from the Go API. The current -examples remain proposals until they are reconciled into that specification. -Adding this deliverable does not claim that the specification has been -written. - -Status on 7 November 2026: partly done. The conformance corpus in -`conformance/stack-encrypt` contains the executable fixtures, and contract -documents for each conformance profile. It covers these parts of the required coverage -below: - -- the current declaration grammar, with accepted and rejected declarations - (#1460); -- runtime values, missing compared with null, and structured diagnostics - (#1461); -- validation stages, and which failures come before key I/O (#1462); -- context ordering and grouping, descriptors, AAD, and index context, for - the current shared context (#1491); -- JSON operations, the predicate grammar, and SQL oracles (#1492). - -These parts are not done: - -- **The format.** No JSON Schema files exist. The declaration grammar - contract explains that a schema of shapes alone would not enforce the - semantic rules. -- **Versions.** The conformance profiles are marked experimental: they do not certify a whole SDK. The contract is not frozen. The `v1` in the corpus is a corpus format version, not a - protocol version. -- **Roles.** The grammar for search scope, primary keys, and scope keys - does not exist. The context contract says that this split is separate - work. -- **Other runners and exchange.** Python and TypeScript runners, fresh - cross-language exchange of JSON values, and a release compatibility - matrix are not done. - -Recommended format: a versioned specification package that contains: - -- [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) - schemas; -- normative rules for semantics and wire format; -- executable conformance fixtures. - -JSON Schema describes the structure of declarations, requests, and -responses. It is not the whole specification of behaviour or of the binary -transport. Keep application declarations of semantic types distinct from -JSON Schema's own `type`, `$defs`, and `$ref` vocabulary. - -Required coverage: - -- **Definition grammar.** Named ordinary types need no role. Naming a type - does not make it eligible for scope. The grammar covers: - - built-in and named types; - - the optional `types` section; - - inline role wrappers; - - context selectors and scope eligibility; - - passthrough and context storage; - - targets and options; - - query capabilities and defaults; - - handling of unknown fields and extensions. -- **Semantic validation.** Resolve named types and field references, and - reject invalid or cyclic references where they are unsupported. Enforce - the capabilities of primary keys and scope keys recursively. Check that - sources and outputs are compatible, and derive agreement between the write - scope and the query scope. Do not imply that generic JSON Schema - validators perform the engine's reference resolution or capability checks. -- **Operation contracts.** The contracts cover: - - input presence; - - the roles of context and scope; - - target and definition identity; - - results; - - stable error codes and locations, without plaintext values; - - validation stages; - - which failures come before key I/O. -- **Wire mapping.** Specify the application binary interface (ABI) between - host and guest, or reference its versioned specification. JSON authoring - syntax must not silently redefine the binary transport. The mapping - covers: - - JSON and frame representations; - - integer widths and ranges, and lossless numeric transport; - - encodings of bytes and text; - - missing compared with null; - - context ordering and grouping; - - the lowering of descriptors, AAD, and index scope; - - normative references to the ciphertext and SQL envelope, and to tag - extraction. -- **Versioning.** Explicit protocol and schema identifiers, handling of - supported versions and capabilities, and rejection of unsupported - features. Define future evolution without adding compatibility obligations for the prototype API. The rc.3 stored format is the first version that later versions must keep reading. - -Conformance fixtures include accepted and rejected definitions and requests, -expected error codes and paths, normalised definitions, and deterministic -vectors for context and encoding. Run the same fixtures through Rust and -through the Go guest path, and through SQL extraction where relevant. -Compare randomised ciphertexts by cross-decryption and query behaviour, not -byte equality. Schema validation is one layer of the suite. Semantic -rejection tests are required even for structurally valid JSON. - -Keep a single maintained contract. Prefer generating structural schemas and -capability metadata from the shared declaration models and catalog, where -that faithfully represents the specified grammar. Commit the generated -artefacts and check them for drift in CI. - -Review generated schemas against the normative rules, instead of treating -Serde derives as a complete specification. This interface does not use HTTP. -Do not introduce OpenAPI, or a new transport, only to document it. - -## Coverage of the complete review - -The configured-target work above covers review issue 13 in depth. The full -API redesign also needs the work in this table. Each row names a concrete -deliverable. Recommendations do not silently settle the open decisions. - -| Work area | Review issues | Required deliverable | Status on 7 November 2026 | -| --- | --- | --- | --- | -| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One map of signatures, imports, and receivers; operation-trait design; and migration of dependencies | Partly done (#1404, #1501). The structural API and `encrypt_as` remain | -| Context contract | 1, 3, 4, 11, 12, 14 | Context rules for scalars and records, explicit optionality, and a matching dynamic representation | Partly done (#1404, #1491). Directional selectors and records remain | -| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde and Base64 behaviour, and a defined boundary with structural trees | Partly done (#1404). Typed text encodings and encoded field targets remain | -| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | Still open. #1503 is in progress | -| Mixed records and collections | 7, 8, 11 | Derive support for passthrough, renaming, and context fields, and retained collection batching | Collections done (#1501). Record derives are in #1504, which is in progress | -| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion, and Go coverage | Still open for Rust. Go coverage gained two JSON queries (#1451, #1452) | -| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | Partly done. Format fixtures are in the conformance corpus. The public examples become tests in #1504, which is in progress | -| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348 to #1352 with this contract, and reuse compatible implementation work | Unchanged since 27 October 2026. #1353 is still open | -| Prior query design | 16 | Reuse the supplied field-owned query model, and reconcile handles, configuration ownership, JSON capabilities, and portability | Unchanged | -| Consistent definition selector | 17 | Choose one method name for encrypt and query. Recommended: `using`, with bounds specific to each operation | Still open | -| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode` and `OpaqueDecode`, consuming `encrypt` and `encrypt_into`, guarded lifetimes for sources and buffers, and an explicit cleanup scope | Partly done (#1404, #1493, #1501). Blob cleanup and partial decoding remain | - -### A. Consolidate the public Rust method and trait structure - -This is the workstream for the Rust facade. Under the latest customer -ordering, make only the engine and contract changes that Go needs before its -release. Defer standalone Rust convenience work and the move of traits. The -sections below organise the full scope of the review, not the critical path -for customers. - -Keep the agreed ordinary keyset receiver and the `EncryptInto` conversion -forms. Use the accepted destination-directed decrypt pair. - -Write out all -signatures, imports, bounds, and return types before renaming traits. Keep -the ordinary keyset `decrypt` and `decrypt_with_context` for opaque -ciphertexts. Remove redundant prototype conveniences without compatibility shims. The APIs can break. The stored format must not. - -Separate plaintext representation, target operation declarations, and -execution conveniences explicitly. Resolve two inconsistencies: - -- `EncryptFrom` declares `encryption()`, while `encrypt_from` is on - `EncryptInto`; -- `DecryptFrom` supplies methods, while `DecryptInto` describes opening. - -A rename without a map of ownership and signatures leaves the discovery -problem unsolved. - -Recommended: Stack owns the public model for encrypting values, and Vitamin -C keeps general primitives. Specify the dependency boundary before moving -code. Keep one canonical identity for each trait, and keep the builds -without HTTP and for WebAssembly. Reassess the original mismatch with -Vitamin C's cipher trait in that architecture. Do not add a public trait -implementation only for symmetry, if it cannot pass structured KMS context -or support the operations it advertises. - -Completion criteria for work area A: representative scalar, blob, typed -record, searchable record, and JSON calls compile with the documented -imports, including from generic callers. The method inventory has no -unexplained duplicate execution path. Compile-fail tests cover the trait -capabilities that are not supported. - -Status on 7 November 2026: partly done. - -- **Done.** #1404 separates the declarations (`EncryptTarget`, - `DecryptSource`) from the conversion methods (`EncryptInto`, - `DecryptFrom`) for opaque values. #1501 extends them to `Vec`. Scalar - calls compile with the crate-root imports, and compile-fail tests cover - unsupported inputs and context. -- **Not done.** Blob, typed record, searchable record, and JSON calls do - not yet compile through the owned API. Pull requests #1503 and #1504 are in progress. -- **Duplicate path.** The method inventory still has a duplicate path. The - structural `EncryptFrom` and `DecryptInto`, `encrypt_as`, and the helpers - in `target::legacy` remain beside the owned API. Issue #1488 tracks their - removal. - -### B. Turn the context scenarios into a normative contract - -The nine scenarios in the [context -examples](2026-10-24-stack-encrypt-context-examples.md) are the acceptance -matrix, not optional examples. Cover: - -- a scalar's static context plus a caller extension; -- selecting a source field; -- caller input; -- storing context in the output; -- recovering context, or checking expected context, on decrypt. - -Keep the accepted `field = "id"` shorthand as a selection on both inputs: -the plaintext on encrypt, and the stored output on decrypt. The shorthand -does not copy anything implicitly. - -Fix the composition order and grouping once: the literal or field base, then -the composite row context, then any caller extension. Specify the exact -descriptor and AAD encoding for each case. An expected context checks the -stored context, and is not another extension. A caller must never need to -open a ciphertext to discover the context required to open that ciphertext. - -Recommended: represent no input, required input, optional extension, and -optional expectation as explicit capabilities. Do not infer optionality from -the application's `Default` implementation. Settle the wrapper or trait -mechanism together with the public signatures. Resolve the existing mismatch -in empty context between ordinary scalar encryption and ciphertext targets. -Do not weaken the requirement for nonempty context on search targets while -doing so. - -Give each rule for source, storage, and composition a dynamic representation -and a way to write it in Go. Reconcile the current EQL identifier -constraints with row and caller context explicitly. Do not bypass those -constraints, or drop context silently. - -#### Separate ciphertext binding from search scope - -Status: recommendation for the next discussion, not yet a decision. - -Distinguish ciphertext binding from search scope. A row ID can bind a -ciphertext to its record without making every index term specific to that -row. The target definition should describe both scopes explicitly. Query -construction uses the same declared search scope as index generation. - -For example, a searchable email can bind its ciphertext to -`(users.email, tenant_id, user_id)`, while its equality index is scoped to -`(users.email, tenant_id)`. A query can then run before the user ID is -known. Do not silently remove components from one generic context argument. -Index-key derivation and KMS descriptors must respect this separation too. -Changing only the leaf AAD is not a complete design. - -Settle the conceptual split before choosing attribute names or exact tuple -encodings, and represent it in the dynamic definition as well. - -The user's direction: allow ciphertext bound to a row, and document that -reads must supply the row ID to decrypt. Search scope should cover several -rows, such as a tenant. Including a row ID automatically would defeat -ordinary search. - -#### Misuse-resistant API (recommendation, not accepted) - -- Give the existing record `context(field = "id")` and its directional - variants the meaning of ciphertext binding. Do not implicitly feed those - selected fields or caller extensions into index keys or terms. This is a - proposed clarification for searchable targets. Older examples did not - distinguish the roles. Update them all together before implementation. -- Declare search scope separately, on the shared target definition. A named - application scope input, such as `TenantScope`, can come from operation - configuration or caller input. The capability proposal below also allows - record-field selectors, when their types explicitly qualify as scopes. -- Writing index terms and constructing queries both require the same typed - scope. A cipher bound to a tenant can supply the scope only through an - explicit declared mapping. Never guess the scope from unrelated metadata. -- A binding input and a search-scope input are distinct capabilities, even - when their underlying values have the same Rust type. Do not provide a - blanket implicit conversion between the roles. If one operation needs both - caller inputs, use named members in its typed input, not one unlabelled - value routed to both operations. -- Missing required scope prevents awaiting a Rust request. Dynamic and Go - validation reject it before key I/O. A default column-wide scope is not - substituted when a tenant scope is missing. Searches without scope use an - explicitly declared column-wide policy, and still bind the column - identity. -- The effective ciphertext binding should include the declared search scope - as well as the record binding. Then changing a supplied tenant also - prevents opening the ciphertext. Index derivation uses only the column - identity plus the search scope. Query requests never require - record-binding input. -- Encrypted output must expose whatever the decrypt selector needs, or - require it from the caller. Examples that project records must select the - ID. No decrypt fallback drops row binding. Selecting a field as context - does not implicitly store that field. - -Working declaration spelling: -`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. The -spelling is illustrative. The spelling keeps the agreed directional-context -shorthand, and gives search an explicitly separate role. Dynamic definitions -must keep this separation, instead of providing one shared collection of -context values. - -This design stops row IDs from spreading into search terms by accident. The -design cannot prove that the application chose a scope that contains several -rows. A user can deliberately put a row ID inside `TenantScope`. Do not -claim otherwise, or try to infer meaning from field names such as `id`. Use -explicit types, declarations, and documentation for that remaining -application choice. - -#### Scope capability and role wrappers - -Status: proposal under discussion. - -Use both ideas together. Scope is the subset of context that search terms -are allowed to use. Semantic wrappers provide positive trait implementations -that identify that subset. The working capability names are -`CiphertextContext`, and `SearchScope: CiphertextContext`. These names -describe eligibility, not source selection, optionality, or encoding. - -Integrate these capabilities with the existing context representation, -instead of introducing a competing serialiser. The previously agreed -`ContextInput` capabilities still describe whether the caller must supply a -value. - -For example, `PrimaryKey` implements eligibility for ciphertext context, -but not for search scope. `ScopeKey` implements both, subject to the -inner value's supported context encoding. Generic parameters are Rust types, -such as `PrimaryKey`. By contrast, `PrimaryKey(42)` is a value -constructor. - -The name `ScopeKey` is preferred for the second wrapper. The user -accepted it in place of `Collection`, because `Collection` can read as -a container. These wrappers need the appropriate `Zeroize` support when they -are owned plaintext fields, consistent with the existing ownership decision. - -Neither wrapper automatically adds the field to context or scope, stores it -in the encrypted output, or changes the plaintext's encryption layout. The -target declaration still selects sources. An illustrative declaration is -`context(field = "id"), search_scope(field = "tenant_id")`. With -`id: PrimaryKey` and `tenant_id: ScopeKey`, the selection is -valid, and selecting `id` as search scope is a compile-time error. This -makes typed field selection a reasonable alternative to the earlier -recommendation to accept only search scope supplied by the caller. - -Requirements for misuse resistance: - -- No blanket `SearchScope` implementation for every context type, - serialisable type, raw number, or raw string. Using a value as scope is an - explicit classification. -- A composite scope qualifies only when every component is a scope. A tuple - that contains a primary key cannot become a scope only because it is a - tuple. Context composites can contain either category. -- Do not pass scope eligibility through `Deref`, `AsRef`, or implicit - conversions that remove a primary-key wrapper. Typed target and query - handles keep the definition's concrete scope type, and do not accept any - other type that implements `SearchScope` in its place. Distinct - application newtypes can tell tenant and project scopes apart, even when - their wire types are equal. -- Allow explicit opt-in by applications through a documented trait or derive - contract. This is a semantic capability, not an unsafe-memory contract. - Sealing every implementation cannot prove that a value groups several - records, because a caller can still explicitly rewrap a row identifier as - a scope. Guarantee that accidental mixing of roles is rejected, not - business semantics. - -Dynamic definitions must keep field roles, and the choice between context -and scope for each selection. Validate those roles before execution: a field -explicitly declared as a primary key is not eligible for scope. A plain JSON -number has no role by itself. Do not infer a role from the field name or the -wire encoding. Generated Go types can represent the distinction, while the -engine validates dynamic declarations. - -Rust and dynamic lowering must produce the same canonical context and scope -values. Never use Rust type names as wire domains. Classifying a value as -scope is not evidence of authorisation to access a tenant. - -Acceptance work: - -- valid typed field scopes and caller scopes; -- rejected primary-key scopes and mixed-tuple scopes; -- no automatic selection of marked fields; -- exact agreement of scope type between write and query; -- equivalent dynamic validation. - -#### Dynamic scope roles and validation - -Recommended design: declare semantic types and source selectors in the -definition, validate them once on load, and supply values separately for -each operation. Do not let an individual request assert `scope_key` on an -arbitrary value to override the loaded definition. A definition author can -explicitly classify values, in the same way that a Rust author can construct -wrappers. This resists misuse. It is not authorisation, or proof that a -scope covers several rows. - -Accepted clarification: `types` is optional, and contains reusable named -types. Fields can use built-in types or inline role wrappers. Ordinary -built-in or named values need no `kind`. Supported primitives remain -eligible for ciphertext context. Eligibility for search requires explicit -`scope_key` classification, or another explicitly supported type that can -act as a scope. - -Illustrative declaration fragment. This is a proposed schema, not today's -wire API: - -```json -{ - "types": { - "UserId": { "kind": "primary_key", "value": "u64" }, - "TenantId": { "kind": "scope_key", "value": "u64" } - }, - "fields": { - "id": { "type": "UserId", "passthrough": true }, - "tenant_id": { "type": "TenantId", "passthrough": true } - }, - "context": { "field": "id" }, - "search_scope": { "field": "tenant_id" } -} +```sql +SELECT id, profile FROM customers WHERE profile @> $1::eql_v4.query_json; ``` -This is the context-related part of a record target definition. Encrypted -email targets and their search options are declared separately, as usual. -The shared `field` shorthand selects plaintext fields on encrypt, and stored -fields on decrypt. It does not imply that the field exists in a query -request. - -For a query on the derived email handle, the definition exposes a required -`TenantId` scope argument. No `UserId` argument is required, or accepted as -an alternative scope. A named or composite scope uses the same ordered -schema on the write and query paths, even though their source selectors -differ. - -Loading a definition: - -- resolves field references, and source and output types; -- computes the context and scope capabilities; -- verifies all composite components; -- rejects a primary-key search selector before key I/O; -- verifies that decrypt selectors are available from stored fields or - declared caller input. - -Compile the validated definition into an immutable handle that encrypt, -decrypt, and query operations share. There is no second scope policy -specific to queries. - -At execution, validate these against the selected definition: presence, -nominal type when a typed handle supplies it, physical representation, -range, and composite shape. Raw wire values get their expected semantic type -from the definition. A raw value cannot prove where it came from in the -application. Unknown role overrides and mismatched nominal value handles are -rejected. A query never falls back to unscoped querying when a required -scope is absent. - -When JSON is the authoring or request format, keep full-width integers -through lossless parsing. The frame transport already distinguishes the -supported integer widths. - -Recommended Go generation uses distinct named ID types, and query methods -that take the required scope argument (`TenantID`, not `uint64` or an -untyped context map). Generated wrappers remain conveniences over the same -validated runtime definition. FFI callers that do not use generated code get -runtime errors for invalid declarations and requests. Do not claim that Go -shares Rust's ownership or trait-bound guarantees. - -Lower the chosen scope and binding through the existing canonical context -machinery, keeping scalar widths, ordering, and grouping. The current -dynamic `context.rs` proves parity between runtime `ContextPiece` trees and -Rust context derivations. It does not distinguish the primary-key role from -the scope role. Add role validation before lowering removes those -distinctions. Keep the resolved scope and the resolved binding available -separately to their own operations. - -Do not store role labels in every row only to recover them at runtime. -Definition identity and deployment remain explicit concerns of loading the -same policy. Add Rust, Go, and dynamic fixtures for valid scopes, -primary-key refusals, nested scopes, missing inputs, and identical derived -terms and AAD. - -Completion criteria for work area B: all nine cases run through Rust and -dynamic declarations. Required input is enforced, altered or wrong context -is refused, and existing context encodings agree across all new -implementations. - -Status on 7 November 2026: partly done. - -- **Composition order.** #1491 fixes it for the current shared context. A - field's base comes first, then each extension, appended in order. A - grouped extension stays one part. An empty extension adds nothing. - Before #1491, repeated Rust extensions replaced earlier ones. -- **Evidence.** #1491's conformance cases check the descriptor, AAD, and - index context bytes in native Rust. They check the plan that Go sends, - and cross-opening between the two languages. Other cases check - stored-context expectations, altered context, and refusals before key - I/O. -- **Required input.** #1404 enforces it for opaque values without using - `Default`. #1504 adds optional extensions and expected context as inputs. #1504 is in progress. -- **Empty context.** #1404 resolves the mismatch for the new `Ciphertext` - target. The target accepts an empty context through both APIs. EQL targets - still require a nonempty context. -- **Still open.** The nine directional scenarios, source-field selection - on records, the dynamic form of grouped extensions, and the split between - ciphertext binding and search scope. - -### C. Finish the ciphertext and whole-value storage contracts - -Accepted direction: a single-value plaintext encoding produces one opaque -`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode and -seal it as one value, or an explicit target for field-level encryption. -Serde alone does not imply automatic blob serialisation. A custom type can -explicitly implement the single-value encoding capability. The wrapper name -and the detailed persistence contract remain proposals. - -Recommended persistence behaviour: - -- canonical byte parsing and encoding; -- human-readable Serde as Base64 of those bytes; -- binary Serde as bytes; -- safe `Debug` behaviour. - -Parsing an envelope does not authenticate it. Authentication happens during -decryption. - -The existing transcoder for encrypted output can implement the adapter. -`Visitor::sealed` consumes `Node::Single(SealedValue)` and moves it into the -opaque wrapper. Keep the sealed value internally, instead of forcing a byte -serialisation at construction. There is no new encryption or KMS request. On -decrypt, the wrapper delegates to the same leaf-opening path, using a -single-node wrapper if the internal API requires one. - -Do not convert maps, sequences, passthrough nodes, or sealed markers for -empty containers into this type only because they have a byte -representation. Serialising a whole encrypted tree keeps its several -ciphertexts and authentication boundaries, and does not produce single-value -encryption. The leaf visitor's errors for unsupported shapes are a defensive -check, not the intended way for users to discover the rule. - -Constrain ordinary encryption to inputs that can be encoded as a single -value. Unsupported structural inputs are then rejected before encryption: -statically in Rust where possible, and during dynamic validation. Settle the -exact plaintext capability together with the representation traits. Do not -discover an unsupported shape only after keys have been minted. - -Keep generic structural encryption and searchable JSON explicit. Specify -whether persistence of structural trees is supported, and in what stable -format. Coordinate any protocol changes with their consumers. - -Status on 7 November 2026: partly done by #1404. - -- **`Ciphertext`.** The new type stores exactly the existing `SealedValue` - v1 envelope. It has `to_bytes` and `from_bytes`. Parsing does not authenticate. Human-readable Serde writes padded standard Base64, and - binary Serde writes bytes. `Debug` uses Vitamin C's `OpaqueDebug`, so it - does not print the ciphertext. -- **Single values only.** `OpaqueEncode` is sealed. A type qualifies only if Vitamin C encrypts it as a single value. So `encrypt` does not compile for a struct, a map, or a `Vec`. -- **Trees.** Structural trees use `encrypt_tree` and `decrypt_tree`, and - their persistence is still unspecified. Nullable values also use the - tree API, because `Option` is not an `OpaqueEncode` type. - -#### Typed encodings and extractable ZeroKMS tags - -Requirements added on 25 October 2026: - -- Converting a ciphertext to text returns the existing concrete encoding - wrappers, not `String`. -- The user also wants to explore selecting an encoded ciphertext as a - field's output type. -- The ZeroKMS key tag must be easy to extract from a stored ciphertext, - including in PL/pgSQL. This supports a refinable-tag mechanism that is - designed separately. - -The storage and authentication contract is not yet fixed. - -The current `eql-bindings::encoding::{Base64Text, Base85Text}` validate -canonical padded standard Base64, and the crate's RFC 1924 Base85 encoding, -respectively. Their private strings, encoding constructors, and validating -deserialisers make decoding infallible. They guarantee the text encoding, -not that the bytes form a ciphertext. Reuse these types, and add ordinary -fallible conveniences for parsing text, instead of requiring Serde to -construct them from text. - -Recommended ownership: a small Vitamin C encoding component, re-exported by -Stack Encrypt and EQL. These byte and text utilities do not depend on -encryption, SQL, or EQL layouts. Keep TypeScript and schema integration -optional in the owning crate, and account for Rust's orphan rules when -moving the existing trait implementations. Do not make Stack Encrypt depend -on EQL only to reuse its text wrappers. Final placement of the component -remains a proposal. - -Proposed ciphertext methods `to_base64` and `to_base85` return `Base64Text` -and `Base85Text`. Matching constructors accept the typed text, and still -return a ciphertext parse error when the decoded envelope is invalid. -Encoding validation, envelope parsing, and authentication are three -different guarantees. Use one canonical byte envelope under both encodings. - -Accepted on 25 October 2026: support explicit encoded ciphertext field -targets. Use a storage wrapper specific to ciphertexts, with the working -names `Base64Ciphertext` and `Base85Ciphertext`, instead of making every -`Base64Text` an encryption target. The wrapper validates both the text and -the envelope, and serialises as the text string. The wrapper composes with -`EncryptTarget` and `DecryptSource`: seal once and then encode, and parse -before opening. - -These names stay provisional until the final naming pass, but the encoded -target, as a capability in its own right, is settled. A structural output -can mix `Ciphertext` fields and encoded ciphertext fields without extra KMS -calls. Do not silently change the prescribed representation of EQL targets. - -The current `SealedValue` v1 envelope already uses a simple prefix: -`version(1) | keyset_id(16) | iv(16) | tag_len(2, LE) | tag | local_ciphertext`. -The envelope's ZeroKMS tag begins at byte offset 35, with its length at -offsets 33 and 34. This shows that extraction can be simple. It is not a -commitment to keep those offsets. - -Document the chosen version's layout, and keep it parseable without the -plaintext codec, key retrieval, or decryption. Prefer a checked borrowed -view of the header or envelope, with an accessor explicitly named `key_tag`. -Do not decompose the whole ciphertext, or confuse the key tag with the AEAD -tag. Extraction does not authenticate the returned tag. - -Release a matching SQL and PL/pgSQL extraction helper, and Rust, Go, and SQL -fixtures, with the storage format. Check versions, lengths, and truncation -before slicing. - -For text storage, prefer Base64 when SQL must inspect the header. -[PostgreSQL's binary string -functions](https://www.postgresql.org/docs/current/functions-binarystring.html) -include Base64 decoding and byte slicing, but not Base85 decoding. If SQL -extraction from a Base85 envelope is supported, supply an explicit decoder. -Do not leave Base85 decoding as an assumed capability. - -PostgreSQL's Base64 output also inserts line breaks, unlike the canonical -Rust wrapper. Normalise the SQL output, or validate it against the same -canonical rules in cross-language tests. JSON-tree storage needs its own -header adapter, because its shared key header is distinct from each node's -ciphertext and parent tag. - -The proposed refinement uses the ZeroKMS tag as an HMAC (hash-based message -authentication code) key, and replaces it with a derived tag. The user -clarified the decryption behaviour: ZeroKMS uses the refined tag for policy -checks and logging, and returns the original tag. The decrypt path uses that -returned original tag to rebuild the leaf AAD. - -This resolves the conceptual conflict with binding the original tag into the -AEAD. After refinement, the stored or request tag and the tag used as -authentication input need not be identical. Keep those roles explicit in the -KMS response and the decrypt implementation. Tag extraction returns the tag -actually stored in the envelope. That tag may be the refined one. - -Protocol details remain deferred. This is the intended behaviour, not a -claim that the current client or engine already implements it. Add -integration coverage for refinement and decryption when that work is merged. - -Status on 7 November 2026: partly done. - -- **Done.** #1404 adds `Ciphertext::key_tag`. The accessor reads the - ZeroKMS key tag from the envelope without decrypting, and its - documentation says the tag is untrusted until the ciphertext opens. -- **Still open.** `to_base64` and `to_base85` with the typed text wrappers. - The encoded ciphertext field targets. The SQL and PL/pgSQL extraction - helper, and its Rust, Go, and SQL fixtures. The refined-tag flow. - -#### Single values for encrypt, collections for EncryptInto - -Decided on 25 October 2026: direct `encrypt` handles one single-value input, -and `EncryptInto` handles collection composition and batching. This replaces -the earlier open question about applying encryption to collections -automatically. The agreed public examples are: +#### Encrypt a whole value as one ciphertext + +`OpaqueBlob` encodes any Serde value and seals it as one ciphertext. Go +writes the same format for its opaque structs. So each language can read +the blobs that the other one writes. ```rust -let scalar: Ciphertext = keyset.encrypt(1_i32).await?; +use stack_encrypt::blob::OpaqueBlob; -let batch: Vec = vec![1_i32, 3_i32] - .encrypt_into(&keyset) +// Not on `next` yet: `OpaqueBlob`. +let sealed = keyset + .encrypt_with_context(OpaqueBlob::new(settings), ("users", "settings")) .await?; - -let blob: Ciphertext = keyset - .encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])) +let settings: OpaqueBlob = keyset + .decrypt_with_context(sealed, ("users", "settings")) .await?; ``` -The batch produces two independent ciphertexts through the shared batch -executor. The blob encodes and seals the complete collection as one value. -Do not add implicit collection handling to direct `encrypt`. Do not add a -separate batch entry point only to provide what `EncryptInto` already -covers. Context-taking variants follow the same division. - -Acceptance checks: +#### Encrypt a collection -- reject direct `encrypt(Vec)` before execution; -- support `Vec` to `Vec` without context; -- keep target-directed opening and batching of collections; -- recover the blob as a whole collection. +A `Vec` of values encrypts into a `Vec` of targets, with one batch of key +requests for the whole collection. Each element has its own data key, so +each one decrypts on its own. The order and the length of the `Vec` are +not authenticated, because a database can return rows in any order. -Existing collection composition is kept. Opaque `Ciphertext`, and its target -support without context, are new work. - -The current prototype treats `Vec` as one byte value. On 25 October 2026, that was not a compatibility constraint. After the decision of 7 November 2026, it is a constraint if Go rc.3 stores byte strings that way. This was not checked. - -The latest recommendation is uniform collection -behaviour. `Vec` to `Vec` encrypts individual bytes, while -`OpaqueBlob>` encodes the complete vector as one value. Do not add a -conversion from `Vec` to `Ciphertext`, or a bytes wrapper, only to keep -prototype behaviour. - -A distinct raw-byte capability would need its own use -case. The blob codec need not reproduce the prototype's raw-byte encoding. A byte string that Go rc.3 stores must still open, since 7 November 2026. - -Status on 7 November 2026: partly done, with one difference from this -plan. - -| Acceptance check | Status | -| --- | --- | -| Reject direct `encrypt(Vec)` before execution | Done. `Vec` is not an `OpaqueEncode` type, and a compile-fail test in #1501 checks it | -| Support `Vec` to `Vec` without context | Done (#1501). Encrypting makes one `generate_keys` call, and decrypting makes one `retrieve_keys` call for each keyset | -| Keep target-directed opening and batching of collections | Done (#1501) | -| Recover the blob as a whole collection | Still open. #1503 is in progress | +```rust +let column: Vec = vec![1_i32, 3, 5].encrypt_into(&keyset).await?; +let values = Vec::::decrypt_from(column, &keyset).await?; -The difference is in byte vectors. Pull request #1501 makes `Vec` and `[u8; N]` encrypt into one `Ciphertext`. In #1501, `u8` is not an `OpaqueEncode` type. So `Vec` never encrypts byte by byte. That is the opposite of the recommendation above. +let rows: Vec = users.encrypt_into(&keyset).await?; +``` -The reason given in #1501 is that Vitamin C has no single value for `u8`. - -Pull request #1501 also leaves the order and length of a `Vec` unauthenticated. ADR-0010 decides this for stored rows. +### Go and the dynamic interface -#### Blob codec, format, and timing requirements - -For `OpaqueBlob`, select: +Go and other languages use the same engine through the dynamic interface. +A Go program declares its records with struct tags. The `stashgen` +generator reads the tags, and writes the encrypted type and its functions. +This section shows how a customer uses the Go SDK today, on Go +`v0.1.0-rc.3`. -- the codec, owned by the library; -- an authenticated codec or type discriminator, and a version; -- the schema-evolution policy; -- the error behaviour; -- the supported source and decoded types. +#### Declare a record -Authenticate before deserialising plaintext. Do not silently reinterpret an -ordinary byte ciphertext as a different encoding. Keep an explicit wrapper, -not an implicit Serde fallback. +The `context=` tag gives the record's context. The tag on each field says +how the field is stored: as a passthrough field, as an EQL type, or with +the indexes it names. -The constant-time question needs a concrete conclusion. Assess: +```go +//go:generate go tool stashgen -type User +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt_into=TextEq"` + Name string `stash:"name,encrypt_into=TextMatch"` + Age int32 `stash:"age,encrypt_into=Integer"` +} -- `serdect`; -- the codec; -- both sides of Serde: the value traits, and the serialiser and - deserialiser; -- disclosure of length and shape; -- which custom implementations are allowed. +//go:generate go tool stashgen -type Customer +type Customer struct { + _ struct{} `stash:"context=customers"` + ID int64 `stash:"id,passthrough"` + Profile Profile `stash:"profile,encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)"` +} -If arbitrary Serde types remain supported, do not promise constant-time -serialisation from end to end. If that property is required, select a restricted set of supported types separately. Constant-time behaviour cannot be inferred -from the wrapper or its encryption primitive. +//go:generate go tool stashgen -type Settings +type Settings struct { + _ struct{} `stash:"context=settings,opaque"` // sealed as one blob + Theme string `json:"theme"` +} +``` -Completion criteria for work area C: scalar and blob examples persist, and -reload through a new cipher instance. The examples reject invalid encodings -and verify fixtures for the selected format. The supported timing contract -and the codec version are documented. +#### Encrypt, query, and decrypt -Status on 7 November 2026: partly done. The scalar half is merged (#1404). -The `record-scalars-v1` profile (#1460) checks that frozen Rust scalar -ciphertexts open in Go, and that fresh Go ciphertexts open in Rust. The blob half, its codec version, and its timing contract are in #1503. #1503 is in progress. +A `Cipher` is the client bound to one keyset. `Extend` adds a part to the +context of every field, such as a tenant ID. The generated `Encrypt` and +`Decrypt` functions take a slice. They send one ZeroKMS request for each +500 values. -### D. Complete record derives and preserve collection composition +```go +client, err := encrypt.NewClient(ctx) +defer client.Close() +cipher := client.DefaultKeyset().Extend("tenant-42") -Implement explicit passthrough and renaming of source fields, independently -of context selection and storage. Generate decrypt reconstruction for the -same record. Support nested targets with and without configuration. Report -incompatible context or source types at their declarations. +encrypted, err := Encrypt(ctx, cipher, people) // []EncryptedUser +opened, err := Decrypt(ctx, cipher, encrypted) // []User -Keep the existing collection behaviour: `Vec` to `Vec` composes the -target operations into one logical batch, instead of awaiting each row -separately. Keep the key granularity per document and per leaf. One logical -batch does not promise one physical ZeroKMS request beyond the backend's -limits. +term, err := Fields.Email.Eq(ctx, cipher, "bob@example.com") +// SELECT id, email, name, age FROM users WHERE email = $1::eql_v4.query_text_eq -Completion criteria for work area D: the public examples of mixed records -and searchable records run. These examples include batch encrypt and -decrypt, nested targets, renamed passthrough fields, and stored context. -Dynamic and Go declarations produce equivalent semantics. A plaintext-side -derive that generates output types is a deferred convenience, not a -prerequisite for these workflows. +needle, err := Fields.Profile.Contains(ctx, cipher, map[string]any{"name": "Ada"}) +older, err := Fields.Profile.At.Age.Gt(ctx, cipher, 30) +visit, err := Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.JSONReadOptions{}, + encrypt.AtPath(Paths.Profile.Visits.Place).Eq("home"), + encrypt.AtPath(Paths.Profile.Visits.At).Gt(15), +) +``` -Status on 7 November 2026: partly done. +A field has a query method only for the operators it supports, so an +unsupported query does not compile. The generator asks the engine which +operators each field supports. It does not keep its own copy of the rules. -- **Collections.** #1501 composes an owned `Vec` into one logical batch. - Lifecycle tests check that an empty collection makes no provider call. -- **Records.** No record derive for the owned API is merged. Pull request #1504 adds one, with passthrough, renaming, nested targets, and stored context. Pull request #1504 is in progress. -- **Configured nested targets.** #1504 defers them. The JSON plaintext type is `serde_json::Value`. That type does not implement `Zeroize`. +Two parts of the target state are not in Go yet: -### E. Finish the configured policy's lifetime and query interface +- A Go program cannot set match options. +- A Go program cannot open one child of a JSON node, bound to its parent. -Use the query-design baseline and reconciliation in [Prior query -design](#prior-query-design-baseline-and-reconciliation). The query model is -not an empty design space. A detailed proposal already exists for -field-owned index selection, typed handles, and shared derivation. +Both are in [Remaining work](#remaining-work). -The direction of typed factories is established. The [target definition -section](#recommended-owner-a-reusable-typed-target-definition) now proposes -their owner. The owner is a shared immutable fixed definition or an explicit -runtime instance, with generated typed field handles. Review that proposal, -and prove its initialisation and lifetime behaviour, before treating the -claim of reuse after one resolution as implemented. Runtime configuration -with several fields, and the exact generated signatures, still need to be -specified. +#### What crosses the boundary -Write complete static and Go examples for equality, JSON containment, -ordered paths, and extraction, all using the write policy. Specify query -results and the errors for unsupported operations. Keep those examples free -of mandatory hand-written plans. Keep direct typed EQL conversion, and -support in the catalog and guest, as deliverables of the configured-target -sequence. +The generated code hands the engine a declaration as data. The engine +turns it into the same plan that the Rust derive builds. So Go and Rust +write the same bytes for the same declaration. -Status on 7 November 2026: still open. No merged pull request implements -typed factories, a definition lifecycle, or typed EQL conversion in Rust. -The catalog gained two JSON query roles in #1445 and #1446. +```json +{ + "id": { "context": ["users", "id"], "outputs": ["passthrough"], "type": "int64" }, + "email": { "context": ["users", "email"], "target": "TextEq" }, + "age": { "context": ["users", "age"], "outputs": ["c", "eq", "ore"], "type": "int32" }, + "profile": { "context": ["users", "profile"], + "target": { "name": "JsonSearch", + "options": { "sealed_root": true, "ordered": ["$.age"] } } } +} +``` -### F. Make naming and coordinated integration part of completion +Each row crosses the boundary as one transport frame. A query names a field +and an operator, such as `eq` or `contains`. The engine returns the query +value, with its EQL type. -Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for: +The output keys and the type names are part of the stored format, so they +cannot change. The output keys are `c`, `eq`, `match`, `ore`, `ope`, and +`passthrough`. The type names include `int64` and `string`. -- the proposed role-based JSON names; -- the retained `OpaqueBlob` and `Ciphertext` spellings; -- the shared format crate; -- the split between primitives and the application layer. - -Keep the accepted names of the declaration and convenience traits. -Recommended: rename the plan builder's output-selection method to `target`, -and distinguish it from executing `EncryptInto`. New spellings and crate -moves still need review. - -Remove replaced APIs directly, and update the EQL, Go, and guest consumers -together. There is no requirement to keep old imports. From 7 November 2026, ciphertext written by Go rc.3 must stay readable and searchable. Specify the selected encoding and cryptographic semantics, and set -up fixtures for that contract from now on. +### Rules in the target state -#### Name context parameters `context` +#### The stored format stays compatible with Go rc.3 -Decided on 7 November 2026: a parameter that takes the caller's context is -named `context`. The name `aad` is kept only for a value that is an AAD, -such as the AAD that Vitamin C derives for one leaf. The private functions -`seal_leaf`, `open_leaf`, and `leaf_aad`, and those in `json/derive.rs`, -already follow the rule. +The Go customer already uses Go `v0.1.0-rc.3`. So the data that rc.3 +writes must stay readable and searchable. The stored format is everything +that Stack Encrypt writes to the database: -On `next` at `e4990e49b`, these public methods in -`packages/stack-encrypt/src/cipher.rs` take a context through a parameter -named `aad`: +- ciphertexts, with their envelopes and headers; +- search terms; +- the entries of JSON documents; +- the declaration names that a stored row depends on. -| Method | Line | -| --- | --- | -| `KeysetCipher::encrypt_tree` | 1021 | -| `KeysetCipher::decrypt_tree` | 1036 | -| `StackCipher::decrypt` | 1105 | -| `PendingStackCipherText::plan`, whose bound is `IntoAad` | 1485 | -| `PendingStackCipherText::seal` | 1520 | -| `PendingStackCipherText::into_pending` | 1554 | - -Rename each of these parameters to `context`. This is a breaking change to -the Rust API only. The decision of 7 November 2026 allows it. The change -does not change the stored format. - -Vitamin C's `IntoAad` trait also takes a context. Renaming it is a change -in Vitamin C. Record it there as a follow-up. It is not done in this -repository. - -The Go SDK and the dynamic interface in `packages/stack-encrypt/src/dynamic` -have no public field or parameter named `aad`. I checked with `git grep` -on `next`. - -Completion criteria for work area F: - -- all public examples are executable tests; -- cross-opening and querying between the static and dynamic paths pass - against the selected storage format; -- batching checks pass; -- the guest builds remain supported. - -No new API is described as implemented until it meets its Rust and Go -acceptance criteria. These are the completion criteria for the full -redesign. The earlier Go release uses the gates for Go, the engine, and -storage in the [implementation -sequence](#configured-target-implementation-sequence-and-completion-criteria). -Those gates do not depend on the facade. - -Status on 7 November 2026: partly done. - -- **Cross-opening.** The conformance corpus checks cross-opening between - native Rust and the Go guest for scalars, records, context, and JSON - (#1460, #1491, #1492). -- **Batching.** #1493 and #1501 check batching. -- **Guest builds.** CI builds the guests for the conformance jobs. -- **Not done.** The public examples are not yet tests. Pull request #1504 makes the record examples into tests. Pull request #1504 is in progress. The plan builder method is not renamed - to `target`. The replaced APIs are not removed. - -## Recommended order for the remaining design decisions - -### Decision checkpoint on 25 October 2026 - -These directions are settled: - -- single-value encryption directed by the keyset; -- collection batching directed by the target; -- explicit `OpaqueBlob`; -- conversion methods separate from the `EncryptTarget` and `DecryptSource` - declarations; -- explicit context-input capabilities instead of `Default`; -- typed Rust options factories, and dynamic deserialisation into the same - options; -- static and dynamic support together; -- shared typed definitions and field handles; -- a consistent `using` selector, with strict typestate and definition-first - query spelling; -- `OpaqueEncode` and `OpaqueDecode`; -- consuming `encrypt` and `encrypt_into`, with guarded cleanup; -- ordinary keyset decrypt alongside the destination-directed decrypt pair, - both with context variants, and both returning ordinary owned `T: Zeroize` - results; -- separate ciphertext context and search scope, with `PrimaryKey` and - `ScopeKey` eligibility; -- validation of roles when a dynamic definition is loaded, optional - named-type declarations, and explicit scope eligibility; -- a versioned interoperability specification with shared conformance - coverage; -- no backward compatibility with the prototype API. On 7 November 2026, the user added that the stored format of Go rc.3 must not break. - -Do not reopen these choices only because older examples or descriptions of -work areas still use provisional wording. - -### Remaining questions - -The remaining product and API decisions are narrower. Keep them separate -from the obligations to specify and implement: - -| Remaining question | Decision required | -| --- | --- | -| JSON contract completion | The main Go interface is implemented, and #1492 enrols it in conformance. Close the gap in opening a child bound to its parent, and keep the current semantics | -| Byte vectors | #1501 encrypts `Vec` as one value, unlike this plan. Accept that behaviour, or change the code | -| Key selection and optional provider policy | Review keyset selection derived from plaintext, routed execution scoped to a client, and ZeroKMS lock context, as optional provider capabilities. In the provider-capabilities companion, assess the neutral boundary between data keys and index keys, and the deliberate non-use of Vitamin C's `AsyncCipher` | -| Customer storage formats | Go opaque storage is integrated. For the final release, replace the Go pins, and freeze the post-cascade format | -| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenisers and filters remain, including #1112. Reconcile the earlier privacy concern explicitly. Merging support does not prove that leakage is unchanged | -| Packaging and names (review in progress) | Review the packaging companion's shared format crate, value execution owned by Stack, a Vitamin C boundary of primitives only, retirement of `AsyncCipher`, the ZeroKMS client rename, and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again | +The baseline is the tag `languages/golang/v0.1.0-rc.3`, at commit +`114e08448`. It includes the `eql-v4-install.sql` bundle attached to that +release. -### Next steps +The code and the APIs can still change in ways that break callers. A change +to the stored format must keep rc.3 data readable and searchable, or add a +new format version beside it. See [Changes that would alter the stored +format](#changes-that-would-alter-the-stored-format). -Next, close the remaining customer gaps in configuration and reads. Finish the specification around the merged conformance corpus. Then freeze the release candidate, and resolve the pins for the final release. The Rust conveniences for blobs and the crate moves can follow after the customer storage formats are final. +#### Context and AAD -The original end-to-end JSON implementation is already done. The first conformance corpus is also done. Packaging remains deferred. Keep the initial ORE equality policy unchanged. +The context is the data that a ciphertext is tied to, such as a column +name or a row ID. The field's declaration supplies part of the context, +such as `users/age`. The caller can supply the rest, such as a row ID. -Specification and implementation work still includes: +Stack Encrypt uses the context in three ways: -- exact grouping and encoding of context and scope; -- the dynamic grammar and version identifiers; -- error codes; -- binary envelope fields; -- Rust, Go, and SQL fixtures; -- the flow of the refined tag and the original tag. +1. **To derive the data key.** Stack Encrypt sends the context to ZeroKMS + as a text descriptor, such as `users/age`. ZeroKMS needs the same + descriptor to derive the key again. +2. **To build the AAD of each ciphertext.** The AAD is data that + encryption checks without encrypting it. Stack Encrypt builds it from + the context and the place of the value in its record. The AAD also + includes the format version, the keyset ID, and the key tag. +3. **To derive the search terms** for the field. -On 7 November 2026, the conformance corpus covers the current context -grouping, the current dynamic grammar, and Rust and Go fixtures. It does -not cover scope, version identifiers, stable error codes as a frozen list, -or the refined tag. +A different context gives a different key and a different AAD, so +decryption fails. -Record candidate choices in the formal specification for review. Do not -treat every byte offset or adapter signature as a new architectural -decision. The initial recommendation for persistence remains ordinary Serde -composition for explicit layouts, without a new wire format for arbitrary -trees. +A parameter that takes the caller's context is named `context`. The name +`aad` is used only for a value that is an AAD. Some public methods do not +follow this rule yet. See [Name context parameters +`context`](#name-context-parameters-context). -Cleanup implementation still needs: +Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` +describes the descriptor and the AAD. The module documentation of +`packages/stack-encrypt/src/target/context.rs` describes the search terms. -- guarded encoder sinks; -- handling of partial decoding; -- checks for cancellation and drop; -- an explicit adapter for drop-guarded results. +#### Other rules + +- **Rust and Go behave the same.** Every built-in capability has the same + meaning and the same stored bytes in Rust and in Go. The conformance + profiles test this. A custom Rust query stage can be Rust-only. It is + then marked as not portable. +- **Checks run before keys are used.** Every check that needs no key runs + before Stack Encrypt asks ZeroKMS for a key, or loads an index key. +- **Encryption takes ownership.** `encrypt` and `encrypt_into` take the + plaintext and wipe it, including when they fail. A decrypted value is an + ordinary owned value. The caller is responsible for wiping it. +- **Context is explicit.** A target says whether the caller must supply a + context, can supply one, or must not. Stack Encrypt does not use a type's + `Default` implementation to decide. +- **Each element of a collection stands alone.** Each element has its own + data key. Each element is authenticated on its own. +- **Order terms do not answer equality.** An order-revealing encryption + (ORE) term does not answer an equality query. A field that needs both + declares both indexes. + +## Remaining work + +The work is in the order that the customer needs it. The Go customer work +comes first. The Rust API work follows, and the crate moves come after the +customer release. + +### 1. The Go customer release + +#### Finish the specification around the conformance profiles + +- **What to build.** JSON Schema files for declarations, requests, and + responses. Protocol version identifiers. A release compatibility matrix. + Tests that exchange new JSON values between languages. +- **Why.** The specification lets another language implement the dynamic + interface and get the same results. +- **Done already.** Conformance profiles exist for these areas (#1460, + #1461, #1462, #1469, #1491, #1492, #1493): + - the declaration grammar; + - runtime values and value kinds; + - stored reads and context; + - queries; + - EQL and JSON; + - the lifecycle of values and keys. + + The profiles are experimental. They do not certify a whole SDK. +- **Finished when** these are all true: + - the profiles are no longer experimental; + - the schemas exist; + - the version identifiers exist; + - the compatibility matrix lists rc.3. + +#### Open one child of a JSON node, bound to its parent + +- **What to build.** An operation that opens one child value of a JSON + node, and checks that the child belongs to that parent. This is J17 in + the [JSON operations](2026-10-25-stack-encrypt-json-operations.md) + document. J14 is the matching SQL. +- **Why.** Today a program must read and open the parent's whole subtree to + see one child. +- **Done already.** Nothing. +- **Finished when** Go can open one child, and the `eql-json-v1` + conformance profile covers it. + +#### Configurable match options in Go + +- **What to build.** Match options that a Go program can set: the + tokeniser, the filters, and the n-gram length. Issue #1112 tracks the + part in the generator. +- **Why.** Today Go uses only the default match options. +- **Done already.** Words of two letters can be searched (#1397). +- **Finished when** a Go field can declare match options, and its queries + use the same options as its stored terms. + +#### Freeze the release candidate + +- **What to build.** Fixed test vectors for the JSON storage format and the + matching EQL bundle, recorded as the rc.3 baseline. +- **Why.** The vectors make sure that later work keeps rc.3 data readable + and searchable. +- **Done already.** The conformance jobs run in CI. +- **Finished when** CI fails on any change to the stored bytes of rc.3 + data. + +#### Replace the temporary Vitamin C pins in Go + +- **What to build.** Replace the Go module's pseudo-versions of Vitamin C + with released module tags. +- **Why.** A final Go release cannot depend on unreleased commits. +- **Done already.** Prereleases can keep the pins, so rc.3 uses them. +- **Finished when** the release workflow accepts a final version. + +### 2. The stored format + +#### Changes that would alter the stored format + +Each change below would alter the stored format if it were built as first +planned. Each one must keep rc.3 data readable and searchable, or add a new +format version beside it. + +| Planned change | What it would alter | +| --- | --- | +| Refined tags | The tag stored in the envelope, and what the AAD binds | +| Separate row binding and search scope | The AAD, and the context that search terms are derived under | +| Text encodings as field targets (Base64 and Base85) | How a ciphertext is written as text in a column | +| The `SealedValue` envelope and the JSON header version | The offsets of the key tag, and header version 1, which rc.3 writes | +| Byte vectors | Whether a byte string is one value, or one value for each byte | +| Configurable match options | The terms of columns written with the rc.3 defaults | + +#### Opaque blobs + +- **What to build.** `OpaqueBlob` in Rust, using Vitamin C's storage + profile. The storage profile is Vitamin C's versioned byte format for an + opaque value that is stored. +- **Why.** Go rc.3 already writes opaque structs in this format, so Rust + must read them. +- **Done already.** Go writes and reads the format (#1369, #1370). #1503 + adds `OpaqueBlob`, and is in review. +- **Finished when** Rust and Go each open blobs that the other wrote, and + the conformance profiles cover it. + +#### Typed text encodings and key tag extraction + +- **What to build.** + - `to_base64` and `to_base85` methods on `Ciphertext`. They return the + existing `Base64Text` and `Base85Text` types, not `String`. + - Field targets that store a ciphertext as text. Their working names are + `Base64Ciphertext` and `Base85Ciphertext`. Each one checks both the text + and the envelope. + - A SQL and PL/pgSQL function that reads the ZeroKMS key tag from a stored + ciphertext. It must check the version and the lengths before it reads. +- **Why.** Some columns store ciphertext as text. SQL needs to read the key + tag for refined tags. +- **Limits.** PostgreSQL can decode Base64, but not Base85. So SQL can read + the key tag from Base85 text only if we supply a decoder. Reading the tag + does not authenticate it. +- **Done already.** `Ciphertext` is stored as bytes, and Serde writes it as + Base64. `Ciphertext::key_tag` reads the tag in Rust (#1404). +- **Finished when** the methods and targets exist, and Rust, Go, and SQL + test vectors show the same key tag for the same stored value. + +### 3. The Rust API + +#### Record derives + +- **What to build.** The `EncryptTarget` and `DecryptSource` derives for + records, with passthrough fields, renamed fields, nested records, and + stored context. +- **Why.** The derives give the [record example](#encrypt-a-record). +- **Done already.** One value (#1404) and collections (#1501). #1504 adds + the record derives, and is in review. +- **Finished when** the public examples for mixed and searchable records + run as tests. + +#### EQL types as fields + +- **What to build.** EQL types such as `TextEq` and `JsonTreeDocument` as + targets of the record derive. Today they work only with the older + structural API, and need a `NonEmpty` context. +- **Why.** This gives the [EQL field example](#store-a-field-as-an-eql-type) + and the [JSON example](#encrypt-a-json-document). +- **Done already.** Nothing. +- **Finished when** a record with EQL fields encrypts, decrypts, and + queries through the new API. + +#### Definitions, field handles, and queries + +- **What to build.** A shared definition for each record type, built once + from its declaration, with a method for each field that returns its + handle. Queries select a field with `using`. Typed Rust option + factories, and the same options read from data in the dynamic interface. +- **Why.** A query must use the same declaration as the stored values. The + caller must not write a second plan by hand. +- **Done already.** Queries work through a plan written by hand: + `cipher.query(op, value).on(&plan)`. +- **Finished when** the [query example](#query) compiles, and tests show + that a query cannot run without its definition or its required context. + +#### Context from fields and from the caller + +- **What to build.** The ways a record gets its context, as the nine + scenarios in the [context + examples](2026-10-24-stack-encrypt-context-examples.md) describe them. A + record can take context from one of its fields, from the caller, or from + both. A record can also store its context in the output. On decrypt, a + record can read its stored context, or check it against an expected context. The + same rules need a form in the dynamic interface and in Go. +- **Why.** A program must never need to open a ciphertext to learn the + context that opens it. +- **Done already.** The order in which context parts are joined is fixed + and tested in Rust and Go (#1491). The context inputs `RequiredContext`, + `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds + `ContextExtension` and `ExpectedContext`, and is in review. +- **Finished when** all nine scenarios run as tests in Rust and Go. -Status on 7 November 2026: +#### Name context parameters `context` -| Cleanup item | Status | -| --- | --- | -| Guarded encoder sinks | Partly done. #1501 makes the frame codec write into a `Buffer`. A `Buffer` wipes each allocation it outgrows | -| Handling of partial decoding | Partly done. #1493 and #1501 wipe the values decoded so far when a later value fails. A partly built Serde value is still open | -| Checks for cancellation and drop | Done (#1493, #1501) | -| An explicit adapter for drop-guarded results | Still open | - -Under the user's latest sequencing decision, implementation improvements can -follow the stable format. Keep the stored contract, and document the actual -guarantees, instead of claiming that the eventual goals are already met. -This plan does not imply a completed constant-time or zeroisation audit. - -The caching of shared definitions, the behaviour on initialisation error and -retry, generic identity, and request lifetimes still need concrete -signatures and proof. That is implementation design within the recorded -ownership model. It is not a reason to reopen whether writes and queries -share a policy. - -Passthrough implementation, transcoder adapters, moving traits, -documentation updates, and acceptance tests remain work items. Collection -composition for the owned API is done (#1501). These items are not reasons to describe the agreed API direction as -undecided. The prototype API needs no backward-compatibility phase. The stored format has needed one since 7 November 2026, because the Go customer builds on rc.3. +These public methods in `packages/stack-encrypt/src/cipher.rs` take a +context through a parameter named `aad`: + +- `KeysetCipher::encrypt_tree`; +- `KeysetCipher::decrypt_tree`; +- `StackCipher::decrypt`; +- `PendingStackCipherText::plan`, whose bound is Vitamin C's `IntoAad`; +- `PendingStackCipherText::seal`; +- `PendingStackCipherText::into_pending`. + +Rename each parameter to `context`. This breaks the Rust API only. It does +not change the stored format. Renaming Vitamin C's `IntoAad` trait is a +follow-up in Vitamin C. The Go SDK and the dynamic interface have no public +parameter named `aad`. + +#### Remove the structural API + +- **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and + the helpers in `target::legacy`. Move EQL, Go, and the guest to the new + API in the same change. Rename the plan builder's output method to + `target`. +- **Why.** Two APIs for the same job make the right one hard to find. +- **Done already.** The new API exists beside the old one. Issue #1488 + tracks the removal. +- **Finished when** the old API is gone, and every consumer uses the new + API. + +#### Wipe plaintext in the remaining cases + +- **What to build.** Wipe the parts of a Serde value that were decoded + before a later part failed. Add an adapter that wipes a decrypted value + when it is dropped. +- **Why.** Plaintext must not stay in freed memory. +- **Done already.** Collections wipe the values decoded before a failure. + Cancellation and drop are tested (#1493, #1501). +- **Finished when** the lifecycle conformance profile covers both cases. + +### 4. After the customer release + +- **Packaging and names.** Crate moves, the shared format crate, and the + final names. The [packaging + document](2026-10-25-stack-encrypt-packaging.md) has the proposal. +- **Key providers.** Choosing keysets from data, and ZeroKMS lock context, + as optional capabilities of a key provider. The [provider capabilities + document](2026-10-25-stack-encrypt-provider-capabilities.md) has the + proposal. + +## Open questions and decisions to be made + +The lead who approves this plan decides each question. + +| Question | Options | Recommendation | +| --- | --- | --- | +| Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | +| Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | +| For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | +| Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | +| How should a ciphertext be bound to its row while its search terms cover a wider scope? | Separate row binding from search scope in the declaration, for example with `PrimaryKey` and `ScopeKey` roles. Or keep one context | Separate them. Never add a row ID to the search scope automatically | +| Should keysets be chosen from data, and should Stack Encrypt use ZeroKMS lock context? | Optional capabilities of a key provider. Or leave them out | Optional capabilities, after the customer release | +| What privacy claim can be made for match terms of short words? | Accept what they reveal, and document it. Or change how short words are stored | No recommendation yet. Merging #1397 did not show that they reveal nothing new | +| Which crate should own `Base64Text` and `Base85Text`? | A small Vitamin C encoding crate. Or keep them in `eql-bindings` | A small Vitamin C crate, re-exported by Stack Encrypt and EQL | +| What names and crate layout should the final API use? | The proposal in the packaging document | Review the proposal after the customer release | diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index 8c07554f7..df80be955 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -51,7 +51,7 @@ From now on, a change that would alter the stored format must keep data written by rc.3 readable and searchable. Otherwise it must add a new, versioned format beside the rc.3 format. The Rust and Go code and APIs can still change in breaking ways. The [API -plan](2026-10-24-stack-encrypt-api-plan.md#work-that-the-stored-format-decision-constrains) +plan](2026-10-24-stack-encrypt-api-plan.md#changes-that-would-alter-the-stored-format) lists the planned work that this decision constrains. ### History: no backward compatibility with the prototype @@ -235,8 +235,9 @@ discussion of batching). Vitamin C's trait definitions are in - **Text encodings, decided on 25 October 2026.** Text conversions must use the existing `Base64Text` and `Base85Text` wrappers from the EQL bindings. The - [persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) - record the proposals and the limit on decoding Base85 in SQL. + [text encodings item](2026-10-24-stack-encrypt-api-plan.md#typed-text-encodings-and-key-tag-extraction) + in the API plan records the remaining work and the limit on decoding + Base85 in SQL. The follow-up of 25 October 2026 also set these directions: @@ -1460,10 +1461,11 @@ Do not reuse that mapping unchanged, for three reasons: - A successful `to_vec` returns a vector with no guard. The -[codec assessment](2026-10-24-stack-encrypt-api-plan.md#blob-codec-assessment-vitamin-c-frames) -lists the work needed, and Borsh as the fallback codec. Neither the codec -choice nor the `Zeroize` bound establishes constant-time behaviour, or -cleanup after partial decoding. +[opaque blobs](2026-10-24-stack-encrypt-api-plan.md#opaque-blobs) item in +the API plan records the codec now chosen: Vitamin C's storage profile. +Go rc.3 already writes that format. Neither the codec choice nor the `Zeroize` +bound establishes constant-time behaviour, or cleanup after partial +decoding. ### Priority update diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 45310cc9d..468a5810d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -33,7 +33,7 @@ Still open, or in progress: From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not break. So a new selector must produce the same context bytes as rc.3 for data that rc.3 can write. See the [API -plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). This document compares what happens to encryption context in each useful combination of diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index 36bcbe97f..f829fefc7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -47,7 +47,7 @@ from now on, because the Go customer builds on Go `v0.1.0-rc.3`. The stored format is what is written to the database, such as ciphertext envelopes and search terms. The APIs can still change. This replaces the decision of 25 October 2026 for stored data. See the [API -plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). ## Terms used in this document @@ -140,7 +140,7 @@ ciphertext for the encoded collection instead. The latest recommendation applies the same rule to byte vectors. No wrapper or special case is needed to keep the prototype's behaviour. Pull request #1501 differs: it encrypts `Vec` and `[u8; N]` as one value. The [API -plan](2026-10-24-stack-encrypt-api-plan.md#differences-between-merged-code-and-this-plan) +plan](2026-10-24-stack-encrypt-api-plan.md#open-questions-and-decisions-to-be-made) records this as a question for the user. These calls need the proposed opaque ciphertext and support for targets that take no context. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 415ca7079..9266b0cc7 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -22,7 +22,7 @@ in Rust, Go, and PostgreSQL tests. From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not break. The stored format is what is written to the database, such as JSON tree entries and their search terms. See the [API -plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). The Rust example in this document uses the proposed public derive and facade. That example is not executable code for the current engine. The diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 0438e1519..fe57aa1ac 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -28,7 +28,7 @@ contract is under separate user review. | --- | --- | | Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | | AAD | Additional authenticated data: data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad) | -| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026) | +| Stored format | What is written to the database, as Go `v0.1.0-rc.3` writes it. See [the stored-format decision](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3) | ## Update on 7 November 2026 diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index 6f490033d..860b9bb46 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -16,7 +16,7 @@ break. The stored format is what is written to the database, such as ciphertext envelopes and search terms. The refined-tag work in item 6 below changes the stored tag, so it must keep rc.3 data readable. See the [API -plan](2026-10-24-stack-encrypt-api-plan.md#stored-format-must-stay-compatible-from-7-november-2026). +plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). The user who requested this assessment asked for three things: From bbbca0bb5a79d0cc4bc3409120e7a0716b934147 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:14:02 +1100 Subject: [PATCH 08/42] docs: pass context by reference in the Stack Encrypt plan Add a target-state rule and a work item: methods that take a context borrow it for the request, so one context serves a write, a query, and a read without a clone. Update the EQL example to pass &column. --- docs/plans/2026-10-24-stack-encrypt-api-plan.md | 17 +++++++++++++++++ .../2026-10-24-stack-encrypt-public-examples.md | 12 +++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index d96c1d3e7..42d14a26d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -442,6 +442,10 @@ describes the descriptor and the AAD. The module documentation of - **Encryption takes ownership.** `encrypt` and `encrypt_into` take the plaintext and wipe it, including when they fail. A decrypted value is an ordinary owned value. The caller is responsible for wiping it. +- **Context is passed by reference.** A method that takes a context + borrows it while the request runs. So a program can use one context for + a write, a query, and a read without cloning it. An owned context also + works. - **Context is explicit.** A target says whether the caller must supply a context, can supply one, or must not. Stack Encrypt does not use a type's `Default` implementation to decide. @@ -642,6 +646,19 @@ not change the stored format. Renaming Vitamin C's `IntoAad` trait is a follow-up in Vitamin C. The Go SDK and the dynamic interface have no public parameter named `aad`. +#### Pass context by reference + +- **What to build.** Let every method that takes a context accept a + borrowed context, such as `&Identifier`. The request borrows the context + for as long as it runs. `eql-bindings` needs `IntoContext` for + `&Identifier`. +- **Why.** On `next`, every context is taken by value. A program that uses + one column context for a write and a query must clone it for each call. +- **Done already.** Stack Encrypt implements `IntoContext` for `&Label`. + The context inputs, such as `RequiredContext`, still need an owned value. +- **Finished when** the public examples pass a context by reference, with + no clone, and compile. + #### Remove the structural API - **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index f829fefc7..bcbe02739 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -761,7 +761,7 @@ async fn main() -> Result<(), Box> { let email = "alice@example.com".to_owned(); let encrypted: TextEq = email - .encrypt_into_with_context(&keyset, column.clone()) + .encrypt_into_with_context(&keyset, &column) .await?; let directory = tempfile::tempdir()?; @@ -771,13 +771,13 @@ async fn main() -> Result<(), Box> { let reader = StackCipher::new().await?; let query_keyset = reader.default_keyset(); let query: TextEqQuery = email - .encrypt_into_with_context(&query_keyset, column.clone()) + .encrypt_into_with_context(&query_keyset, &column) .await?; let loaded: TextEq = serde_json::from_slice(&std::fs::read(&path)?)?; assert_eq!(loaded.hm, query.hm); // Supply the expected column, so stored identifier metadata is checked too. - let restored: String = loaded.decrypt_into(&reader, column).await?; + let restored: String = loaded.decrypt_into(&reader, &column).await?; assert_eq!(restored, email); Ok(()) } @@ -790,6 +790,12 @@ the program needs `stack_encrypt::target::legacy::{DecryptFrom, EncryptInto}`, or `keyset.encrypt_as`, as `eql-bindings` uses. This was checked by reading the source, not by compiling. +The program passes the column by reference, as `&column`, so it does not +clone it for each call. That is the target form. On `next`, every context +is taken by value, so the program would need `column.clone()`. The [API +plan](2026-10-24-stack-encrypt-api-plan.md#pass-context-by-reference) +lists this as remaining work. + `TextEq` includes its EQL version, its identifier, the ciphertext, and the equality term. Its context type is `NonEmpty`. The application supplies the typed identifier for the expected table and column, on both From 7370751341f84581f0496cb79c3f0ec589479533 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:16:33 +1100 Subject: [PATCH 09/42] docs: plan IdentifiedBy constructors for keyset selection Replace IdentifiedBy::Name("acme".to_string().into()) in the examples with IdentifiedBy::name("acme")?, and add the constructors to the remaining work. They belong in zerokms-protocol, which defines the type. The From conversion the old example used skips name validation. --- .../2026-10-24-stack-encrypt-api-plan.md | 28 +++++++++++++++++-- ...-25-stack-encrypt-provider-capabilities.md | 7 +++-- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 42d14a26d..85defa3ea 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -58,9 +58,10 @@ use stack_encrypt::{kms::IdentifiedBy, StackCipher}; let cipher = StackCipher::new().await?; let keyset = cipher.default_keyset(); -let tenant = cipher - .keyset(IdentifiedBy::Name("acme".to_string().into())) - .await?; +// Not on `next` yet: the `IdentifiedBy::name` and `IdentifiedBy::uuid` +// constructors. `name` checks the name, so it returns a `Result`. +let tenant = cipher.keyset(IdentifiedBy::name("acme")?).await?; +let by_id = cipher.keyset(IdentifiedBy::uuid(keyset_id)).await?; ``` #### Encrypt and decrypt one value @@ -659,6 +660,27 @@ parameter named `aad`. - **Finished when** the public examples pass a context by reference, with no clone, and compile. +#### Constructors for `IdentifiedBy` + +- **What to build.** Two constructors that select a keyset: + - `IdentifiedBy::name(&str)`. It checks the name, so it returns a + `Result`; + - `IdentifiedBy::uuid(Uuid)`. +- **Why.** On `next`, selecting a keyset by name needs + `IdentifiedBy::Name("acme".to_string().into())`. The `.into()` converts + the `String` into the `Name` type. That conversion does not check the + name. Only `Name::try_from(&str)` checks it: at most 64 characters, not + empty, and only `A-Z a-z 0-9 _ - /`. +- **Where.** `IdentifiedBy` and `Name` are in the `zerokms-protocol` crate. + The `cipherstash-suite` repository publishes that crate. Rust allows inherent methods only in + the crate that defines the type. So the constructors go in + `zerokms-protocol`. Stack Encrypt re-exports `IdentifiedBy` from + `stack_encrypt::kms`, so callers get them through that path. +- **Done already.** `StackCipher::keyset` takes `impl Into`, + and `IdentifiedBy` implements `From` and `From`. +- **Finished when** a released `zerokms-protocol` has both constructors, + and the examples in this plan compile. + #### Remove the structural API - **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index 860b9bb46..381d3d7d2 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -176,10 +176,13 @@ execution or provider layer, not in the derive. ### Selecting the keyset by hand and by declaration The existing form, where the caller selects the keyset by hand, already -fits the agreed API: +fits the agreed API. This example uses the `IdentifiedBy::name` +constructor that the [API +plan](2026-10-24-stack-encrypt-api-plan.md#constructors-for-identifiedby) +proposes: ```rust -let tenant = cipher.keyset(IdentifiedBy::Name(user.tenant_id.clone().into())).await?; +let tenant = cipher.keyset(IdentifiedBy::name(&user.tenant_id)?).await?; let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; ``` From 95bf77496612f52ec35826a0f4f18c83f7a85b02 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:19:48 +1100 Subject: [PATCH 10/42] docs: let Stack Encrypt encrypt borrowed values such as &str Show keyset.encrypt("secret") in the target state, and add the work to accept borrowed single values. Encryption still wipes an owned plaintext; for a borrowed one the caller keeps that responsibility. --- .../2026-10-24-stack-encrypt-api-plan.md | 39 +++++++++++++++---- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 85defa3ea..e349e94e5 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -66,18 +66,23 @@ let by_id = cipher.keyset(IdentifiedBy::uuid(keyset_id)).await?; #### Encrypt and decrypt one value -`encrypt` takes ownership of the value and wipes it. It returns one -`Ciphertext`. A context ties the ciphertext to where it belongs, so it -cannot be decrypted anywhere else. +`encrypt` returns one `Ciphertext`. A context ties the ciphertext to where +it belongs, so it cannot be decrypted anywhere else. + +`encrypt` accepts an owned value or a borrowed one. It wipes an owned +value, such as a `String`. It only reads a borrowed value, such as a +`&str`, so the caller is responsible for wiping the original. ```rust use stack_encrypt::Ciphertext; -let secret: Ciphertext = keyset.encrypt("secret".to_owned()).await?; +// Not on `next` yet: borrowed values such as `&str`. +let secret: Ciphertext = keyset.encrypt("secret").await?; let plaintext: String = keyset.decrypt(secret).await?; +let number: String = read_card_number()?; let card: Ciphertext = keyset - .encrypt_with_context("4111 1111 1111 1111".to_owned(), "users/42/card") + .encrypt_with_context(number, "users/42/card") // wipes `number` .await?; // Store the ciphertext as bytes, or as Base64 text through Serde. @@ -440,9 +445,11 @@ describes the descriptor and the AAD. The module documentation of then marked as not portable. - **Checks run before keys are used.** Every check that needs no key runs before Stack Encrypt asks ZeroKMS for a key, or loads an index key. -- **Encryption takes ownership.** `encrypt` and `encrypt_into` take the - plaintext and wipe it, including when they fail. A decrypted value is an - ordinary owned value. The caller is responsible for wiping it. +- **Encryption wipes an owned plaintext.** `encrypt` and `encrypt_into` wipe an + owned plaintext, including when they fail. They only read a borrowed + plaintext, so the caller is responsible for wiping it. Stack Encrypt + wipes every copy it makes. A decrypted value is an ordinary owned value. + The caller is responsible for wiping it. - **Context is passed by reference.** A method that takes a context borrows it while the request runs. So a program can use one context for a write, a query, and a read without cloning it. An owned context also @@ -660,6 +667,22 @@ parameter named `aad`. - **Finished when** the public examples pass a context by reference, with no clone, and compile. +#### Encrypt borrowed values + +- **What to build.** Let `encrypt` and `encrypt_with_context` accept a + borrowed single value, such as `&str`, `&[u8]`, or `&i64`. Keep the + current behaviour for an owned value. +- **Why.** A program that encrypts a string literal must write + `"secret".to_owned()` on `next`. The copy adds nothing, because a literal + cannot be wiped. +- **How.** The engine already reads an owned value through a borrow + (`ScalarRef` in `scalar.rs`). But `OpaqueEncode` requires `Zeroize` and + `'static`, so `&str` does not meet it. A borrowed value needs its own + bound, and the request borrows the value for as long as it runs. +- **Done already.** Owned single values (#1404). +- **Finished when** `keyset.encrypt("secret")` compiles, and a test shows + that Stack Encrypt wipes its own copies of a borrowed value. + #### Constructors for `IdentifiedBy` - **What to build.** Two constructors that select a keyset: From 56413926fd27910834d5293a2e5c7115831e860e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:46:03 +1100 Subject: [PATCH 11/42] docs: say Go already stores EQL types as record fields The dynamic interface already lets a record field name its EQL type, and Go uses it. Only the Rust record derive lacks EQL field types, so the example and the work item now say that. --- .../2026-10-24-stack-encrypt-api-plan.md | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index e349e94e5..22b93aba7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -159,6 +159,10 @@ A field can use an EQL type from the `eql-bindings` crate as its target. The field is then stored as one EQL column, such as `public.eql_v4_text_eq`. +Go records already do this through the dynamic interface. A field there +names its EQL type, such as `"target": "TextEq"`, and the engine runs that +type for the field. The Rust record derive does not support it yet. + ```rust use eql_bindings::v4::{integer::IntegerOrd, text::TextEq}; @@ -169,7 +173,8 @@ struct Member { age: i32, } -// Not on `next` yet: EQL types as fields of a record. +// Not on `next` yet in Rust: EQL types as fields of a derived record. +// Go records do this today through the dynamic interface. #[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] #[stash(struct = Member, context = "members")] struct EncryptedMember { @@ -599,13 +604,22 @@ format version beside it. #### EQL types as fields - **What to build.** EQL types such as `TextEq` and `JsonTreeDocument` as - targets of the record derive. Today they work only with the older - structural API, and need a `NonEmpty` context. + field types of the Rust record derive. In Rust today, an EQL type works + only on its own, through the older structural API, with a + `NonEmpty` context. The structural record derive gives each + field a `CallerContext`, and `NonEmpty` has no conversion + from it. So no Rust record can have an EQL field yet. - **Why.** This gives the [EQL field example](#store-a-field-as-an-eql-type) and the [JSON example](#encrypt-a-json-document). -- **Done already.** Nothing. -- **Finished when** a record with EQL fields encrypts, decrypts, and - queries through the new API. +- **Done already.** The dynamic interface supports EQL fields, so Go + records use them today. A field there names its EQL type, and the engine + looks the type up by name through `eql-bindings` (the `TargetResolver` + in `dynamic/record.rs`). `Identifier` already gives the same context as + the `(table, column)` pair that a record derive builds. +- **Finished when** these are all true: + - a Rust record with EQL fields encrypts, decrypts, and queries through + the new API; + - it writes the same bytes as a Go record with the same declaration. #### Definitions, field handles, and queries From 616be5fdd2a1fc68305fd31c23dd379ec727ce43 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 12:53:52 +1100 Subject: [PATCH 12/42] docs: show static Rust queries before the dynamic interface Split the target state into Rust with static types and a dedicated dynamic interface section. Static queries encrypt a value into the field's EQL query type. The dynamic section shows the declaration as data, queries through a definition, and the Go SDK. Record why a Rust record cannot have an EQL field on next (a test record with a TextEq field fails to compile), and add a ready-made EQL resolver for the dynamic interface to the remaining work. --- .../2026-10-24-stack-encrypt-api-plan.md | 217 ++++++++++++------ 1 file changed, 146 insertions(+), 71 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 22b93aba7..fb23000e1 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -42,9 +42,10 @@ This section shows what a developer writes when every item in this plan is built. A comment marks each piece of code that does not compile on `next` today. [Remaining work](#remaining-work) says what each of those needs. -### Rust +### Rust with static types -These examples use the `stack-encrypt` crate directly. +These examples use the `stack-encrypt` crate directly, with a Rust type for +each encrypted value. #### Connect and choose a keyset @@ -185,29 +186,29 @@ struct EncryptedMember { } ``` -#### Query +#### Query a field -A query encrypts the search value with the same keyset and the same field -declaration as the stored values. `definition()` returns the record's -declaration, built once. It has one method for each field. That method -returns the field's handle. The query value binds to SQL as a parameter. +A query encrypts the search value into the query type of the field. It uses +the same keyset and the same context as the stored value. Each EQL type has +a query type, such as `TextEqQuery` for `TextEq` and `IntegerOrdQuery` for +`IntegerOrd`. The query value binds to SQL as a parameter, and the SQL +operator chooses the comparison. ```rust -use stack_encrypt::op; +use eql_bindings::v4::{integer::IntegerOrdQuery, text::TextEqQuery}; +use eql_bindings::Identifier; -// Not on `next` yet: `definition()`, field handles, and `using`. -// Today a query names a plan written by hand: -// `cipher.query(op::Eq, value).on(&plan)`. -let members = EncryptedMember::definition()?; +let email = Identifier::for_column("members", "email")?; +let age = Identifier::for_column("members", "age")?; -let by_email = cipher - .using(&members.email()) - .query(op::Eq, "alice@example.com") +// Not on `next` yet: the new traits for EQL types, and a borrowed value +// and context. Today this needs `target::legacy::EncryptInto`, an owned +// value, and an owned context. +let by_email: TextEqQuery = "alice@example.com" + .encrypt_into_with_context(&keyset, &email) .await?; -let older = cipher.using(&members.age()).query(op::Gt, &30).await?; -let adults = cipher - .using(&members.age()) - .query(op::Between, &(18, 65)) +let older: IntegerOrdQuery = 30_i32 + .encrypt_into_with_context(&keyset, &age) .await?; ``` @@ -216,16 +217,19 @@ SELECT id, email, age FROM members WHERE email = $1::eql_v4.query_text_eq; SELECT id, email, age FROM members WHERE age > $1::eql_v4.query_integer_ord; ``` -A match query (`op::Matches`) works in the same way on a field with a match -index. Its answer can include rows that do not match. So the caller -decrypts the rows, and checks each one. +The context `Identifier::for_column("members", "email")` is the same as the +context that the record derive gives the `email` field of +`EncryptedMember`. So the query matches the stored values. + +A match query works in the same way, on a field with a match index. Its +answer can include rows that do not match. So the caller decrypts the rows, +and checks each one. #### Encrypt a JSON document A JSON field is stored as one encrypted document in a `public.eql_v4_json_search` column. Typed Rust options say which paths can -be ordered. A query can test containment, compare the value at a path, -match one element of an array, or run a jsonpath predicate. +be ordered. ```rust use eql_bindings::v4::json_tree::JsonTreeDocument; @@ -247,17 +251,13 @@ struct EncryptedProfile(JsonTreeDocument); let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?; let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; - -let profiles = EncryptedProfile::definition()?; -let needle = cipher - .using(&profiles.document()) - .query(op::Contains, &json!({ "name": "Ada" })) - .await?; +let opened = JsonPlaintext::decrypt_from(encrypted, &keyset).await?; ``` -```sql -SELECT id, profile FROM customers WHERE profile @> $1::eql_v4.query_json; -``` +A JSON query can test containment, compare the value at a path, match one +element of an array, or run a jsonpath predicate. Each of +these takes an operator and a path. So JSON queries use the [dynamic +interface](#query-through-a-definition). #### Encrypt a whole value as one ciphertext @@ -291,15 +291,95 @@ let values = Vec::::decrypt_from(column, &keyset).await?; let rows: Vec = users.encrypt_into(&keyset).await?; ``` -### Go and the dynamic interface +### The dynamic interface -Go and other languages use the same engine through the dynamic interface. -A Go program declares its records with struct tags. The `stashgen` -generator reads the tags, and writes the encrypted type and its functions. -This section shows how a customer uses the Go SDK today, on Go -`v0.1.0-rc.3`. +The dynamic interface takes a declaration and values as data, not as Rust +types. A program uses it in two cases: + +- when it learns the declaration only at run time, such as a query that + chooses its operator and path from user input; +- when it is written in another language, such as Go. -#### Declare a record +The engine turns a declaration into the same plan that the Rust derive +builds. So the dynamic interface writes the same bytes as the static API +for the same declaration. + +#### A declaration as data + +A declaration names each field and its context. It also says how the field +is stored: as a passthrough field, with the indexes it lists, or as an EQL +type. + +```json +{ + "id": { "context": ["users", "id"], "outputs": ["passthrough"], "type": "int64" }, + "email": { "context": ["users", "email"], "target": "TextEq" }, + "age": { "context": ["users", "age"], "outputs": ["c", "eq", "ore"], "type": "int32" }, + "profile": { "context": ["users", "profile"], + "target": { "name": "JsonSearch", + "options": { "sealed_root": true, "ordered": ["$.age"] } } } +} +``` + +Each row crosses the boundary as one transport frame. A query names a field +and an operator, such as `eq` or `contains`, and gives a value. The engine +returns the query value, with its EQL type. + +The output keys and the type names are part of the stored format, so they +cannot change. The output keys are `c`, `eq`, `match`, `ore`, `ope`, and +`passthrough`. The type names include `int64` and `string`. + +#### Query through a definition + +In Rust, `definition()` returns the declaration of a derived record, built +once, as data. It has one method for each field. That method returns the +field's handle. `using` selects a field, and `query` takes an operator and a value. +A program can also build a definition from a declaration that it reads at +run time. + +```rust +use stack_encrypt::op; + +// Not on `next` yet: `definition()`, field handles, and `using`. +// Today a query names a plan written by hand: +// `cipher.query(op::Eq, value).on(&plan)`. +let members = EncryptedMember::definition()?; + +let by_email = cipher + .using(&members.email()) + .query(op::Eq, "alice@example.com") + .await?; +let adults = cipher + .using(&members.age()) + .query(op::Between, &(18, 65)) + .await?; + +let profiles = EncryptedProfile::definition()?; +let ada = cipher + .using(&profiles.document()) + .query(op::Contains, &json!({ "name": "Ada" })) + .await?; +``` + +```sql +SELECT id, profile FROM customers WHERE profile @> $1::eql_v4.query_json; +``` + +Below the definition, `stack_encrypt::dynamic::record` has the functions +that take a declaration as a frame: `plan_with`, `encrypt_with`, `query`, +and `decrypt_with`. Each takes a `TargetResolver`. The resolver runs the EQL +type that a field names. + +On `next`, each program writes its own resolver over +`eql-bindings`. In the target state, `eql-bindings` provides it. See [A +ready-made EQL resolver](#a-ready-made-eql-resolver). + +#### Call it from Go + +A Go program declares its records with struct tags. The `stashgen` +generator reads the tags, and writes the encrypted type, its functions, and +the declaration it sends to the engine. The examples show how a customer +uses the Go SDK today, on Go `v0.1.0-rc.3`. The `context=` tag gives the record's context. The tag on each field says how the field is stored: as a passthrough field, as an EQL type, or with @@ -329,8 +409,6 @@ type Settings struct { } ``` -#### Encrypt, query, and decrypt - A `Cipher` is the client bound to one keyset. `Extend` adds a part to the context of every field, such as a tenant ID. The generated `Encrypt` and `Decrypt` functions take a slice. They send one ZeroKMS request for each @@ -366,31 +444,6 @@ Two parts of the target state are not in Go yet: Both are in [Remaining work](#remaining-work). -#### What crosses the boundary - -The generated code hands the engine a declaration as data. The engine -turns it into the same plan that the Rust derive builds. So Go and Rust -write the same bytes for the same declaration. - -```json -{ - "id": { "context": ["users", "id"], "outputs": ["passthrough"], "type": "int64" }, - "email": { "context": ["users", "email"], "target": "TextEq" }, - "age": { "context": ["users", "age"], "outputs": ["c", "eq", "ore"], "type": "int32" }, - "profile": { "context": ["users", "profile"], - "target": { "name": "JsonSearch", - "options": { "sealed_root": true, "ordered": ["$.age"] } } } -} -``` - -Each row crosses the boundary as one transport frame. A query names a field -and an operator, such as `eq` or `contains`. The engine returns the query -value, with its EQL type. - -The output keys and the type names are part of the stored format, so they -cannot change. The output keys are `c`, `eq`, `match`, `ore`, `ope`, and -`passthrough`. The type names include `int64` and `string`. - ### Rules in the target state #### The stored format stays compatible with Go rc.3 @@ -606,9 +659,17 @@ format version beside it. - **What to build.** EQL types such as `TextEq` and `JsonTreeDocument` as field types of the Rust record derive. In Rust today, an EQL type works only on its own, through the older structural API, with a - `NonEmpty` context. The structural record derive gives each - field a `CallerContext`, and `NonEmpty` has no conversion - from it. So no Rust record can have an EQL field yet. + `NonEmpty` context. The EQL types do derive the structural + `EncryptFrom` and `DecryptInto`. A record still cannot use them as + fields. + A record with a `TextEq` field, built against `next` on 7 November 2026, + fails with three errors. `NonEmpty` and + `ExpectedContext` do not implement `From`, and + the record derive gives each field a `CallerContext`. +- **The smallest fix.** Convert a `CallerContext` of two parts into an + `Identifier`, with the table first and the column second. `Identifier` + already gives the same context as that pair, so the stored bytes do not + change. - **Why.** This gives the [EQL field example](#store-a-field-as-an-eql-type) and the [JSON example](#encrypt-a-json-document). - **Done already.** The dynamic interface supports EQL fields, so Go @@ -621,6 +682,20 @@ format version beside it. the new API; - it writes the same bytes as a Go record with the same declaration. +#### A ready-made EQL resolver + +- **What to build.** A `TargetResolver` in `eql-bindings`, behind its + `targets` feature. It runs the EQL type that a field of a dynamic + declaration names, for encryption, decryption, and queries. +- **Why.** On `next`, every Rust program that uses + `stack_encrypt::dynamic::record` with EQL types writes its own resolver. + The EQL tests and the Go guest each have a copy. +- **Done already.** The functions it calls exist in `eql-bindings`: + `encrypt_with`, `decrypt`, and `query` in `v4::targets` and + `encryption::targets`. +- **Finished when** the tests and the Go guest use the resolver from + `eql-bindings`, and their own copies are deleted. + #### Definitions, field handles, and queries - **What to build.** A shared definition for each record type, built once @@ -631,7 +706,7 @@ format version beside it. caller must not write a second plan by hand. - **Done already.** Queries work through a plan written by hand: `cipher.query(op, value).on(&plan)`. -- **Finished when** the [query example](#query) compiles, and tests show +- **Finished when** the [definition example](#query-through-a-definition) compiles, and tests show that a query cannot run without its definition or its required context. #### Context from fields and from the caller From 128d7f98a6b179d7e3946af72d023ce34d57622e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:10:25 +1100 Subject: [PATCH 13/42] docs: show encrypting straight into JsonTreeDocument first JsonTreeDocument already encrypts a document with the default options (no ordered paths, root sealed) under the column's Identifier. Show that form first, then the named type for custom options. --- .../2026-10-24-stack-encrypt-api-plan.md | 31 ++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index fb23000e1..b5fe6e433 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -228,12 +228,35 @@ and checks each one. #### Encrypt a JSON document A JSON field is stored as one encrypted document in a -`public.eql_v4_json_search` column. Typed Rust options say which paths can -be ordered. +`public.eql_v4_json_search` column. + +The simplest form encrypts straight into `JsonTreeDocument`, with the +default options. The context is the column. The default options order no +paths. They also store the whole document in its root entry, so it +decrypts in one step. + +```rust +use eql_bindings::{v4::json_tree::JsonTreeDocument, Identifier}; +use stack_encrypt::json::JsonPlaintext; + +let column = Identifier::for_column("customers", "profile")?; +let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?; + +// Not on `next` yet: the new traits for EQL types. Today this needs +// `target::legacy::EncryptInto`, a `serde_json::Value`, and an owned +// context. +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, &column) + .await?; +let opened = JsonPlaintext::decrypt_from_with_context(encrypted, &keyset, &column).await?; +``` + +With the default options, no path has an order term. So a query cannot +compare the order of the values at a path. To order some paths, declare a +named type with typed Rust options: ```rust -use eql_bindings::v4::json_tree::JsonTreeDocument; -use stack_encrypt::json::{JsonOptions, JsonPlaintext}; +use stack_encrypt::json::JsonOptions; fn profile_options() -> Result { JsonOptions::new().ordering(["$.age", "$.visits[*].at"]) From 7aaa90b8283517b760135bf7314e68ecdc3e8612 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:17:14 +1100 Subject: [PATCH 14/42] docs: convert serde_json::Value into JsonPlaintext in the plan JsonPlaintext exists on next only inside the dynamic interface. Plan a public stack_encrypt::json::JsonPlaintext with From, a strict parse, and value(). Encryption into a JSON target also accepts a borrowed &serde_json::Value, as for any borrowed value. --- .../2026-10-24-stack-encrypt-api-plan.md | 38 +++++++++++++++++-- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index b5fe6e433..ddc3f4e8e 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -237,20 +237,29 @@ decrypts in one step. ```rust use eql_bindings::{v4::json_tree::JsonTreeDocument, Identifier}; +use serde_json::json; use stack_encrypt::json::JsonPlaintext; let column = Identifier::for_column("customers", "profile")?; -let profile = JsonPlaintext::parse(r#"{"name":"Ada","age":35}"#)?; -// Not on `next` yet: the new traits for EQL types. Today this needs -// `target::legacy::EncryptInto`, a `serde_json::Value`, and an owned -// context. +// Not on `next` yet: the new traits for EQL types, and the public +// `JsonPlaintext`. Today this needs `target::legacy::EncryptInto`, a +// `serde_json::Value`, and an owned context. +let profile: JsonPlaintext = json!({ "name": "Ada", "age": 35 }).into(); let encrypted: JsonTreeDocument = profile .encrypt_into_with_context(&keyset, &column) .await?; + let opened = JsonPlaintext::decrypt_from_with_context(encrypted, &keyset, &column).await?; +let name = &opened.value()["name"]; // `opened` wipes the document when it is dropped ``` +`JsonPlaintext` contains a `serde_json::Value`. It wipes the value when it +is dropped. It converts from a `serde_json::Value`, or parses JSON text with +the strict checks on JSON input. A program can also encrypt a borrowed +`&serde_json::Value`. Then the program is responsible for wiping it, as for +any [borrowed value](#encrypt-borrowed-values). + With the default options, no path has an order term. So a query cannot compare the order of the values at a path. To order some paths, declare a named type with typed Rust options: @@ -795,6 +804,27 @@ parameter named `aad`. - **Finished when** `keyset.encrypt("secret")` compiles, and a test shows that Stack Encrypt wipes its own copies of a borrowed value. +#### A public `JsonPlaintext` + +- **What to build.** A public `stack_encrypt::json::JsonPlaintext`, with: + - `From`; + - `parse(&str)`, with the strict checks on JSON input; + - `value()`, to read the document by reference. + + Encryption into a JSON target accepts a `JsonPlaintext`, or a borrowed + `&serde_json::Value`. Decryption returns a `JsonPlaintext`. +- **Why.** The new traits need a plaintext that can be wiped. + `serde_json::Value` does not implement `Zeroize`. Rust's orphan rule + stops this repository adding that, so a wrapper is needed. A program + that already has a `serde_json::Value` must not have to print it and + parse it again. +- **Done already.** `stack_encrypt::dynamic::json::JsonPlaintext` wraps a + `serde_json::Value`, and wipes it when it is dropped. It is part of the + dynamic interface. It has only `value()`. The strict parser exists as + `stack_encrypt::json::parse_text`. +- **Finished when** the JSON examples in this plan compile, and a test + shows that a dropped `JsonPlaintext` leaves no plaintext behind. + #### Constructors for `IdentifiedBy` - **What to build.** Two constructors that select a keyset: From b66a5d7343543bd80a5d8f856f9dc41cfe620929 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:22:50 +1100 Subject: [PATCH 15/42] docs: show JSON queries in static Rust, and plan typed operands JSON queries do not need the dynamic interface. On next, Rust builds containment, comparisons at a path, and selectors with typed operators over a plan. Show those with their SQL. List the three kinds that exist only in the dynamic module (element match, jsonpath predicates, children), and propose typed operand types that take the column and options from the target type, with a work item and an open question. --- .../2026-10-24-stack-encrypt-api-plan.md | 119 +++++++++++++++++- 1 file changed, 115 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index ddc3f4e8e..40d29371b 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -286,10 +286,103 @@ let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; let opened = JsonPlaintext::decrypt_from(encrypted, &keyset).await?; ``` -A JSON query can test containment, compare the value at a path, match one -element of an array, or run a jsonpath predicate. Each of -these takes an operator and a path. So JSON queries use the [dynamic -interface](#query-through-a-definition). +#### Query a JSON document + +A JSON query does not need the dynamic interface. It needs two things from +the field's declaration: + +- **The column context.** Each path's selector is derived under it, so the + query must use the same context as the stored documents. +- **The options.** They say which paths have order terms, and how far a + wildcard such as `[*]` expands. An order query on a path without order + terms would find no rows, so Stack Encrypt refuses it. + +On `next`, Rust passes the declaration as a plan, and uses the typed +operators in `stack_encrypt::op`: + +```rust +use stack_encrypt::{json::JsonPath, op, NonEmpty}; + +// Containment: documents that contain this document. +let ada = cipher + .query(op::Contains, &json!({ "name": "Ada" })) + .on(&plan) + .await?; + +// A comparison at a path. `$.age` must be an ordered path. +let older = cipher + .query(op::AtPath(op::Gt), &("$.age", 30)) + .on(&plan) + .await?; +let adults = cipher + .query(op::AtPath(op::Between), &("$.age", (18, 65))) + .on(&plan) + .await?; + +// The selector of a path, for `->` in SQL. +let age = keyset + .json_selector(&JsonPath::parse("$.age")?, NonEmpty::new(column.clone())?) + .await?; +``` + +```sql +SELECT id FROM customers WHERE profile @> $1::eql_v4.query_json; + +-- $1 is the selector of `$.age`. $2 and $3 are the bounds. +SELECT id FROM customers +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text); +``` + +The query values convert to the EQL query types: `JsonTreeQuery` for +containment, and `JsonTreeLeafQuery` for a comparison at a path. + +Three kinds of JSON query have no typed Rust API on `next`. They exist +only in the dynamic interface, in `stack_encrypt::dynamic`: + +| Query | What it finds | SQL | +| --- | --- | --- | +| Element match | Rows where one element of an array meets several conditions together | `eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)` | +| Jsonpath predicate | Rows where a SQL/JSON path predicate is true | `profile @@ $1::eql_v4.query_json_predicate` | +| Children and child count | The entries directly below one node, or how many there are | `eql_v4.json_tree_children(profile, $1::text)` | + +Go has typed methods for element match and jsonpath predicates, generated +from the Go type, such as `Fields.Profile.Elements.Visits.Match`. So a +static form of these queries is possible. + +#### Query a JSON document without a plan + +In the target state, a JSON query works like a [scalar +query](#query-a-field). An operand type encrypts into its EQL query type. +The target type of the field supplies the column and the options. So the +program does not write a plan. An order query on a path that the target +does not order is refused. + +```rust +use eql_bindings::v4::json_tree::{JsonTreeElementsQuery, JsonTreeLeafQuery, JsonTreeQuery}; +use stack_encrypt::json::{AtPath, Contains, ElementMatch}; + +// Not on `next` yet: JSON operand types. The names are a proposal. +let ada: JsonTreeQuery = Contains(json!({ "name": "Ada" })) + .encrypt_into_for::(&keyset) + .await?; +let older: JsonTreeLeafQuery = AtPath::gt("$.age", 30) + .encrypt_into_for::(&keyset) + .await?; +let home: JsonTreeElementsQuery = ElementMatch::at("$.visits[*]") + .eq("place", "home") + .gt("at", 15) + .encrypt_into_for::(&keyset) + .await?; +``` + +The names and the way a query names its target are an [open +question](#open-questions-and-decisions-to-be-made). + +The dynamic interface is still needed when the program learns the query +only at run time, such as a jsonpath predicate that a user types. See +[Query through a definition](#query-through-a-definition). #### Encrypt a whole value as one ciphertext @@ -728,6 +821,23 @@ format version beside it. - **Finished when** the tests and the Go guest use the resolver from `eql-bindings`, and their own copies are deleted. +#### Static JSON queries + +- **What to build.** A typed Rust operand type for each kind of JSON + query: containment, comparison at a path, element match, jsonpath + predicate, and reads of children. Each one encrypts into its EQL query + type. The target type of the field supplies the column and the + options. +- **Why.** On `next`, a Rust program needs a plan for any JSON query, and + needs the dynamic interface for element match, predicates, and children. + Go already has typed methods for these. +- **Done already.** Typed containment and comparison at a path, through a + plan (`op::Contains`, `op::AtPath`). The EQL query types exist in + `eql-bindings`. +- **Finished when** the [static JSON query + example](#query-a-json-document-without-a-plan) compiles, and an order + query on a path the target does not order is refused. + #### Definitions, field handles, and queries - **What to build.** A shared definition for each record type, built once @@ -884,6 +994,7 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | +| How should a static JSON query name its target and operator? | Operand types such as `Contains` and `AtPath` that encrypt into the EQL query types for a named target type. Or query methods on the target type, such as `EncryptedProfile::query().at("$.age").gt(30)` | Operand types, to match scalar queries such as `TextEqQuery`. Decide the names in the naming review | | Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | From 249a465f276eeffdfe0e21478470d873698a29b6 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:38:20 +1100 Subject: [PATCH 16/42] docs: drop encrypt_into_for; JSON operands use encrypt_into A JSON query operand keeps the one verb. With default options the caller passes the column as context, as a scalar query does. With custom options the operand is generic over the field's target type, which supplies the column and the options. --- .../2026-10-24-stack-encrypt-api-plan.md | 42 +++++++++++++------ 1 file changed, 29 insertions(+), 13 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 40d29371b..414b9142b 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -354,30 +354,46 @@ static form of these queries is possible. #### Query a JSON document without a plan In the target state, a JSON query works like a [scalar -query](#query-a-field). An operand type encrypts into its EQL query type. -The target type of the field supplies the column and the options. So the -program does not write a plan. An order query on a path that the target -does not order is refused. +query](#query-a-field). An operand type encrypts into its EQL query type +with `encrypt_into`, the same verb as every other value. The program does +not write a plan. + +With the default options, the operand needs only the column. The caller +passes it as the context, as a scalar query does: ```rust -use eql_bindings::v4::json_tree::{JsonTreeElementsQuery, JsonTreeLeafQuery, JsonTreeQuery}; -use stack_encrypt::json::{AtPath, Contains, ElementMatch}; +use eql_bindings::v4::json_tree::{JsonTreeLeafQuery, JsonTreeQuery}; +use stack_encrypt::json::{AtPath, Contains}; // Not on `next` yet: JSON operand types. The names are a proposal. let ada: JsonTreeQuery = Contains(json!({ "name": "Ada" })) - .encrypt_into_for::(&keyset) + .encrypt_into_with_context(&keyset, &column) + .await?; +``` + +With custom options, the operand names the target type of the field. The +target's declaration supplies the column and the options. An order query +on a path that the target does not order is refused. + +```rust +use eql_bindings::v4::json_tree::JsonTreeElementsQuery; +use stack_encrypt::json::ElementMatch; + +// Not on `next` yet: JSON operand types. The names are a proposal. +let ada: JsonTreeQuery = Contains::(json!({ "name": "Ada" })) + .encrypt_into(&keyset) .await?; -let older: JsonTreeLeafQuery = AtPath::gt("$.age", 30) - .encrypt_into_for::(&keyset) +let older: JsonTreeLeafQuery = AtPath::::gt("$.age", 30) + .encrypt_into(&keyset) .await?; -let home: JsonTreeElementsQuery = ElementMatch::at("$.visits[*]") +let home: JsonTreeElementsQuery = ElementMatch::::at("$.visits[*]") .eq("place", "home") .gt("at", 15) - .encrypt_into_for::(&keyset) + .encrypt_into(&keyset) .await?; ``` -The names and the way a query names its target are an [open +The names of the operand types are an [open question](#open-questions-and-decisions-to-be-made). The dynamic interface is still needed when the program learns the query @@ -994,7 +1010,7 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | -| How should a static JSON query name its target and operator? | Operand types such as `Contains` and `AtPath` that encrypt into the EQL query types for a named target type. Or query methods on the target type, such as `EncryptedProfile::query().at("$.age").gt(30)` | Operand types, to match scalar queries such as `TextEqQuery`. Decide the names in the naming review | +| What are the names of the JSON operand types? | `Contains`, `AtPath`, and `ElementMatch` in `stack_encrypt::json`, generic over the target type when the field has custom options. Or names chosen in the naming review | Keep one verb, `encrypt_into`, and put the target type on the operand. Decide the names in the naming review | | Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | From 09820b99ffef37a729808a457a52f241a1de34ec Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:41:42 +1100 Subject: [PATCH 17/42] docs: put the static JSON query first, and the plan form in the dynamic section Passing a plan is the dynamic interface with typed operators, so the plan-based JSON queries move to a "Query through a plan" section there. The static section shows only the operand types, with the SQL and a table of every kind of JSON query. --- .../2026-10-24-stack-encrypt-api-plan.md | 148 ++++++++++-------- 1 file changed, 81 insertions(+), 67 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 414b9142b..c54d7367c 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -288,8 +288,9 @@ let opened = JsonPlaintext::decrypt_from(encrypted, &keyset).await?; #### Query a JSON document -A JSON query does not need the dynamic interface. It needs two things from -the field's declaration: +A JSON query works like a [scalar query](#query-a-field). An operand type +encrypts into its EQL query type with `encrypt_into`, the same verb as +every other value. A query needs two things from the field's declaration: - **The column context.** Each path's selector is derived under it, so the query must use the same context as the stored documents. @@ -297,67 +298,6 @@ the field's declaration: wildcard such as `[*]` expands. An order query on a path without order terms would find no rows, so Stack Encrypt refuses it. -On `next`, Rust passes the declaration as a plan, and uses the typed -operators in `stack_encrypt::op`: - -```rust -use stack_encrypt::{json::JsonPath, op, NonEmpty}; - -// Containment: documents that contain this document. -let ada = cipher - .query(op::Contains, &json!({ "name": "Ada" })) - .on(&plan) - .await?; - -// A comparison at a path. `$.age` must be an ordered path. -let older = cipher - .query(op::AtPath(op::Gt), &("$.age", 30)) - .on(&plan) - .await?; -let adults = cipher - .query(op::AtPath(op::Between), &("$.age", (18, 65))) - .on(&plan) - .await?; - -// The selector of a path, for `->` in SQL. -let age = keyset - .json_selector(&JsonPath::parse("$.age")?, NonEmpty::new(column.clone())?) - .await?; -``` - -```sql -SELECT id FROM customers WHERE profile @> $1::eql_v4.query_json; - --- $1 is the selector of `$.age`. $2 and $3 are the bounds. -SELECT id FROM customers -WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf - AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf -ORDER BY eql_v4.ord_term(profile -> $1::text); -``` - -The query values convert to the EQL query types: `JsonTreeQuery` for -containment, and `JsonTreeLeafQuery` for a comparison at a path. - -Three kinds of JSON query have no typed Rust API on `next`. They exist -only in the dynamic interface, in `stack_encrypt::dynamic`: - -| Query | What it finds | SQL | -| --- | --- | --- | -| Element match | Rows where one element of an array meets several conditions together | `eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements)` | -| Jsonpath predicate | Rows where a SQL/JSON path predicate is true | `profile @@ $1::eql_v4.query_json_predicate` | -| Children and child count | The entries directly below one node, or how many there are | `eql_v4.json_tree_children(profile, $1::text)` | - -Go has typed methods for element match and jsonpath predicates, generated -from the Go type, such as `Fields.Profile.Elements.Visits.Match`. So a -static form of these queries is possible. - -#### Query a JSON document without a plan - -In the target state, a JSON query works like a [scalar -query](#query-a-field). An operand type encrypts into its EQL query type -with `encrypt_into`, the same verb as every other value. The program does -not write a plan. - With the default options, the operand needs only the column. The caller passes it as the context, as a scalar query does: @@ -386,6 +326,9 @@ let ada: JsonTreeQuery = Contains::(json!({ "name": "Ada" })) let older: JsonTreeLeafQuery = AtPath::::gt("$.age", 30) .encrypt_into(&keyset) .await?; +let adults: JsonTreeLeafQuery = AtPath::::between("$.age", 18, 65) + .encrypt_into(&keyset) + .await?; let home: JsonTreeElementsQuery = ElementMatch::::at("$.visits[*]") .eq("place", "home") .gt("at", 15) @@ -393,12 +336,37 @@ let home: JsonTreeElementsQuery = ElementMatch::::at("$.visits .await?; ``` +```sql +SELECT id FROM customers WHERE profile @> $1::eql_v4.query_json; + +-- $1 is the selector of `$.age`. $2 and $3 are the bounds. +SELECT id FROM customers +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text); + +SELECT id FROM customers +WHERE eql_v4.json_tree_element_match(profile, $1::eql_v4.query_json_elements); +``` + +The kinds of JSON query, and what each one finds: + +| Query | What it finds | EQL query type | +| --- | --- | --- | +| Containment | Rows whose document contains the given document, or is contained by it | `JsonTreeQuery` | +| Comparison at a path | Rows where the value at an ordered path equals, orders against, or lies between the given values | `JsonTreeLeafQuery` | +| Element match | Rows where one element of an array meets several conditions together | `JsonTreeElementsQuery` | +| Jsonpath predicate | Rows where a SQL/JSON path predicate is true | `JsonTreePredicateQuery` | +| Children and child count | The entries directly below one node, or how many there are | A node token | + The names of the operand types are an [open question](#open-questions-and-decisions-to-be-made). -The dynamic interface is still needed when the program learns the query -only at run time, such as a jsonpath predicate that a user types. See -[Query through a definition](#query-through-a-definition). +None of the operand types exists on `next`. Today a Rust program queries +a JSON document [through a plan](#query-through-a-plan). A plan is part +of the dynamic interface. The dynamic interface stays the right tool for a +query the program learns only at run time, such as a jsonpath predicate +that a user types. #### Encrypt a whole value as one ciphertext @@ -470,6 +438,52 @@ The output keys and the type names are part of the stored format, so they cannot change. The output keys are `c`, `eq`, `match`, `ore`, `ope`, and `passthrough`. The type names include `int64` and `string`. +#### Query through a plan + +On `next`, a Rust program queries a JSON document through a plan. The plan +is the declaration as data, so this is the dynamic interface with typed +operators. `stack_encrypt::op` names the operator. `on` names the plan, and the plan +supplies the column and the options: + +```rust +use stack_encrypt::{json::JsonPath, op, NonEmpty}; + +// Containment: documents that contain this document. +let ada = cipher + .query(op::Contains, &json!({ "name": "Ada" })) + .on(&plan) + .await?; + +// A comparison at a path. `$.age` must be an ordered path. +let older = cipher + .query(op::AtPath(op::Gt), &("$.age", 30)) + .on(&plan) + .await?; +let adults = cipher + .query(op::AtPath(op::Between), &("$.age", (18, 65))) + .on(&plan) + .await?; + +// The selector of a path, for `->` in SQL. +let age = keyset + .json_selector(&JsonPath::parse("$.age")?, NonEmpty::new(column.clone())?) + .await?; +``` + +The query values convert to the EQL query types: `JsonTreeQuery` for +containment, and `JsonTreeLeafQuery` for a comparison at a path. + +Three kinds of JSON query have no typed operator on `next`. They exist +only as data, through `stack_encrypt::dynamic::query_term`, with the +operator named as text: + +- element match: `"element_match"`; +- jsonpath predicates: `"match"`; +- children and child count: `"children"` and `"child_count"`. + +Go has generated methods for the first two, such as +`Fields.Profile.Elements.Visits.Match`. + #### Query through a definition In Rust, `definition()` returns the declaration of a derived record, built @@ -851,7 +865,7 @@ format version beside it. plan (`op::Contains`, `op::AtPath`). The EQL query types exist in `eql-bindings`. - **Finished when** the [static JSON query - example](#query-a-json-document-without-a-plan) compiles, and an order + example](#query-a-json-document) compiles, and an order query on a path the target does not order is refused. #### Definitions, field handles, and queries From d5fa385638ba446dffbdab5b2b578dcb496b37ce Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:41:56 +1100 Subject: [PATCH 18/42] docs: shorten a sentence in the JSON query section --- docs/plans/2026-10-24-stack-encrypt-api-plan.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index c54d7367c..7eb4cc8b9 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -364,9 +364,9 @@ question](#open-questions-and-decisions-to-be-made). None of the operand types exists on `next`. Today a Rust program queries a JSON document [through a plan](#query-through-a-plan). A plan is part -of the dynamic interface. The dynamic interface stays the right tool for a -query the program learns only at run time, such as a jsonpath predicate -that a user types. +of the dynamic interface. The dynamic interface is still the right tool for a +query known only at run time, such as a jsonpath predicate that a user +types. #### Encrypt a whole value as one ciphertext From 314efb8e747141d4d246a47bb751e19259d063f9 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:50:06 +1100 Subject: [PATCH 19/42] docs: drop the Serde derives from the plan's examples The examples only encrypt and decrypt, which needs no Serde. Say once that a program adds Serialize and Deserialize when it stores the value as JSON. --- docs/plans/2026-10-24-stack-encrypt-api-plan.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 7eb4cc8b9..372b14961 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -118,7 +118,7 @@ struct User { } // A searchable email: its ciphertext and an equality term. -#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[derive(EncryptTarget, DecryptSource)] #[stash(plaintext = String)] struct SearchableEmail { ciphertext: Ciphertext, @@ -126,7 +126,7 @@ struct SearchableEmail { } // Not on `next` yet: the record derive and `passthrough`. -#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[derive(EncryptTarget, DecryptSource)] #[stash(struct = User, context = "users")] struct EncryptedUser { #[stash(passthrough)] @@ -141,6 +141,10 @@ let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; let user = User::decrypt_from(encrypted, &cipher).await?; ``` +The derives need no Serde. A program that stores an encrypted value as +JSON adds `Serialize` and `Deserialize` to the encrypted type. That is +the same as for any other type. + To tie every field to one row, the caller supplies the row ID as a context extension, and supplies it again to decrypt. The extension also changes the search terms, so a query must supply the same row ID. How to bind a @@ -176,7 +180,7 @@ struct Member { // Not on `next` yet in Rust: EQL types as fields of a derived record. // Go records do this today through the dynamic interface. -#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[derive(EncryptTarget, DecryptSource)] #[stash(struct = Member, context = "members")] struct EncryptedMember { #[stash(passthrough)] @@ -272,8 +276,7 @@ fn profile_options() -> Result { } // Not on `next` yet: a JSON target with typed options. -#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] -#[serde(transparent)] +#[derive(EncryptTarget, DecryptSource)] #[stash( plaintext = JsonPlaintext, transparent, From 8db9ee7069f32e51f0f3481ab1dc836dcc4741f0 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 13:57:08 +1100 Subject: [PATCH 20/42] docs: JSON operand types are the dynamic interface's input types The dynamic module already has one private input type per JSON operator, which its parser reads a binding's JSON object into. The static form makes those types public with constructors, so one type serves both forms and both share one set of input rules. --- .../2026-10-24-stack-encrypt-api-plan.md | 38 +++++++++++++------ 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 372b14961..3b29ce5a6 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -365,8 +365,20 @@ The kinds of JSON query, and what each one finds: The names of the operand types are an [open question](#open-questions-and-decisions-to-be-made). -None of the operand types exists on `next`. Today a Rust program queries -a JSON document [through a plan](#query-through-a-plan). A plan is part +The operand types are not new. The dynamic interface already has one +private input type for each operator, in `stack_encrypt::dynamic`: + +- `DocumentInput` for containment; +- `AtInput` for a comparison at a path; +- `ElementsInput` for element match. + +The dynamic parser reads the JSON object a binding sends into one of +them. The engine then derives the operand from it. The static form makes +those types public, with constructors. The parser reads into the same +types, so both forms share one set of input rules. + +The public forms do not exist on `next`. Today a Rust program queries a +JSON document [through a plan](#query-through-a-plan). A plan is part of the dynamic interface. The dynamic interface is still the right tool for a query known only at run time, such as a jsonpath predicate that a user types. @@ -856,17 +868,21 @@ format version beside it. #### Static JSON queries -- **What to build.** A typed Rust operand type for each kind of JSON - query: containment, comparison at a path, element match, jsonpath - predicate, and reads of children. Each one encrypts into its EQL query - type. The target type of the field supplies the column and the - options. +- **What to build.** Make the dynamic interface's input types public. + There is one for each kind of JSON query: containment, comparison at a + path, element match, jsonpath predicate, and reads of children. Give + each one + constructors, and an `EncryptTarget` implementation that encrypts it + into its EQL query type. The target type of the field supplies the + column and the options. The dynamic parser keeps reading into the same + types, so there is one definition of each input. - **Why.** On `next`, a Rust program needs a plan for any JSON query, and needs the dynamic interface for element match, predicates, and children. Go already has typed methods for these. -- **Done already.** Typed containment and comparison at a path, through a - plan (`op::Contains`, `op::AtPath`). The EQL query types exist in - `eql-bindings`. +- **Done already.** The input types and their parsing rules, as private + types in `stack_encrypt::dynamic::json_query`. Typed containment and + comparison at a path, through a plan (`op::Contains`, `op::AtPath`). The + EQL query types exist in `eql-bindings`. - **Finished when** the [static JSON query example](#query-a-json-document) compiles, and an order query on a path the target does not order is refused. @@ -1027,7 +1043,7 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | -| What are the names of the JSON operand types? | `Contains`, `AtPath`, and `ElementMatch` in `stack_encrypt::json`, generic over the target type when the field has custom options. Or names chosen in the naming review | Keep one verb, `encrypt_into`, and put the target type on the operand. Decide the names in the naming review | +| What are the public names of the JSON operand types? | `Contains`, `AtPath`, and `ElementMatch` in `stack_encrypt::json`, generic over the target type when the field has custom options. Or the dynamic module's current names, such as `ElementsInput` | Keep one verb, `encrypt_into`, and put the target type on the operand. One type serves the static form and the dynamic parser. Decide the names in the naming review | | Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | From 04d1498b597b7a9a690588b2354a63e13c1b6cd8 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:01:47 +1100 Subject: [PATCH 21/42] docs: settle the JSON operand names Contains, AtPath, and ElementMatch are agreed, with Predicate and Children for the remaining two kinds. Remove the open question. --- .../2026-10-24-stack-encrypt-api-plan.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 3b29ce5a6..aa9b28535 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -308,7 +308,7 @@ passes it as the context, as a scalar query does: use eql_bindings::v4::json_tree::{JsonTreeLeafQuery, JsonTreeQuery}; use stack_encrypt::json::{AtPath, Contains}; -// Not on `next` yet: JSON operand types. The names are a proposal. +// Not on `next` yet: the public JSON operand types. let ada: JsonTreeQuery = Contains(json!({ "name": "Ada" })) .encrypt_into_with_context(&keyset, &column) .await?; @@ -322,7 +322,7 @@ on a path that the target does not order is refused. use eql_bindings::v4::json_tree::JsonTreeElementsQuery; use stack_encrypt::json::ElementMatch; -// Not on `next` yet: JSON operand types. The names are a proposal. +// Not on `next` yet: the public JSON operand types. let ada: JsonTreeQuery = Contains::(json!({ "name": "Ada" })) .encrypt_into(&keyset) .await?; @@ -362,8 +362,9 @@ The kinds of JSON query, and what each one finds: | Jsonpath predicate | Rows where a SQL/JSON path predicate is true | `JsonTreePredicateQuery` | | Children and child count | The entries directly below one node, or how many there are | A node token | -The names of the operand types are an [open -question](#open-questions-and-decisions-to-be-made). +The operand types are named `Contains`, `AtPath`, and `ElementMatch`, +with `Predicate` for a jsonpath predicate and `Children` for the entries +below a node. The user agreed these names on 7 November 2026. The operand types are not new. The dynamic interface already has one private input type for each operator, in `stack_encrypt::dynamic`: @@ -871,11 +872,12 @@ format version beside it. - **What to build.** Make the dynamic interface's input types public. There is one for each kind of JSON query: containment, comparison at a path, element match, jsonpath predicate, and reads of children. Give - each one - constructors, and an `EncryptTarget` implementation that encrypts it - into its EQL query type. The target type of the field supplies the - column and the options. The dynamic parser keeps reading into the same - types, so there is one definition of each input. + each one constructors, and an `EncryptTarget` implementation that + encrypts it into its EQL query type. The target type of the field + supplies the column and the options. The dynamic parser keeps reading + into the same types, so there is one definition of each input. +- **Names.** `Contains`, `AtPath`, `ElementMatch`, `Predicate`, and + `Children`, in `stack_encrypt::json`. Agreed on 7 November 2026. - **Why.** On `next`, a Rust program needs a plan for any JSON query, and needs the dynamic interface for element match, predicates, and children. Go already has typed methods for these. @@ -1043,7 +1045,6 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | -| What are the public names of the JSON operand types? | `Contains`, `AtPath`, and `ElementMatch` in `stack_encrypt::json`, generic over the target type when the field has custom options. Or the dynamic module's current names, such as `ElementsInput` | Keep one verb, `encrypt_into`, and put the target type on the operand. One type serves the static form and the dynamic parser. Decide the names in the naming review | | Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | From da63a38a298ecf0a7949b469b4e4fd1cfed69caa Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:08:05 +1100 Subject: [PATCH 22/42] docs: a byte string encrypts as one value Record the decision of 7 November 2026: Vec and [u8; N] encrypt into one Ciphertext, as #1501 built, and a Vec of byte strings is a collection of ciphertexts with no wrapper. Add the example, and remove the open question and the stored-format constraint it raised. --- .../2026-10-24-stack-encrypt-api-plan.md | 29 +++++++++++++++++-- .../2026-10-24-stack-encrypt-api-review.md | 5 ++-- ...026-10-24-stack-encrypt-public-examples.md | 14 ++++----- 3 files changed, 36 insertions(+), 12 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index aa9b28535..c9d18e628 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -416,6 +416,33 @@ let values = Vec::::decrypt_from(column, &keyset).await?; let rows: Vec = users.encrypt_into(&keyset).await?; ``` +#### Encrypt a byte string + +A `Vec` is the exception to the collection rule. It encrypts as one +value, into one `Ciphertext`, because a byte string is almost always one +thing: a file, a key, or an image. A fixed-size array `[u8; N]` is a byte +string too. `u8` on its own is not a value that encrypts, so a `Vec` +cannot encrypt byte by byte. Go treats a `[]byte` field the same way, so +the stored format is the same in both languages. + +A `Vec` of byte strings follows the collection rule. It encrypts into a +`Vec`, one for each byte string, with no wrapper type. + +```rust +let file: Ciphertext = std::fs::read("photo.jpg")?.encrypt_into(&keyset).await?; +let photo = Vec::::decrypt_from(file, &keyset).await?; + +let key: Ciphertext = [7u8; 16].encrypt_into(&keyset).await?; + +let files: Vec = vec![vec![1u8, 2, 3], vec![], vec![9]] + .encrypt_into(&keyset) + .await?; +let opened = Vec::>::decrypt_from(files, &keyset).await?; +``` + +The user agreed this behaviour on 7 November 2026. It is what pull +request #1501 built. + ### The dynamic interface The dynamic interface takes a declaration and values as data, not as Rust @@ -777,7 +804,6 @@ format version beside it. | Separate row binding and search scope | The AAD, and the context that search terms are derived under | | Text encodings as field targets (Base64 and Base85) | How a ciphertext is written as text in a column | | The `SealedValue` envelope and the JSON header version | The offsets of the key tag, and header version 1, which rc.3 writes | -| Byte vectors | Whether a byte string is one value, or one value for each byte | | Configurable match options | The terms of columns written with the rc.3 defaults | #### Opaque blobs @@ -1045,7 +1071,6 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | -| Should a `Vec` encrypt as one value, or as one value for each byte? | One value, as #1501 built it. Or one value for each byte, with `OpaqueBlob>` for a whole vector | First check whether Go rc.3 stores a byte string as one value. If it does, keep one value | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index df80be955..f01f34f7a 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1741,8 +1741,9 @@ description was not enough. It ran no test suite. #### Differences from the plan -#1501 encrypts `Vec` as one value. The plan recommends encrypting it -byte by byte. The API plan records this as a question for the user. +#1501 encrypts `Vec` as one value. The plan had recommended encrypting +it byte by byte. On 7 November 2026, the user agreed with #1501: a byte +string is one value, and a `Vec` of byte strings is a collection. #### What the reconciliation of 27 October 2026 checked diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index bcbe02739..15d708f87 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -137,13 +137,11 @@ for example: `keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` produces one ciphertext for the encoded collection instead. -The latest recommendation applies the same rule to byte vectors. No wrapper -or special case is needed to keep the prototype's behaviour. Pull request -#1501 differs: it encrypts `Vec` and `[u8; N]` as one value. The [API -plan](2026-10-24-stack-encrypt-api-plan.md#open-questions-and-decisions-to-be-made) -records this as a question for the user. These calls -need the proposed opaque ciphertext and support for targets that take no -context. +A byte string is the one exception. `Vec` and `[u8; N]` encrypt as one +value, as pull request #1501 built them. The user agreed this on 7 November +2026. The [API plan](2026-10-24-stack-encrypt-api-plan.md#encrypt-a-byte-string) +shows it. These calls need the proposed opaque ciphertext and support for +targets that take no context. ### Status of each API surface @@ -159,7 +157,7 @@ context. | `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the canonical sealed-leaf encoding | Done (#1404). The envelope is unchanged | | `Ciphertext: Serialize + Deserialize` | Proposed: Base64 for human-readable formats, bytes for binary formats | Done (#1404) | | Encoded ciphertext field targets | Accepted; final names remain for the naming pass | Still open | -| `Ciphertext` as an encryption or decryption target | Proposed single-value target | Done (#1404). `Vec` is one value, unlike the recommendation (#1501) | +| `Ciphertext` as an encryption or decryption target | Proposed single-value target | Done (#1404). `Vec` is one value (#1501), agreed on 7 November 2026 | | Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path | Still open. In progress in #1503 | | Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair | Done for one value (#1404) | | `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair | Done for one value and `Vec` (#1404, #1501). Records are in #1504 | From 72ff215caee9b33eee4e21e3f4ed05490f8e45ad Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:21:26 +1100 Subject: [PATCH 23/42] docs: a Go byte string takes the same engine path as a Rust one The dynamic interface and the Rust API share one engine, so they write the same bytes for the same declaration. Say so instead of implying the two agree by chance. --- docs/plans/2026-10-24-stack-encrypt-api-plan.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index c9d18e628..b6be1870f 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -422,8 +422,16 @@ A `Vec` is the exception to the collection rule. It encrypts as one value, into one `Ciphertext`, because a byte string is almost always one thing: a file, a key, or an image. A fixed-size array `[u8; N]` is a byte string too. `u8` on its own is not a value that encrypts, so a `Vec` -cannot encrypt byte by byte. Go treats a `[]byte` field the same way, so -the stored format is the same in both languages. +cannot encrypt byte by byte. + +A Go `[]byte` field takes the same path. The Go SDK sends the bytes to +the engine as one bytes value. The engine seals them with the same +Vitamin C code that seals a Rust `Vec`. The dynamic interface and the +Rust API share one engine, so they write the same bytes for the same +declaration (ADR-0007). + +The byte string is not a special case of either language. It is a rule of +the engine. A `Vec` of byte strings follows the collection rule. It encrypts into a `Vec`, one for each byte string, with no wrapper type. From 6c7bb9d5db8ce89d7a780aa7f3adb465229c19e9 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:25:03 +1100 Subject: [PATCH 24/42] docs: plan an opt-in wrapper that authenticates a collection's order The default collection does not authenticate order or length. Add Positioned>, which seals each element under its position and the length, as ADR-0010 foresaw: opt-in, by type. With an example, a work item, and the name as an open question. --- .../2026-10-24-stack-encrypt-api-plan.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index b6be1870f..1576e0d45 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -416,6 +416,26 @@ let values = Vec::::decrypt_from(column, &keyset).await?; let rows: Vec = users.encrypt_into(&keyset).await?; ``` +That is the default, because it is the common case. A caller who needs +the order and the length authenticated asks for it by type. This is the +same way that `OpaqueBlob` changes how a value is sealed. The wrapper seals each +element under its position and the length of the `Vec`, as part of the +context. An element that is swapped, dropped, or added then fails to open. + +```rust +use stack_encrypt::Positioned; + +// Not on `next` yet: `Positioned`. The name is a proposal. +let steps: Positioned> = Positioned(vec![1_i32, 3, 5]) + .encrypt_into(&keyset) + .await?; +let values = Positioned::>::decrypt_from(steps, &keyset).await?; +``` + +`Positioned` is a new stored form, so it does not change the data that +Go rc.3 writes. ADR-0010 records that stricter authentication of a +container is opt-in, by type. + #### Encrypt a byte string A `Vec` is the exception to the collection rule. It encrypts as one @@ -1041,6 +1061,20 @@ parameter named `aad`. - **Finished when** a released `zerokms-protocol` has both constructors, and the examples in this plan compile. +#### Authenticate the order and length of a collection + +- **What to build.** A wrapper type, `Positioned>`, that seals each + element under its position and the length of the collection. Opening + checks both, so a swapped, dropped, or added element fails. The dynamic + interface needs the same form, so that Go can declare it. +- **Why.** The default collection does not authenticate order or length, + because a database returns rows in any order. Some callers store a + sequence whose order matters, such as the steps of a workflow. ADR-0010 + says this is opt-in, by type. It does not exist yet. +- **Done already.** Nothing. The default collection (#1501). +- **Finished when** a test shows that each of swapping, dropping, and + adding an element fails to open, in Rust and in Go. + #### Remove the structural API - **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and @@ -1079,6 +1113,7 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | +| What is the name of the wrapper that authenticates a collection's order and length? | `Positioned>`. Or a name chosen in the naming review | `Positioned`. Decide in the naming review | | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | From 40a3b68df7a9a556db3b30324bc9633d6be2f311 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:30:33 +1100 Subject: [PATCH 25/42] docs: show encrypting a Vec byte by byte through a named type A newtype for one byte is a single value, so a Vec of it is an ordinary collection and encrypts into one Ciphertext per element. Add the example, and a work item for a derive that gives a newtype the three implementations a single value needs. --- .../2026-10-24-stack-encrypt-api-plan.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 1576e0d45..ed723c4e3 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -468,6 +468,26 @@ let files: Vec = vec![vec![1u8, 2, 3], vec![], vec![9]] let opened = Vec::>::decrypt_from(files, &keyset).await?; ``` +To encrypt each byte of a `Vec` as its own ciphertext, name a type +for one byte. A `Vec` of that type is an ordinary collection, so it +encrypts into one `Ciphertext` for each element. The named type is a +single value because its borrow implements Vitamin C's `Scalar`. That is +the same rule every built-in single value follows. + +```rust +use zeroize::Zeroize; + +/// One byte, sealed on its own. +#[derive(Zeroize)] +struct Byte(u8); + +// Not on `next` yet: a derive for a single-value type. Today `Byte` needs +// `Encrypt` and `Scalar` on `&Byte`, and `FromScalar` on `Byte`, by hand. +let bytes: Vec = vec![0u8, 1, 255].into_iter().map(Byte).collect(); +let each: Vec = bytes.encrypt_into(&keyset).await?; +let opened = Vec::::decrypt_from(each, &keyset).await?; +``` + The user agreed this behaviour on 7 November 2026. It is what pull request #1501 built. @@ -1075,6 +1095,23 @@ parameter named `aad`. - **Finished when** a test shows that each of swapping, dropping, and adding an element fails to open, in Rust and in Go. +#### A derive for a single-value type + +- **What to build.** A derive, or a `#[stash(scalar)]` attribute, for a + newtype around one built-in single value, such as `struct Byte(u8)` or + `struct Token(String)`. It gives the type what `OpaqueEncode` and + `OpaqueDecode` need: `Encrypt` and `Scalar` on its borrow, and + `FromScalar` on the type. +- **Why.** A newtype is how a caller changes the rule for a value, such as + encrypting a `Vec` byte by byte. On `next`, the three + implementations are written by hand. `FromScalar` is not documented for + that use. +- **Done already.** `OpaqueEncode` and `OpaqueDecode` already accept any + type that meets those bounds. Vitamin C documents a hand-written + `Blob(Vec)` newtype. +- **Finished when** the [byte example](#encrypt-a-byte-string) compiles + with the derive and no hand-written implementation. + #### Remove the structural API - **What to build.** Remove `EncryptFrom`, `DecryptInto`, `encrypt_as`, and From 8701a254f076527782295b586c9f8b0a664a44f5 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 14:34:57 +1100 Subject: [PATCH 26/42] docs: show #[derive(Scalar)] for a single-value newtype The target interface is a derive on the newtype, beside Zeroize. It forwards the three implementations a single value needs to the one field. --- .../plans/2026-10-24-stack-encrypt-api-plan.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index ed723c4e3..f810f97de 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -475,19 +475,25 @@ single value because its borrow implements Vitamin C's `Scalar`. That is the same rule every built-in single value follows. ```rust +use stack_encrypt::Scalar; use zeroize::Zeroize; /// One byte, sealed on its own. -#[derive(Zeroize)] +// Not on `next` yet: `#[derive(Scalar)]`. Today `Byte` needs `Encrypt` +// and `Scalar` on `&Byte`, and `FromScalar` on `Byte`, by hand. +#[derive(Zeroize, Scalar)] struct Byte(u8); -// Not on `next` yet: a derive for a single-value type. Today `Byte` needs -// `Encrypt` and `Scalar` on `&Byte`, and `FromScalar` on `Byte`, by hand. let bytes: Vec = vec![0u8, 1, 255].into_iter().map(Byte).collect(); let each: Vec = bytes.encrypt_into(&keyset).await?; let opened = Vec::::decrypt_from(each, &keyset).await?; ``` +`#[derive(Scalar)]` works on a struct with one field, whose type is a +built-in single value. The derive encrypts the struct as that field, and +decrypts the field back into the struct. So `Byte` is sealed exactly as +the `u8` inside it would be, if `u8` were a single value on its own. + The user agreed this behaviour on 7 November 2026. It is what pull request #1501 built. @@ -1097,11 +1103,11 @@ parameter named `aad`. #### A derive for a single-value type -- **What to build.** A derive, or a `#[stash(scalar)]` attribute, for a - newtype around one built-in single value, such as `struct Byte(u8)` or +- **What to build.** `#[derive(Scalar)]`, for a struct with one field + whose type is a built-in single value, such as `struct Byte(u8)` or `struct Token(String)`. It gives the type what `OpaqueEncode` and `OpaqueDecode` need: `Encrypt` and `Scalar` on its borrow, and - `FromScalar` on the type. + `FromScalar` on the type. Each one forwards to the field. - **Why.** A newtype is how a caller changes the rule for a value, such as encrypting a `Vec` byte by byte. On `next`, the three implementations are written by hand. `FromScalar` is not documented for From de3a40e58a92f75f46fdba36ec46f553c520c0b2 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 15:51:09 +1100 Subject: [PATCH 27/42] docs: one context, two scopes Record the decision of 7 November 2026 from the review of the context artifact. The stored context is every part; the query scope is a prefix of it. The key and the AAD derive under the stored context, and search terms under the query scope. A part is scoped, binding, or post-seal. Access intelligence is the use case. The API for marking a binding part is open, pending the rest of the context review. --- .../2026-10-24-stack-encrypt-api-plan.md | 72 +++++++++++++++++-- 1 file changed, 65 insertions(+), 7 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index f810f97de..7e9a9d863 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -32,7 +32,9 @@ log](2026-10-24-stack-encrypt-api-review.md). | Keyset | A set of keys in ZeroKMS. Every data key belongs to one keyset. A tenant can have its own keyset | | Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | | Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | +| Query scope | The prefix of the stored context that search terms derive under. A query supplies it. See [Two scopes of one context](#two-scopes-of-one-context) | | Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | +| Stored context | Every part of a value's context. The data key and the AAD derive under it | | Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | | Target | The encrypted type that a value is stored as, such as `Ciphertext`, an EQL type like `TextEq`, or a record type | @@ -145,11 +147,15 @@ The derives need no Serde. A program that stores an encrypted value as JSON adds `Serialize` and `Deserialize` to the encrypted type. That is the same as for any other type. -To tie every field to one row, the caller supplies the row ID as a context -extension, and supplies it again to decrypt. The extension also changes the -search terms, so a query must supply the same row ID. How to bind a -ciphertext to its row while its search terms cover the whole column is an -[open question](#open-questions-and-decisions-to-be-made). +To tie every field to one row, the caller supplies the row ID as a part of +the context, and supplies it again to decrypt. The row ID is a binding +part. It is in the stored context, so the key and the AAD depend on it. It +is not in the query scope, so a query still finds the row across the +column. + +[Two scopes of one context](#two-scopes-of-one-context) gives the rule. +The API for declaring a binding part is an [open +question](#open-questions-and-decisions-to-be-made). ```rust let encrypted: EncryptedUser = user @@ -747,6 +753,41 @@ Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` describes the descriptor and the AAD. The module documentation of `packages/stack-encrypt/src/target/context.rs` describes the search terms. +#### Two scopes of one context + +Decided on 7 November 2026. A context has two scopes: + +- **The stored context** is every part. The data key and the AAD derive + under it. +- **The query scope** is a prefix of the stored context. Search terms + derive under it. A query supplies the query scope, and matches every row + stored under it. + +A part of the context is one of three kinds. The declaration says which, +and fixes the order: the query scope first, then the binding parts. + +| Kind of part | Example | In the query scope | In the descriptor | In the AAD | Verified by | +| --- | --- | --- | --- | --- | --- | +| Scoped | The label and the field, or a tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | +| Binding | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | +| Post-seal | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | + +A part is scoped unless the declaration marks it, because that is what Go +rc.3 writes. A binding part can be stored in the row as a context field, +so the reader takes it from the row. ZeroKMS logs the descriptor at every +key retrieval, so a binding part such as the writer's IP address is in the +access log. The descriptor has a length limit. It is also sent to ZeroKMS, so the +part leaves the customer's system. + +A post-seal part is added after sealing, by a database trigger that +refines the stored tag. ZeroKMS verifies the refined tag and returns the +original. The AAD binds the original. The design allows for this kind. The work +depends on ZeroKMS, and is the [refined tags](#changes-that-would-alter-the-stored-format) +item. + +The use case is access intelligence. Each value is bound to its row by a +client-generated UUID. A query by column or by tenant still finds it. + #### Other rules - **Rust and Go behave the same.** Every built-in capability has the same @@ -855,7 +896,7 @@ format version beside it. | Planned change | What it would alter | | --- | --- | | Refined tags | The tag stored in the envelope, and what the AAD binds | -| Separate row binding and search scope | The AAD, and the context that search terms are derived under | +| Search terms under the query scope | A new form of search term, beside the rc.3 form, where terms derive under every part | | Text encodings as field targets (Base64 and Base85) | How a ciphertext is written as text in a column | | The `SealedValue` envelope and the JSON header version | The offsets of the key tag, and header version 1, which rc.3 writes | | Configurable match options | The terms of columns written with the rc.3 defaults | @@ -999,6 +1040,23 @@ format version beside it. `ContextExtension` and `ExpectedContext`, and is in review. - **Finished when** all nine scenarios run as tests in Rust and Go. +#### Two scopes of one context + +- **What to build.** The binding kind of part, in the declaration and in + the dynamic grammar. Search terms derive under the query scope, and the + key and the AAD under the stored context. A binding part can be a context + field stored in the row. The order is fixed: the query scope, then the + binding parts. +- **Why.** Access intelligence needs each value bound to its row, and found + by a column or tenant query. [Two scopes of one + context](#two-scopes-of-one-context) has the rule. +- **Done already.** Nothing. Today every part is in both scopes. +- **Finished when** these are all true, in Rust and in Go: + - a record with a binding row ID encrypts; + - a column query finds it; + - it fails to open under another row ID; + - the stored bytes of a record with no binding part are unchanged. + #### Name context parameters `context` These public methods in `packages/stack-encrypt/src/cipher.rs` take a @@ -1160,7 +1218,7 @@ The lead who approves this plan decides each question. | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | -| How should a ciphertext be bound to its row while its search terms cover a wider scope? | Separate row binding from search scope in the declaration, for example with `PrimaryKey` and `ScopeKey` roles. Or keep one context | Separate them. Never add a row ID to the search scope automatically | +| How does a declaration mark a binding part of the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences two to seven of the context artifact | | Should keysets be chosen from data, and should Stack Encrypt use ZeroKMS lock context? | Optional capabilities of a key provider. Or leave them out | Optional capabilities, after the customer release | | What privacy claim can be made for match terms of short words? | Accept what they reveal, and document it. Or change how short words are stored | No recommendation yet. Merging #1397 did not show that they reveal nothing new | | Which crate should own `Base64Text` and `Base85Text`? | A small Vitamin C encoding crate. Or keep them in `eql-bindings` | A small Vitamin C crate, re-exported by Stack Encrypt and EQL | From 8a53d3ccdf20a7a01c6d8ee099f43db23a0c4236 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 15:54:32 +1100 Subject: [PATCH 28/42] docs: refine and bind are the context verbs Record the vocabulary decided on 7 November 2026: a refinement narrows the data and the queries, a binding part ties a ciphertext to its row. "Extend" is retired: Go Extend becomes Refine with a deprecated alias until 0.1.0, and Bind is added. The frozen vectors do not change. --- .../2026-10-24-stack-encrypt-api-plan.md | 43 ++++++++++++++----- 1 file changed, 32 insertions(+), 11 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 7e9a9d863..ea12131ac 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -23,6 +23,7 @@ log](2026-10-24-stack-encrypt-api-review.md). | Term | Meaning | | --- | --- | | AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | +| Binding part | A part of the context that ties a ciphertext to its row, such as a row ID, and is not in the query scope. Added with `bind` | | Ciphertext | One encrypted value, as the Rust `Ciphertext` type. It is stored as bytes, or as Base64 text | | Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Rust and Go run the same cases. The files are in `conformance/stack-encrypt/v1/` | | Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | @@ -33,6 +34,7 @@ log](2026-10-24-stack-encrypt-api-review.md). | Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | | Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | | Query scope | The prefix of the stored context that search terms derive under. A query supplies it. See [Two scopes of one context](#two-scopes-of-one-context) | +| Refinement | A part of the context that narrows the data and the queries over it, such as a tenant ID. Added with `refine`. See [Two scopes of one context](#two-scopes-of-one-context) | | Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | | Stored context | Every part of a value's context. The data key and the AAD derive under it | | Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | @@ -667,15 +669,17 @@ type Settings struct { } ``` -A `Cipher` is the client bound to one keyset. `Extend` adds a part to the -context of every field, such as a tenant ID. The generated `Encrypt` and +A `Cipher` is the client bound to one keyset. `Refine` adds a refinement +to the context of every field, such as a tenant ID. `Bind` adds a binding +part, such as a row ID. The generated `Encrypt` and `Decrypt` functions take a slice. They send one ZeroKMS request for each 500 values. ```go client, err := encrypt.NewClient(ctx) defer client.Close() -cipher := client.DefaultKeyset().Extend("tenant-42") +// Not on rc.3 yet: `Refine` and `Bind`. Today `Refine` is `Extend`. +cipher := client.DefaultKeyset().Refine("tenant-42") encrypted, err := Encrypt(ctx, cipher, people) // []EncryptedUser opened, err := Decrypt(ctx, cipher, encrypted) // []User @@ -766,14 +770,23 @@ Decided on 7 November 2026. A context has two scopes: A part of the context is one of three kinds. The declaration says which, and fixes the order: the query scope first, then the binding parts. -| Kind of part | Example | In the query scope | In the descriptor | In the AAD | Verified by | -| --- | --- | --- | --- | --- | --- | -| Scoped | The label and the field, or a tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | -| Binding | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | -| Post-seal | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | +| Kind of part | Verb | Example | In the query scope | In the descriptor | In the AAD | Verified by | +| --- | --- | --- | --- | --- | --- | --- | +| Refinement | `refine` | A tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | +| Binding part | `bind` | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | +| Post-seal part | None. The database adds it | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | -A part is scoped unless the declaration marks it, because that is what Go -rc.3 writes. A binding part can be stored in the row as a context field, +The words were decided on 7 November 2026. A caller refines a context +with a refinement. A refinement narrows the data and the queries over it. +A caller binds a context with a binding part. A binding part ties the +ciphertext to its row and leaves the queries as they are. + +"Extend" and "extension" are no longer used. Go `Extend` becomes `Refine`, with +`Extend` kept as a deprecated alias until Go 0.1.0. `Bind` is added to +Go. + +A part is a refinement unless the declaration marks it as binding, +because that is what Go rc.3 writes. A binding part can be stored in the row as a context field, so the reader takes it from the row. ZeroKMS logs the descriptor at every key retrieval, so a binding part such as the writer's IP address is in the access log. The descriptor has a length limit. It is also sent to ZeroKMS, so the @@ -1038,6 +1051,8 @@ format version beside it. and tested in Rust and Go (#1491). The context inputs `RequiredContext`, `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds `ContextExtension` and `ExpectedContext`, and is in review. + `ContextExtension` is renamed for the refinement vocabulary; its new + name is decided with the field reference design. - **Finished when** all nine scenarios run as tests in Rust and Go. #### Two scopes of one context @@ -1050,6 +1065,12 @@ format version beside it. - **Why.** Access intelligence needs each value bound to its row, and found by a column or tenant query. [Two scopes of one context](#two-scopes-of-one-context) has the rule. +- **Vocabulary.** The Rust chain's `.extend(..)` becomes `.refine(..)`, + and `.bind(..)` is added. Go `Extend` becomes `Refine`, with `Extend` as + a deprecated alias until 0.1.0. `Bind` is added to Go. The context contract + and the conformance README say "refinement" instead of "extension". The + frozen vectors do not change, because no key on the wire names the + operation. - **Done already.** Nothing. Today every part is in both scopes. - **Finished when** these are all true, in Rust and in Go: - a record with a binding row ID encrypts; @@ -1218,7 +1239,7 @@ The lead who approves this plan decides each question. | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | -| How does a declaration mark a binding part of the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences two to seven of the context artifact | +| How does a declaration mark a binding part of the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences three to seven of the context artifact | | Should keysets be chosen from data, and should Stack Encrypt use ZeroKMS lock context? | Optional capabilities of a key provider. Or leave them out | Optional capabilities, after the customer release | | What privacy claim can be made for match terms of short words? | Accept what they reveal, and document it. Or change how short words are stored | No recommendation yet. Merging #1397 did not show that they reveal nothing new | | Which crate should own `Base64Text` and `Base85Text`? | A small Vitamin C encoding crate. Or keep them in `eql-bindings` | A small Vitamin C crate, re-exported by Stack Encrypt and EQL | From dd9017eef619983a8085333fbc83d5fca8d66d45 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 16:24:25 +1100 Subject: [PATCH 29/42] docs: the plan builder is the one source of a field's context Record difference three of the context review, decided on 7 November 2026. A field's context comes from a literal, from a row field stored unsealed, or from the caller on every call. Parts are separate and ordered. Scenarios 3 and 8 of the context examples are removed, because each seals a value that is also another field's context. Scenario 7 stays, with its type name noted as wrong. --- .../2026-10-24-stack-encrypt-api-plan.md | 43 +++++++++++++------ ...26-10-24-stack-encrypt-context-examples.md | 27 ++++++++++++ 2 files changed, 58 insertions(+), 12 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index ea12131ac..7684580c1 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1038,22 +1038,41 @@ format version beside it. #### Context from fields and from the caller -- **What to build.** The ways a record gets its context, as the nine - scenarios in the [context - examples](2026-10-24-stack-encrypt-context-examples.md) describe them. A - record can take context from one of its fields, from the caller, or from - both. A record can also store its context in the output. On decrypt, a - record can read its stored context, or check it against an expected context. The - same rules need a form in the dynamic interface and in Go. +- **What to build.** One mechanism for where a field's context comes + from: the plan builder. The derive and the dynamic grammar lower to it. + A field's context comes from one of three sources: + - a literal, through `Plan::context(label)`, the only way to put a label + in a plan; + - a row field stored unsealed in the record, so the reader takes it + from the row, and `open` may state the label it expects; + - the caller, on every write, query, and read, as a refinement or a + binding part. + + Parts are separate and ordered: the label, the field, the refinements, + then the binding parts. Grouping is available inside one part. Nothing + is read from the plaintext on one side and from the caller on the + other. Decided on 7 November 2026, with the context artifact. - **Why.** A program must never need to open a ciphertext to learn the - context that opens it. -- **Done already.** The order in which context parts are joined is fixed - and tested in Rust and Go (#1491). The context inputs `RequiredContext`, + context that opens it. One mechanism keeps Rust and Go writing the same + bytes. +- **Scenarios.** The [context + examples](2026-10-24-stack-encrypt-context-examples.md) are the + acceptance matrix. Scenarios 3 and 8 are removed. Each seals a value + that is also the context of another field. Nothing can open without the + plaintext, so the sealed copy protects nothing. Scenario 7 stays: a caller-supplied part + that is never stored, such as a host ID, so the data opens only on that + host. Its type name `PartiallyStoredUser` is wrong, because the record + is fully stored. Only part of the context is not. +- **Done already.** The order in which parts are joined is fixed and + tested in Rust and Go (#1491). The context inputs `RequiredContext`, `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds `ContextExtension` and `ExpectedContext`, and is in review. `ContextExtension` is renamed for the refinement vocabulary; its new - name is decided with the field reference design. -- **Finished when** all nine scenarios run as tests in Rust and Go. + name is decided with the field reference design. The plan builder's + `.context(..)` and `.context_field(..)` and the derive's + `#[stash(context_field)]` are the forms this replaces. +- **Finished when** scenarios 0, 1, 2, 4, 5, 6, and 7 run as tests in + Rust and Go, through the builder. #### Two scopes of one context diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 468a5810d..ca37fbd92 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -7,6 +7,33 @@ Date: 24 October 2026. This design exploration goes with the **Everything introduced here is proposed syntax, not an implemented API.** None of the code below compiles against Stack Encrypt today. +## Decisions on 7 November 2026 + +The [API plan](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) +records three decisions that change this document. The scenarios stay as +the acceptance matrix for the Rust API. The prose and the attribute +spellings will be rewritten after the plan builder's entry points are +settled. + +- **The plan builder is the one mechanism.** The derive's attributes lower + to it, as the dynamic grammar does. A field's context comes from a + literal, from a row field carried unsealed, or from the caller on every + call. The directional selectors `encrypt(field = ..)` and + `decrypt(caller = ..)` do not survive as a mechanism of their own. +- **Scenarios 3 and 8 are removed.** Each seals a value that is also the + context of another field. A reader must know the plaintext before it + can open anything, so the sealed copy protects nothing. A value the + caller already knows is a caller-supplied part on both sides. It is + not stored. +- **Scenario 7 stays, renamed.** A caller-supplied part that is never + stored, such as a host or application ID, means the data opens only + where that part is known. The type name `PartiallyStoredUser` is wrong: + the record is fully stored, and only part of its context is not. +- **Parts are separate and ordered.** Refinements, then binding parts. + "Extension" is now "refinement". A binding part is a new kind. See [Two + scopes of one + context](2026-10-24-stack-encrypt-api-plan.md#two-scopes-of-one-context). + ## Implementation status on 7 November 2026 This document was checked against `next` at commit `e4990e49b`. The From bf4b3fff6373edabbabf76ee23ee980fc5977ed8 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 16:39:36 +1100 Subject: [PATCH 30/42] docs: a declaration lists the parts a caller must supply Record difference four of the context review, decided on 7 November 2026. Rust keeps three states: required, optional, and none, with none as the default, because an optional part a query omits matches nothing silently. The dynamic grammar can declare required parts; its default stays "any" for Go rc.3 callers. --- .../2026-10-24-stack-encrypt-api-plan.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 7684580c1..56e5f3928 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -818,9 +818,16 @@ client-generated UUID. A query by column or by tenant still finds it. borrows it while the request runs. So a program can use one context for a write, a query, and a read without cloning it. An owned context also works. -- **Context is explicit.** A target says whether the caller must supply a - context, can supply one, or must not. Stack Encrypt does not use a type's - `Default` implementation to decide. +- **Context is explicit.** A declaration lists the parts the caller must + supply, each with its kind and type. A part that is not declared is + refused. A declaration can mark a part as optional. That is the rare + case, because a query that omits an optional part matches nothing and + gives no error. Stack Encrypt does not use a type's `Default` + implementation to decide. Decided on 7 November 2026. +- **The dynamic grammar can declare required parts too,** with their kind + and type, so the Go generator can require them when it generates the + code. Its default stays "any part or none", because Go rc.3 callers use + that, and the stored bytes are the same either way. - **Each element of a collection stands alone.** Each element has its own data key. Each element is authenticated on its own. - **Order terms do not answer equality.** An order-revealing encryption @@ -1071,8 +1078,17 @@ format version beside it. name is decided with the field reference design. The plan builder's `.context(..)` and `.context_field(..)` and the derive's `#[stash(context_field)]` are the forms this replaces. -- **Finished when** scenarios 0, 1, 2, 4, 5, 6, and 7 run as tests in - Rust and Go, through the builder. +- **Required parts.** A Rust declaration lists the parts the caller must + supply, with kind and type, as the artifact's `refined_by::()` does, + extended to binding parts. An undeclared part is refused. Optional is + the marked case. The dynamic grammar can declare required parts; its + default stays "any", and the Go generator requires a declared part at + compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` + (#1404) are the three states this keeps. +- **Finished when** these are both true: + - scenarios 0, 1, 2, 4, 5, 6, and 7 run as tests in Rust and Go, + through the builder; + - a Rust call with an undeclared part fails to compile. #### Two scopes of one context From 9bb92ba9325a4d7818be66b7da09df1a39faad4b Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 17:43:03 +1100 Subject: [PATCH 31/42] docs: scope and bind, through paired entry points Record differences two and five of the context review, decided on 7 November 2026. The verbs are scope and bind, and the parts are a scope and a binding; Go Extend becomes Scope with a deprecated alias, and Bind is added. The context argument is Scope::new(..), Bind::new(..), or Scope::new(..).bind(..), with the order fixed by type, and a query accepts a Scope only. Both the typed API and the chain take it through paired entry points; the chain's open becomes decrypt and no chain state offers context, extend, or keyset. A stored part passed on decrypt is checked against the row, which retires ExpectedContext as a public type. --- .../2026-10-24-stack-encrypt-api-plan.md | 144 +++++++++++++----- 1 file changed, 102 insertions(+), 42 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 56e5f3928..6b5db109e 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -23,7 +23,7 @@ log](2026-10-24-stack-encrypt-api-review.md). | Term | Meaning | | --- | --- | | AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | -| Binding part | A part of the context that ties a ciphertext to its row, such as a row ID, and is not in the query scope. Added with `bind` | +| Binding | A part of the context that ties a ciphertext to its row, such as a row ID, and is not in the query scope. Passed as `Bind::new(..)` | | Ciphertext | One encrypted value, as the Rust `Ciphertext` type. It is stored as bytes, or as Base64 text | | Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Rust and Go run the same cases. The files are in `conformance/stack-encrypt/v1/` | | Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | @@ -34,7 +34,7 @@ log](2026-10-24-stack-encrypt-api-review.md). | Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | | Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | | Query scope | The prefix of the stored context that search terms derive under. A query supplies it. See [Two scopes of one context](#two-scopes-of-one-context) | -| Refinement | A part of the context that narrows the data and the queries over it, such as a tenant ID. Added with `refine`. See [Two scopes of one context](#two-scopes-of-one-context) | +| Scope | A part of the context that narrows the data and the queries over it, such as a tenant ID. Passed as `Scope::new(..)`. See [Two scopes of one context](#two-scopes-of-one-context) | | Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | | Stored context | Every part of a value's context. The data key and the AAD derive under it | | Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | @@ -156,14 +156,15 @@ is not in the query scope, so a query still finds the row across the column. [Two scopes of one context](#two-scopes-of-one-context) gives the rule. -The API for declaring a binding part is an [open +The API for declaring a binding is an [open question](#open-questions-and-decisions-to-be-made). ```rust +// Not on `next` yet: `Bind`. Today the row ID is passed bare. let encrypted: EncryptedUser = user - .encrypt_into_with_context(&keyset, user_id) + .encrypt_into_with_context(&keyset, Bind::new(user_id)) .await?; -let user = User::decrypt_from_with_context(encrypted, &keyset, user_id).await?; +let user = User::decrypt_from_with_context(encrypted, &keyset, Bind::new(user_id)).await?; ``` #### Store a field as an EQL type @@ -217,10 +218,10 @@ let age = Identifier::for_column("members", "age")?; // and context. Today this needs `target::legacy::EncryptInto`, an owned // value, and an owned context. let by_email: TextEqQuery = "alice@example.com" - .encrypt_into_with_context(&keyset, &email) + .encrypt_into_with_context(&keyset, Scope::new(&email)) .await?; let older: IntegerOrdQuery = 30_i32 - .encrypt_into_with_context(&keyset, &age) + .encrypt_into_with_context(&keyset, Scope::new(&age)) .await?; ``` @@ -259,10 +260,11 @@ let column = Identifier::for_column("customers", "profile")?; // `serde_json::Value`, and an owned context. let profile: JsonPlaintext = json!({ "name": "Ada", "age": 35 }).into(); let encrypted: JsonTreeDocument = profile - .encrypt_into_with_context(&keyset, &column) + .encrypt_into_with_context(&keyset, Scope::new(&column)) .await?; -let opened = JsonPlaintext::decrypt_from_with_context(encrypted, &keyset, &column).await?; +// On decrypt the column is checked against the one stored in the document. +let opened = JsonPlaintext::decrypt_from_with_context(encrypted, &keyset, Scope::new(&column)).await?; let name = &opened.value()["name"]; // `opened` wipes the document when it is dropped ``` @@ -318,7 +320,7 @@ use stack_encrypt::json::{AtPath, Contains}; // Not on `next` yet: the public JSON operand types. let ada: JsonTreeQuery = Contains(json!({ "name": "Ada" })) - .encrypt_into_with_context(&keyset, &column) + .encrypt_into_with_context(&keyset, Scope::new(&column)) .await?; ``` @@ -578,6 +580,21 @@ let age = keyset The query values convert to the EQL query types: `JsonTreeQuery` for containment, and `JsonTreeLeafQuery` for a comparison at a path. +A chain call that needs a scope or a binding takes it through the paired +entry point, as the typed API does: + +```rust +// Not on `next` yet: the paired chain entry points and `Scope`. +let by_tenant = cipher + .query_with_context(op::Eq, "alice@example.com", Scope::new(tenant)) + .on(&plan) + .await?; +let user: User = cipher + .decrypt_with_context(encrypted, Scope::new(tenant).bind(row_id)) + .using(&plan) + .await?; +``` + Three kinds of JSON query have no typed operator on `next`. They exist only as data, through `stack_encrypt::dynamic::query_term`, with the operator named as text: @@ -669,17 +686,17 @@ type Settings struct { } ``` -A `Cipher` is the client bound to one keyset. `Refine` adds a refinement -to the context of every field, such as a tenant ID. `Bind` adds a binding -part, such as a row ID. The generated `Encrypt` and +A `Cipher` is the client bound to one keyset. `Scope` adds a scope to the +context of every field, such as a tenant ID. `Bind` adds a binding, such +as a row ID. The generated `Encrypt` and `Decrypt` functions take a slice. They send one ZeroKMS request for each 500 values. ```go client, err := encrypt.NewClient(ctx) defer client.Close() -// Not on rc.3 yet: `Refine` and `Bind`. Today `Refine` is `Extend`. -cipher := client.DefaultKeyset().Refine("tenant-42") +// Not on rc.3 yet: `Scope` and `Bind`. Today `Scope` is `Extend`. +cipher := client.DefaultKeyset().Scope("tenant-42") encrypted, err := Encrypt(ctx, cipher, people) // []EncryptedUser opened, err := Decrypt(ctx, cipher, encrypted) // []User @@ -768,27 +785,27 @@ Decided on 7 November 2026. A context has two scopes: stored under it. A part of the context is one of three kinds. The declaration says which, -and fixes the order: the query scope first, then the binding parts. +and fixes the order: the query scope first, then the bindings. | Kind of part | Verb | Example | In the query scope | In the descriptor | In the AAD | Verified by | | --- | --- | --- | --- | --- | --- | --- | -| Refinement | `refine` | A tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | -| Binding part | `bind` | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | +| Scope | `scope` | A tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | +| Binding | `bind` | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | | Post-seal part | None. The database adds it | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | -The words were decided on 7 November 2026. A caller refines a context -with a refinement. A refinement narrows the data and the queries over it. -A caller binds a context with a binding part. A binding part ties the -ciphertext to its row and leaves the queries as they are. +The words were decided on 7 November 2026. A caller scopes a value with a +scope. A scope narrows the data and the queries over it. A caller binds a +value with a binding. A binding ties the ciphertext to its row and leaves +the queries as they are. -"Extend" and "extension" are no longer used. Go `Extend` becomes `Refine`, with +"Extend" and "extension" are no longer used. Go `Extend` becomes `Scope`, with `Extend` kept as a deprecated alias until Go 0.1.0. `Bind` is added to Go. -A part is a refinement unless the declaration marks it as binding, -because that is what Go rc.3 writes. A binding part can be stored in the row as a context field, +A part is a scope unless the declaration marks it as a binding, because +that is what Go rc.3 writes. A binding can be stored in the row as a context field, so the reader takes it from the row. ZeroKMS logs the descriptor at every -key retrieval, so a binding part such as the writer's IP address is in the +key retrieval, so a binding such as the writer's IP address is in the access log. The descriptor has a length limit. It is also sent to ZeroKMS, so the part leaves the customer's system. @@ -1052,11 +1069,11 @@ format version beside it. in a plan; - a row field stored unsealed in the record, so the reader takes it from the row, and `open` may state the label it expects; - - the caller, on every write, query, and read, as a refinement or a - binding part. + - the caller, on every write, query, and read, as a scope or a + binding. - Parts are separate and ordered: the label, the field, the refinements, - then the binding parts. Grouping is available inside one part. Nothing + Parts are separate and ordered: the label, the field, the scopes, then + the bindings. Grouping is available inside one part. Nothing is read from the plaintext on one side and from the caller on the other. Decided on 7 November 2026, with the context artifact. - **Why.** A program must never need to open a ciphertext to learn the @@ -1074,13 +1091,13 @@ format version beside it. tested in Rust and Go (#1491). The context inputs `RequiredContext`, `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds `ContextExtension` and `ExpectedContext`, and is in review. - `ContextExtension` is renamed for the refinement vocabulary; its new + `ContextExtension` is renamed for the scope vocabulary; its new name is decided with the field reference design. The plan builder's `.context(..)` and `.context_field(..)` and the derive's `#[stash(context_field)]` are the forms this replaces. - **Required parts.** A Rust declaration lists the parts the caller must supply, with kind and type, as the artifact's `refined_by::()` does, - extended to binding parts. An undeclared part is refused. Optional is + extended to bindings. An undeclared part is refused. Optional is the marked case. The dynamic grammar can declare required parts; its default stays "any", and the Go generator requires a declared part at compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` @@ -1094,24 +1111,67 @@ format version beside it. - **What to build.** The binding kind of part, in the declaration and in the dynamic grammar. Search terms derive under the query scope, and the - key and the AAD under the stored context. A binding part can be a context + key and the AAD under the stored context. A binding can be a context field stored in the row. The order is fixed: the query scope, then the - binding parts. + bindings. - **Why.** Access intelligence needs each value bound to its row, and found by a column or tenant query. [Two scopes of one context](#two-scopes-of-one-context) has the rule. -- **Vocabulary.** The Rust chain's `.extend(..)` becomes `.refine(..)`, - and `.bind(..)` is added. Go `Extend` becomes `Refine`, with `Extend` as - a deprecated alias until 0.1.0. `Bind` is added to Go. The context contract - and the conformance README say "refinement" instead of "extension". The - frozen vectors do not change, because no key on the wire names the - operation. +- **Vocabulary.** The Rust chain's `.extend(..)` is removed; the context + argument carries the scopes and bindings instead. Go `Extend` becomes + `Scope`, with `Extend` as a deprecated alias until 0.1.0. `Bind` is added + to Go. The context contract and the conformance README say "scope" + instead of "extension". The frozen vectors do not change, because no key + on the wire names the operation. - **Done already.** Nothing. Today every part is in both scopes. - **Finished when** these are all true, in Rust and in Go: - a record with a binding row ID encrypts; - a column query finds it; - it fails to open under another row ID; - - the stored bytes of a record with no binding part are unchanged. + - the stored bytes of a record with no binding are unchanged. + +#### One context argument, and paired entry points + +Decided on 7 November 2026, as differences two and five of the context +artifact review. + +- **What to build.** The context argument is one of three values, built + before the call: `Scope::new(..)`, `Bind::new(..)`, or + `Scope::new(..).bind(..)`. Several of one kind chain: + `Scope::new(tenant).scope(region).bind(id)`. The order is fixed by type: + after the first `bind`, `scope` is not offered. `Scope::new((a, b))` is + one grouped part. That is a different context from two parts. + Both layers take it through paired entry points: + - the typed API: `encrypt` with `encrypt_with_context`; `encrypt_into` + with `encrypt_into_with_context`; `decrypt_from` with + `decrypt_from_with_context`; + - the chain: `encrypt` with `encrypt_with_context`; `query` with + `query_with_context`; `decrypt` with `decrypt_with_context`. The + chain's `open` is renamed `decrypt`. No chain state offers + `.context(..)`, `.extend(..)`, `.scope(..)`, `.bind(..)`, or + `.keyset(..)`. The keyset is chosen before the chain starts. + + A query entry point accepts a `Scope` and nothing else, so a binding in + a query does not compile. A part whose source is a row field is read + from the plaintext on encrypt. On decrypt, a stored part may be passed, + and is then checked against the row before any key is fetched. That + check is what `ExpectedContext` does today; it stops being a public + type. +- **Why.** The typed API already pairs its entry points (#1404). A + value for the context reads the same on both layers. A builder + state for the context, and a part added later in a chain, are the + mistakes the artifact describes. +- **Done already.** The scalar pairs (#1404). The record pairs, in + #1504. The chain's `encrypt`, `query`, and `open`, with `.extend(..)` + and `.context(..)` on several states. +- **Rename.** `Scope<'a, K>` in `dynamic::record` and `CipherScope` in the + owned API both mean the cipher a call runs under. Both are renamed in + the naming review, so that `Scope` means a part of the context. +- **Finished when** these are all true: + - the examples in [Target state](#target-state) compile; + - the compile-fail cases in `tests/ui` cover a binding in a query, and + a scope after a bind; + - the chain has no `.context`, `.extend`, or `.keyset` state. #### Name context parameters `context` @@ -1274,7 +1334,7 @@ The lead who approves this plan decides each question. | Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | -| How does a declaration mark a binding part of the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences three to seven of the context artifact | +| How does a declaration mark a binding in the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences three to seven of the context artifact | | Should keysets be chosen from data, and should Stack Encrypt use ZeroKMS lock context? | Optional capabilities of a key provider. Or leave them out | Optional capabilities, after the customer release | | What privacy claim can be made for match terms of short words? | Accept what they reveal, and document it. Or change how short words are stored | No recommendation yet. Merging #1397 did not show that they reveal nothing new | | Which crate should own `Base64Text` and `Base85Text`? | A small Vitamin C encoding crate. Or keep them in `eql-bindings` | A small Vitamin C crate, re-exported by Stack Encrypt and EQL | From 5cb144573058137669f987d8bf74495eea5cb519 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 18:37:49 +1100 Subject: [PATCH 32/42] docs: field markers and encryption_plan() replace definition() Record difference six of the context review, decided on 7 November 2026. The derive emits one marker per field, such as EncryptedMember::EMAIL, which the plan builder and a query both take, and encryption_plan(), which builds the record's Plan once through the builder. `using` selects a plan or a marker for both encryption and queries. Whether `using` comes before or after `query` is deferred to the query design. --- .../2026-10-24-stack-encrypt-api-plan.md | 83 ++++++++++++------- ...026-10-25-stack-encrypt-json-operations.md | 4 +- .../2026-10-25-stack-encrypt-packaging.md | 2 +- 3 files changed, 57 insertions(+), 32 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 6b5db109e..c58b1eac6 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -606,42 +606,48 @@ operator named as text: Go has generated methods for the first two, such as `Fields.Profile.Elements.Visits.Match`. -#### Query through a definition +#### Query through a field marker -In Rust, `definition()` returns the declaration of a derived record, built -once, as data. It has one method for each field. That method returns the -field's handle. `using` selects a field, and `query` takes an operator and a value. -A program can also build a definition from a declaration that it reads at -run time. +The derive emits one marker for each field of a record, such as +`EncryptedMember::EMAIL`. A marker names the field, its plaintext type, +and its accessor. It is the value the plan builder takes to declare the +field, and the value a query takes to select it. + +The derive also emits `encryption_plan()`. It builds the record's `Plan` +through the plan builder, once, and keeps it. A query through a marker +reaches that plan itself. Decided on 7 November 2026. ```rust use stack_encrypt::op; -// Not on `next` yet: `definition()`, field handles, and `using`. -// Today a query names a plan written by hand: +// Not on `next` yet: field markers, `encryption_plan()`, and `using` for +// a query. Today a query names a plan written by hand: // `cipher.query(op::Eq, value).on(&plan)`. -let members = EncryptedMember::definition()?; - -let by_email = cipher - .using(&members.email()) +let by_email = keyset + .using(EncryptedMember::EMAIL) .query(op::Eq, "alice@example.com") .await?; -let adults = cipher - .using(&members.age()) +let adults = keyset + .using(EncryptedMember::AGE) .query(op::Between, &(18, 65)) .await?; - -let profiles = EncryptedProfile::definition()?; -let ada = cipher - .using(&profiles.document()) +let ada = keyset + .using(EncryptedProfile::DOCUMENT) .query(op::Contains, &json!({ "name": "Ada" })) .await?; + +// The same plan, for the chain and the dynamic interface. +let plan: &Plan = EncryptedMember::encryption_plan(); ``` ```sql SELECT id, profile FROM customers WHERE profile @> $1::eql_v4.query_json; ``` +Whether `using` comes before `query`, as above, or after it, is decided +with the query design. A program can also build a `Plan` by hand, from a +declaration it reads at run time, and select it with `using`. + Below the definition, `stack_encrypt::dynamic::record` has the functions that take a declaration as a frame: `plan_with`, `encrypt_with`, `query`, and `decrypt_with`. Each takes a `TargetResolver`. The resolver runs the EQL @@ -1047,18 +1053,38 @@ format version beside it. example](#query-a-json-document) compiles, and an order query on a path the target does not order is refused. -#### Definitions, field handles, and queries +#### Field markers and the encryption plan + +Decided on 7 November 2026, as difference six of the context artifact +review. -- **What to build.** A shared definition for each record type, built once - from its declaration, with a method for each field that returns its - handle. Queries select a field with `using`. Typed Rust option - factories, and the same options read from data in the dynamic interface. +- **What to build.** The derive emits one marker for each field, as an + associated constant such as `EncryptedMember::EMAIL`, with the field's + name, its plaintext type, and its accessor. The marker is the `FieldRef` + the plan builder takes, and the handle a query takes. The derive also + emits `encryption_plan()`. It returns the record's `Plan`, built once + through the plan builder with the markers as its arguments. `using` + selects a plan or a marker, for encryption and for a query; `on` goes. + Typed Rust option factories, and the same options read from data in the + dynamic interface. - **Why.** A query must use the same declaration as the stored values. The - caller must not write a second plan by hand. -- **Done already.** Queries work through a plan written by hand: - `cipher.query(op, value).on(&plan)`. -- **Finished when** the [definition example](#query-through-a-definition) compiles, and tests show - that a query cannot run without its definition or its required context. + caller must not write a second plan by hand. One marker type serves the + builder and the query, so the derive lowers to the builder with no + second mechanism. +- **Done already.** `FieldRef` and `pick` exist in `plan/field_ref.rs`. + The derive already lowers a structural record to the builder. Queries + work through a plan written by hand: `cipher.query(op, value).on(&plan)`. +- **Cannot fail.** `build()` refuses a repeated name or a shared identity. + The derive sees every field, so it refuses those at compile time, and + `encryption_plan()` returns `&'static Plan`, not a `Result`. The + hand-written builder keeps the runtime check. The dynamic grammar needs + it. +- **Later.** A compile-time duplicate-field check for the hand-written + builder, through a type-level list of markers, stays a follow-up, as the + artifact says. +- **Finished when** the [marker example](#query-through-a-field-marker) + compiles, and tests show that a query cannot run without its plan or its + required context. #### Context from fields and from the caller @@ -1331,7 +1357,6 @@ The lead who approves this plan decides each question. | Question | Options | Recommendation | | --- | --- | --- | | What is the name of the wrapper that authenticates a collection's order and length? | `Positioned>`. Or a name chosen in the naming review | `Positioned`. Decide in the naming review | -| Should encryption and queries both select a definition with `using`? | `using` for both. Or keep `on` for queries | `using` for both, with both call orders. An earlier decision list calls this settled, but its own section calls it a proposal. Confirm it | | For each change that would alter the stored format, should it keep the rc.3 format or add a new version? | Keep rc.3 data readable and searchable. Or add a version beside it | Decide for each change when it is designed | | Should Stack Encrypt use refined tags? | Store a tag derived from the ZeroKMS key tag with a hash-based message authentication code (HMAC). Or keep the original tag | No recommendation yet. The protocol is not designed | | How does a declaration mark a binding in the context, in Rust and in the dynamic grammar? | A kind on each part of the declaration, in the derive, the plan builder, and the Go struct tag. The spelling depends on the context vocabulary and the field reference design, decided with the context artifact | Decide with differences three to seven of the context artifact | diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 9266b0cc7..38c2ef186 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -326,7 +326,7 @@ struct EncryptedProfile(JsonTreeDocument); async fn main() -> Result<(), Box> { let client = StackCipher::new().await?; let keyset = client.default_keyset(); - let _definition = EncryptedProfile::definition()?; + let _plan = EncryptedProfile::encryption_plan(); let profile = JsonPlaintext::parse( r#"{"age":42,"name":"Alice","visits":[{"date":"2026-10-01"}]}"#, @@ -366,7 +366,7 @@ For that type: 1. Select those fields through the context and scope attributes described in the API plan. 2. Get its JSON field handle from - `EncryptedUser::definition()?.profile()`, and require `TenantId` on + `EncryptedUser::PROFILE`, and require `TenantId` on query requests. 3. Store and return both ID fields for decryption. 4. If the caller supplies input, use the agreed `_with_context` variants, diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index fe57aa1ac..f62d3f4f3 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -315,7 +315,7 @@ combined trait. Keep these agreed spellings: - `.using(...)` -- `definition()` +- `encryption_plan()` - `encrypt` - `encrypt_with_context` - `encrypt_into` From 3e183db1d2b27f0062db81b6c0a42b2fd199f2ed Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 18:50:43 +1100 Subject: [PATCH 33/42] docs: query context, stored context, and context name Record differences seven and eight of the context review, decided on 7 November 2026. Every part of a context is a scope or a binding. The query context is all the scopes, declared then caller-supplied; the stored context is the query context followed by the bindings. A field's context name is its last declared scope, pinned on a rename, and never shared by two fields of one record (ADR-0006). "Label", "identity", and "query scope" are retired. The keyset is the receiver of every entry point. --- .../2026-10-24-stack-encrypt-api-plan.md | 79 +++++++++++-------- ...26-10-24-stack-encrypt-context-examples.md | 8 +- 2 files changed, 52 insertions(+), 35 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index c58b1eac6..9db268ab0 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -23,9 +23,10 @@ log](2026-10-24-stack-encrypt-api-review.md). | Term | Meaning | | --- | --- | | AAD (additional authenticated data) | Data that encryption checks for one ciphertext, but does not encrypt. Stack Encrypt builds it from the context. See [Context and AAD](#context-and-aad) | -| Binding | A part of the context that ties a ciphertext to its row, such as a row ID, and is not in the query scope. Passed as `Bind::new(..)` | +| Binding | A part of the context that ties a ciphertext to its row, such as a row ID. It is in the stored context and not in the query context. Passed as `Bind::new(..)` | | Ciphertext | One encrypted value, as the Rust `Ciphertext` type. It is stored as bytes, or as Base64 text | | Conformance profile | A named, versioned set of test cases for one area, such as `record-contexts-v1`. Rust and Go run the same cases. The files are in `conformance/stack-encrypt/v1/` | +| Context name | The last declared scope of a field: its name by default, pinned when the Rust field is renamed. Two fields of one record never share one | | Context | Data that a ciphertext is tied to, such as a column name or a row ID. Decryption needs the same context. See [Context and AAD](#context-and-aad) | | Dynamic interface | The engine's interface for languages other than Rust. It takes a declaration and values as data, not as Rust types | | EQL (Encrypt Query Language) | The PostgreSQL types and functions in `packages/eql` that store encrypted values and answer queries on them | @@ -33,8 +34,8 @@ log](2026-10-24-stack-encrypt-api-review.md). | Keyset | A set of keys in ZeroKMS. Every data key belongs to one keyset. A tenant can have its own keyset | | Opaque blob | A whole value, such as a struct, encoded and sealed as one ciphertext. Nothing inside it can be searched | | Passthrough field | A field stored unencrypted next to the encrypted fields. It is not authenticated | -| Query scope | The prefix of the stored context that search terms derive under. A query supplies it. See [Two scopes of one context](#two-scopes-of-one-context) | -| Scope | A part of the context that narrows the data and the queries over it, such as a tenant ID. Passed as `Scope::new(..)`. See [Two scopes of one context](#two-scopes-of-one-context) | +| Query context | All the scopes of a context, declared and caller-supplied. Search terms derive under it. See [Query context and stored context](#query-context-and-stored-context) | +| Scope | A part of the context that narrows the data and the queries over it. The declaration supplies the first scopes, such as `users` and `email`; the caller adds more, such as a tenant ID, with `Scope::new(..)` | | Search term | A value stored next to a ciphertext, so that a query can find it without decrypting it. Equality, order, and match each use their own term | | Stored context | Every part of a value's context. The data key and the AAD derive under it | | Stored format | Everything Stack Encrypt writes to the database. See [The stored format stays compatible with Go rc.3](#the-stored-format-stays-compatible-with-go-rc3) | @@ -152,10 +153,10 @@ the same as for any other type. To tie every field to one row, the caller supplies the row ID as a part of the context, and supplies it again to decrypt. The row ID is a binding part. It is in the stored context, so the key and the AAD depend on it. It -is not in the query scope, so a query still finds the row across the +is not in the query context, so a query still finds the row across the column. -[Two scopes of one context](#two-scopes-of-one-context) gives the rule. +[Query context and stored context](#query-context-and-stored-context) gives the rule. The API for declaring a binding is an [open question](#open-questions-and-decisions-to-be-made). @@ -556,17 +557,17 @@ supplies the column and the options: use stack_encrypt::{json::JsonPath, op, NonEmpty}; // Containment: documents that contain this document. -let ada = cipher +let ada = keyset .query(op::Contains, &json!({ "name": "Ada" })) .on(&plan) .await?; // A comparison at a path. `$.age` must be an ordered path. -let older = cipher +let older = keyset .query(op::AtPath(op::Gt), &("$.age", 30)) .on(&plan) .await?; -let adults = cipher +let adults = keyset .query(op::AtPath(op::Between), &("$.age", (18, 65))) .on(&plan) .await?; @@ -585,11 +586,11 @@ entry point, as the typed API does: ```rust // Not on `next` yet: the paired chain entry points and `Scope`. -let by_tenant = cipher +let by_tenant = keyset .query_with_context(op::Eq, "alice@example.com", Scope::new(tenant)) .on(&plan) .await?; -let user: User = cipher +let user: User = keyset .decrypt_with_context(encrypted, Scope::new(tenant).bind(row_id)) .using(&plan) .await?; @@ -780,22 +781,33 @@ Sources: the module documentation of `packages/stack-encrypt/src/cipher.rs` describes the descriptor and the AAD. The module documentation of `packages/stack-encrypt/src/target/context.rs` describes the search terms. -#### Two scopes of one context +#### Query context and stored context + +Decided on 7 November 2026. Every part of a context is a scope or a +binding. The declaration supplies the first scopes: the record name, then +the field's context name. The caller supplies the rest. + +- **The query context** is all the scopes: the declared ones, then the + caller's. Search terms derive under it. A query supplies the caller's + scopes, and matches every row stored under the same query context. +- **The stored context** is the query context followed by the bindings. + The data key and the AAD derive under it. -Decided on 7 November 2026. A context has two scopes: +A field's **context name** is the last declared scope. It is the field's +name by default. It can be pinned when the Rust field is renamed, so that +stored data keeps opening. -- **The stored context** is every part. The data key and the AAD derive - under it. -- **The query scope** is a prefix of the stored context. Search terms - derive under it. A query supplies the query scope, and matches every row - stored under it. +The context name is always the last declared scope, so two fields of one +record never share a context. If they did, they would share a key and an +AAD, and a ciphertext copied between the two columns would open. ADR-0006 +records this. A part of the context is one of three kinds. The declaration says which, -and fixes the order: the query scope first, then the bindings. +and fixes the order: the scopes first, then the bindings. -| Kind of part | Verb | Example | In the query scope | In the descriptor | In the AAD | Verified by | +| Kind of part | Verb | Example | In the query context | In the descriptor | In the AAD | Verified by | | --- | --- | --- | --- | --- | --- | --- | -| Scope | `scope` | A tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | +| Scope | `scope` | A record name, a field's context name, or a tenant ID | Yes | Yes | Yes | ZeroKMS and the AEAD | | Binding | `bind` | A row's UUID, or the IP address of the writer | No | Yes | Yes | ZeroKMS and the AEAD | | Post-seal part | None. The database adds it | A key the database generates after the insert | No | Through the refined tag | No | ZeroKMS only | @@ -939,7 +951,7 @@ format version beside it. | Planned change | What it would alter | | --- | --- | | Refined tags | The tag stored in the envelope, and what the AAD binds | -| Search terms under the query scope | A new form of search term, beside the rc.3 form, where terms derive under every part | +| Search terms under the query context | A new form of search term, beside the rc.3 form, where terms derive under every part | | Text encodings as field targets (Base64 and Base85) | How a ciphertext is written as text in a column | | The `SealedValue` envelope and the JSON header version | The offsets of the key tag, and header version 1, which rc.3 writes | | Configurable match options | The terms of columns written with the rc.3 defaults | @@ -1074,7 +1086,7 @@ review. - **Done already.** `FieldRef` and `pick` exist in `plan/field_ref.rs`. The derive already lowers a structural record to the builder. Queries work through a plan written by hand: `cipher.query(op, value).on(&plan)`. -- **Cannot fail.** `build()` refuses a repeated name or a shared identity. +- **Cannot fail.** `build()` refuses a repeated field name or a shared context name. The derive sees every field, so it refuses those at compile time, and `encryption_plan()` returns `&'static Plan`, not a `Result`. The hand-written builder keeps the runtime check. The dynamic grammar needs @@ -1091,15 +1103,16 @@ review. - **What to build.** One mechanism for where a field's context comes from: the plan builder. The derive and the dynamic grammar lower to it. A field's context comes from one of three sources: - - a literal, through `Plan::context(label)`, the only way to put a label - in a plan; + - the declaration, through `Plan::context(record_name)`, the only way + to put the record name in a plan, with each field's context name + after it; - a row field stored unsealed in the record, so the reader takes it - from the row, and `open` may state the label it expects; + from the row. A reader may pass it on decrypt to check it; - the caller, on every write, query, and read, as a scope or a binding. - Parts are separate and ordered: the label, the field, the scopes, then - the bindings. Grouping is available inside one part. Nothing + Parts are separate and ordered: the declared scopes, the caller's + scopes, then the bindings. Grouping is available inside one part. Nothing is read from the plaintext on one side and from the caller on the other. Decided on 7 November 2026, with the context artifact. - **Why.** A program must never need to open a ciphertext to learn the @@ -1133,16 +1146,16 @@ review. through the builder; - a Rust call with an undeclared part fails to compile. -#### Two scopes of one context +#### Query context and stored context - **What to build.** The binding kind of part, in the declaration and in - the dynamic grammar. Search terms derive under the query scope, and the + the dynamic grammar. Search terms derive under the query context, and the key and the AAD under the stored context. A binding can be a context - field stored in the row. The order is fixed: the query scope, then the + field stored in the row. The order is fixed: the scopes, then the bindings. - **Why.** Access intelligence needs each value bound to its row, and found by a column or tenant query. [Two scopes of one - context](#two-scopes-of-one-context) has the rule. + context](#query-context-and-stored-context) has the rule. - **Vocabulary.** The Rust chain's `.extend(..)` is removed; the context argument carries the scopes and bindings instead. Go `Extend` becomes `Scope`, with `Extend` as a deprecated alias until 0.1.0. `Bind` is added @@ -1190,6 +1203,10 @@ artifact review. - **Done already.** The scalar pairs (#1404). The record pairs, in #1504. The chain's `encrypt`, `query`, and `open`, with `.extend(..)` and `.context(..)` on several states. +- **Order of calls.** The keyset is chosen first. It is the receiver of + every entry point: `keyset.encrypt(..)`, `keyset.using(..)`. Whether + `using` comes before or after `query` is decided with the query design. + Difference seven of the context artifact review, 7 November 2026. - **Rename.** `Scope<'a, K>` in `dynamic::record` and `CipherScope` in the owned API both mean the cipher a call runs under. Both are renamed in the naming review, so that `Scope` means a part of the context. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index ca37fbd92..9cbffea9d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -29,10 +29,10 @@ settled. stored, such as a host or application ID, means the data opens only where that part is known. The type name `PartiallyStoredUser` is wrong: the record is fully stored, and only part of its context is not. -- **Parts are separate and ordered.** Refinements, then binding parts. - "Extension" is now "refinement". A binding part is a new kind. See [Two - scopes of one - context](2026-10-24-stack-encrypt-api-plan.md#two-scopes-of-one-context). +- **Parts are separate and ordered.** Scopes, then bindings. "Extension" + is now "scope". A binding is a new kind. See [Query + context and stored + context](2026-10-24-stack-encrypt-api-plan.md#query-context-and-stored-context). ## Implementation status on 7 November 2026 From 06e0eb6fe6d757f940823dd1a763dca9cb97ecb6 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 18:57:32 +1100 Subject: [PATCH 34/42] docs: rewrite the context scenarios as seven short examples Replace the directional-selector design with the decided one: every part of a context is a scope or a binding, declared or supplied by the caller through Scope::new(..) and Bind::new(..), with a stored part checked when a reader passes it on decrypt. Seven scenarios, each a use case and one code block. The two scenarios that sealed a value which was also another field's context are removed. From 913 lines to 221. --- .../2026-10-24-stack-encrypt-api-plan.md | 14 +- ...26-10-24-stack-encrypt-context-examples.md | 1012 +++-------------- 2 files changed, 166 insertions(+), 860 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 9db268ab0..56318452a 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1120,12 +1120,10 @@ review. bytes. - **Scenarios.** The [context examples](2026-10-24-stack-encrypt-context-examples.md) are the - acceptance matrix. Scenarios 3 and 8 are removed. Each seals a value - that is also the context of another field. Nothing can open without the - plaintext, so the sealed copy protects nothing. Scenario 7 stays: a caller-supplied part - that is never stored, such as a host ID, so the data opens only on that - host. Its type name `PartiallyStoredUser` is wrong, because the record - is fully stored. Only part of the context is not. + acceptance matrix: seven scenarios, from a static context alone to a + scope that is never stored. Two earlier scenarios were removed, because + each sealed a value that was also another field's context, so nothing + could open without the plaintext. - **Done already.** The order in which parts are joined is fixed and tested in Rust and Go (#1491). The context inputs `RequiredContext`, `NoCallerContext`, and `OptionalContext` exist (#1404). #1504 adds @@ -1142,8 +1140,8 @@ review. compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` (#1404) are the three states this keeps. - **Finished when** these are both true: - - scenarios 0, 1, 2, 4, 5, 6, and 7 run as tests in Rust and Go, - through the builder; + - all seven scenarios run as tests in Rust and Go, through the + builder; - a Rust call with an undeclared part fails to compile. #### Query context and stored context diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 9cbffea9d..b596d4316 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -1,913 +1,221 @@ -# Context sources and storage: directional API examples - -Date: 24 October 2026. This design exploration goes with the -[API review log](2026-10-24-stack-encrypt-api-review.md) and the -[complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md). - -**Everything introduced here is proposed syntax, not an implemented API.** -None of the code below compiles against Stack Encrypt today. - -## Decisions on 7 November 2026 - -The [API plan](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) -records three decisions that change this document. The scenarios stay as -the acceptance matrix for the Rust API. The prose and the attribute -spellings will be rewritten after the plan builder's entry points are -settled. - -- **The plan builder is the one mechanism.** The derive's attributes lower - to it, as the dynamic grammar does. A field's context comes from a - literal, from a row field carried unsealed, or from the caller on every - call. The directional selectors `encrypt(field = ..)` and - `decrypt(caller = ..)` do not survive as a mechanism of their own. -- **Scenarios 3 and 8 are removed.** Each seals a value that is also the - context of another field. A reader must know the plaintext before it - can open anything, so the sealed copy protects nothing. A value the - caller already knows is a caller-supplied part on both sides. It is - not stored. -- **Scenario 7 stays, renamed.** A caller-supplied part that is never - stored, such as a host or application ID, means the data opens only - where that part is known. The type name `PartiallyStoredUser` is wrong: - the record is fully stored, and only part of its context is not. -- **Parts are separate and ordered.** Scopes, then bindings. "Extension" - is now "scope". A binding is a new kind. See [Query - context and stored - context](2026-10-24-stack-encrypt-api-plan.md#query-context-and-stored-context). - -## Implementation status on 7 November 2026 - -This document was checked against `next` at commit `e4990e49b`. The -directional selectors are not implemented. These parts of the model are -now in the code: - -- **Required input.** Pull request #1404 adds `RequiredContext`, - `NoCallerContext`, and `OptionalContext` for the owned API. A required - `u64` stays required, even though `u64` implements `Default`. -- **Composition.** Pull request #1491 makes repeated Rust extensions append - in order. A grouped extension stays one part. This matches the - composition rule below. -- **Conformance.** Pull request #1491 adds shared Rust and Go cases for a - base with extensions, for stored-context expectations, and for altered - context. The nine scenarios below are not among those cases. - -Still open, or in progress: - -- `ContextExtension` and `ExpectedContext` as caller inputs. Pull request - #1504 adds them, and is in progress. -- `context(...)`, `from_context`, `prefix`, and `tuple(...)`. No pull - request adds them. - -From 7 November 2026, the stored format of Go `v0.1.0-rc.3` must not -break. So a new selector must produce the same context bytes as rc.3 for -data that rc.3 can write. See the [API -plan](2026-10-24-stack-encrypt-api-plan.md#the-stored-format-stays-compatible-with-go-rc3). - -This document -compares what happens to encryption context in each useful combination of -context source and context storage. That comparison comes before anyone -chooses attributes or changes traits. The `Ciphertext` and `passthrough` -proposals come from the workflow draft. The workflow draft is the complete -public workflow examples document linked above. - -## Terms used in this document - -These terms appear throughout. Each one is also explained where it first -matters: - -- **Encryption context**, or context, is data that a ciphertext is tied - to, such as a column name or a row ID. Decryption needs the same context. - The context also scopes a field's search terms. See [Context and - AAD](2026-10-24-stack-encrypt-api-plan.md#context-and-aad). -- **AAD** means additional authenticated data. Encryption checks it for one ciphertext, but does not encrypt it. Stack Encrypt builds it from - the context. -- **Static prefix** is the fixed string in the declaration, such as - `"users"`. Together with the field name it forms a field's static base. -- **Logical row context**, written `R`, is the part of the context that - changes from record to record, such as a user id. -- **Effective context** is the complete context one field is encrypted - under: its static base combined with `R`. -- **Selector** is a declaration inside `context(...)` that says where to - get `R`, for one direction or for both. -- **Caller argument** is a context value the application passes to a - `*_with_context` method, rather than one read from a record. -- **Passthrough field** is an output field copied in cleartext from the - input, with `#[stash(passthrough)]`. A passthrough field is not encrypted - and not authenticated. -- **`from_context` field** is an output field that stores the computed `R` - in cleartext, so that decryption can read it back. -- **Caller extension** is an existing feature of the simple static form. - The caller adds an extra context value to the declared static context. -- **Keyset** is a set of encryption keys held in ZeroKMS, CipherStash's key - management service. `cipher.default_keyset()` chooses which keyset - encrypts. -- **The derive** means the `EncryptFrom` and `DecryptInto` derive macros in - `packages/stack-encrypt-derive`. - -## Recommendation: directional sources, separate storage - -Name each direction, and say for each one where to get context. Store -the context with a separate field annotation. - -The nested direction style comes from Serde. Serde's `rename` attribute can -contain separate names for serialisation and deserialisation. See -[Serde's field attributes](https://serde.rs/field-attrs.html). - -For encryption, the direction names say **where to get context**: +# Context scenarios for the Stack Encrypt API + +Date: 7 November 2026. + +This document lists the seven ways a record can get its context, each as a +code example with its use case. Together they are the acceptance matrix +for the work item [Context from fields and from the +caller](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) +in the API plan. The plan's section [Query context and stored +context](2026-10-24-stack-encrypt-api-plan.md#query-context-and-stored-context) +gives the rules, and its terms table defines every term used here. + +In short: every part of a context is a scope or a binding. The query +context is all the scopes. The stored context is the scopes followed by +the bindings. Search terms derive under the query context, and the data +key and the AAD derive under the stored context. + +The code below is proposed. None of it compiles on `next` today. The +rules are decided; the attribute spellings, such as `scope(tenant: u64)`, +are illustrative. + +## How to read the examples + +- `scope(tenant: u64)` declares a scope the caller supplies on every call. + `bind(row: Uuid)` declares a binding the caller supplies. +- `scope(field = "tenant")` declares a scope stored in the row, in the + passthrough field `tenant`. `bind(field = "id")` declares a stored + binding in the same way. +- A caller passes its parts as `Scope::new(..)`, `Bind::new(..)`, or + `Scope::new(..).bind(..)`. A query accepts a `Scope` only. +- A stored part is read from the plaintext on encrypt. On decrypt, the + reader takes it from the row. If the reader passes it as well, Stack + Encrypt checks it against the row before it fetches any key. + +## 1. A static context only + +The record's own name and each field's name make up the whole context. +Nothing comes from the caller or the row. This is the default for a +record that has no tenant and no row binding, such as a settings table. ```rust -#[stash(context( - prefix = "users", - encrypt(field = "id"), - decrypt(field = "id") -))] -``` - -The two `field` selectors refer to different objects. On encrypt, the -selector reads the plaintext input. On decrypt, it reads the encrypted -input. Both must produce the same typed context value. - -Unlike the two names in a Serde rename, the two selectors are not -independent cryptographic names. If the context value or its shape differs -between the two directions, decryption fails. - -Both directions often use the same Rust field name. For that case, the -recommended shorthand is: - -```rust -#[stash(context(prefix = "users", field = "id"))] -``` - -This shorthand expands exactly to -`encrypt(field = "id"), decrypt(field = "id")`. It still reads two -different objects: the plaintext input on encrypt, and the encrypted input -on decrypt. The shorthand does not copy or store the field by itself. The -output layout must make that value available, normally with passthrough. -Use the explicit directional form when the names or sources differ. - -To **store the computed context**, use a separate field annotation. This is -needed only when ordinary passthrough fields do not already contain the -context value: - -```rust -#[stash(from_context)] -scope: u64, -``` - -A `from_context` field is an output metadata field filled from the -computed logical context. It does not choose where the context comes from. -A passthrough source field can already contain the context value. In that -case, no `from_context` field is needed. This proposal adds no new -source-side encryption methods and no public `encrypt_as` entry point. - -These names are better than bare `context(in)` and `context(out)`. Each of -those terms could mean any of three things: - -- an operation direction; -- an input object or an output object; -- storing a value. - -The names `encrypt`, `decrypt`, and `from_context` give those three roles -separate names. - -## Proposed vocabulary and semantics - -The table lists each proposed form inside `context(...)` and what it means. - -| Syntax inside `context(...)` | Meaning | -| --- | --- | -| `prefix = "users"` | Static base for this layout's field contexts | -| `field = "id"` | Shorthand for both `encrypt(field = "id")` and `decrypt(field = "id")` | -| `encrypt(caller = u64)` | Get the logical context from a required `u64` argument | -| `encrypt(field = "id")` | Get it from the plaintext input's `id` | -| `decrypt(caller = u64)` | Get it from a required `u64` argument on decrypt | -| `decrypt(field = "scope")` | Get it from the encrypted input's cleartext `scope` | -| `encrypt(tuple(field = "tenant_id", field = "id"))` | Build one ordered, typed pair from plaintext fields | -| `decrypt(tuple(field = "id", caller = u64))` | Build one pair from a stored field and the decrypt argument | - -`tuple(...)` is a declarative selector. It is not a Rust expression or a -function that gets access to keys. A `tuple(...)` can contain another -`tuple(...)`. - -The derive resolves `field` names at compile time. The first design -supports direct fields only, not arbitrary paths or callbacks. Each -direction has at most one caller argument selector. When that argument -needs several values, make it a tuple or a typed context object. - -A declaration uses either the shared `field` shorthand, or both explicit -directional selectors. For the initial API, the derive rejects a mix of the -shorthand and directional selectors. Rejecting the mix avoids inventing -precedence rules, and keeps the expansion predictable. A record whose names -differ between directions uses the explicit form. - -`context = "users"` -remains the existing simple spelling for a static prefix with optional -caller extensions. Scenario 0 uses it. - -### Composition rule - -Call the logical row context `R`. A field named `name` has the static base -`B = ("users", "name")`. Its effective context is `(B, R)`. With only -a static prefix and no extension, the effective context is `B`, not -`(B, ())`. - -The selector's result is **one structured value**. A pair -`(tenant_id, id)` produces `(("users", "name"), (tenant_id, id))`, not -`((("users", "name"), tenant_id), id)`. A `from_context` field stores `R`, -not the effective context. The effective context differs from field to -field. Static prefixes and field identities are rebuilt from the -declaration. - -The names in this proposal do not change existing byte encodings. When new -selectors are mapped onto the existing context code, they must preserve -types, order, grouping, and descriptor rendering. A new explicit selector -is a new declaration. It must not silently reinterpret data that is already -stored. - -### Storage and expected context - -Sometimes stored fields contain all of `R`. Then the default decrypt call -rebuilds `R` from those fields. The caller can also supply an expected `R` -through the existing `ExpectedContext` pattern. Decryption then checks -that the two are equal, and uses the stored value. That expected value is -**not another extension**. - -A decrypt selector can need a caller argument. That argument is then -required to build `R`, and no call without a context is available. It is -not a check of an expected value against context the record already -contains in full. - -A future caller could need both a required input and a separate check of -the assembled `R`. That caller would use a distinct checked-context input -or a named API. Do not make one unlabelled argument mean both. This -proposal does not include that combined-check API. Scenario 7 shows where -the need arises. - -## Scenario matrix - -Each scenario combines one context source with one way of storing it. The -table shows where each scenario gets and stores the context. - -| Scenario | Encrypt gets context from | Context available in stored output | Decrypt gets context from | -| ---: | --- | --- | --- | -| 0 | Static prefix; no row context | Declaration | Declaration | -| 1 | Caller argument | Not stored | Caller argument | -| 2 | Caller argument | Dedicated metadata field | Metadata field | -| 3 | Plaintext `id` | No cleartext copy; `id` itself is encrypted | Caller argument | -| 4 | Plaintext `id` | Passthrough `id` | Stored `id` | -| 5 | Plaintext `(tenant_id, id)` | Two passthrough fields | Same pair of stored fields | -| 6 | Plaintext `id` plus caller scope | Dedicated pair-valued metadata field | Stored pair | -| 7 | Plaintext `id` plus caller scope | Only `id` is stored in cleartext | Stored `id` plus caller scope | -| 8 | Plaintext `(tenant_id, id)` | Neither component is stored in cleartext | Caller supplies the pair | - -An encrypted copy of a context field does not make that context available -before decryption. Scenarios 3 and 8 keep encrypted copies so that -decryption can recover the complete plaintext object. Callers still need to -supply the context separately. - -## Shared code for the examples - -Each scenario defines an output type and an async function. Together, this -prelude, the scenario blocks, and the final `main` form one candidate -program. Every example stores its output in a real file and reads it back -through the same helper. So no example depends on a context value hidden -inside an in-memory handle. All the examples use proposed APIs, so they -cannot be typechecked against Stack yet. - -The dependencies match the workflow draft: Stack Encrypt, Serde with -derives, serde_json, tempfile, and Tokio. Once these APIs exist, the -examples need ZeroKMS credentials. In the caller examples, `scope = 7` is -external application context. It is not a keyset selector or an -authorisation credential. - -```rust -use serde::{de::DeserializeOwned, Deserialize, Serialize}; -use stack_encrypt::{ - Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, NonEmpty, - StackCipher, -}; -use std::error::Error; - -type Result = std::result::Result>; - -#[derive(Debug, PartialEq)] -struct User { - id: u64, - name: String, - email: String, - created_at: i64, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Settings, context = "settings")] +struct EncryptedSettings { + theme: Ciphertext, } -fn sample_user() -> User { - User { - id: 42, - name: "Alice".to_owned(), - email: "alice@example.com".to_owned(), - created_at: 1_700_000_000, - } -} - -// The candidate Ciphertext Serde contract encodes its canonical envelope. -fn store_and_reload(value: &T) -> Result { - let directory = tempfile::tempdir()?; - let path = directory.path().join("record.json"); - std::fs::write(&path, serde_json::to_vec(value)?)?; - Ok(serde_json::from_slice(&std::fs::read(path)?)?) -} +let encrypted: EncryptedSettings = settings.encrypt_into(&keyset).await?; +let settings = Settings::decrypt_from(encrypted, &keyset).await?; ``` -## 0. Static prefix only +## 2. A scope the caller supplies and does not store -This example shows the baseline: a static prefix and nothing else. There -is no row context to store or supply. The example keeps the existing -static-context form, and uses the proposed passthrough support. +The caller knows the tenant from the request, and passes it on every +write, query, and read. Nothing about the tenant is stored with the row. +A query for one tenant matches only that tenant's rows. ```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User, context = "users")] -struct StaticUser { - #[stash(passthrough)] - id: u64, - name: Ciphertext, - email: Ciphertext, - #[stash(passthrough)] - created_at: i64, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(tenant: u64))] +struct EncryptedUser { + email: SearchableEmail, } -async fn example_0() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, Scope::new(tenant)) + .await?; +let user = User::decrypt_from_with_context(encrypted, &keyset, Scope::new(tenant)).await?; - let stored: StaticUser = user.encrypt_into(&keyset).await?; - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from(loaded, &cipher).await?; - assert_eq!(restored, user); - Ok(()) -} +let by_email = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(tenant)) + .await?; ``` -With this shorthand, caller extensions stay available, as previously -agreed. Encrypt and decrypt must receive the same extension. The remaining -examples use explicit directional selectors. Their argument requirements -come from those selectors alone. No implicit extension is added on top. +## 3. A scope the caller supplies and the record stores -## 1. Caller context, not stored - -This example shows a context that the caller supplies and the output does -not store. `scope` is absent from the output, so the caller must provide it -again on decrypt. The plaintext `id` in this layout has nothing to do with -the chosen context source. +The tenant is a field of the plaintext, and the record stores it in the +clear. The declaration marks that field as the stored copy, so encrypt +reads it from the plaintext. A reader takes the tenant from the row, so a +batch job can decrypt rows from many tenants without knowing each one +first. A reader that serves one +tenant passes it as well, to check that the row belongs to that tenant. ```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context( - prefix = "users", - encrypt(caller = u64), - decrypt(caller = u64) -))] -struct ExternalScopeUser { - #[stash(passthrough)] - id: u64, - name: Ciphertext, - email: Ciphertext, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"))] +struct EncryptedUser { #[stash(passthrough)] - created_at: i64, + tenant: u64, + email: SearchableEmail, } -async fn example_1() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - let scope = 7_u64; - - let stored: ExternalScopeUser = user - .encrypt_into_with_context(&keyset, scope) - .await?; - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; - assert_eq!(restored, user); - Ok(()) -} -``` - -Logical context: `7_u64`. Effective context for `name`: -`(("users", "name"), 7_u64)`. - -## 2. Caller context, stored in an output metadata field - -This example shows a caller-supplied context that the output stores, so -decrypt needs no argument. The source is still the function argument. The -`from_context` field stores its value, and the decrypt selector explicitly -chooses that stored field. `scope` is output metadata. It is not a field to -add to the recovered `User`. +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; // tenant comes from `user.tenant` -```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context( - prefix = "users", - encrypt(caller = u64), - decrypt(field = "scope") -))] -struct StoredScopeUser { - #[stash(from_context)] - scope: u64, - #[stash(passthrough)] - id: u64, - name: Ciphertext, - email: Ciphertext, - #[stash(passthrough)] - created_at: i64, -} - -async fn example_2() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - - let stored: StoredScopeUser = user - .encrypt_into_with_context(&keyset, 7_u64) - .await?; - assert_eq!(stored.scope, 7); - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from(loaded, &cipher).await?; - assert_eq!(restored, user); - Ok(()) -} +let user = User::decrypt_from(encrypted, &keyset).await?; // tenant comes from the row +let user = User::decrypt_from_with_context(encrypted, &keyset, Scope::new(session.tenant)).await?; ``` -The logical context is still `7_u64`. The output stores it once, and -decrypt does not append it a second time. This example extends the -existing case, where the caller supplies the context and it is stored, to -the field-by-field layout. +The second decrypt is refused with `ContextMismatch` if the row's tenant +is not `session.tenant`, before any key is fetched. -## 3. Plaintext field as context, no cleartext context stored +## 4. A binding stored in the row -This example shows a context read from a plaintext field that is then -encrypted, so the output has no cleartext copy. The encrypt selector reads -`User.id`, so encrypt needs no argument. The output's `id` is encrypted. An -encrypted `id` cannot supply the context needed to decrypt itself. The -reader gets the id from its own record lookup, outside the record. +The row's own ID is part of the context, so each ciphertext opens only +in its row. A ciphertext copied into another row fails to open. The ID is +stored in the clear, as any primary key is. A query is not affected, +because a binding is not in the query context. ```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context( - prefix = "users", - encrypt(field = "id"), - decrypt(caller = u64) -))] -struct HiddenIdUser { - id: Ciphertext, - name: Ciphertext, - email: Ciphertext, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", bind(field = "id"))] +struct EncryptedUser { #[stash(passthrough)] - created_at: i64, -} - -async fn example_3() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - let lookup_id = user.id; - - let stored: HiddenIdUser = user.encrypt_into(&keyset).await?; - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from_with_context(loaded, &cipher, lookup_id).await?; - assert_eq!(restored, user); - Ok(()) + id: Uuid, + email: SearchableEmail, } -``` - -Both directions build `R = 42_u64`. If the reader supplies a different id, -the encrypted fields fail to decrypt. Rebuilding `User` does not drop the -id or use a default. Decryption opens the encrypted copy of the id with -the context the reader supplied. -## 4. Plaintext field as context, stored as the corresponding output field +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &keyset).await?; -This example shows a context read from a plaintext field and stored as a -passthrough copy of the same field. It is the obvious id-bound form of -workflow example 2. The context selector and the passthrough annotation -have separate jobs: the selector chooses `id`, and passthrough copies it. -The example uses the shared `field` shorthand. That shorthand is the same -as the explicit pair of directional selectors introduced above. - -```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context(prefix = "users", field = "id"))] -struct IdBoundUser { - #[stash(passthrough)] - id: u64, - name: Ciphertext, - email: Ciphertext, - #[stash(passthrough)] - created_at: i64, -} - -async fn example_4() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - - let stored: IdBoundUser = user.encrypt_into(&keyset).await?; - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from(loaded, &cipher).await?; - assert_eq!(restored, user); - - // Alternatively, require the stored context to match the lookup's id. - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from_with_context( - loaded, - &cipher, - NonEmpty::from(user.id), - ) - .await?; - assert_eq!(restored, user); - Ok(()) -} +// A query covers the whole column. It does not take the row ID. +let by_email = keyset.using(EncryptedUser::EMAIL).query(op::Eq, "alice@example.com").await?; ``` -The second decrypt call uses the `ExpectedContext` conversion. That -call checks the stored id. It does not build `(stored_id, expected_id)`. - -Changing `id` alone breaks decryption. Moving a whole consistent record is -detected only if the reader supplies the expected id, or checks placement -another way. `created_at` stays ordinary passthrough metadata. Passthrough -metadata is not authenticated. - -## 5. Several plaintext fields as context, stored in separate fields +## 5. A stored scope and a stored binding -This example shows a context built from two plaintext fields, both stored -as passthrough fields. The input gains a tenant id, and the context is the -ordered pair `(tenant_id, id)`. The output renames `id` to `user_id`. The -rename shows why separate field selectors for each direction are useful. -No extra context metadata field is needed. +The tenant and the row ID are both stored in the row. A query for one +tenant matches that tenant's rows, and each ciphertext opens only in its +own row. This is the shape for access intelligence, where each decrypted +value must be tied to the row it came from. ```rust -#[derive(Debug, PartialEq)] -struct TenantUser { - tenant_id: u64, - id: u64, - name: String, - email: String, - created_at: i64, -} - -fn sample_tenant_user() -> TenantUser { - TenantUser { - tenant_id: 7, - id: 42, - name: "Alice".to_owned(), - email: "alice@example.com".to_owned(), - created_at: 1_700_000_000, - } -} - -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = TenantUser)] -#[stash(context( - prefix = "users", - encrypt(tuple(field = "tenant_id", field = "id")), - decrypt(tuple(field = "tenant_id", field = "user_id")) -))] -struct TenantBoundUser { +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"), bind(field = "id"))] +struct EncryptedUser { #[stash(passthrough)] - tenant_id: u64, - #[stash(passthrough, from = "id")] - user_id: u64, - name: Ciphertext, - email: Ciphertext, + tenant: u64, #[stash(passthrough)] - created_at: i64, + id: Uuid, + email: SearchableEmail, } -async fn example_5() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_tenant_user(); +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; +let user = User::decrypt_from(encrypted, &keyset).await?; - let stored: TenantBoundUser = user.encrypt_into(&keyset).await?; - let loaded = store_and_reload(&stored)?; - let restored = TenantUser::decrypt_from(loaded, &cipher).await?; - assert_eq!(restored, user); - Ok(()) -} -``` +// Check both stored parts on decrypt. +let user = User::decrypt_from_with_context( + encrypted, + &keyset, + Scope::new(session.tenant).bind(requested_id), +) +.await?; -Both directions build `(7_u64, 42_u64)`. The selector sets the order of -the context, not the order of the Rust fields. A Serde rename of the -stored property also leaves the selector unchanged. Selectors name Rust -fields, not serialised property names. +let by_email = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(session.tenant)) + .await?; +``` -## 6. Plaintext field plus caller argument, store the combined context +## 6. A stored part and a caller-supplied part together -This example shows a context built from a plaintext field and a caller -argument, with the whole pair stored. Here -`R = (User.id, caller_scope)`, and the `binding` field stores that -complete pair. The output also keeps the original id as ciphertext, so -that decryption can rebuild `User`. Without that copy, the derive would -need a new rule that maps pieces of metadata into plaintext fields. The -id is stored twice on purpose, so that this example stays about how the -context is put together, not about mapping metadata to fields. +The tenant is stored in the row. The row ID comes from the caller on +every call, and is never stored. The row ID is known from elsewhere, such +as the request path. A reader passes the row ID, and may pass the tenant +as well to check it. ```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context( - prefix = "users", - encrypt(tuple(field = "id", caller = u64)), - decrypt(field = "binding") -))] -struct StoredCombinedUser { - #[stash(from_context)] - binding: (u64, u64), - id: Ciphertext, - name: Ciphertext, - email: Ciphertext, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = User, context = "users", scope(field = "tenant"), bind(row: Uuid))] +struct EncryptedUser { #[stash(passthrough)] - created_at: i64, + tenant: u64, + email: SearchableEmail, } -async fn example_6() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - - let stored: StoredCombinedUser = user - .encrypt_into_with_context(&keyset, 7_u64) - .await?; - assert_eq!(stored.binding, (42, 7)); - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from(loaded, &cipher).await?; - assert_eq!(restored, user); - Ok(()) -} -``` - -The encryption function accepts only the part the caller supplies (`7`). -It does not accept the whole pair, because the derive can build the pair -itself. A checked decrypt would expect the whole rebuilt pair `(42, 7)`, -not only the caller's part. - -## 7. Plaintext field plus caller argument, store only the field component - -This example shows a context built from a plaintext field and a caller -argument, with only the field stored. The output stores `id` as a normal -passthrough field, and leaves the external scope out. Decryption combines -the stored id with the scope the reader supplies. +let encrypted: EncryptedUser = user + .encrypt_into_with_context(&keyset, Bind::new(row_id)) + .await?; -```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = User)] -#[stash(context( - prefix = "users", - encrypt(tuple(field = "id", caller = u64)), - decrypt(tuple(field = "id", caller = u64)) -))] -struct PartiallyStoredUser { - #[stash(passthrough)] - id: u64, - name: Ciphertext, - email: Ciphertext, - #[stash(passthrough)] - created_at: i64, -} +// The tenant comes from the row. The row ID must be passed. +let user = User::decrypt_from_with_context(encrypted, &keyset, Bind::new(row_id)).await?; -async fn example_7() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_user(); - let scope = 7_u64; - - let stored: PartiallyStoredUser = user - .encrypt_into_with_context(&keyset, scope) - .await?; - let loaded = store_and_reload(&stored)?; - let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; - assert_eq!(restored, user); - Ok(()) -} +// Passing the tenant as well checks it against the row. +let user = User::decrypt_from_with_context( + encrypted, + &keyset, + Scope::new(session.tenant).bind(row_id), +) +.await?; ``` -This example also builds `(42_u64, 7_u64)` in both directions. It shows -why one yes-or-no flag cannot always describe whether context is stored. -Some parts may be stored, and others supplied from outside the record. -The application can also need to check which record it has. If so, it can -compare `loaded.id` with an expected id before decrypting. - -## 8. Several plaintext fields as context, keep neither component in cleartext +## 7. A scope the caller supplies on every call and never stores -This example shows a context built from two plaintext fields, with neither -stored in cleartext. The encrypt side gets `(tenant_id, id)` from the -source. The reader supplies the complete pair from outside the encrypted -record. Once that context is supplied, decryption recovers both source -fields from their ciphertexts. +The scope is a value that exists only where the data may be used, such as +a host or application ID. The record is fully stored, but part of its +context is not. So the data opens only on a host that has the value. A +row copied to another host fails to open, and a query from another host +matches nothing. ```rust -#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] -#[stash(struct = TenantUser)] -#[stash(context( - prefix = "users", - encrypt(tuple(field = "tenant_id", field = "id")), - decrypt(caller = (u64, u64)) -))] -struct HiddenTenantUser { - tenant_id: Ciphertext, - id: Ciphertext, - name: Ciphertext, - email: Ciphertext, - #[stash(passthrough)] - created_at: i64, +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Secret, context = "secrets", scope(host: String))] +struct EncryptedSecret { + value: Ciphertext, } -async fn example_8() -> Result<()> { - let cipher = StackCipher::new().await?; - let keyset = cipher.default_keyset(); - let user = sample_tenant_user(); - let lookup_context = (user.tenant_id, user.id); - - let stored: HiddenTenantUser = user.encrypt_into(&keyset).await?; - let loaded = store_and_reload(&stored)?; - let restored = - TenantUser::decrypt_from_with_context(loaded, &cipher, lookup_context).await?; - assert_eq!(restored, user); - Ok(()) -} +let host = host_id()?; +let encrypted: EncryptedSecret = secret + .encrypt_into_with_context(&keyset, Scope::new(&host)) + .await?; +let secret = Secret::decrypt_from_with_context(encrypted, &keyset, Scope::new(&host)).await?; ``` -This is a new contract for a typed selector. It does not mean flattening -the pair into text. It also does not make separately chained extensions -interchangeable with one argument whose value is a tuple. - -## Candidate program entry point - -This `main` runs all nine scenarios in order. - -```rust -#[tokio::main] -async fn main() -> Result<()> { - example_0().await?; - example_1().await?; - example_2().await?; - example_3().await?; - example_4().await?; - example_5().await?; - example_6().await?; - example_7().await?; - example_8().await?; - Ok(()) -} -``` +## Scenarios that were removed -## Consequences for the traits and derive - -The logical context `R` is not always the value a caller passes. The table -compares `R` with the caller's input in the scenarios that have a row -context: - -| Scenario | Logical `R` | Caller input on encrypt | Caller input on decrypt | -| ---: | --- | --- | --- | -| 1 | `u64` | Required `u64` | Required `u64` | -| 2 | `u64` | Required `u64` | Optional expectation of `u64` | -| 3 | `u64` | None | Required `u64` | -| 4 | `u64` | None | Optional expectation of `u64` | -| 6 | `(u64, u64)` | Required `u64` component | Optional expectation of the whole pair | -| 7 | `(u64, u64)` | Required `u64` component | Required `u64` component | -| 8 | `(u64, u64)` | None | Required pair | - -Today, `EncryptFrom::Context` already belongs to a different trait from -`DecryptInto

::Context`. The two types can differ. - -Each `Context` type describes the input the caller needs to supply. That -input is not always the final context every operation uses. The documentation should state that difference. The name -`ContextInput` might be clearer, but this document does not decide on a -rename. - -To support `encrypt_into`, a selector that needs no caller input can -expose an input that has a default, such as `()`. A decrypt that reads -stored context uses the existing `ExpectedContext` idea. A required -caller selector must stay required, even if its Rust type implements -`Default`. A required `u64` must not silently become zero only because -`u64: Default`. - -That rule exposed a real limit in the convenience methods of 24 October -2026. Their only test was `Context: Default`. Since #1404, the owned -methods use the explicit capabilities `AllowsNoContext` and -`AcceptsContext` instead. The structural methods in `target::legacy` still -use `Default`. An implementation of this proposal would need -one of two things: - -- a `RequiredContext` input that has no default, with an easy - conversion from `T`; -- a separate trait capability that controls calls without a context. - -The examples pass bare caller values at call sites. They do not assume -that the associated input type can be a bare `u64`. Choose that mechanism -before claiming the directional API enforces required inputs. - -The current public `Encryption` constructors must also gain or expose the -source-selection code the derive needs, owned by the core crate. - -Plan context fields are the existing context fields of the plan builder, -in `packages/stack-encrypt/src/plan/build.rs`. They already show context -that depends on the source. But plan context fields are based on labels, -carry the field, and use different composition rules. So they do not -already implement this proposal's general `R`: a typed value that can be -stored. - -## Rules the implementation should enforce - -The implementation should enforce these rules: - -- Selectors for both directions must produce the same context type and - ordered structure. Different Rust field names are fine. Different values - or grouping are not a supported way to encrypt and decrypt one record. -- Resolve field names and types at compile time. A decrypt selector cannot - read an encrypted field to get the context needed to decrypt it. -- A `from_context` field is cleartext output metadata with the logical - context's type. It cannot also be encrypted, set to a default, or copied - separately from an unrelated source. An ordinary passthrough field does - not become context only because its name is `id` or `tenant_id`. -- Recovery does not invent extra metadata as a plaintext field. Every - original plaintext field must have a defined way to be recovered. No - field is silently left out, or set to a default, because a source was - used only for context. -- Check supplied expectations before fetching keys, where possible. A - context value read from storage stays untrusted until the encrypted - fields authenticate under it. An expectation check alone is not - authentication. -- A selected source value can appear both in stored context metadata and - in a separately recovered plaintext field. If so, check that the two - agree before returning the rebuilt object. Never return an id different - from the one the encrypted fields were bound to. -- Stored context should be required in the serialised schema. A missing - field must not silently fall back to a default context. The policy for - empty context must be explicit. That policy must keep the current rule - that a target's context is nonempty. -- An indexed target is a field that also produces search index terms. For - an indexed target, the selected context also scopes those terms, unless - a future API explicitly separates index context from ciphertext - context. So a row-specific id changes how search works. These - examples are about ciphertext layouts. They do not silently promise - queries that match across rows. - -The required tests are: - -- storage round trips for every scenario; -- missing required arguments; -- wrong expected values; -- altered stored context; -- reordered or regrouped tuples; -- renamed source or output fields; -- attempts to make decryption depend on its own output in a circle. - -Two fields of the same type can be swapped without failing a type check. -So round-trip and mismatch tests are still needed, even with typed -selectors. - -## Alternatives worth comparing - -1. **Field-level `context(encrypt, decrypt)` flags.** These are concise for - one id. But they refer to the output declaration, while the encrypt - source is declared elsewhere. They leave the order of several fields - unstated, and cannot easily describe a context built from a caller - argument and a field. Prefer explicit selectors on the container first. -2. **One `context = expression` callback.** This is flexible, but it hides - dependencies. It makes rebuilding the context in reverse harder, and - makes cycles harder to detect. It also encourages arbitrary application - code in the declaration. Keep simple declarative selectors as the - initial API. -3. **A typed context object with a dedicated derive.** This could reduce - repetition for large composite contexts, and give a stronger encoding - contract. Consider it once the single-value and tuple cases have settled - the intended behaviour. - -The recommendation is the model of directional selectors with separate -storage. It is not a commitment to every spelling shown here. In -particular, these remain choices for review: `from_context`, `prefix`, the -tuple notation, and how required inputs are represented. - -## Evidence and validation - -The current behaviour was checked in these files: - -- [context types](../../packages/stack-encrypt/src/target/context.rs); -- [target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs); -- [derive validation](../../packages/stack-encrypt-derive/src/shape.rs); -- [plan context fields](../../packages/stack-encrypt/src/plan/build.rs). - -Serde inspired the directional nesting, but Serde does not provide it. - -These are candidate API examples. Checks of syntax and links do not show -that the macros support them, that the types are correct, or that -encryption works as described. This document claims no live encryption -or database tests, and changes no library code. - -The following validation was completed. `rustfmt` parsed the combined -candidate program for the nine scenarios, using edition 2024. It also -parsed the introductory fragments, wrapped as Rust items. Local links, -Markdown fences, and trailing whitespace were checked in this document and -both companion documents. No proposed macro was expanded or typechecked. +Two earlier scenarios sealed a value that was also the context of another +field, and stored no copy of it in the clear. A reader needed the +plaintext before it could open anything, so the sealed copy protected +nothing. Both are gone. A value the caller already knows is a +caller-supplied part, as in scenarios 2, 6, and 7. From 310d4a132a8cf5526ad496ffe90ccbfaf63c00cf Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:11:35 +1100 Subject: [PATCH 35/42] docs: a developer can see the context a declaration or call uses Add contexts() on a Plan and context() on a prepared call, which show the query context and the stored context with each part's type, before any key is fetched. A wrong context loses data silently, so the output is pinned for the context scenarios and matched between Rust and Go. --- .../2026-10-24-stack-encrypt-api-plan.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 56318452a..97e0b362e 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -836,6 +836,38 @@ item. The use case is access intelligence. Each value is bound to its row by a client-generated UUID. A query by column or by tenant still finds it. +#### A developer can see the context + +Decided on 7 November 2026. A wrong context is silent: a query matches +nothing, or a row opens somewhere it should not. So every declaration +and every call can show the context it uses, before any key is fetched. + +```rust +// Not on `next` yet: `contexts()` and `context()`. + +// For a declaration: each field, its query context, and its stored +// context, with a placeholder for every part the caller supplies. +println!("{}", EncryptedUser::encryption_plan().contexts()); +// email +// query context: users / email / {tenant: u64} +// stored context: users / email / {tenant: u64} / {row: Uuid} +// search terms: under the query context + +// For a call: the exact context, with the caller's values filled in. +let call = keyset + .using(EncryptedUser::EMAIL) + .query_with_context(op::Eq, "alice@example.com", Scope::new(42_u64)); +println!("{}", call.context()); +// query context: users / email / 42u64 +``` + +The output shows each part's type, because `7u64`, `7i64`, the text `7`, +and the bytes `37` are four different contexts. The descriptor that +ZeroKMS logs is coarser, so it is not enough on its own. A call's context +can include the caller's values, such as a tenant ID or an IP address. +The plan form shows placeholders instead, so it is the one to put in +logs. + #### Other rules - **Rust and Go behave the same.** Every built-in capability has the same @@ -1214,6 +1246,35 @@ artifact review. a scope after a bind; - the chain has no `.context`, `.extend`, or `.keyset` state. +#### Show the context + +Decided on 7 November 2026. + +- **What to build.** Two ways to see a context before any key is fetched: + - `contexts()` on a `Plan`, listing each field's query context and + stored context, with a typed placeholder for every caller-supplied + part. The derive's `encryption_plan()` gives it to every record, and + a hand-built plan has it too. + - `context()` on a prepared call. It gives the exact query context and + stored context the call will use, with the caller's values filled in. + Each part shows its type. + + Both render through the same code that derives the key descriptor and + the AAD, so what a developer sees is what the engine uses. The dynamic + interface has the same two forms. So `stashgen` can print a Go record's + contexts. A Go call can show its context. +- **Why.** A wrong context loses data silently. A query under the wrong + context matches nothing and gives no error. A developer who cannot see + the context cannot find the mistake. +- **Done already.** `Descriptor` renders a context as text, and + `ContextMismatch` reports the stored descriptor. Neither shows the + types, and neither is reachable from a plan or a call. +- **Finished when** these are both true: + - a test pins the output of `contexts()` for each of the [context + scenarios](2026-10-24-stack-encrypt-context-examples.md); + - a conformance case shows that Rust and Go print the same context for + the same declaration. + #### Name context parameters `context` These public methods in `packages/stack-encrypt/src/cipher.rs` take a From 9ef00b7524779e3ff6256eccca4b8374e9f6e1f1 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:19:02 +1100 Subject: [PATCH 36/42] docs: assertions over the context for unit tests A testing feature of stack-encrypt gives assert_context! over a field marker, a plan, or a call, and the Go module gets the same in a testing package. Both read the output of contexts() and context(), so a test and a log never disagree. Each context scenario carries one. --- .../2026-10-24-stack-encrypt-api-plan.md | 37 ++++++++++++++++++- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 97e0b362e..4d813400b 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -868,6 +868,28 @@ can include the caller's values, such as a tenant ID or an IP address. The plan form shows placeholders instead, so it is the one to put in logs. +The same output serves a unit test. A testing feature of the crate gives assertions over it. A developer can +pin the context of a field or a call, and the build fails when it +changes. The Go module has the same +assertions, in a testing package. + +```rust +// Not on `next` yet: the `testing` feature and `assert_context!`. +#[cfg(test)] +mod tests { + use stack_encrypt::testing::assert_context; + + #[test] + fn email_is_scoped_to_the_tenant_and_bound_to_the_row() { + assert_context!( + EncryptedUser::EMAIL, + query = ["users", "email", scope::()], + stored = ["users", "email", scope::(), bind::()], + ); + } +} +``` + #### Other rules - **Rust and Go behave the same.** Every built-in capability has the same @@ -1263,9 +1285,18 @@ Decided on 7 November 2026. the AAD, so what a developer sees is what the engine uses. The dynamic interface has the same two forms. So `stashgen` can print a Go record's contexts. A Go call can show its context. +- **Assertions for tests.** A `testing` feature of `stack-encrypt`, with + `assert_context!` over a field marker, a plan, or a prepared call. It + compares the query context and the stored context, with types, against + what the test states, and names the first part that differs. The Go + module has the same in a testing package, such as `encrypttest`, so a + Go test can pin a record's contexts the same way. Both read the same + output as `contexts()` and `context()`, so a test and a log never + disagree. - **Why.** A wrong context loses data silently. A query under the wrong context matches nothing and gives no error. A developer who cannot see - the context cannot find the mistake. + the context cannot find the mistake. A test that cannot state the + context cannot catch a change to it. - **Done already.** `Descriptor` renders a context as text, and `ContextMismatch` reports the stored descriptor. Neither shows the types, and neither is reachable from a plan or a call. @@ -1273,7 +1304,9 @@ Decided on 7 November 2026. - a test pins the output of `contexts()` for each of the [context scenarios](2026-10-24-stack-encrypt-context-examples.md); - a conformance case shows that Rust and Go print the same context for - the same declaration. + the same declaration; + - the [context scenarios](2026-10-24-stack-encrypt-context-examples.md) + each carry an `assert_context!` test, in Rust and in Go. #### Name context parameters `context` From e5a8d555ed412e3d964f57b9acb5a6f670a35b57 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:26:31 +1100 Subject: [PATCH 37/42] docs: two more context scenarios Scenario 8 shows an EQL type with its column identifier as the context, taken from the stored value on decrypt, and checked before any key is fetched when the reader passes it. Scenario 9 shows the bindings access intelligence stores in the row (row ID, address, time), read back without the reader knowing them, and logged by ZeroKMS at every key retrieval. --- .../2026-10-24-stack-encrypt-api-plan.md | 6 +- ...26-10-24-stack-encrypt-context-examples.md | 95 ++++++++++++++++++- 2 files changed, 97 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 4d813400b..19f2681a0 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1174,8 +1174,8 @@ review. bytes. - **Scenarios.** The [context examples](2026-10-24-stack-encrypt-context-examples.md) are the - acceptance matrix: seven scenarios, from a static context alone to a - scope that is never stored. Two earlier scenarios were removed, because + acceptance matrix: nine scenarios, from a static context alone to the + bindings that access intelligence stores. Two earlier scenarios were removed, because each sealed a value that was also another field's context, so nothing could open without the plaintext. - **Done already.** The order in which parts are joined is fixed and @@ -1194,7 +1194,7 @@ review. compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` (#1404) are the three states this keeps. - **Finished when** these are both true: - - all seven scenarios run as tests in Rust and Go, through the + - all nine scenarios run as tests in Rust and Go, through the builder; - a Rust call with an undeclared part fails to compile. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index b596d4316..746e54a91 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -2,7 +2,7 @@ Date: 7 November 2026. -This document lists the seven ways a record can get its context, each as a +This document lists nine ways a record can get its context, each as a code example with its use case. Together they are the acceptance matrix for the work item [Context from fields and from the caller](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) @@ -212,6 +212,99 @@ let encrypted: EncryptedSecret = secret let secret = Secret::decrypt_from_with_context(encrypted, &keyset, Scope::new(&host)).await?; ``` +## 8. An EQL type, with its column as the context + +An EQL type such as `TextEq` is one column's value, stored in a +PostgreSQL column of type `public.eql_v4_text_eq`. Its context is the +table and column, as an `Identifier`. The type stores the identifier in +its `i` field, so a reader takes it from the stored value. A reader that +knows which column it is reading passes the identifier as well. If the +stored identifier differs, the value is refused before any key is +fetched. + +Inside a record, the derive supplies the identifier from the record's +name and the field's context name, so none of this is written by hand. + +```rust +use eql_bindings::{v4::text::{TextEq, TextEqQuery}, Identifier}; + +let column = Identifier::for_column("users", "email")?; + +// Encrypt one value for the column. The identifier is the context, and +// `TextEq` stores it in `i`. +let stored: TextEq = "alice@example.com" + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; + +// Decrypt, taking the column from the stored value. +let email = String::decrypt_from(stored, &keyset).await?; + +// Decrypt, checking that the stored column is the one expected. A value +// from `users.name` is refused with `ContextMismatch` before any key +// is fetched. +let email = String::decrypt_from_with_context(stored, &keyset, Scope::new(&column)).await?; + +// A query for the column uses the same context, so its term matches. +let query: TextEqQuery = "alice@example.com" + .encrypt_into_with_context(&keyset, Scope::new(&column)) + .await?; +``` + +## 9. Bindings stored for access intelligence + +Access intelligence needs to know which value was read, and the facts +about its writing: the row, the address it was written from, and when. +Each of those is a binding, stored in the clear in the row. Encrypt reads +them from the plaintext. A reader takes them from the row, so it needs to +know none of them. + +Each binding is in the descriptor, so ZeroKMS logs them at every key +retrieval. That log is where the access record comes from. A binding is +not in the query context, so queries are unchanged. + +```rust +#[derive(EncryptTarget, DecryptSource)] +#[stash( + struct = Note, + context = "notes", + scope(field = "tenant"), + bind(field = "id"), + bind(field = "written_from"), + bind(field = "written_at"), +)] +struct EncryptedNote { + #[stash(passthrough)] + tenant: u64, + #[stash(passthrough)] + id: Uuid, + #[stash(passthrough)] + written_from: IpAddr, + #[stash(passthrough)] + written_at: Timestamp, + body: Ciphertext, +} + +let note = Note { tenant, id: Uuid::new_v4(), written_from: request.ip(), written_at: now(), body }; +let encrypted: EncryptedNote = note.encrypt_into(&keyset).await?; + +// The reader passes nothing. Every part comes from the row, and ZeroKMS +// logs the whole context when it returns the key. +let note = Note::decrypt_from(encrypted, &keyset).await?; + +// A reader that serves one tenant and one row checks both. +let note = Note::decrypt_from_with_context( + encrypted, + &keyset, + Scope::new(session.tenant).bind(requested_id), +) +.await?; +``` + +Two limits apply. The descriptor has a length limit at ZeroKMS, so the +bindings must stay short. And the descriptor is sent to ZeroKMS, so the +address and the time leave the customer's system. For access +intelligence that is the point. A customer should still know it. + ## Scenarios that were removed Two earlier scenarios sealed a value that was also the context of another From a541941e9c834b87d1544f99286fcdb95a7f2ea7 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:37:54 +1100 Subject: [PATCH 38/42] docs: bindings for access intelligence live inside the EQL payload Scenario 9 now stores the address and the time in an authenticated metadata field of the EQL payload, so one column is enough to decrypt and the reader passes nothing. Add the EQL work item for that optional field, and note that ZeroKMS v2 is planned to remove the descriptor limit and encrypt the access log. --- .../2026-10-24-stack-encrypt-api-plan.md | 25 +++++++ ...26-10-24-stack-encrypt-context-examples.md | 69 ++++++++----------- 2 files changed, 55 insertions(+), 39 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 19f2681a0..c7bc97e3e 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -827,6 +827,9 @@ key retrieval, so a binding such as the writer's IP address is in the access log. The descriptor has a length limit. It is also sent to ZeroKMS, so the part leaves the customer's system. +ZeroKMS version 2 is planned to remove the limit and to encrypt the +access log. + A post-seal part is added after sealing, by a database trigger that refines the stored tag. ZeroKMS verifies the refined tag and returns the original. The AAD binds the original. The design allows for this kind. The work @@ -1022,6 +1025,28 @@ format version beside it. - **Finished when** Rust and Go each open blobs that the other wrote, and the conformance profiles cover it. +#### Authenticated metadata in an EQL payload + +- **What to build.** An optional field, `m`, in every EQL payload type, + for authenticated metadata. It contains named bindings, stored in the clear inside the payload, such as the address a value was written from + and when. The + Rust declaration marks a binding as stored `in = metadata`, and the + derive writes it into the field's payload. The dynamic grammar has the + same form, so Go can declare it. The EQL bundle accepts the field in + every domain and ignores it in every operator. +- **Why.** Access intelligence needs each value's writing facts bound to + it. As row columns, a reader would have to select every one of them to + decrypt the value. Inside the payload, one column stores the value and its bindings, so + `SELECT body` is enough. Scenario 9 of the [context + scenarios](2026-10-24-stack-encrypt-context-examples.md) shows it. +- **Stored format.** The field is optional, so a payload without it is + unchanged. A payload with it is a new form beside the rc.3 form. +- **Done already.** Nothing. +- **Finished when** these are all true: + - scenario 9 encrypts, and is read back from one column; + - it fails to open when the metadata is changed; + - the EQL conformance profile covers the field in Rust and in Go. + #### Typed text encodings and key tag extraction - **What to build.** diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 746e54a91..fc5e96bc9 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -250,60 +250,51 @@ let query: TextEqQuery = "alice@example.com" .await?; ``` -## 9. Bindings stored for access intelligence +## 9. Bindings stored inside the EQL payload, for access intelligence Access intelligence needs to know which value was read, and the facts -about its writing: the row, the address it was written from, and when. -Each of those is a binding, stored in the clear in the row. Encrypt reads -them from the plaintext. A reader takes them from the row, so it needs to -know none of them. +about its writing: the address it was written from, and when. Each fact +is a binding. If each one were its own column, a reader would have to +select every one of them to decrypt the value. So the bindings travel +inside the EQL payload instead, in an authenticated metadata field. One column then stores the value and its bindings, so `SELECT body` is +enough to decrypt it. -Each binding is in the descriptor, so ZeroKMS logs them at every key -retrieval. That log is where the access record comes from. A binding is -not in the query context, so queries are unchanged. +The metadata is stored in the clear. It is part of the stored context, so +the AAD and the descriptor cover it. A changed address fails to open. +Each binding is in the descriptor, so ZeroKMS logs it at every key +retrieval. That log is where the access record comes from. + +EQL has no such field today; the plan's [stored format +work](2026-10-24-stack-encrypt-api-plan.md#authenticated-metadata-in-an-eql-payload) +adds it. ```rust #[derive(EncryptTarget, DecryptSource)] -#[stash( - struct = Note, - context = "notes", - scope(field = "tenant"), - bind(field = "id"), - bind(field = "written_from"), - bind(field = "written_at"), -)] +#[stash(struct = Note, context = "notes", bind(field = "id"))] struct EncryptedNote { #[stash(passthrough)] - tenant: u64, - #[stash(passthrough)] - id: Uuid, - #[stash(passthrough)] - written_from: IpAddr, - #[stash(passthrough)] - written_at: Timestamp, - body: Ciphertext, + id: Uuid, // the row's primary key column + #[stash(bind(field = "written_from", in = metadata))] + #[stash(bind(field = "written_at", in = metadata))] + body: TextSearch, // one EQL column: ciphertext, terms, and the metadata } -let note = Note { tenant, id: Uuid::new_v4(), written_from: request.ip(), written_at: now(), body }; +let note = Note { id: Uuid::new_v4(), written_from: request.ip(), written_at: now(), body }; let encrypted: EncryptedNote = note.encrypt_into(&keyset).await?; -// The reader passes nothing. Every part comes from the row, and ZeroKMS -// logs the whole context when it returns the key. -let note = Note::decrypt_from(encrypted, &keyset).await?; +// The payload stored in the `body` column, with the metadata in `m`: +// {"v": 1, "i": {"t": "notes", "c": "body"}, "c": "...", "bf": [...], +// "m": {"written_from": "203.0.113.5", "written_at": "2026-11-07T10:15:00Z"}} -// A reader that serves one tenant and one row checks both. -let note = Note::decrypt_from_with_context( - encrypted, - &keyset, - Scope::new(session.tenant).bind(requested_id), -) -.await?; +// `SELECT id, body FROM notes WHERE id = $1` is enough. The reader passes +// nothing: the row ID comes from its column, and the address and the +// time come from the payload. ZeroKMS logs all three with the key. +let note = Note::decrypt_from(encrypted, &keyset).await?; ``` -Two limits apply. The descriptor has a length limit at ZeroKMS, so the -bindings must stay short. And the descriptor is sent to ZeroKMS, so the -address and the time leave the customer's system. For access -intelligence that is the point. A customer should still know it. +ZeroKMS today limits the length of a descriptor, and logs it in the +clear. ZeroKMS version 2 is planned to remove the limit and to encrypt +the access log. ## Scenarios that were removed From c6dcfb38d8918437f94ebdd3ceace74951d48b89 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:39:49 +1100 Subject: [PATCH 39/42] docs: drop the removed-scenarios note from the context examples --- docs/plans/2026-10-24-stack-encrypt-context-examples.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index fc5e96bc9..87c436fff 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -295,11 +295,3 @@ let note = Note::decrypt_from(encrypted, &keyset).await?; ZeroKMS today limits the length of a descriptor, and logs it in the clear. ZeroKMS version 2 is planned to remove the limit and to encrypt the access log. - -## Scenarios that were removed - -Two earlier scenarios sealed a value that was also the context of another -field, and stored no copy of it in the clear. A reader needed the -plaintext before it could open anything, so the sealed copy protected -nothing. Both are gone. A value the caller already knows is a -caller-supplied part, as in scenarios 2, 6, and 7. From 7726403cc30393d85fc08690ac2a62cfc28fc5bf Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:41:37 +1100 Subject: [PATCH 40/42] docs: scenario 10, the access intelligence shape without EQL A target type of our own stores the ciphertext, a match term, and the bindings as passthrough fields, as one column through Serde. EQL adds only the PostgreSQL domain and operators. The plan's metadata item links the two forms. --- .../2026-10-24-stack-encrypt-api-plan.md | 8 ++- ...26-10-24-stack-encrypt-context-examples.md | 51 ++++++++++++++++++- 2 files changed, 56 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index c7bc97e3e..fe6f7cec7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -1039,6 +1039,10 @@ format version beside it. decrypt the value. Inside the payload, one column stores the value and its bindings, so `SELECT body` is enough. Scenario 9 of the [context scenarios](2026-10-24-stack-encrypt-context-examples.md) shows it. +- **Without EQL.** The same shape works in a target type of our own, with + the bindings as passthrough fields of the target. Scenario 10 shows it. + That needs bindings declared on a target type, from the [record + derives](#record-derives) item, and no change to EQL. - **Stored format.** The field is optional, so a payload without it is unchanged. A payload with it is a new form beside the rc.3 form. - **Done already.** Nothing. @@ -1199,7 +1203,7 @@ review. bytes. - **Scenarios.** The [context examples](2026-10-24-stack-encrypt-context-examples.md) are the - acceptance matrix: nine scenarios, from a static context alone to the + acceptance matrix: ten scenarios, from a static context alone to the bindings that access intelligence stores. Two earlier scenarios were removed, because each sealed a value that was also another field's context, so nothing could open without the plaintext. @@ -1219,7 +1223,7 @@ review. compile time. `RequiredContext`, `OptionalContext`, and `NoCallerContext` (#1404) are the three states this keeps. - **Finished when** these are both true: - - all nine scenarios run as tests in Rust and Go, through the + - all ten scenarios run as tests in Rust and Go, through the builder; - a Rust call with an undeclared part fails to compile. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 87c436fff..ec8480147 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -2,7 +2,7 @@ Date: 7 November 2026. -This document lists nine ways a record can get its context, each as a +This document lists ten ways a record can get its context, each as a code example with its use case. Together they are the acceptance matrix for the work item [Context from fields and from the caller](2026-10-24-stack-encrypt-api-plan.md#context-from-fields-and-from-the-caller) @@ -295,3 +295,52 @@ let note = Note::decrypt_from(encrypted, &keyset).await?; ZeroKMS today limits the length of a descriptor, and logs it in the clear. ZeroKMS version 2 is planned to remove the limit and to encrypt the access log. + +## 10. The same bindings, in a type of our own instead of EQL + +The shape of scenario 9 does not depend on EQL. A target type of our +own can store the ciphertext, its search terms, and its bindings as one +value. It needs only the derives Stack Encrypt already has or plans. + +The bindings are passthrough fields of the target, stored in the clear inside +it. They are part of the stored context. One column stores +the whole value, through Serde, and the reader passes nothing. + +What EQL adds is the PostgreSQL side: a domain for the column, and the +operators that run a search on it. A type of our own has neither, so a +query compares the stored term in SQL that the application writes. + +```rust +#[derive(Zeroize)] +struct Written { + text: String, + from: IpAddr, + at: Timestamp, +} + +/// One stored value: the ciphertext, a match term, and two bindings. +#[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] +#[stash(plaintext = Written, bind(field = "from"), bind(field = "at"))] +struct SealedNote { + ciphertext: Ciphertext, + terms: MatchTerm, + #[stash(passthrough)] + from: IpAddr, + #[stash(passthrough)] + at: Timestamp, +} + +#[derive(EncryptTarget, DecryptSource)] +#[stash(struct = Note, context = "notes", bind(field = "id"))] +struct EncryptedNote { + #[stash(passthrough)] + id: Uuid, + body: SealedNote, // one column, stored as JSON through Serde +} + +let note = Note { id: Uuid::new_v4(), body: Written { text, from: request.ip(), at: now() } }; +let encrypted: EncryptedNote = note.encrypt_into(&keyset).await?; + +// `SELECT id, body FROM notes WHERE id = $1` is enough, as in scenario 9. +let note = Note::decrypt_from(encrypted, &keyset).await?; +``` From 65a2c4428d5c04f77ca6fe22fb129a7492d9b53f Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:43:56 +1100 Subject: [PATCH 41/42] docs: drop the match term from scenario 10 --- .../2026-10-24-stack-encrypt-context-examples.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index ec8480147..7d080720b 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -299,16 +299,16 @@ the access log. ## 10. The same bindings, in a type of our own instead of EQL The shape of scenario 9 does not depend on EQL. A target type of our -own can store the ciphertext, its search terms, and its bindings as one -value. It needs only the derives Stack Encrypt already has or plans. +own can store the ciphertext and its bindings as one value. It needs only the derives Stack Encrypt already has or plans. The bindings are passthrough fields of the target, stored in the clear inside it. They are part of the stored context. One column stores the whole value, through Serde, and the reader passes nothing. What EQL adds is the PostgreSQL side: a domain for the column, and the -operators that run a search on it. A type of our own has neither, so a -query compares the stored term in SQL that the application writes. +operators that run a search on it. A type of our own has neither. Add a +search term to it only if the application writes the SQL that compares +the term. ```rust #[derive(Zeroize)] @@ -318,12 +318,11 @@ struct Written { at: Timestamp, } -/// One stored value: the ciphertext, a match term, and two bindings. +/// One stored value: the ciphertext and two bindings. #[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] #[stash(plaintext = Written, bind(field = "from"), bind(field = "at"))] struct SealedNote { ciphertext: Ciphertext, - terms: MatchTerm, #[stash(passthrough)] from: IpAddr, #[stash(passthrough)] From 7af365d43fda8c840b52d29ba21cf8a6573076e9 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 7 Nov 2026 19:45:40 +1100 Subject: [PATCH 42/42] docs: rename SealedNote to EncryptedNoteBody in scenario 10 --- docs/plans/2026-10-24-stack-encrypt-context-examples.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md index 7d080720b..e7461bda5 100644 --- a/docs/plans/2026-10-24-stack-encrypt-context-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -321,7 +321,7 @@ struct Written { /// One stored value: the ciphertext and two bindings. #[derive(EncryptTarget, DecryptSource, Serialize, Deserialize)] #[stash(plaintext = Written, bind(field = "from"), bind(field = "at"))] -struct SealedNote { +struct EncryptedNoteBody { ciphertext: Ciphertext, #[stash(passthrough)] from: IpAddr, @@ -334,7 +334,7 @@ struct SealedNote { struct EncryptedNote { #[stash(passthrough)] id: Uuid, - body: SealedNote, // one column, stored as JSON through Serde + body: EncryptedNoteBody, // one column, stored as JSON through Serde } let note = Note { id: Uuid::new_v4(), body: Written { text, from: request.ip(), at: now() } };