feat(golang, stack-encrypt): optional sealed fields, stored as a sealed null or SQL NULL by coderdan · Pull Request #1328 · cipherstash/stack

coderdan · 2026-10-21T09:27:05Z

Summary

The Go SDK can now encrypt an optional field. A field declared over a pointer, such as DOB *encrypt.Date tagged stash:"dob,encrypt_into=Date", generates code, and Decrypt returns nil for a nil. Until now stashgen refused such a field as a composite, as #1293 showed. stashgen is the code generator that reads stash:"…" struct tags.

There are two ways to store nil, and they differ in what someone reading the database can see:

Sealed (null=sealed): nil is encrypted like a value. vitaminc 0.8 pads it to the length of a value of the field's type, so a row with no value looks like any other row.
- Beside an equality index, a null stores its own equality term, the one term every null of the field shares. The absent rows are then one equality group, as rows holding one repeated value are, and a query for null finds them: the generated EqNull, the eq query of nil.
- This is the default for a field whose type has a fixed width (bools, numbers, dates, timestamps, decimals, the 128-bit integers) and whose only index, if it has one, is equality.
SQL NULL (null=sql): nothing is stored, and the field's columns are NULL. Anyone reading the table can see which rows have no value.
- A field with an order or match index, and a field of text or bytes, must write null=sql explicitly. A null has no order or match term, and text shows its length anyway.
- stashgen never chooses a visible NULL silently.

Stacked on #1327 (the vitaminc 0.8 / ORE upgrade, now on the released clww-ore 0.6.0 and ore-rs 0.10.0). When #1327 merges, this PR's base becomes next.

Changes

Stack Encrypt (Rust engine)

A null has an equality term. vitaminc-normalize 0.8 derives one per context, shared by every null, typed or plain. chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327 dropped it to keep the old behaviour; this PR keeps it. It applies on the typed path (EqualityTerm of a None) and the dynamic one (IndexSpec::Equality on a null). No other index has a term for a null.
A data plan field spec takes "null": "sealed" or "null": "sql" (NullMode, FieldPlan::with_null).
"sealed" seals a null as a typed null of the field's declared type, and stores the null's equality term beside it when the field has an equality index. It is refused on:
- a field, or an EQL type, with any index other than equality (SealedNullIndex);
- a field with no type, or a type of variable width (SealedNullType);
- a passthrough field (NullPassthrough).
"sql" leaves the field out of the stored record. A source may omit the field, and a stored record without it opens it as null.
An eq query whose input is a null derives the null term, through query_term or record::query, on a "sealed" field. Any other null query is refused with the new NullQuery; a "sql" field's nulls are SQL NULL, which IS NULL finds.
Rows of a batch are grouped by which fields they leave out; one lowered plan runs per group, and all groups' requests go in one ZeroKMS call.
A field with no "null" behaves exactly as before.

eql-bindings (EQL v4 types in Rust)

Two kinds of type now also derive EncryptFrom and DecryptInto for Option<plaintext>: the storage-only types (Date, Integer, Text, …) and the types whose only term is equality (DateEq, IntegerEq, TextEq, …). The equality types' query twins also take Option<plaintext>. These come from a codegen change; the generated files were regenerated.
The by-name dispatch that the Go guest runs (encryption::targets) encrypts, opens and queries these types through that Option.
- A value seals exactly the leaf it sealed before.
- A null seals as a typed null, with the null's hm where the type has one, and opens as a null.
- The query of a null derives that hm.
eql-bindings' own Date, Timestamp and Decimal plaintexts now declare their fixed width to vitaminc, so their None is padded too.

Go SDK

stashgen:
- takes *T for a sealed scalar field;
- reads null=sealed|sql, picks sealed where it hides the absence (fixed width, equality the only index), and otherwise refuses with a reason;
- generates EqNull(ctx, c) on a sealed nullable field that answers eq, through the new gensupport.Field.NullTerm;
- still refuses **T, pointers to structs and slices, and null= on a field that is not a pointer or on a passthrough.
For a null=sql field, the generated member is a pointer (*eql.DateEq). Seal and Open skip the field when it is nil, and Value reads it with the new gensupport.GetPtr.
gensupport gains Declaration.Null, Deref, GetPtr and Field.NullTerm. internal/record and the encrypt record path carry the mode and accept a null=sql field that is left out.
A stored null read into a field that is not a pointer is a Decrypt error, from the engine's kind check.
-model does not map a null=sql field yet, and stashgen says so instead of generating code that would dereference nil.

Docs

stashgen README: a new "Optional fields" section, with a table of what each mode stores and what a reader of the column can see.
encrypt.Date and encrypt.Ciphertext docs, the stack-encrypt and Go CHANGELOGs, and a @cipherstash/eql patch changeset (eql-bindings ships at the npm package's version).

Verification

Root workspace: cargo nextest run --workspace --all-features passes (1267 tests), and clippy -D warnings is clean. New engine tests:
- plan parsing and every refusal;
- a null's equality term is one term for every type and differs from every value's, on both paths, and a null has no order or match term;
- a sealed null beside an equality index stores the shared term, and an eq query of null derives it; null queries on a "sql" field, on an order operator, or through another index are refused;
- a sealed null is exactly as long as a value for a date, a float64 and a bool, and opens as null;
- a "sql" batch of four rows with mixed nulls stores no entry for the left-out field, makes one key request and opens nulls;
- a row with every field left out;
- a stored record may leave out only a "sql" field.
EQL workspace:
- cargo test -p eql-codegen passes, with the dispatch tests updated; so do -p eql-bindings --all-features and -p eql-encryption-tests.
- mise run types:check shows no drift.
- New tests:
  - a Date stores None as long as a value on the typed and by-name paths, and each path opens the other's;
  - a DateEq null has the null hm, which the query of a None derives, and a type with an order term refuses a null;
  - an end-to-end record through the resolver the Go guest installs (Date sealed null, DateEq SQL null).
PostgreSQL 15, mise run test:encryption:postgres, which includes the new database test:
- public.eql_v4_date and public.eql_v4_date_eq accept a value whose c is a sealed null, and it reads back and opens as None. Go SDK: stashgen refuses a pointer field such as *encrypt.Date, so an optional value cannot be encrypted #1298 had listed this as unverified.
- WHERE due = <query of null>::eql_v4.query_date_eq returns exactly the rows whose value was absent.
Go: all four guests rebuilt; mise run go:test and mise run go:lint pass, and go generate ./... changes no committed file.
- cmd/stashgen: the test(stashgen): reproduce nullable DOB generation failure #1293 reproduction, now skipped when the guest is not built, passes for encrypt.Date and *encrypt.Date.
- encrypt: TestOptionalFieldsStoreNilAsTheirTagsSay round-trips a Person with both modes over the deterministic EQL guest.
  - A nil DOB's, Born's and Age's ciphertexts are as long as present ones.
  - Every nil Born (DateEq) and Age (index=equality) shares one equality term that no value has, and EqNull derives it.
  - The null=sql members are nil, and every nil decrypts to nil.
  - Reading the sealed null into a non-pointer encrypt.Date is an error.
- stashgen: 12 new refusal cases.
cargo mutants --in-diff for stack-encrypt:
- Against chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327: 81 mutants, 50 caught, 29 unviable, 2 missed. Both survivors were real gaps: a sealed null on an EQL type with a term was never shown refused, and a stored null was never read through a plan with no "null". Each now has a test, and a rerun of the two catches both.
- On the null-term commit: 21 mutants, 12 caught, 6 unviable, 3 missed, all in record::query's null check, which no test reached. A new test catches all three on a rerun.

Review notes

Start with packages/stack-encrypt/src/dynamic/record.rs (NullMode, source_row, by_absent and in_row_order, and the grouped encrypt_with / decrypt_with), then languages/golang/stashgen/read.go (nullMode) and the generated encrypt/internal/testusers/person_stash.go.
A null's equality term is in; order terms for a null are deliberately not. Fields with an order index take null=sql, and that is the intended design, not a gap to fill later. vitaminc-ore 0.8 can place nulls at one end of an order space, but a sealed null there buys nothing:
- Every null gets the same order term at one end of the order, so anyone holding the column can see the absent rows as the tied group at that end. That is what SQL NULL shows.
- Range queries on the near side would then match the nulls, so each one would need a bound that excludes the null's term, and that bound in every query confirms which group is the nulls.
- Filtering on the client instead gives wrong counts and LIMITs, and fails when the column is not selected.
What a sealed null beside equality reveals: the absent rows share one equality term, so they are one group, as any repeated value is. Someone who knows one row is absent learns which others are. That is less than SQL NULL, which shows every absent row with no inference.
Variable-width types (text, bytes) take null=sql only, as Go SDK: stashgen refuses a pointer field such as *encrypt.Date, so an optional value cannot be encrypted #1298 scoped. eql-bindings' Text does accept Option<String>, because every storage-only type is generated alike, but the engine refuses a sealed null on a variable-width type.
Mutants: KindOf::typed_null's default mutant is excluded in chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327 as equivalent. It is reachable now, since ValueFrame::typed_null frames reach it, and it is still equivalent.
A stored null in a field whose plan has no "null" is refused when opened (FieldType), so reading a nullable column through a declaration that is not nullable fails loudly.

https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

changeset-bot · 2026-10-21T09:27:13Z

🦋 Changeset detected

Latest commit: 12f9e83

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Patch
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-10-21T09:27:15Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-21T09:42:33Z

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
62	0	32	0

Every mutant in the changed lines was caught by a test.

coderdan · 2026-10-23T01:24:12Z

Rebased onto #1327, which now takes clww-ore 0.6.0 and ore-rs 0.10.0 from crates.io instead of git. The commits on this branch are unchanged apart from their hashes. Rerun after the rebase: root nextest (1267 passed, 1 skipped), clippy, the EQL crate tests, all four Go guests and go:test; go generate ./... changes nothing.

coderdan · 2026-10-23T01:56:45Z

Two pushes since the last note:

test(stack-encrypt): a sealed null is as long as a value of every fixed-width kind fixes the failing CRAP gate. ValueKind::sealed_null (complexity 18) had only its date, float64 and bool arms tested. The length test now seals a value and a null of every kind it maps, plus a decimal given in 64 bits, whose null pads to 128 bits as the value does. The local CRAP run passes, with no function over 30.
Rebased again onto chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327, after its review fixes. One conflict, in the stashgen README, was resolved by taking this branch's "Optional fields" wording together with chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327's corrected Stack Encrypt cannot order floats, dates, timestamps or decimals, and its integer order keys predate ADR-0002: derive every order term over vitaminc-ore #1236 sentence. Rerun after the rebase: root nextest (1267 passed, 1 skipped), clippy, all four guests and go:test; go generate ./... changes nothing.

coderdan · 2026-10-23T02:39:37Z

Rebased onto #1327 again, after its review changes: stack-encrypt now re-exports vitaminc for bindings, and the Go guest takes vitaminc only through those re-exports. There was one conflict, in the stack-encrypt CHANGELOG, and both entries are kept. Rerun after the rebase: root nextest (1267 passed, 1 skipped), clippy, mise run wasm:guest:test, all four guests and go:test; go generate ./... changes nothing.

A plan field may now carry "null": "sealed" or "sql" (#1298), so an optional value can be encrypted without a sentinel. - "sealed" seals a null like a value, under the field's label, as vitaminc 0.8's typed null of the field's type, so its ciphertext is as long as a present value's and the stored record does not show which rows are absent. It opens as a null. The field must declare a type of fixed width (a bool's plain null is already as long as its value) and derive no term, since a null has none yet. - "sql" stores nothing for a null, or for a row with no entry for the field: the stored record has no entry, so the host writes SQL NULL. A stored record with no entry for the field opens it as a null. Rows of a batch are run by one lowered plan per set of fields they leave out, and the groups' requests are merged into the one call. A field with no "null" behaves as before: every row must have it, and a typed field refuses a null. Refs #1298. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

…a value A storage-only EQL v4 type (Date, Integer, Text, ...) carries no term, so an absent value has nothing that would show it but the length of `c`. Each now also derives EncryptFrom and DecryptInto for an Option of its plaintext, and the by-name dispatch the Go guest runs reads and opens a storage-only type through that Option: a value seals exactly the leaf the plaintext seals (Some walks the value itself), and a null seals as vitaminc 0.8's typed null, as long as a value, and opens as a null. A type with a term still refuses a null. eql-bindings' own Date, Timestamp and Decimal plaintexts declare their fixed width to vitaminc, so their None is padded too. The codegen emits the second plaintext attribute and the Option dispatch; the generated files are regenerated with `mise run types:generate`. Whether a field may hold a null is its plan's to say (#1298): the engine refuses a null for a target field without "null". Refs #1298. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

A sealed field may now be declared over a pointer to one scalar (`DOB *encrypt.Date` with `stash:"dob,encrypt_into=Date"`, which #1293 reproduced being refused as a composite), and Decrypt gives nil back for a nil (#1298). A new tag part, `null=sealed` or `null=sql`, says how nil is stored, and reaches the engine as the data plan's "null": - sealed: nil is sealed as a typed null as long as a value, so the column does not show which rows are absent. stashgen picks it for a field with no index whose type has a fixed width, sealed with encrypt or into a storage-only EQL type. - sql: nothing is stored, so the field's columns hold SQL NULL and its member in the generated type is a pointer. A field with an index, or of text or bytes, has to say so: a null has no term yet, and text shows its length, so stashgen will not choose a visible NULL silently. gensupport gains Declaration.Null, Deref and GetPtr; the record model and the encrypt record path carry the mode and accept a SQLNull field left out. A stored null read into a field that is not a pointer is a Decrypt error, from the engine's kind check. -model does not map a null=sql field yet, and stashgen says so. The reproduction from #1293 is folded into cmd/stashgen's tests, with the guest-built skip the other engine test has. Closes #1293. Refs #1298. Co-authored-by: Jonathon Walker <1989282+jonathon-walker@users.noreply.github.com> Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

The domain check of eql_v4_date takes a value whose c is a typed null, as it takes a date, and it reads back and opens as None (#1298). Runs in mise run test:encryption:postgres with the file's other database tests. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

…y through a nullable plan The two survivors of cargo mutants --in-diff on the null modes (#1298): no test refused "null": "sealed" on an EQL type with a term, and none read a stored null through a plan with no "null", where the field's kind check must refuse it rather than skip. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

vitaminc-normalize 0.8 derives an equality term for a null: one term per context, shared by every null, typed or plain. #1327 dropped it to keep the old behaviour; this keeps it (#1298). - stack-encrypt: `EqualityTerm` of a `None`, and `IndexSpec::Equality` on a null, now derive that term. No other index has one. - Data plans: `"null": "sealed"` takes a field whose only index is equality, on the field or on the EQL type it names. The null is sealed as long as a value and stores the null term beside it, so the absent rows are one equality group, as any repeated value is, and the column holds no SQL NULL. An `eq` query whose input is a null derives that term (`query_term`, `record::query`); a null query of any other field or operator is the new `Reason::NullQuery`. - eql-bindings: a type whose only term is equality's (`DateEq`, `IntegerEq`, ...) and its query twin take an `Option` of the plaintext, as storage-only types do, and the by-name dispatch encrypts, opens and queries them through it. - Go: stashgen defaults to `null=sealed` for a fixed-width field whose only index is equality, and generates `EqNull`, the `eq` query of nil, through the new `gensupport.Field.NullTerm`. Verified in PostgreSQL: `=` with the query of a null finds exactly the rows of a DateEq column whose value was a sealed null. Refs #1298. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

…d equality The three survivors of cargo mutants on the null term: record::query's refusal of a null was reached by no test, since query_term refuses the same input before it. Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

…ed-width kind The length test covered a date, a float64 and a bool, so half of `ValueKind::sealed_null`'s arms ran in no test and the CRAP gate failed on it (CC 18, 47.6% covered). It now seals a value and a null of every kind the function maps, each integer and float width, a decimal, a date and a timestamp, and checks each pair is the same length and opens back. A decimal given in 64 bits is included: its null is padded to 128 bits, so the value had to be too. It is, and it opens in 128 bits. Refs #1298 Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

… an absent value Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

coderdan mentioned this pull request Oct 21, 2026

chore(stack-encrypt)!: move to vitaminc 0.8, clww-ore 0.6 and ore-rs 0.10 #1327

Open

coderdan force-pushed the feat/nullable-sealed-fields branch from 1701131 to 7938f8f Compare October 23, 2026 01:24

coderdan force-pushed the feat/nullable-sealed-fields branch from 7601d3f to 368f303 Compare October 23, 2026 01:56

coderdan force-pushed the feat/nullable-sealed-fields branch from 368f303 to 0deb2b1 Compare October 23, 2026 02:39

coderdan and others added 9 commits October 23, 2026 13:46

docs(stack-encrypt): point from_bytes at the plan null mode that pads…

12f9e83

… an absent value Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

coderdan force-pushed the feat/nullable-sealed-fields branch from 0deb2b1 to 12f9e83 Compare October 23, 2026 02:46

coderdan mentioned this pull request Oct 23, 2026

feat(stack-encrypt, eql-bindings, golang): order dates, timestamps and decimals #1337

Draft

Navigation Menu

Repository navigation

feat(golang, stack-encrypt): optional sealed fields, stored as a sealed null or SQL NULL - #1328

feat(golang, stack-encrypt): optional sealed fields, stored as a sealed null or SQL NULL#1328
coderdan wants to merge 9 commits into
chore/vitaminc-0-8-ore-0-10from
feat/nullable-sealed-fields

coderdan commented Oct 21, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 21, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

coderdan commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

Related

Review notes

Uh oh!

changeset-bot Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Oct 21, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

coderdan commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderdan commented Oct 21, 2026 •

edited

Loading

changeset-bot Bot commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)