Repository navigation
Conversation
🦋 Changeset detectedLatest commit: 488a100 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 63 | 0 | 29 | 0 |
Every mutant in the changed lines was caught by a test.
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Review of 4799dd1 against feat/eql-v4-text-search-optional-terms (4430c0c). No blocking defect found. The SQL comparator reproduces ore-rs 0.9.0's comparators exactly, and every check I could run without credentials passes, including the PostgreSQL-backed suite and the timing harness on a scratch PostgreSQL 15.12. Three low-severity notes follow, each inline; nothing here needs to hold the PR.
Findings (most severe first)
-
Low — the shape check is about half of every comparison.
ore_block_v2_schemeruns twice percompare_ore_block_v2_termcall and each run issues two SPI sub-selects (the symbol scan and the padding-bucketEXISTS). Measured here on ani32-shaped term: scheme check 3.6 µs, hash bit 2.6 µs, whole comparison 14.6 µs (20,000 calls each, PG 15.12, Apple silicon); v3's comparator on a 408-byte term takes 3.9 µs on its equal path. A sort or btree build re-validates the same stored term at every comparison it takes part in. Both scans can be plain expressions; details inline. -
Low — the 1023-byte ceiling is enforced at encryption only; the comparator admits any count the policy allows, up to the wire maximum. That is what ore-rs does too, so this is a note rather than a divergence. A well-formed chained term of 65535 blocks (1,638,398 bytes) needs no key to craft, since every shape check is on public structure; one comparison of it takes ~20 ms here, and every comparison a btree takes against that row pays it. It needs write access to the table, which already allows worse, so this is informational unless you want
ore_block_v2_schemeto bound the count to what Stack Encrypt can write (1,366 blocks for a 1,024-byte bucket). -
Low — residual data-dependent work in the comparator loop, already acknowledged in the doc comment.
substr(...) <> substr(...)is amemcmpthat stops at the first differing byte, and the two post-loop reads atfirst_diffare at a secret-dependent offset. Your harness and my run of it both put every |t| under 5, so this is a suggestion: the tags can be compared as twobiginthalves XORed (constant work), and the right block and tag can be latched arithmetically inside the loop so nothing after it is indexed by the first difference, which is what ore-rs'sFirstDiffdoes. Inline on the loop.
Housekeeping, not a defect: tests/timing/ore_block_v2/ctgen is a fourth detached Cargo workspace with its own Cargo.lock; AGENTS.md's list of detached workspaces (three fuzz crates and two Go guests) and .github/dependabot.yml do not know about it yet. Inline on its Cargo.toml.
What I verified as sound
Comparator against ore-rs 0.9.0 (read src/scheme/bit2_w6.rs, chained.rs, width.rs, primitives/hash.rs, ciphertext.rs, vitaminc.rs from the registry copy the lockfile names, and the head functions.sql side by side):
- Layout and offsets: header 4 (
02 02) or 7 (02 03+ 3-byte policy), thencountsymbols,count16-byte tags, 16-byte nonce,count8-byte right blocks;nonce_b = header + 17*cb, right block k at 1-basednonce_b + 17 + 8k, tag k ofaatheader + ca + 16k + 1. All matchtotal_len,f_at,nonce_at,right_at. - Lewi-Wu split:
a'sxt/f,b's nonce and right block, asFirstDiff::latch(a[n], left_block(a_f,n), right_block(b_right,n), first). - Bit extraction: PostgreSQL
get_bit(bytea, n)is byten/8, bitn%8LSB-first, which isct_select_byte(right, xt/8)thenct_bit(byte, xt%8), and the same orderlsb_mask/indicator_mask_xorwrite. - σ-MMO: GF(2^128) doubling big-endian with the lo top bit carried into hi and
0x87folded on hi's top bit, XOR nonce, AES-128-ECB under the publicORE-rs.v2.H-pi.1,lsb(π(m)) ⊕ lsb(m);aes-ecb/pad:noneon an exact 16-byte block. The 12 vectors pass here. - Refusal set equals ore-rs's: version/scheme id,
count ≥ 1,≤ 14for the fixed scheme, exact length,Padding::from_wire(Noneonly withn = 0,padded_len(0) ≤ 49151) andadmits_count(kin0..=15,bucket > min_bytes,ceil(8n/6) = (8n+5)/6), every symbol< 64, different schemes, different fixed counts, different policies. ore-rs returnsNone; the SQL raises, naming only the length. - Chained: lexicographic over
min(ca, cb), shorter first on a full match (ca.cmp(&cb)=sign(ca - cb)); for the fixed schemeca = cbis enforced so that term is 0. - Result selection and the latch are arithmetic; the hash runs on every path.
Security
- Key derivation:
ORE_BLOCK_KEY_DOMAIN(stack-encrypt/sem/ore-block-key/v1) is distinct from the CLLW domains; the 32-byte index key goes toBit6BlockOre::new/ChainedBlockOre::new, which derive their 16-byte ORE keys under separate labels (block-ore-bit6/key/v2,chained-block-ore/key/v2) and bind scheme, target and context into the term salt through ore-rs'sinit_with_saltstages. Consistent with ADR-0002 as amended;left_halves_are_pinnedandterms_are_keyed_by_their_contexthold it. - Nonce: 16 bytes from
ChaCha20Rng::from_entropy()inside ore-rs, per term. - Wiping: the plaintext travels as
Carried<T>; ore-rs frames text/bytes in aZeroizingcopy and scrubs the stack;heap_hygienereads the producible text targets from the table, soTextOrdOreandTextSearchOreare covered without an edit. - Limits: 1023 =
DEFAULT_MAX_STR_BYTES− the marker byte; trailing NUL refused because NUL padding would alias"a\0"and"a". Both are enforced at derivation (ore-rs'sTooLong/TrailingNul, mapped to the newTermErrors) and by the key-freeore_block_textcheck on the query path, both bounds included. Neither message names the value. - Error messages in SQL name a length or a cardinality, never bytes; the test asserts it.
Dependencies and release
ore-rs = "0.9.0"from crates.io (default-features = false,vitaminc), not git or path. TheCargo.lockchurn in the root, EQL, Go encrypt guest and stack-encrypt fuzz workspaces is ore-rs and its dependencies (block-modes,byteorder,num*,orderable-bytes,subtle-ng) plusvitaminc-*0.6.0→0.6.1, which ore-rs 0.9.0 requires. protect-ffi's and the auth guest's lockfiles are untouched, so no WASM-unknown build picks upgetrandom.cargo metadata --lockedpasses in the root,packages/eql, both Go guests,packages/stack-encrypt/fuzzandctgen;packages/stack-auth/fuzzfails on the base too, as the PR says.- eql-bindings pins
stack-encrypt = "0.3.0", which crates.io does not have yet (max 0.2.0); pre-existing on the base, and theIndexSpec::OreBlockbreak sits under[Unreleased]in stack-encrypt's CHANGELOG, so the ordering stays "stack-encrypt 0.3.0 before eql-bindings".
v3 bundle and v4 objects
mise run --force buildat base and head:cipherstash-encrypt.sqlandcipherstash-encrypt-uninstall.sqlare byte-identical (8eccb1cb…,b1b5131b…). The v4 bundle namesore_block_256only in@filepath comments (5) andore_block_v2173 times; installed,eql_v4_internalhas noore_block_256object, andore_block_v2_operator_classis the default btree opclass for the type.{{ore_block}}refusal:the_block_ore_name_follows_the_versionpasses (literal refused except assem/ore_block_256/).
Breaking surface and docs: CHANGELOGs (stack-encrypt breaking entry under Unreleased, eql-bindings, Go), ADR-0002 and ADR-0009 amendments, the @cipherstash/eql minor changeset. The skills name only v3 objects, so leaving them is right. stored_rows.rs and record_lowering.rs are not in the diff and the suite passes.
Checks run
mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 695 passed, 1 skipped.packages/eql:cargo test --locked -p eql-codegen(162),-p eql-bindings --all-features,-p eql-encryption-tests;mise run types:check.go test ./stashgen/... ./cmd/stashgen/... ./internal/...: ok.- Scratch PostgreSQL 15.12 (mise's; Docker was down) with the head v4 bundle: the
--ignoredsuitesv4_ore_block(5),text_eq,v4_types,v4_order,v4_text_searchall pass, includingEXPLAINshowing the index for<andORDER BYthroughord_term_ore. tests/timing/ore_block_v2/run.sh 20000: 0 wrong of 1,500; t = +0.93/+1.21 (fixed first vs last), −2.07/−3.41 (equal vs first), +2.48/+2.41 (chained first vs last), −2.14/−0.73 (prefix vs first).- CI: all checks green at this head. No prior reviews or review comments on the PR.
| END IF; | ||
| END IF; | ||
| IF EXISTS ( | ||
| SELECT 1 FROM generate_series(header, header + count - 1) AS i |
There was a problem hiding this comment.
Low (performance). This function runs twice per comparison, and each run does two SPI sub-selects: this symbol scan and the EXISTS (... generate_series(0, 15)) bucket check above. Measured on PG 15.12 with an i32-shaped term, 20,000 calls each: ore_block_v2_scheme 3.6 µs, ore_block_v2_hash_bit 2.6 µs, the whole compare_ore_block_v2_term 14.6 µs, so the two shape checks are roughly half of each comparison, and a sort or index build re-validates the same stored term every time it is compared.
Both can be plain expressions with no sub-select: the symbol check as encode(substr(t, header + 1, count), 'hex') ~ '^([0-3][0-9a-f])*$' (a byte is below 64 exactly when its high hex digit is 0–3), and the power-of-two admission as arithmetic on the one candidate length: n := 6 * count / 8 is the largest length with (8n+5)/6 = count when it exists, so count = (8*policy+5)/6 OR (n > policy AND n & (n - 1) = 0 AND count = (8*n+5)/6). Not a blocker; v4 is new and the opclass path was never cheap.
There was a problem hiding this comment.
Not changed in this PR. Both suggestions look right: a regex over the hex for the symbol check, and arithmetic on the one candidate length for the bucket check. But they change the comparator's shape checks, and those need the PostgreSQL pairwise and timing runs again. I've left the call to Dan: take it here, or as a follow-up.
There was a problem hiding this comment.
Done in 79b5689. Both shape checks are now expressions, with no sub-select:
- Symbols:
encode(substr(t, header + 1, count), 'hex') !~ '^([0-3][0-9a-f])*$'. - Power-of-two admission:
ceil(8n / 6)grows withn, so the one length that could givecountblocks isfloor(6 * count / 8). The check is that this length is above the floor, is a power of two, and gives backcount. The floor's own bucket and the other kinds are unchanged.
To check that it admits exactly what the old one did, I installed the old function beside the new one and ran both over 1,946,610 headers: every padding kind; policies 0 to 600, each power of two up to 2^15 ±2, and 49151, 49152 and 65535; and counts 0 to 1500 plus large ones. They agree everywhere, except the 2,032 terms past the new 1,366-block bound (see the next thread). They also agree on 20,000 fixed-width terms with random symbol bytes (8,381 admitted, 11,619 refused). The new test postgres_admits_the_block_counts_ore_rs_does_up_to_the_bound uses ore-rs's own parser as the oracle over a grid like that.
Measured on PG 17.5, best of 7 interleaved rounds of 5,000 calls, old against new (the machine was loaded, so the ratios matter more than the absolutes):
| old | new | |
|---|---|---|
| fixed-width shape check | 2.21 µs | 0.78 µs |
| fixed-width comparison | 8.81 µs | 5.72 µs |
| chained shape check (43 blocks) | 7.36 µs | 1.18 µs |
| chained comparison | 23.66 µs | 10.87 µs |
The timing harness is unchanged: 0 wrong, and |t| ≤ 5.42 at 20k samples (≤ 5.99 before, with the same prefix-against-first-block pattern).
| scheme := get_byte(t, 1); | ||
| header := CASE scheme WHEN 2 THEN 4 ELSE 7 END; | ||
| count := get_byte(t, 2) * 256 + get_byte(t, 3); | ||
| IF count < 1 OR (scheme = 2 AND count > 14) OR len <> header + 25 * count + 16 THEN |
There was a problem hiding this comment.
Low (informational). For the chained scheme this admits every count the policy allows up to the wire maximum, as OreAes128Bit6Chained::compare_raw_slices does, while the 1023-byte ceiling lives only in ore-rs's encryptor and ore_block_text. Every check here is on public structure, so a well-formed term of 65,535 blocks (1,638,398 bytes, random tags and right blocks, symbols below 64) needs no key to craft. I built one on the scratch database: ore_block_v2_scheme accepts it and one compare_ore_block_v2_term(t, t) takes about 20 ms, paid by every comparison a btree takes against that row. It needs write access to the table, which already allows worse, so I am not asking for a change; if you want a bound, the largest term Stack Encrypt writes is a 1,024-byte bucket, 1,366 blocks (Padding::None for i128 is 22).
There was a problem hiding this comment.
Noted, no change. As you say, the largest well-formed term needs no key but does need write access to the table, and the comparator admits exactly what compare_raw_slices does. A cap at the largest bucket Stack Encrypt writes (1,366 blocks) is easy to add if Dan wants one.
There was a problem hiding this comment.
Added in 79b5689. A chained term over 1,366 blocks is now refused as malformed. 1,366 blocks is a 1,024-byte string: ore-rs's default ceiling, which is what Stack Encrypt runs with (it counts the marker byte before text and bytes, so 1,023-byte text, padded to the 1,024 bucket). That covers i128 under Padding::None (22 blocks) too. The doc comment states the bound and why.
Tests (b4acdbf):
postgres_compares_the_longest_term_and_refuses_one_past_it: 1,023-byte text throughTextOrdOreis exactly 1,366 blocks and compares both ways. A well-formed 1,367-block term, which ore-rs's parser accepts, raisesMalformed ORE block termon either side.- The admission grid test requires SQL to admit exactly what ore-rs admits with 1 to 1,366 blocks, and nothing past that.
EQL v4 still refuses a chained term of 0 blocks, as before, though ore-rs parses one. Stack Encrypt never writes one, and the test says so.
| latch := latch + (latch = 0)::int * (block + 1) * ( | ||
| (get_byte(ab, header + block) <> get_byte(bb, header + block))::int | ||
| | (substr(ab, header + ca + 16 * block + 1, 16) | ||
| <> substr(bb, header + cb + 16 * block + 1, 16))::int); |
There was a problem hiding this comment.
Low (suggestion; the doc comment above already names both effects). Two things inside the per-block statement still depend on the data at the C level: bytea <> bytea is a memcmp that stops at the first differing byte, so equal tags cost 16 bytes and unequal ones about one; and after the loop substr(bb, nonce_b + 17 + 8 * first_diff, 8) and substr(ab, header + ca + 16 * first_diff + 1, 16) read at an offset that is the secret first difference, which is the load ore-rs's FirstDiff latch was added to remove. Your harness and my run of it (20k samples, PG 15.12: |t| ≤ 3.41) agree it is below one statement's cost, so this is optional.
If you want the loop fully arithmetic: compare each tag as two bigint halves, (ha # hb) | (la # lb) <> 0, which does the same work whatever the bytes; and latch the values rather than the index, first := (latch = 0)::int * differs, right_l := right_l + first * right_k, tag_hi := tag_hi + first * ha, tag_lo := tag_lo + first * la, xt_l := xt_l + first * xt_k, with right_k/ha/la read at the loop's public offset block on every iteration. The hash then takes int8send(tag_hi) || int8send(tag_lo) and the bit comes from int8send(right_l), and nothing after the loop is indexed by the first difference. It adds about four encode calls per block, so weigh it against finding 1.
There was a problem hiding this comment.
Leaving the loop as is for now. Your run and the harness agree the remaining data-dependent costs are below one statement's cost (|t| ≤ 3.41). The fully arithmetic version would add about four encode calls per block, so it is best weighed together with the shape-check suggestion in the other thread. That is Dan's call.
There was a problem hiding this comment.
Not changed here. Moving the loop fully to arithmetic, so that nothing after it is indexed by the first difference and the tag comparison does the same work whatever the bytes, is tracked in #1281 for the timing-hardening pass. It needs a fresh timing run, and it weighs against the speedup from the first thread.
| rand = "0.8" | ||
| hex = "0.4" | ||
|
|
||
| [workspace] |
There was a problem hiding this comment.
Housekeeping. This is now a fourth detached Cargo workspace with its own Cargo.lock (it passes cargo metadata --locked at this head). AGENTS.md's "Three Cargo workspaces, not one" paragraph lists the three fuzz crates and the two Go guests as the detached ones, and .github/dependabot.yml has no cargo entry for this directory. Since run.sh runs cargo run without --locked, the lock is advisory either way; a line in that AGENTS.md list (and a decision on dependabot) would keep the next reader from wondering which it is.
There was a problem hiding this comment.
Fixed in 55d3d13. AGENTS.md's workspace paragraph now lists the generator (packages/eql/tests/timing/ore_block_v2/ctgen) among the detached workspaces, and says none of them has a Dependabot entry. test-eql.yml's Rust job checks its lock with cargo metadata --locked, since no job builds it, and run.sh builds it with --locked, so the lock is authoritative rather than advisory.
4430c0c to
9048302
Compare
4799dd1 to
59087de
Compare
|
Restacked onto the new #1274 and force-pushed. Besides hashes: smallint has a Rust plaintext now (#1267), so |
59087de to
2189d5a
Compare
9048302 to
30d3a35
Compare
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Incremental re-review of the restack to 2189d5a (onto #1274's 30d3a35). Range-diff against 4799dd1: commits 1 (the engine) and 3 (Go) are identical; commit 2 (the SQL comparator and harness) and commit 4 (the bindings) changed.
Substantive, commit 2, responses to review 5566990967.
- Housekeeping, the
ctgendetached workspace: addressed. AGENTS.md lists it,test-eql.yml'srust-cratesjob runscargo metadata --lockedagainst its manifest (reachable:packages/eql/tests/**is in both the push filter and therelevant:block, and the job's working directory ispackages/eql), andrun.shbuilds it with--locked. One sentence of the AGENTS.md text is wrong; finding 1 below. - The three LOWs on the comparator (the shape check runs twice with two SPI sub-selects each; no upper bound on a chained term's block count; the residual
memcmp/substrdata dependence) are unchanged. The author left each to the maintainer (1, 2, 3). Still open; the comparator body infunctions.sqlis byte-identical to4799dd1.
Rebase adaptations, commit 4 (smallint). SmallintOrdOre is producible, deliberately and consistently. The engine already takes i16 for a block ORE term (sem/mod.rs:1217, i16 => I16, under Bit6BlockOre for targets of at most ten bytes); ENCRYPTION_DOMAINS gains ("smallint","ord_ore") in the same position as the other families; v4/smallint.rs derives it with #[stash(plaintext = i16)]; v4/targets.rs has the TARGETS row (indexes: ["ore_block"], producible) and all three _named arms; eql_gen.go agrees (mise run types:check no drift at this head); the guest's targets.rs and native_ops.rs, eql_test.go, query_test.go (families now includes Smallint), the codegen tests, tests/targets.rs, both READMEs, both CHANGELOGs and the changeset list it. The v4_ore_block.rs changes are this adaptation, not a review response: SmallintOrdOre joins the shape table, and an i16 PostgreSQL agreement check runs over [MIN, -300, -1, 0, 1, 63, 64, 300, MAX], which crosses a 6-bit block boundary at 63/64. The refusal tests move to DateOrdOre with the base's "no Rust plaintext type" wording. The run.sh change is the --locked flag only. No fixture changed.
New findings
- LOW, docs (inline on AGENTS.md): "None of them has a Dependabot entry" is wrong for five of the six detached workspaces.
.github/dependabot.yml(lines 254–260) lists/packages/stack-auth/fuzz,/packages/stack-kms/fuzz,/packages/stack-encrypt/fuzz,/languages/golang/encrypt/guestand/languages/golang/auth/guestunder a lockfile-only cargo entry, held byscripts/__tests__/dependabot-cargo-cross-workspace.test.mjs. Onlyctgenis unlisted, and that file's comment still counts "five detached workspaces". A reader of AGENTS.md would either add a duplicate entry for the fuzz crates or take ctgen's omission as the convention. Either add/packages/eql/tests/timing/ore_block_v2/ctgento thatdirectories:list (it shares no crate by path, so the cross-workspace rule does not bite) and update the count, or reword the sentence to say ctgen alone is unlisted and why. - LOW, coverage (inline on
query_test.go): no Go test seals or queries a block ORE type through the guest.TestTheCatalogListsEachTypesOperatorsreads the catalog only, and notestusersmodel carries an*OrdOreorTextSearchOrefield, so there is no Go-sealed row and noLtterm forobchecked from Go, where the OPE types haveRankingandTestV4Ordered*andTextSearchhasTestV4TextSearchStoresEveryTermAndQueriesCarryOne. The guest's own tests (native_ops.rs) run natively, so ore-rs is never executed on wasm32 anywhere in CI; a wasm32-only panic or a term the guest writes differently from native Rust would surface in a customer program first. A model with anIntegerOrdOreand aTextOrdOrefield, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL the wayRankingis, would close it (the nonce makesobnon-deterministic, so the fixture would compare everything butcandoband checkobthrough the comparator).
Nit: the PR body still says "Smallint's OrdOre is not produced here" and "the six types are producible"; the restack comment has it right.
Checks (at this head, no credentials): cargo test --locked -p eql-codegen -p eql-bindings --all-features (162 + 116 unit, all integration suites pass), -p eql-encryption-tests (v4_ore_block 1 passed, 5 ignored without a database); mise run types:check no drift; mise x --env test -- cargo nextest run -p stack-encrypt --all-features 695 passed, 1 skipped; go test ./stashgen/... ./cmd/stashgen/... ok; go vet clean. CI is green at this head.
| three fuzz crates, the two Go guests and the EQL v4 block ORE timing | ||
| harness's ciphertext generator | ||
| (`packages/eql/tests/timing/ore_block_v2/ctgen`) are detached workspaces | ||
| too, each with its own `Cargo.lock`. None of them has a Dependabot entry. |
There was a problem hiding this comment.
LOW (docs): this is wrong for five of the six. .github/dependabot.yml lines 254–260 list /packages/stack-auth/fuzz, /packages/stack-kms/fuzz, /packages/stack-encrypt/fuzz, /languages/golang/encrypt/guest and /languages/golang/auth/guest under a lockfile-only cargo entry (held by scripts/__tests__/dependabot-cargo-cross-workspace.test.mjs). Only ctgen is unlisted, and that file's own comment still says "five detached workspaces". Either add /packages/eql/tests/timing/ore_block_v2/ctgen to that directories: list (it shares no crate by path, so the cross-workspace rule does not apply) and fix the count there, or reword this to say ctgen alone has no entry and why.
| for _, family := range families { | ||
| orderIndex[family+"Ord"] = record.Ope | ||
| orderIndex[family+"OrdOpe"] = record.Ope | ||
| orderIndex[family+"OrdOre"] = record.OreBlock |
There was a problem hiding this comment.
LOW (coverage): this test reads the catalog only, and no testusers model carries an *OrdOre or TextSearchOre field, so no Go test seals or queries a block ORE type through the guest. The OPE types have Ranking and TestV4Ordered*, and TextSearch has TestV4TextSearchStoresEveryTermAndQueriesCarryOne; the block ORE types have nothing on the Go side. The guest's native_ops.rs tests run natively, so ore-rs is never executed on wasm32 in CI; a wasm32-only panic, or a term the guest writes differently from native Rust, would show up in a customer program first. A model with an IntegerOrdOre and a TextOrdOre field, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL as Ranking is, would close it (the nonce makes ob non-deterministic, so the fixture would compare everything but c and ob and check ob through the comparator).
2189d5a to
b4acdbf
Compare
|
Force-pushed b4acdbf (from 2189d5a). Two content changes, folded into their owning commits: the v4 shape check no longer uses sub-selects, which makes a comparison about 1.5–2× faster, and it now refuses chained terms over 1,366 blocks (79b5689). There are two new PostgreSQL tests (b4acdbf). Details are in the review threads. v3's install and both uninstall bundles are byte-identical. |
30d3a35 to
61eb180
Compare
b4acdbf to
b079a84
Compare
|
Rebased onto #1274's new head (61eb180), after #1262, #1263 and #1264 merged into
The lockfile conflicts resolved to the new base plus ore-rs 0.9 and its dependencies. Every |
61eb180 to
422f4a1
Compare
b079a84 to
380c49a
Compare
|
Rebased onto #1274's new head (b079a84 → 380c49a). The block ORE commit changed with it: |
422f4a1 to
bd9091a
Compare
380c49a to
0d96d0a
Compare
…index `IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block ORE under its own key domain, over the order keys the CLLW indexes use: `ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and the 128-bit integers. The legacy 8-bit scheme is not used. `sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so it compares by the scheme's comparator, never by its bytes, and `from_bytes` checks a stored term's structure. A query is the full ciphertext too: EQL v4's comparator reads the right half of the operand a btree index passes second. Text whose fold is longer than 1023 bytes or ends in NUL is refused, on the query path before any key is touched. The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte. BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a `match` over it needs an arm for it.
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:
- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
doubling on two bigint halves with no branch on the data, then AES under
the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
(header, block count, exact length, padding policy and the block counts
it can produce, every symbol below 64). A malformed term raises, naming
its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
fixed-width terms of different block counts and chained terms padded
under different policies. For terms of one shape it runs a constant
operation count: one statement per block evaluates both comparisons and
latches the first difference arithmetically, the hash runs once on every
path, and the result is selected without a branch. Chained terms compare
lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.
The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.
`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block` (`record.OreBlock`) in `se_targets`, and the engine answers its orderings through it. The checker accepts it in a type's indexes and as an operator's index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a field reaches it only by naming such a type with `encrypt_into=`, so it has no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still refuse it. The static test engine produces `TextOrdOre`, `TextSearchOre` and `IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`, `DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their `ob` term is Stack Encrypt's block ORE term, one hex element holding the whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its stored value takes the empty filter, and its query carries only the term its operator reads. In the target table an `ob` term is the index `ore_block`, not `ore`, which is CLLW ORE and has no EQL domain, and the block-ORE reason in `encryption_gap` is gone. The unproducible types are the date, timestamp and numeric families, which have no Rust plaintext type yet, and the two JSON types. `tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts in PostgreSQL, both ways round, against the plaintext order and ore-rs's own comparator: equal values, a first difference in every block, shared prefixes and text in three padding buckets. It checks the hash bit against ore.rs's vectors, that malformed and mismatched terms raise, and that a btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres` runs it. The Go catalog, the guest's tests and the Go operator test follow. ADR-0009 is amended.
bd9091a to
92b0e1c
Compare
0d96d0a to
488a100
Compare
|
Rebased onto |
This branch has not been deployed
1 participant
Summary
Stack Encrypt can now write the block ORE EQL v4 types:
IntegerOrdOre,BigintOrdOre,RealOrdOre,DoubleOrdOre,TextOrdOreandTextSearchOre. Before this, those column types were listed but refused, because EQL stored one kind of order-revealing ciphertext and the engine made another. (ORE, order-revealing encryption, lets PostgreSQL sort and range-query encrypted values without decrypting them. EQL is the SQL extension that stores and compares those values;eql_v4is the version Stack Encrypt writes.)The engine now derives ore-rs's 6-bit block ORE, and the EQL v4 SQL gets a comparator for that format, under names of its own (
ore_block_v2), so nothing in v4 reuses the name of EQL v3's older 8-bit format. The EQL v3 SQL bundle is unchanged, byte for byte.Changes
stack-encrypt (breaking)
IndexSpec::OreBlock(wire keyore_block), termsem::OreBlockTerm:ore_rs::vitaminc::Bit6BlockOrefor values up to ten bytes (integers to 64 bits, floats),ore_rs::vitaminc::ChainedBlockOrefor text, bytes and 128-bit integers. Never the legacy 8-bitBlockOre.TermErrorvariants.IndexSpecis exhaustive, so the new variant breaks amatchover it.OreandOpeindexes are untouched.EQL SQL (v4 only)
sem/ore_block_256/functions.sqlholds the v4 comparator,compare_ore_block_v2_term: shape checks, the σ-MMO hash bit through pgcrypto, a constant operation count for terms of one shape, and errors for malformed or mismatched terms.{{ore_block}}. It rendersore_block_256in v3 andore_block_v2in v4, and a literalore_block_256in the source is refused. The hand-written SQL, the pin script, the generated extractors and the ORE fallback use it.tests/timing/ore_block_v2/: a hand-run timing harness with a detached ciphertext generator.eql-bindings / eql-codegen
OreBlock256implementsEncryptFrom, andTextSearchOrefollowsTextSearch(empty filter when stored, one term per query).obterm maps to the indexore_block, notore(CLLW).Go
record.OreBlockandstashgen.IndexOreBlock(not a tag word), checker parsing, the static test engine, and the regeneratedencrypt/eqlcatalog.Docs
@cipherstash/eqlchangeset.stash-postgresandstash-indexingdescribe the v3 surface only and name no v4 object, so they are unchanged.Verification
Run locally on macOS (Apple silicon), on the branch as pushed:
mise run --force buildat the base (4e44e5ba6, before the latest test-only base move) and at this branch:cmpofrelease/cipherstash-encrypt.sqlandrelease/cipherstash-encrypt-uninstall.sqlpasses (SHA-2568eccb1cb…andb1b5131b…both sides). In the v4 bundleore_block_256now appears only in source file paths, andore_block_v2in 173 places.mise run test:encryption:postgrespasses, including the newtests/encryption/tests/v4_ore_block.rs:compare_ore_block_v2_termboth ways round, and must match both the plaintext order and ore-rs's own comparator. Fixed-width:i32andi64with a first difference in every block, extremes and neighbours; floats including ±0, ±∞ and NaN; two writes of each value, which compare equal. Chained: text with shared prefixes across three padding buckets and a difference at each of 40 positions; folded equivalents compare equal.Malformed ORE block term: N bytes, naming no bytes: truncated, too long, empty, header only, wrong version, unknown scheme, symbol 64, a count/length mismatch, an unknown padding kind, a count the padding cannot produce. Fixed-width against chained, 6 blocks against 11, two padding policies and a two-term array each raise.<,>=,=andORDER BYoninteger_ord_oreandtext_ord_orecolumns return the right rows through a btree index, andEXPLAINshows the index.tests/timing/ore_block_v2/run.sh 100000, PostgreSQL 17.5 native, two runs; 0 wrong of 1,500 checks): Welch t for first- vs last-block difference, fixed −1.22 / +0.33 and chained +0.74 / +0.09; equal vs first block −0.65 / −0.40; chained prefix (block counts settle it) vs first block −2.33 / −4.01. All under |t| = 5. The prefix figure is about 40 ns a comparison, below one statement; see the harness README.tests-crates.yml):cargo fmt --check, clippy-D warnings,cargo nextest run --workspace --all-features,mise run test:doc,mise run doc, the examples build and thecargo publish --dry-runof the stack-* crates all pass.cargo mutants --in-diffon stack-encrypt found 91 mutants, 21 missed. Tests were added for every one (OreBlockTerm's views,DebugandPartialOrd, the 1023-byte and trailing-NUL bounds, the query path's early refusal,IndexSpec::parse,indexes()). A re-run of those: 48 mutants, 33 caught, 15 unviable, 0 missed.test-eql.ymltasks):test:crates,test:encryption,check:encryption:wasi,types:check,codegen:parity,test:schemas:parity,test:self_contained_v3,test:installer_complete,test:symbol_order_v3,test:v4_isolation,test:build_ordering_helpers, the four matrix inventories,test:matrix:catalog-coverage,docs:validate:sourceandtest:schemapass.test:docs_v3_grepdid not run here: it needs bash 4 (mapfile), and this machine has bash 3.2, so CI covers it.heap_hygienepasses withTextOrdOreandTextSearchOrethrough every index and no new exclusion. The block ORE visitor carries its plaintext inCarried<T>, as the CLLW visitors do.go:test,go:lintandwasm:guest:testpass, andgo generate ./...leaves the tree clean.cargo metadata --lockedpasses on everyCargo.lockit touches.packages/stack-auth/fuzzandpackages/stack-kms/fuzzalready fail--lockedon the base; this branch does not touch them.test:docs_v3_grep(bash 4).pnpm run test:scriptshit 5-second timeouts in release-gate and Go-proxy tests under load, a different set each run, in files this branch does not touch.Related
Closes #1132
Review notes
ore_block_256, the name of cipherstash-client's 8-bit format (256-way blocks). v4 now holds ore-rs's 6-bit format, wire version 2, so it isore_block_v2:eql_v4_internal.ore_block_v2_term,compare_ore_block_v2_term(s),ore_block_v2_hash_bit,ore_block_v2_schemeand the operator class. The name is a render-time placeholder, so the v3 output stays identical. The source directory keeps its name (sem/ore_block_256/), since it is a path, not an object. No released v4 bundle stored anobterm, so nothing installed is renamed under data.compare_{{ore_block}}_term. Each term's shape is checked first (header, block count, exact length, padding policy and the block counts it allows, every symbol below 64). It raises on different schemes, on fixed-width terms of different block counts and on different padding policies. Then one statement per compared block evaluates both comparisons (non-short-circuiting|) and latches the first difference arithmetically. The hash runs exactly once on every path, equal terms included, and the result is selected arithmetically. Chained terms compare lexicographically: when the shorter one's blocks all match, it sorts first. PL/pgSQL is not constant-time at the C level; the timing results are below.OrdOreis not produced here. The smallint family has no plaintext in this branch, and the work that adds it is separate.https://claude.ai/code/session_01NmUu6jin2Kna7qF8WFSfft