feat(stack-encrypt): order text over ADR-0002's folded order key

Cargo.toml

@@ -81,7 +81,10 @@ stack-profile = { path = "./packages/stack-profile", version = "0.43.0" }
 cts-common = { version = "0.43.0", default-features = false }
 zerokms-protocol = "0.12.31"
 recipher = "=0.3.1"
-cllw-ore = { version = "=0.5.0", default-features = false }
+# cllw-ore's `vitaminc` feature implements vitaminc-ore's `OrderScheme`,
+# which Stack Encrypt's order terms are encrypted through. It first ships
+# in a pre-release, and Cargo resolves a pre-release only when named.
+cllw-ore = { version = "=0.5.2-rc.1", default-features = false, features = ["vitaminc"] }
 
 # External dependencies, with the suite's feature lists.
 base64 = "0.22.0"
@@ -130,5 +133,8 @@ vitaminc-hmac = "0.6.1"
 # Normalized equality terms (Unicode 16 NFC text, one integer encoding for
 # every width), which `vitaminc-prf`'s `normalize` feature derives over.
 vitaminc-normalize = "0.6.1"
+# The text order key: the system ADR-0002's fold, pinned to Unicode 16,
+# which text ORE and OPE terms are derived over (ADR-0009).
+vitaminc-ore = "0.6.1"
 vitaminc-prf = { version = "0.6.1", features = ["normalize"] }
 vitaminc-protected = "0.6.1"

docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md

@@ -203,9 +203,18 @@ transform and its version is named in the domain label, for example
   UTC calendar day.
 - **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt`
   statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of
-  every code point with General_Category `Mn`, then NFC. The order of the
-  steps is part of the transform, because folding can produce text that is
-  no longer normalised and the steps do not commute.
+  every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc`
+  or `Me`) with a non-zero canonical combining class: the acute in `é`, the
+  umlaut in `ö`, a Devanagari virama. A mark of class zero is a letter of
+  its own in its script, such as a Thai or Devanagari vowel sign (U+0E31,
+  U+0901), and is kept. The order of the steps is part of the transform,
+  because folding can produce text that is no longer normalised and the
+  steps do not commute.
+  *Amended 2026-10-19:* this step said "every code point with
+  General_Category `Mn`", which would also drop class-zero vowel signs and
+  so tie different words. The fold that ships, `vitaminc-ore`'s
+  `vitaminc/ore/value/text-folded/unicode-16/v1`, keeps them, and this
+  bullet now describes it.
 - **Prefix truncation**: a number of code points kept after any other
   transform. Alphabet packing, if ever built, is a transform of the same
   kind.
@@ -219,6 +228,13 @@ write. The cost is that two strings using characters added after the pin,
 which a later Unicode would treat as equivalent, may not match. Moving the
 pin is a new label: existing columns keep the label they were written under
 and stay queryable, and a column moves only by re-encryption.
+*Amended 2026-10-18:* vitaminc 0.6 refuses a code point unassigned in the
+pinned version instead of passing it through. `vitaminc-normalize` (equality
+and match terms) and `vitaminc-ore`'s fold (order terms) both return
+`UnassignedCodePoint`, so text holding one has no term, and writing it to an
+indexed column or querying with it is refused. The refusal is the same on
+every path and in every language, so the transform is still a fixed
+function of the text it accepts, and moving the pin is still a new label.
 
 Order terms fold case and accents because code-point order puts `é` after
 `z`; a fixed fold approximates the first level of Unicode collation without

languages/golang/CHANGELOG.md

@@ -35,6 +35,20 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   implemented outside this module (a wrapper, or a fake in a generator's
   tests) changes its `Check` to return them; `enginetest.Static` shows the
   table.
+- **An `int8`, `uint8` or `byte` field takes no `ore` or `ope` index; drop
+  the index and run stashgen again.** With 256 values, an order term ranks
+  every row among 256, which a few known rows resolve to the values, the
+  leak a `bool`'s order term is already refused for. The engine refuses the
+  declaration as it refuses one on a `bool`, so stashgen stops at it. The
+  field keeps `equality` and `Eq`; a value that must order declares a wider
+  type.
+- **Every ORE and OPE term changes bytes; encrypt indexed rows again.**
+  Order terms now encrypt through cllw-ore's vitaminc schemes, for every
+  type: integers of every width, floats, text and bytes. A term v0.1.0-alpha.1
+  stored does not compare with one this release derives, and nothing
+  refuses it: `Lt`, `Between` or an `ORDER BY` over a column holding the
+  alpha's terms silently misses or misplaces rows. Re-derive a column's
+  order terms (encrypt its rows again) before querying it.
 
 ### Added
 
@@ -47,8 +61,8 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   `Decimal` are vitaminc's `vcvalue` types, with `encrypt.DateOf`,
   `ParseInt128`, `ParseUint128` and `ParseDecimal` (which refuses NaN and
   the infinities with `ErrNonFiniteDecimal`). Equality applies to all of
-  them, ORE and OPE to every integer width. stashgen refuses an order index
-  on a decimal, date or timestamp, naming #1236. The value-kinds fixture
+  them, ORE and OPE to every integer width but 8 bits. stashgen refuses an
+  order index on a decimal, date or timestamp, naming #1236. The value-kinds fixture
   test opens a record the Rust lowering sealed and derives the same terms.
 - **Any built-in Go integer field takes an integer EQL type** (ADR-0002):
   an `int16` or a `uint16` into `Integer`, an `int` into `Bigint`. The
@@ -74,9 +88,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   type, range checked as for `Integer`) tagged `encrypt_into=` one of them
   seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as
   an integer of the same value under the same column.
+- **Ordering methods on text.** A string field with `index=ore` or
+  `index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
+  folded form (ADR-0002: case and accents folded, under Unicode 16), so
+  `Café` and `cafe` tie and `alice` orders before `Zoe`.
 - **stashgen notices an index no generated method queries**: an `ore` or
-  `ope` index on text, bytes or a float (the engine answers the orderings
-  over integers only), or the second of two order indexes on an integer.
+  `ope` index on bytes or a float (the engine answers the orderings over
+  integers and text only), or the second of two order indexes on a field.
   The index is still stored; the notice is on stderr and in the file.
 
 ### Changed

languages/golang/cmd/stashgen/README.md

@@ -124,7 +124,7 @@ For `-type User`, the file `user_stash.go` holds:
 - `Fields`, with one entry for each sealed field.
   An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried.
   An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`.
-  stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both.
+  stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer or a string the four orderings and `Between`, through ORE when the field declares both. A string orders by its folded form: case and accents folded, under Unicode 16.
   `encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`.
   `encrypt_into=TextMatch` answers `Matches`, which returns an `encrypt.EQLMatchQuery`: the `eql.TextMatchQuery` operand, with `Verify` and `TargetLimit`.
   The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither.

languages/golang/encrypt/README.md

@@ -229,20 +229,21 @@ or a type defined over one (`type Email string`):
 
 | Go field | Seals as | Indexes it can store | Query methods |
 |---|---|---|---|
-| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches` |
+| `string` | text | `equality`, `match`, `ore`, `ope` | `Eq`, `Matches`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
 | `[]byte` | bytes | `equality`, `ore`, `ope` | `Eq` |
 | `bool` | a boolean | none | none |
-| `int8`, `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
-| `uint8` (`byte`), `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
+| `int16`, `int32`, `int64`, `int` | an integer of the field's own width (`int` is 64-bit) | `equality`, `ore`, `ope` | `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` |
+| `uint16`, `uint32`, `uint64`, `uint` | the same, unsigned | `equality`, `ore`, `ope` | the same |
+| `int8`, `uint8` (`byte`) | an 8-bit integer | `equality` | `Eq` |
 | `encrypt.Int128`, `encrypt.Uint128` | a 128-bit integer | `equality`, `ore`, `ope` | the same |
 | `float32`, `float64` | a float of its width | `equality`, `ore`, `ope` | `Eq` |
 | `encrypt.Decimal` | a decimal: a 96-bit coefficient and a scale up to 28 | `equality` | `Eq` |
 | `encrypt.Date` | a calendar date | `equality` | `Eq` |
 | `time.Time` | a timestamp, to the nanosecond, in UTC | `equality` | `Eq` |
 
-Each query method needs its index declared. An `ore` or `ope` index on text,
-bytes or a float stores its term, but the engine answers no ordering through
-it yet, so such a field has no `Lt` (see [Queries](#queries)).
+Each query method needs its index declared. An `ore` or `ope` index on bytes
+or a float stores its term, but the engine answers no ordering through it
+yet, so such a field has no `Lt` (see [Queries](#queries)).
 
 `encrypt.Date`, `Int128`, `Uint128` and `Decimal` are vitaminc's `vcvalue`
 types; build them with `encrypt.DateOf`, `ParseInt128`, `ParseUint128` and
@@ -253,8 +254,11 @@ sealed with, but a zero has no sign: `-0.00` opens as `0.00`. The zero
 `ErrEncoding`, naming the field; so is a decimal or a `time.Time` outside
 vitaminc's range. A sealed field cannot be a pointer, so an optional date has
 no sealed form yet: keep it in an opaque struct, or use a sentinel date. Each integer type seals at its own
-width, and each width is passed to the order scheme as itself, so an `int8`
-field's terms are an `int8`'s. Equality follows vitaminc's normalisation: an
+width, and each width is passed to the order scheme as itself, so an `int16`
+field's terms are an `int16`'s. An `int8` or `uint8` field has no order
+index: with 256 values, an order term would rank every row among 256, which
+a few known rows resolve to the values, as a `bool`'s would say which rows
+are true. Equality follows vitaminc's normalisation: an
 integer equals itself at every width, decimals compare at any scale
 (`1.50 = 1.5`), and a timestamp compares to the microsecond. An order index on
 a decimal, date or timestamp is refused for now: those order terms wait for
@@ -298,17 +302,27 @@ rules.
 | The field declares | Methods | Each returns |
 |---|---|---|
 | `index=equality` | `Eq` | `encrypt.EqualityTerm` |
-| `index=ore` or `index=ope`, on an integer of any width | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
+| `index=ore` or `index=ope`, on an integer of 16 bits or wider or a string | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` |
 | the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms |
 | `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` |
 | `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` |
 | `encrypt_into=TextMatch` | `Matches` | `encrypt.EQLMatchQuery[eql.TextMatchQuery, T]` |
 
 A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32`
-field takes a `uint32`. An order index on text, bytes or a float stores its
-term but answers no ordering operator yet, so such a field has no `Lt`; nor
-does the `ope` index of a field that declares `ore` too, since `ore` answers
-the orderings. stashgen prints a notice for each such index, on stderr and in
+field takes a `uint32`.
+
+Text orders by its folded form (ADR-0002): case and accents are folded and
+`ß` becomes `ss`, under Unicode 16. So `Café`, `cafe` and `CAFÉ` are one
+value and tie, `alice` orders before `Zoe`, and `é` orders with `e`, not
+after `z`. `Between` is inclusive at both ends and, as SQL's `BETWEEN`,
+does not refuse a range whose low bound is above its high bound: such a
+range matches nothing. So `Between("b", "A")` matches nothing, since `A`
+folds to `a`, below `b`. Text holding a code point Unicode 16 does not assign has no
+fold, and its query or write is refused with `encrypt.ErrTerm`.
+
+An order index on bytes or a float stores its term but answers no ordering
+operator, so such a field has no `Lt`; nor does the `ope` index of a field
+that declares `ore` too, since `ore` answers the orderings. stashgen prints a notice for each such index, on stderr and in
 the generated file: the terms are stored, and only your own code can compare
 them. A match index on an index-only field (`index=match` with no `encrypt`)
 is refused, because `Verify` checks a match query's rows by decrypting them.

languages/golang/encrypt/doc.go

@@ -153,7 +153,10 @@
 //
 // Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL
 // type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a
-// [Range] of two. Their answers are exact. Matches returns a [MatchQuery],
+// [Range] of two. Their answers are exact. The orderings are answered on
+// integers and on text, which orders by its folded form (case and accents
+// folded, under Unicode 16), so Café and cafe tie and alice orders before
+// Zoe. Matches returns a [MatchQuery],
 // whose answer can hold rows that do not match: size the fetch with
 // [MatchQuery.TargetLimit], then decrypt the rows and keep those
 // [MatchQuery.Verify] reports true, one call into the engine for the page.

languages/golang/encrypt/guest/src/abi.rs

@@ -465,8 +465,9 @@ pub unsafe extern "C" fn se_plan_check(plan_ptr: *const u8, plan_len: u32) -> u6
 /// term's codec-encoded object out. An operator the field does not answer
 /// is `STATUS_ENCODING` naming the field, as a mistyped value is, before
 /// the cipher is consulted; the refusals the typed path makes before a term
-/// exists (a range the wrong way round, match text that cuts no token or
-/// does not normalize) are `STATUS_TERM`, also before the cipher. The input
+/// exists (match text that cuts no token or does not normalize) are
+/// `STATUS_TERM`, also before the cipher. A range the wrong way round is
+/// not refused: it matches no row, as SQL's `BETWEEN` does. The input
 /// buffer is wiped in place.
 ///
 /// # Safety

languages/golang/encrypt/guest/tests/native_ops.rs

@@ -330,15 +330,16 @@ fn guest_terms_match_the_native_sem_derivations() {
     .expect("native");
     assert_eq!(eq_bytes, native_bytes.as_bytes());
 
-    // Variable-width CLLW output for strings.
+    // Variable-width CLLW output for strings: the text behind cllw-ore's
+    // one marker byte, eight bytes a plaintext byte.
     let ore_s = block_on(ops::term(
         &cipher.default_keyset(),
         &encode(s("alice")),
         &encode(s("users/name")),
         TERM_ORE,
     ))
     .expect("string ore");
-    assert_eq!(ore_s.len(), 5 * 8);
+    assert_eq!(ore_s.len(), 6 * 8);
 
     // No ZeroKMS traffic for any of it.
     assert_eq!(cipher.kms().generate_calls.load(Ordering::SeqCst), 0);
@@ -1556,13 +1557,6 @@ fn query_term_validation_refuses_what_the_query_op_refuses() {
             "between",
             STATUS_ENCODING,
         ),
-        (
-            "inverted range",
-            Tree::Array(vec![Tree::UInt32(65), Tree::UInt32(18)]),
-            "age",
-            "between",
-            STATUS_TERM,
-        ),
         ("no token", s("al"), "name", "matches", STATUS_TERM),
     ] {
         let input = encode(input);