feat(eql)!: EQL v4 text search queries carry only the term their operator reads; produce TextSearch

.changeset/eql-v4-text-search-optional-terms.md

@@ -0,0 +1,5 @@
+---
+'@cipherstash/eql': minor
+---
+
+**An EQL v4 text search query carries only the terms its operator reads.** `eql_v4.query_text_search` and `eql_v4.query_text_search_ore` now require at least one of their terms (`hm`, `op` or `ob`, `bf`) instead of all of them, so `col = $1` and `col < $1` work on text too short to cut a 3-gram, which has no match term. Each extractor over such a query raises (`invalid_parameter_value`) wherever it reads the operand when the query lacks the term it reads, or holds it as JSON `null` or an empty array, naming the domain, the term and the operators that compare it: `=` without `hm`, `<` without `op`, `@@` without `bf`. So a missing term raises rather than matching no row silently, or every row; the raise is at evaluation, so a generic plan that never reaches the predicate returns no row and no error. Stored `eql_v4_text_search` values still carry every term. The `eql_v3` domains are unchanged: the v3 bundle is byte for byte what it was. With it, **Stack Encrypt produces `TextSearch`** (`eql-bindings`, `stack-encrypt` feature): `TextSearchQuery`'s terms are `Option`s, built per index through `TextSearchEqualityQuery`, `TextSearchOrderQuery` and `TextSearchMatchQuery`, and the stored value carries every term: text too short for a 3-gram stores the empty filter (`"bf": []`, the new `v4::terms::SearchBloomFilter` field type), so `=` and `<` find it and no `@@` query does, where `TextMatch` still refuses it. `TextSearchQuery`'s `Option` terms and `TextSearch`'s `SearchBloomFilter` change the fields of the unreleased `v4` module only; no published `eql-bindings` has it.

docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md

@@ -443,6 +443,35 @@ without loss, and refuses it otherwise.
 - **During the overlap**, expected to last a quarter or more, cipherstash-client
   takes fixes only. New kinds and domains are produced through Stack Encrypt
   and v4. An exception is a decision written down in its issue.
+- *Amended 2026-10-19:* **a v4 search query carries only the terms its
+  operator reads.** The text search query domains (`eql_v4.query_text_search`
+  and `eql_v4.query_text_search_ore`, whose terms include the match term
+  `bf` beside others) require at least one of their terms rather than all
+  of them. Text shorter than one 3-gram has no match term, so a query that
+  had to carry every term could not express `=` or `<` on short text.
+  Stack Encrypt emits only the term of the index that answers the query's
+  operator. Every extractor over such an operand raises wherever it reads
+  the operand when the term it reads is absent, JSON `null` or an empty
+  array, naming the domain, the term and the operators that compare it, and
+  never the operand's contents, rather than reading it as `NULL` (no row) or
+  as an empty filter. The raise is at evaluation: a literal operand is
+  folded and raises at plan time, while a bound parameter under a generic
+  plan whose predicate is never reached returns no row and no error. A
+  stored search value carries every term, and text too short for a match
+  term stores the empty filter (`"bf": []`), the stored side's answer in
+  Stack Encrypt: `=` and `<` find it, and no `@@` query does, because a
+  query's filter is never empty (the extractor refuses `"bf": []`). That is
+  a claim about queries: `@@` between two stored values (a self-join
+  `a.name @@ b.name`, or `matches(col, <a stored value as jsonb>)`) keeps
+  the shared guard, so two stored empty filters match each other there, as
+  they do in v3. No producer builds that predicate. A match
+  query of such text is refused, as it is for `TextMatch`, whose stored
+  value still refuses it too. The v3 domains are cipherstash-client's format
+  and keep requiring every term, byte for byte, so here the v4 SQL differs
+  from the v3 SQL. The one source holds both forms between version-block directive lines (`-- {{#v3}}` … `-- {{/v3}}`,
+  `-- {{#v4}}` … `-- {{/v4}}`), which the renderer keeps for their own
+  version and drops from the other, so the v3 bundle renders exactly as it
+  did before the block was written.
 
 ### Review
 

languages/golang/CHANGELOG.md

@@ -116,6 +116,15 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   that orders by ADR-0002's order key: floats as PostgreSQL's `float8`
   sorts them, text by its fold. No `Between` yet: it needs EQL v4's range
   query domains.
+- **`TextSearch`**: a `string` field tagged `encrypt_into=TextSearch` gets
+  `Eq`, `Lt`, `Lte`, `Gt`, `Gte` and `Matches`, one column for all of them.
+  The stored value keeps all three terms. Text shorter than three
+  characters, an empty string included, is stored with an empty match
+  filter, where `TextMatch` refuses it with `ErrTerm`: `Eq`, `Lt`, `Lte`,
+  `Gt` and `Gte` find the row, and no `Matches` query does. Each query
+  carries only the term its operator reads, so `Eq` and `Lt` take a
+  two-letter value and `Matches` refuses one (`ErrTerm`). Needs the EQL v4
+  SQL from this release, whose `eql_v4.query_text_search` accepts one term.
 - **Ordering methods on text.** A string field with `index=ore` or
   `index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
   folded form (ADR-0002: case and accents folded, under Unicode 16), so

languages/golang/cmd/stashgen/README.md

@@ -158,7 +158,7 @@ The engine produces these EQL types today, each from one Go field type:
 
 | `encrypt_into` | Go field |
 |---|---|
-| `TextEq`, `Text`, `TextMatch`, `TextOrd`, `TextOrdOpe` | `string` |
+| `TextEq`, `Text`, `TextMatch`, `TextOrd`, `TextOrdOpe`, `TextSearch` | `string` |
 | `Smallint`, `SmallintEq`, `SmallintOrd`, `SmallintOrdOpe` | `int16`, or any Go integer type |
 | `Integer`, `IntegerEq`, `IntegerOrd`, `IntegerOrdOpe` | `int32`, or any Go integer type |
 | `Bigint`, `BigintEq`, `BigintOrd`, `BigintOrdOpe` | `int64`, or any Go integer type |
@@ -169,12 +169,13 @@ The engine produces these EQL types today, each from one Go field type:
 | `Timestamp`, `TimestampEq` | `time.Time` |
 | `Numeric`, `NumericEq` | `encrypt.Decimal` |
 
-The `Eq` types answer `Eq`; the `Ord` and `OrdOpe` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real`, `Double`, `Date`, `Timestamp` and `Numeric` are stored and read only, with no query method.
+The `Eq` types answer `Eq`; the `Ord` and `OrdOpe` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `TextMatch` answers `Matches`, and `TextSearch` all of `Eq`, `Lt`, `Lte`, `Gt`, `Gte` and `Matches`, each query carrying only the term its operator reads; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real`, `Double`, `Date`, `Timestamp` and `Numeric` are stored and read only, with no query method.
 The ordered date, timestamp and numeric types are refused: the engine derives no order term for these kinds yet.
 An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains.
 
 Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`.
 A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too.
+A `TextSearch` field stores such text with an empty match filter, so its `Eq`, `Lt`, `Lte`, `Gt` and `Gte` queries find it and no `Matches` query does; those queries take such text too, and its `Matches` query refuses it.
 Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed.
 `encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.
 Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.

languages/golang/encrypt/README.md

@@ -488,6 +488,7 @@ The EQL types the engine produces, and the Go field types each takes:
 | `TextEq` | `string` | `=`, `<>` |
 | `TextMatch` | `string` | `@@` (substring match, 3-grams, case-insensitive) |
 | `TextOrd`, `TextOrdOpe` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` |
+| `TextSearch` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY`, `@@` |
 | `Text` | `string` | none: stored and read only |
 | `Smallint` | `int16`, or any Go integer type (see below) | none |
 | `SmallintEq` | the same | `=`, `<>` |
@@ -523,7 +524,13 @@ with `ErrEncoding`, naming the field's kind and not the value.
 exactly their own width.
 
 A `TextMatch` value must be at least three characters long: shorter text has
-no 3-grams to index and is refused. The number equality types compare as
+no 3-grams to index and is refused. A `TextSearch` value of such text, an
+empty string included, is stored with an empty match filter: `Eq`, `Lt`,
+`Lte`, `Gt` and `Gte` find it through its other terms, and no `Matches`
+query does. A `TextSearch` query carries only the term its operator reads,
+so `Eq` and `Lt` on two-letter text work and `Matches` on it is refused; EQL
+v4 raises if a query lacks the term its operator compares. The number
+equality types compare as
 PostgreSQL compares the plain types: an integer equals itself whatever its
 width, and for floats `-0.0 = 0.0` and `NaN = NaN`; the stored value still
 decrypts to exactly what was written. The same holds for the date,

languages/golang/encrypt/eql/eql_test.go

@@ -24,7 +24,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) {
 		"Smallint", "SmallintEq", "SmallintOrd", "SmallintOrdOpe",
 		"Bigint", "BigintEq", "BigintOrd", "BigintOrdOpe",
 		"Date", "DateEq", "Timestamp", "TimestampEq", "Numeric", "NumericEq",
-		"Text", "TextEq", "TextMatch", "TextOrd", "TextOrdOpe",
+		"Text", "TextEq", "TextMatch", "TextOrd", "TextOrdOpe", "TextSearch",
 		"Boolean",
 		"Real", "RealEq", "RealOrd", "RealOrdOpe",
 		"Double", "DoubleEq", "DoubleOrd", "DoubleOrdOpe",

languages/golang/encrypt/eql_v4_text_search_test.go

@@ -0,0 +1,159 @@
+package encrypt_test
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/cipherstash/stack/languages/golang/encrypt"
+	"github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers"
+)
+
+// EQL v4's TextSearch, sealed and queried through generated code over the
+// deterministic guest build with the EQL types. A stored value carries all
+// three terms, short text with the empty filter; a query carries only the
+// one its operator reads (ADR-0002, amended 2026-10-19), so Eq and Lt take a
+// two-letter name and Matches refuses one.
+// packages/eql/tests/encryption/tests/v4_text_search.rs holds the same rules
+// against PostgreSQL.
+
+// keysAndTerms is an EQL value's sorted top-level keys, and the value of
+// each key.
+func keysAndTerms(t *testing.T, value []byte) (string, map[string]json.RawMessage) {
+	t.Helper()
+	var raw map[string]json.RawMessage
+	if err := json.Unmarshal(value, &raw); err != nil {
+		t.Fatalf("the EQL value is not JSON: %v", err)
+	}
+	if string(raw["v"]) != "4" {
+		t.Errorf("v = %s, want 4", raw["v"])
+	}
+	keys := make([]string, 0, len(raw))
+	for k := range raw {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	return strings.Join(keys, ","), raw
+}
+
+func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) {
+	c := deterministicEQLClient(t)
+	ctx := context.Background()
+	cipher := c.DefaultKeyset()
+
+	encrypted, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 1, Name: "Alice"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	keys, stored := keysAndTerms(t, encrypted[0].Name)
+	if keys != "bf,c,hm,i,op,v" {
+		t.Errorf("stored keys %s, want bf,c,hm,i,op,v", keys)
+	}
+	decrypted, err := testusers.DecryptEntry(ctx, cipher, encrypted)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if decrypted[0].Name != "Alice" {
+		t.Errorf("opened %q", decrypted[0].Name)
+	}
+
+	f := testusers.EntryFields.Name
+	// Each method's query carries its operator's term alone, the stored one.
+	for _, tc := range []struct {
+		method string
+		query  func(string) ([]byte, error)
+		term   string
+	}{
+		{"Eq", func(v string) ([]byte, error) { return f.Eq(ctx, cipher, v) }, "hm"},
+		{"Lt", func(v string) ([]byte, error) { return f.Lt(ctx, cipher, v) }, "op"},
+		{"Gte", func(v string) ([]byte, error) { return f.Gte(ctx, cipher, v) }, "op"},
+	} {
+		q, err := tc.query("Alice")
+		if err != nil {
+			t.Fatalf("%s: %v", tc.method, err)
+		}
+		keys, terms := keysAndTerms(t, q)
+		if want := strings.Join(sortedWith("i", "v", tc.term), ","); keys != want {
+			t.Errorf("%s query keys %s, want %s", tc.method, keys, want)
+		}
+		if string(terms[tc.term]) != string(stored[tc.term]) {
+			t.Errorf("%s query's %s is not the stored one", tc.method, tc.term)
+		}
+		// Text too short for a 3-gram still has an equality and an order
+		// term.
+		short, err := tc.query("Al")
+		if err != nil {
+			t.Fatalf("%s on two-letter text: %v", tc.method, err)
+		}
+		if keys, _ := keysAndTerms(t, short); keys != strings.Join(sortedWith("i", "v", tc.term), ",") {
+			t.Errorf("%s query keys on two-letter text %s", tc.method, keys)
+		}
+	}
+
+	m, err := f.Matches(ctx, cipher, "Alice")
+	if err != nil {
+		t.Fatal(err)
+	}
+	keys, terms := keysAndTerms(t, m.Query())
+	if keys != "bf,i,v" {
+		t.Errorf("Matches query keys %s, want bf,i,v", keys)
+	}
+	if string(terms["bf"]) != string(stored["bf"]) {
+		t.Error("the match query's bf is not the stored filter")
+	}
+	if ok, err := m.Verify(ctx, []string{"alice", "Bob"}); err != nil || !ok[0] || ok[1] {
+		t.Errorf("Verify = %v, %v", ok, err)
+	}
+
+	// A match query on text too short for a 3-gram is refused, never sent
+	// as an empty filter.
+	if q, err := f.Matches(ctx, cipher, "Al"); !errors.Is(err, encrypt.ErrTerm) {
+		t.Errorf("Matches on two-letter text = %v, %v, want ErrTerm", q, err)
+	}
+	// Storing such text, an empty string included, keeps every term, the
+	// match term as the empty filter (where TextMatch refuses the text): Eq
+	// and Lt find the row through hm and op, and no Matches query does.
+	short, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 2, Name: "Al"}, {ID: 3, Name: ""}})
+	if err != nil {
+		t.Fatalf("storing short text in a TextSearch: %v", err)
+	}
+	for i, name := range []string{"Al", ""} {
+		keys, terms := keysAndTerms(t, short[i].Name)
+		if keys != "bf,c,hm,i,op,v" {
+			t.Errorf("stored keys for %q %s, want bf,c,hm,i,op,v", name, keys)
+		}
+		if string(terms["bf"]) != "[]" {
+			t.Errorf("stored filter for %q is %s, want []", name, terms["bf"])
+		}
+		eq, err := f.Eq(ctx, cipher, name)
+		if err != nil {
+			t.Fatalf("Eq(%q): %v", name, err)
+		}
+		if _, q := keysAndTerms(t, eq); string(q["hm"]) != string(terms["hm"]) {
+			t.Errorf("Eq(%q)'s hm is not the stored one", name)
+		}
+		lt, err := f.Lt(ctx, cipher, name)
+		if err != nil {
+			t.Fatalf("Lt(%q): %v", name, err)
+		}
+		if _, q := keysAndTerms(t, lt); string(q["op"]) != string(terms["op"]) {
+			t.Errorf("Lt(%q)'s op is not the stored one", name)
+		}
+	}
+	opened, err := testusers.DecryptEntry(ctx, cipher, short)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if opened[0].Name != "Al" || opened[1].Name != "" {
+		t.Errorf("opened %q and %q", opened[0].Name, opened[1].Name)
+	}
+}
+
+// sortedWith is the given keys, sorted.
+func sortedWith(keys ...string) []string {
+	sort.Strings(keys)
+	return keys
+}

languages/golang/encrypt/guest/src/targets.rs

@@ -79,11 +79,13 @@ impl TargetResolver for EqlTargets {
     fn query<'a, K: 'static>(
         &self,
         name: &str,
+        index: &stack_encrypt::target::IndexSpec,
         keyset: &'a stack_encrypt::KeysetCipher<'_, K>,
         label: &stack_encrypt::Label,
         plaintext: stack_encrypt::dynamic::ValueFrame,
     ) -> Result<stack_encrypt::Pending<'a, Vec<u8>, K>, TargetError> {
-        eql_bindings::encryption::targets::query(name, keyset, label, plaintext).map_err(convert)
+        eql_bindings::encryption::targets::query(name, index, keyset, label, plaintext)
+            .map_err(convert)
     }
 }
 
@@ -212,6 +214,7 @@ mod tests {
                 "TextMatch",
                 "TextOrd",
                 "TextOrdOpe",
+                "TextSearch",
                 "Boolean",
                 "Real",
                 "RealEq",
@@ -252,6 +255,16 @@ mod tests {
             }),
             TargetError::Plaintext { target, expected: Some(stack_encrypt::dynamic::ValueKind::String), .. } if target == "TextEq"
         ));
+        // The engine asks a type's query only through an index the type
+        // carries, so one it does not is the engine's failure, not the
+        // caller's input.
+        assert!(matches!(
+            convert(Eql::NoIndex {
+                name: "TextEq",
+                index: "ope",
+            }),
+            TargetError::Other(_)
+        ));
     }
 
     /// serde_json quotes the value it refused, and a stored EQL value holds