feat(stack-encrypt): store short TextMatch text as an empty filter, refuse only short queries

languages/golang/cmd/stashgen/README.md

@@ -96,9 +96,9 @@ A query on such a type derives its term under the context the cipher names, so i
 > To update one column, use its `Fields` entry, such as `Fields.Email.Encrypt`.
 
 The index names are `equality`, `match`, `ore`, `ope` and `json`.
-A `match` index needs text with at least one token: the engine derives no match term for an empty string, separator-only text, or text shorter than the n-gram length (3 characters), because an empty term would match every row.
-`Encrypt` then fails for the whole batch, with an error naming the row, the field and the index.
-So an optional or short value does not belong under `match`: give the field `equality` alone, or make the value required.
+A `match` index stores text with no token (an empty string, separator-only text, or text shorter than the n-gram length of 3 characters) as the empty filter, which no query matches, so an optional or short value can live under `match`; the empty filter tells a reader of the column that the value is one of those.
+A match query of such text is refused, because an empty filter as a query would match every row.
+An EQL `TextMatch` field is stricter: see below.
 An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`.
 This build refuses an index with options: a query term is derived with the default options only, so a stored term with other options would never match one.
 These words are the same as the Rust API's words for the same behaviour, with two that only Go has.

languages/golang/encrypt/README.md

@@ -202,12 +202,13 @@ and `Equality`, `Match`, `Ore` or `Ope` (the term types). A field with
 An `opaque` struct is one `Sealed` field. Every stored type implements
 `driver.Valuer` and `sql.Scanner`.
 
-A `match` index needs text with at least one token. The engine derives no
-match term for an empty string, separator-only text, or text shorter than the
-n-gram length (3 characters), because an empty term would match every row.
-`Encrypt` then fails for the whole batch with `ErrTerm`, naming the row, the
-field and the index. So an optional or short value does not belong under
-`match`: give such a field `equality` alone, or make the value required.
+A `match` index stores text with no token (an empty string, separator-only
+text, or text shorter than the n-gram length of 3 characters) as the empty
+filter: a valid value that no query matches, so an optional or short value can
+live under `match`. The empty filter tells anyone who can read the column that
+the value is one of those. A match *query* of such text is refused with
+`ErrTerm`, because an empty filter as a query would match every row. An EQL
+`TextMatch` field is stricter, as "EQL columns" below says.
 
 Terms are byte-equal to the ones the Rust crate derives, so a term from
 `users.Fields.Email.Equality` compares against a stored term written from any

languages/golang/encrypt/doc.go

@@ -116,12 +116,13 @@
 // [Match], [Ore] and [Ope]; [JSON] is declared and refused until the engine
 // derives it. Every stored type implements driver.Valuer and sql.Scanner.
 //
-// A match index needs text with at least one token: the engine derives no
-// match term for an empty string, separator-only text, or text shorter than
-// the n-gram length (3 characters), because an empty term would match every
-// row. Encrypt then fails for the whole batch with [ErrTerm], naming the row,
-// the field and the index, so an optional or short value does not belong
-// under match.
+// A match index stores text with no token (an empty string, separator-only
+// text, or text shorter than the n-gram length of 3 characters) as the empty
+// filter: a valid value that no query matches, so an optional or short value
+// can live under match. The empty filter tells anyone who can read the
+// column that the value is one of those. A match *query* of such text is
+// refused with [ErrTerm], because an empty filter as a query would match every
+// row.
 //
 // # Transport and auth
 //

languages/golang/encrypt/locate_test.go

@@ -0,0 +1,41 @@
+package encrypt
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"testing"
+
+	"github.com/cipherstash/stack/languages/golang/internal/record"
+)
+
+// A record stores match text that cuts no token as the empty filter, so the
+// term-failure locator must not blame a row for it. Derive asks the query
+// path, which refuses that text; the locator skips that refusal and goes on
+// to the rows after it.
+func TestLocateTermFailureSkipsShortMatchText(t *testing.T) {
+	ctx := context.Background()
+	c, err := NewDeterministicClient(ctx, [32]byte{})
+	if errors.Is(err, ErrDeterministicGuestNotBuilt) {
+		t.Skip(err)
+	}
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = c.Close() })
+	cph := c.DefaultKeyset()
+	plan := usersPlan()
+
+	// The query path refuses the short email, so without the skip the
+	// locator would name row 0 for it.
+	_, derr := cph.Derive(ctx, plan, "email", record.Match, "ab")
+	if !queryOnlyRefusal(record.Match, derr) {
+		t.Fatalf("Derive of a short email: %v, want the empty-term-text refusal", derr)
+	}
+
+	sentinel := fmt.Errorf("%w: from the record call", ErrTerm)
+	rows := []record.Source{usersRow(1, "ab"), usersRow(2, "alice@example.com")}
+	if got := cph.locateTermFailure(ctx, plan, rows, sentinel); got != sentinel {
+		t.Fatalf("locateTermFailure = %v, want the record call's error unchanged", got)
+	}
+}

languages/golang/encrypt/records.go

@@ -470,10 +470,12 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, context string, p
 // locateTermFailure names the row, field and index behind an ErrTerm from a
 // record call. The guest's error says what was wrong with the term but not
 // which value or field it came from, so the host asks the engine again, one
-// term at a time, which costs no key request: a term derives locally. The engine defines no match term for text that yields no
-// token (empty, separator-only, or shorter than the n-gram), because an
-// empty term would match every row; a program hands such a field a value or
-// drops the match index. p is the plan without this cipher's extension,
+// term at a time, which costs no key request: a term derives locally. A
+// stored value is never refused for its text (text with no token is stored
+// as the empty filter), so a term failure on a record is the scheme refusing
+// the value's kind under that index. Derive asks the query path, which does
+// refuse text with no token, so that one refusal is skipped here: it cannot
+// be why the record failed. p is the plan without this cipher's extension,
 // which Derive adds.
 func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error {
 	for i, row := range rows {
@@ -482,15 +484,27 @@ func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows [
 				if !o.IsTerm() {
 					continue
 				}
-				if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) {
-					return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value (a match index needs text with at least one token; an empty or separator-only string has none)", err, i, f.Name, indexWord(o))
+				if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) && !queryOnlyRefusal(o, derr) {
+					return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value", err, i, f.Name, indexWord(o))
 				}
 			}
 		}
 	}
 	return err
 }
 
+// queryOnlyRefusal reports whether derr is the one term refusal the query
+// path makes and the record path does not: match text that cuts no token,
+// which a record stores as the empty filter.
+func queryOnlyRefusal(o record.Output, derr error) bool {
+	var d *Diagnostic
+	return o == record.Match && errors.As(derr, &d) && d.Code == emptyTermTextCode
+}
+
+// emptyTermTextCode is the stable code of the engine's refusal of match text
+// that cuts no token.
+const emptyTermTextCode = "stack_encrypt::empty_term_text"
+
 func indexWord(o record.Output) string {
 	switch o {
 	case record.Equality:

languages/golang/encrypt/roundtrip_test.go

@@ -260,37 +260,56 @@ func TestATamperedRecordDoesNotOpen(t *testing.T) {
 	}
 }
 
-// A match index needs text that yields a token: the engine defines no match
-// term for an empty or separator-only string, because an empty term would
-// match every row, and the typed Rust path refuses it the same way. The Go
-// error names the row, the field and the index. Every other zero value
-// seals: a zero integer under ore and equality, an empty string with no
-// match index.
-func TestAMatchIndexNeedsText(t *testing.T) {
+// A match index stores text that yields no token as the empty filter: an
+// empty or separator-only string, or one shorter than the n-gram, is a valid
+// value that no query matches, and it seals and opens like any other. Only a
+// match *query* of such text is refused, because an empty filter as a query
+// would match every row. Every other zero value seals too: a zero integer
+// under ore and equality, an empty string with no match index.
+func TestAMatchIndexStoresShortText(t *testing.T) {
 	c := deterministicClient(t)
 	ctx := context.Background()
 	cipher := c.DefaultKeyset()
 	for name, u := range map[string]testusers.User{
-		"the zero value":    {},
-		"an empty email":    {ID: 1, Age: 34, Notes: "x"},
-		"a separator email": {ID: 1, Age: 34, Email: " \t", Notes: "x"},
+		"the zero value":     {},
+		"an empty email":     {ID: 1, Age: 34, Notes: "x"},
+		"a separator email":  {ID: 1, Age: 34, Email: " \t", Notes: "x"},
+		"a two-letter email": {ID: 1, Age: 34, Email: "ab", Notes: "x"},
 	} {
-		_, err := testusers.Encrypt(ctx, cipher, []testusers.User{{ID: 9, Age: 1, Email: "ok@example.com", Notes: "x"}, u})
-		if !errors.Is(err, encrypt.ErrTerm) {
-			t.Fatalf("%s: %v, want ErrTerm", name, err)
+		rows := []testusers.User{{ID: 9, Age: 1, Email: "ok@example.com", Notes: "x"}, u}
+		encrypted, err := testusers.Encrypt(ctx, cipher, rows)
+		if err != nil {
+			t.Fatalf("%s: %v, want the row to seal", name, err)
+		}
+		if positions, err := encrypted[1].Email.Match.Positions(); err != nil || len(positions) != 0 {
+			t.Errorf("%s: the stored match term is %v %v, want the empty filter", name, positions, err)
+		}
+		back, err := testusers.Decrypt(ctx, cipher, encrypted)
+		if err != nil {
+			t.Fatalf("%s: decrypt: %v", name, err)
 		}
-		for _, want := range []string{`value 1`, `field "email"`, "no match term"} {
-			if !strings.Contains(err.Error(), want) {
-				t.Errorf("%s: error %q does not name %s", name, err, want)
-			}
+		if back[1].Email != u.Email {
+			t.Errorf("%s: email opened as %q, want %q", name, back[1].Email, u.Email)
+		}
+		// No query matches the empty filter: a query has at least one
+		// position, and the empty filter holds none of them.
+		q, err := testusers.Fields.Email.Match(ctx, cipher, "example")
+		if err != nil {
+			t.Fatal(err)
+		}
+		if positions, _ := q.Positions(); len(positions) == 0 {
+			t.Fatalf("%s: a query with a token sets no bit", name)
 		}
 	}
-	// The field entry says the same for one value.
-	if _, err := testusers.Fields.Email.Encrypt(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) || !strings.Contains(err.Error(), `field "email"`) {
+	// The field entry stores one such value the same way.
+	if _, err := testusers.Fields.Email.Encrypt(ctx, cipher, ""); err != nil {
 		t.Fatalf("Fields.Email.Encrypt(\"\"): %v", err)
 	}
-	if _, err := testusers.Fields.Email.Match(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) {
-		t.Fatalf("Fields.Email.Match(\"\"): %v", err)
+	// A match query with no token is the one value the engine refuses.
+	for _, text := range []string{"", " \t", "ab"} {
+		if _, err := testusers.Fields.Email.Match(ctx, cipher, text); !errors.Is(err, encrypt.ErrTerm) {
+			t.Fatalf("Fields.Email.Match(%q): %v, want ErrTerm", text, err)
+		}
 	}
 	// Zero values the engine does seal.
 	encrypted, err := testusers.Encrypt(ctx, cipher, []testusers.User{{Email: "zero@example.com"}})

packages/eql/crates/eql-bindings/src/encryption.rs

@@ -84,7 +84,7 @@ pub mod targets;
 use base64::{engine::general_purpose::STANDARD, Engine as _};
 #[allow(deprecated)]
 use stack_encrypt::sem::MatchTerm;
-use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, Tokenizer};
+use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, TermError, Tokenizer};
 use stack_encrypt::target::transcode::{Reader, Transcode, Visitor};
 use stack_encrypt::target::{self, AeadContext, CallerContext};
 use stack_encrypt::{
@@ -286,6 +286,14 @@ impl MatchConfig for EqlMatch {
 /// take it from the same type, so they always agree; matching
 /// cipherstash-client's parameters is not a goal, because cipherstash-client
 /// never writes an `eql_v4` column.
+///
+/// Text that cuts no token (empty, separator-only, or shorter than a 3-gram)
+/// is refused with [`TermError::EmptyTermText`], stored or queried. Stack
+/// Encrypt stores such a value as the empty filter, but EQL treats an empty
+/// filter as contained by every stored filter, and this one type builds both
+/// the stored filter and the query operand, so it cannot tell a query from a
+/// stored value: refusing both is the one way a query cannot match every
+/// row.
 #[doc(hidden)]
 pub struct MatchVisitor;
 impl Visitor for MatchVisitor {
@@ -295,6 +303,9 @@ impl Visitor for MatchVisitor {
     // tree: the one spelling that compiles against both.
     #[allow(deprecated)]
     fn matching<O: MatchConfig>(self, terms: MatchTerm<O>) -> Result<Self::Value, Error> {
+        if terms.positions().is_empty() {
+            return Err(Error::Term(TermError::EmptyTermText));
+        }
         terms
             .positions()
             .iter()

packages/eql/tests/encryption/tests/v4_types.rs

@@ -421,11 +421,15 @@ async fn text_match_filters_contain_their_substrings() {
     .await;
     assert_eq!(named.bf, stored.bf);
 
-    // Text shorter than a 3-gram has no tokens. Stack Encrypt 0.2 refuses it
-    // for a stored value as well as a query, so a short value cannot be
-    // written to a TextMatch column yet.
+    // Text shorter than a 3-gram has no tokens. eql-bindings refuses it for a
+    // stored value as well as a query: one filter type builds both, and an
+    // empty filter as a query would match every row. So a short value cannot
+    // be written to a TextMatch column yet.
     let short: Result<text::TextMatch, _> = keyset.encrypt_as(&"ab".to_owned(), column()).await;
     assert!(short.is_err());
+    let short_query: Result<text::TextMatchQuery, _> =
+        keyset.encrypt_as(&"ab".to_owned(), column()).await;
+    assert!(short_query.is_err(), "a short query would match every row");
 }
 
 /// The filter bits for one query under the test key source. The

packages/stack-encrypt/CHANGELOG.md

@@ -98,6 +98,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Breaking
 
+- **A match value with no tokens is stored as the empty filter, so
+  `KeysetCipher::match_terms` is no longer a query.** In 0.2.0 the same call
+  served write time and query time, and refused text that cuts no token
+  under the match index's tokenizer (empty or separator-only text, or text
+  shorter than the n-gram length) with `TermError::EmptyTermText`. Now it
+  derives the stored filter only, and returns the empty filter (`bf: []`)
+  for such text with no error. **Do not send the output of `match_terms` as
+  a query:** EQL treats an empty filter as contained by every stored filter,
+  and its query domains do not refuse one, so that query returns every row
+  with no error anywhere. A query comes from `cipher.query(op::Matches, ..)`,
+  which still refuses such text with `TermError::EmptyTermText`, when the
+  query is awaited, before a term exists and before any keyset is loaded
+  (`op::Supports::check` now takes the query's value); `dynamic::term`
+  refuses it for a match kind the same way. The stored side is the change a
+  caller sees: one two-letter name no longer fails a whole row, the value
+  seals and opens as any other, and the `dynamic` record path stores it the
+  same way. Storing the empty filter tells a reader of the database that the
+  value is empty, separator-only or shorter than the n-gram length
+  (ADR-0009).
+
 - **The query chain starts with the operator.** `cipher.query(op, v)`
   replaces `cipher.query(v)`: `cipher.query(op::Eq, v).on(&plan)` replaces
   `cipher.query(v).using(&plan).equality()`, and `on` replaces `using` for

packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md

@@ -273,11 +273,17 @@ times `want`, so it takes nothing else.
 `verify` is infallible. It runs on a value that is already decrypted and
 already the type the index was declared over, so the only question it can
 answer is "does this row match", and every other outcome is "no": a JSON
-path absent from the document, text the index would have refused to store.
+path absent from the document, text too short to hold the query's tokens.
 A `bool` is what `Iterator::filter` takes; an error would have no recovery
 except dropping the row, which is `false` by another name. The one input the
-library refuses, a needle too short to tokenise, is refused at
-`cipher.query`, before a term exists.
+library refuses, a query value too short to tokenise, is refused when the query
+is awaited, before a term exists. The stored side refuses nothing: a value
+too short to tokenise is stored as the empty filter, which no query matches,
+so a two-letter name is a valid value in a match column and `verify` is
+never asked about it. The empty filter is visible to anyone who can read the
+column: it tells them the value is empty, separator-only or shorter than the
+n-gram length. A filter's fill already leaks its value's approximate length;
+this is an exact class, and the price of storing such values at all.
 
 `Plaintext` is the decrypted value the term is defined over, and that
 is not always the value the user wrote. For `Matches` it is the text. For