From dc3d345f6f5735ba2ff2b3b7e1720311baab8eea Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 19:38:29 +1100 Subject: [PATCH 01/38] docs(adr): ADR-0009, operators select the index and a query term is not the stored term Records the query-side model for the Rust Stack Encrypt library: the caller names an operator type, the domain type picks the best index that answers it, each (index, operator) has its own query term, and the trait a term implements (Term, Verifiable, Ranked) is its fidelity. Verification runs on the client after decrypt through `verify`, with `target_limit` giving the Bloom false-positive bound; the query loop stays a documented sqlx pattern. Supersedes ADR-0004 decision 5, whose premise that a probe and a stored term are the same bytes holds only for HMAC equality. ADR-0004 gets a note. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- ...t-threaded-through-the-declaration-tree.md | 7 + ...and-a-query-term-is-not-the-stored-term.md | 390 ++++++++++++++++++ 2 files changed, 397 insertions(+) create mode 100644 packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md diff --git a/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md b/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md index 19056b7ae..44ec5e88e 100644 --- a/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md +++ b/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md @@ -3,6 +3,7 @@ status: accepted date: 2026-09-13 revised: 2026-10-04 extends: ADR-0003 +superseded-in-part-by: ADR-0009 (decision 5) --- # One context per target, threaded through the declaration tree @@ -197,6 +198,12 @@ documentation" option below, applied to the one place the type system cannot reach: a probe and a stored term are the same bytes, and the term methods cannot tell which they are producing. +> **Superseded by [ADR-0009](0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md) +> (2026-10-13).** The rule stands; the justification does not. A probe and a +> stored term are the same bytes only for HMAC equality. For every other index +> the query term differs from the stored term, and ADR-0009 makes the operator, +> not the index, the thing a caller names. + ### 6. Targets declare their sources; `FfiValue` is not one A target names the plaintext it accepts (`TextEq` from `Protected`), and diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md new file mode 100644 index 000000000..0e0baaa9a --- /dev/null +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -0,0 +1,390 @@ +--- +status: proposed +date: 2026-10-13 +supersedes: ADR-0004 (decision 5) +extends: ADR-0003, ADR-0007 +--- + +# Operators select the index, and a query term is not the stored term + +[ADR-0004](0004-one-context-per-target-threaded-through-the-declaration-tree.md) +decision 5 kept the standalone term methods as the query path on the grounds +that "a probe and a stored term are the same bytes, and the term methods +cannot tell which they are producing". That premise is wrong, and the API +built on it has the caller name an *index* when what they mean is an +*operator*. + +This ADR records the query-side model for the Rust Stack Encrypt library: +what a caller asks for, how the library chooses the index that answers it, +what it hands back, and how far that answer can be trusted. It is a design +record. No code changes with it. + +Scope is the Rust library only. The TypeScript bindings and the ProtectJS FFI +are not covered; ADR-0007 governs how a binding reaches this surface. + +## The problem + +### A probe and a stored term are not the same bytes + +Decision 5 of ADR-0004 assumed they were. They are not, for any index except +HMAC equality: + +| index | stored term | query term | +| --- | --- | --- | +| HMAC | the HMAC | the same HMAC | +| ORE | left and right ciphertext (right only, in future) | the left ciphertext only | +| OPE | the OPE value | the same OPE value | +| Bloom match | one filter over every n-gram of the value | a filter over the n-grams of the needle: a subset, or bits the stored filter lacks | +| Bloom prefix (planned) | an HMAC per edge n-gram | one HMAC | +| JSON path | a selector and, per leaf, that leaf's term | a selector, or a selector plus a leaf query term | +| similarity (research preview) | `[u64; BANDS]` tokens per row | a probe set of `BANDS × PROBES` tokens | + +The term is a function of the index *and the operator*, not of the index +alone. An ORE index answers `=`, `<` and `BETWEEN` with different query shapes; +a Bloom index answers `Matches` and (if used for ranked search) `Nearest` with +the same stored filter and different queries. + +### The caller names an index, not what they want + +Today the query chain is +`cipher.query(value).using(&plan).index(Match(..))`: the caller picks the +index, and the operator is implied. That has three costs. + +- A domain type with two indexes that both answer an operator leaves the choice + to the caller, who has to know that HMAC is the better equality than ORE. + The original CipherStash client made the choice for them, by operator — and + took the *first* index after an alphabetical sort, so `Eq` went to ORE when + both were present. +- An operator an index cannot answer is not a compile error. Nothing in the + types says which operators `Match(..)` supports. +- There is no place to put operator-specific input. `Between` takes two + values; `Nearest` takes an embedding and a fetch size; a JSON path operator + takes a path. + +### The server's answer is sometimes approximate, and the library says nothing + +A Bloom filter is probabilistic. A row whose filter happens to contain every +bit of the query passes `@@` without containing the needle. The lossy +`orderise` transform behind text ordering collapses case and punctuation, so +`<` on text returns false positives at the boundary. A similarity index returns +candidates by construction. + +The library has never said which terms are exact. A caller who wants ten real +matches has no way to know how many rows to fetch, and no help deciding, after +decrypting, which rows to keep. "We have never implemented client-side +filtering for Bloom filters" is a gap the API should at least leave room for. + +### What the library is + +Stack Encrypt is not a query interface. It produces query terms that a caller +binds into SQL through sqlx, Diesel or (through the bindings) Drizzle and +Prisma, and it decrypts what comes back. It never runs a query. Every decision +below respects that line: the library can tell you how many rows to ask for +and whether a decrypted row is a real match, but the SQL, the `LIMIT` and the +loop are yours. + +## Decision + +### 1. The caller names an operator; the operator is a type + +```rust +let term = cipher.query::("bob@example.com").on(&users::email)?; +let term = cipher.query::(40).on(&users::age)?; +let term = cipher.query::((18, 65)).on(&users::age)?; +let term = cipher.query::("ali").on(&users::name)?; +let term = cipher.query::>(("$.address.city", "Sydney")).on(&users::profile)?; +``` + +Operators are unit types, as in the original client, not a runtime enum. An +operator carries its own input type: `Eq` takes one value, `Between` a pair, +`Nearest` an embedding, `AtPath` a path and `Op`'s input. + +Operators come in three kinds. A **predicate** (`Eq`, `Lt`, `Lte`, `Gt`, +`Gte`, `Between`, `Matches`, `StartsWith`, JSON `Contains`, `ContainedBy`) +filters rows. An **accessor** (`AtPath`) applies another operator at a +JSON path. A **ranking** (`Nearest`) orders rows and takes a fetch size +rather than filtering. + +Two naming points, so they are not re-litigated. `Contains` and `ContainedBy` +are JSON containment (`@>` / `<@`); text search is `Matches`, because a Bloom +match is not a containment. The accessor is `AtPath`, not `At`, because `At` +alone does not say what it is at. + +Case is a parameter of the text operators, defaulting to the common case: +`Matches`, `StartsWith`. A Bloom index +implements the variant its configuration fixes (`downcase` on or off), so a +case-sensitive query against a case-folding index is a type error, not an +empty result. + +### 2. An index declares the operators it supports, and the query term for each + +```rust +trait Index { type Stored; } + +trait Supports: Index { + type Input; // what the operator takes: T, (T, T), an embedding … + type Query: Term; // what this index sends for this operator +} +``` + +`Query` is the query term, and decision 3 says what it must be. It is not +`Stored`. An ORE index's `Supports::Query` is the left ciphertext; its +`Stored` is left and right. A Bloom index's `Supports::Query` is a +filter over the needle; its `Stored` is a filter over the value. + +The bound tightens per operator. `Supports` requires +`Query: Verifiable`; `Supports` requires `Query: Ranked`. An index +cannot claim an inexact operator without saying how its answer is checked. + +### 3. The domain type chooses the best index for an operator + +```rust +trait Answers { type Via: Supports; } + +impl Answers for TextSearch { type Via = Hmac; } +impl Answers for TextSearch { type Via = Ore; } +impl Answers for TextSearch { type Via = Bloom; } +``` + +"Best" is defined per domain type, by hand, and means the most efficient or +the most secure index that answers the operator, as the index types in play +dictate. A domain with both HMAC and ORE answers `Eq` with HMAC: cheaper to +compare, and the ORE term reveals order. This is the decision the original +client made by accident (first after a sort) and this ADR makes on purpose. + +A domain that does not answer an operator has no `Answers` impl, and the +query is a compile error. + +### 4. A query term is one of three things, and the trait it implements is its fidelity + +```rust +pub trait Term: Serialize { type Plaintext; } + +pub trait Verifiable: Term { + fn verify(&self, value: &Self::Plaintext) -> bool; + fn target_limit(&self, want: u64, stats: &Stats) -> Limit; +} + +pub trait Ranked: Term { + fn score(&self, value: &Self::Plaintext) -> Score; + fn fetch_for(&self, want: u64) -> u64; +} +``` + +- A term that implements `Term` only is **exact**. The server's answer is the + answer. HMAC `Eq`, ORE and OPE on numbers (`Lt`, `Between`), JSON + `AtPath`. +- A `Verifiable` term can return **false positives and no false negatives**. + `verify` is the operator's own meaning, run on a decrypted value: it + decides whether a row the server returned is a real match. Bloom `Matches` + and `StartsWith`, and `Lt` on text through the lossy `orderise` (the ties). +- A `Ranked` term returns **candidates in approximate order**. `score` is the + exact score on a decrypted value, for re-ranking. `Nearest` over a + similarity index, and `Nearest` over a Bloom index if used for ranked text + search. + +There is no `Fidelity` enum and no `verify` on exact terms. A caller cannot +write `term.verify(..)` for an HMAC term, because there is nothing to verify, +and the compiler says so. + +`AtPath` inherits the fidelity of the leaf: `AtPath` on a text leaf is +`Verifiable`, as `Lt` on a text column is. + +### 5. Verification is the caller's, after decrypt; the library supplies the predicate and the bound + +The library never runs a query, so it cannot filter the result. What it gives +a caller of a `Verifiable` term is two things. + +`verify` is the predicate. For `Matches`, it folds case, +tokenises both sides with the index's own tokenizer and checks that the +needle's n-grams occur in the value. It lives on the term because the term +already carries the operator's input and the index's configuration, so it +cannot disagree with what was stored. + +`target_limit` is how many rows to ask for to expect `want` real matches. +For a Bloom filter it is computable from what the client knows, plus one +number the client cannot know: + +- `m` bits and `k` hashes: the index configuration. +- `b`: the distinct bits set in the *query* filter. The client built it, so + this is exact, not `k × n-grams`. +- `f`: the fill of a *stored* filter, `1 − e^(−k·n/m)` for `n` stored tokens. + The client cannot see `n` per row; it takes either a declared estimate or a + measured fill. +- `N`: the rows the predicate scans. + +A non-matching row passes when every one of the `b` query bits is set in its +filter: `p ≈ f^b`. False positives are then `Poisson(λ = p·N)`, and the +worst case that still satisfies the caller is exactly `want` true matches, so + +``` +target_limit(want, N) = want + Poisson_quantile(λ = p·N, confidence) +``` + +If fewer than `want` verified rows come back, there were fewer than `want` +matches and the caller has them all. + +With the EQL v4 defaults (3-grams, `k = 6`, `m = 2048`), a million rows and 99% +confidence: + +| stored tokens | needle | `b` | `p` | `λ` | `LIMIT` | +| --- | --- | --- | --- | --- | --- | +| 20 (a name) | `ali` | 6 | 3.4e-8 | 0.03 | `want + 1` | +| 200 (a paragraph) | `ali` | 6 | 7.3e-3 | 7300 | `want + ~7500` | +| 200 | `alice` | 18 | 3.8e-7 | 0.4 | `want + 2` | + +The bad case is a short needle against long stored text, and the library can +say so before the query runs. `λ` is also the number of decrypts that will be +wasted, which is a better signal for tuning `m` and `k` than anything the +library reports today. + +`N` and `f` come from the server, always. The library takes them as numbers in +`Stats` and does not care where they came from. The documented sources are: + +- `SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass` — a catalog + read, approximate, cached at startup. Staleness only moves the confidence; + `verify` keeps the result correct. +- `SELECT count(*) FROM users WHERE name @@ $1` — the candidate count + `C = T + FP`, index-only. `C` is the true minimum: fetch `C` rows and every + real match is among them. `p·N` then says how much of `C` is noise. +- `SELECT avg(cardinality(bf)) FROM users` — `bf` stores set-bit positions, + so this is the mean fill directly, and removes the token estimate. + +`Lt` on text has no `p`: the false positives are the rows that collapse onto +the boundary key, which depends on the data. Its `target_limit` is +`Limit::Unbounded`, and the caller uses the loop below. A `Ranked` term has no +`target_limit`; `fetch_for` applies the index's configured recall factor. + +### 6. The loop is a documented pattern, not a library helper + +A `Verifiable` term composes with the caller's query library like this. One +shot, with a bound: + +```rust +let term = cipher.query::("ali").on(&users::name)?; +let fetch = term.target_limit(10, &stats).fetch(); + +let rows: Vec = sqlx::query_as( + "SELECT id, name FROM users WHERE name @@ $1 ORDER BY id LIMIT $2", +) +.bind(sqlx::types::Json(&term)) +.bind(fetch as i64) +.fetch_all(&pool).await?; + +let users = cipher.decrypt_all(rows).await?; +let hits: Vec = users.into_iter().filter(|u| term.verify(&u.name)).take(10).collect(); +``` + +Keyset paging, which needs no `Stats` and works for `Limit::Unbounded`: + +```rust +let term = cipher.query::("ali").on(&users::name)?; +let (mut hits, mut after, page) = (Vec::new(), 0_i64, 50_i64); + +while hits.len() < 10 { + let rows: Vec = sqlx::query_as( + "SELECT id, name FROM users WHERE name @@ $1 AND id > $2 ORDER BY id LIMIT $3", + ) + .bind(sqlx::types::Json(&term)).bind(after).bind(page) + .fetch_all(&pool).await?; + if rows.is_empty() { break; } + after = rows.last().unwrap().id; + hits.extend(cipher.decrypt_all(rows).await?.into_iter().filter(|u| term.verify(&u.name))); +} +hits.truncate(10); +``` + +The library ships `verify` and `target_limit`. It does not ship the loop, +because the loop runs queries. That may change when Stack Similarity lands and +the ranked loop is worth packaging; it is deferred, not refused. + +### 7. `Between` is one query + +`cipher.query::((lo, hi))` produces one term carrying both ORE left +ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. It is +one operand in SQL. Two `Gte`/`Lte` terms would be two round trips through +the term path and two operands the planner cannot see as a range. + +### 8. One query operand domain per (family, term) in EQL v4 + +The EQL v4 side carries one query domain per (type family, term), such as +`eql_v4.query_text_match` and `eql_v4.query_integer_range`, rather than one per +operator or one per stored domain. This is the "option B" of the design +discussion and is recorded here so the Rust side's `Supports::Query` has a +fixed target: the query term for an (index, operator) serialises to exactly +one of those domains. + +## Considered options + +**Keep the index as the thing the caller names** (status quo). Rejected for +the three costs in the problem statement. It is also what the review of +#1164 ran into: storing short `TextMatch` text needs the stored term and the +query term for a Bloom filter to differ, and the API had no place to say so. + +**Operators as a runtime enum**, as `cipherstash-client` did. Rejected: an +unsupported operator becomes a runtime error, and operator-specific input has +to be a tagged union. Rust can carry both in the type. + +**First index that supports the operator**, as `cipherstash-client` did. +Rejected: the choice is per domain type and must be deliberate, because it +decides what the query term reveals. + +**A `Fidelity` enum on every term, and `verify` that returns `true` for exact +terms**. Considered and dropped: a method that is a no-op on most terms is +noise, and a runtime enum says at runtime what the trait says at compile +time. + +**The library returns candidates and verifies them itself** +(`cipher.query(..).fetch(&pool)`). Rejected: the library would run queries, +own `LIMIT` and paging, and need a driver. That is a query interface, which +this library is not. + +**Two `Gte` and `Lte` terms for a range.** Rejected, decision 7. + +## Consequences + +ADR-0004 decision 5 is superseded. Its rule — a stored term goes through a +target, a probe comes from the query path — stands, but its justification was +wrong and is replaced: the two are different bytes for every index but HMAC, +and the query path is `cipher.query::(..)`, which produces a term the +stored side never would. + +A caller who wants a real-match count or offset paging on a `Verifiable` term +cannot have either from the server. `COUNT(*)` is an upper bound, and +`OFFSET` shifts with the false positives. Keyset paging only. This is the most +visible change for users and is the reason to settle it now. + +Only the inexact terms in a `WHERE` need verifying. `age > $2 AND name @@ $1` +verifies `name` and not `age`. + +Verification costs decrypts, not database work, and tells the server nothing +it does not already know. The cost is `λ` ZeroKMS calls per query, so the +Bloom parameters now have a number to be tuned against. + +A verified row needs its ciphertext. EQL v4 values carry `c` beside their +terms. A similarity index stores its vector separately, so a `Nearest` query +must select it too. + +The lossy text-order case falls out without a special case: EQL's ADR-0002 +(the value model, #1139) already refuses equality through `orderise`, and `Lt` +on text is now simply a `Verifiable` term. + +It is a breaking change to the query chain. `QueryUsing::index(..)` goes; +`QueryUsing::equality()` becomes `query::`. The write side (`encrypt`, +targets, plans) does not change. + +## Deferred + +- The SQL spelling of `Between`: `age <@ $1` through an operator on the range + domain, or `eql_v4.between(age, $1)`. Either must stay inlinable. +- Whether `Nearest` is `cipher.query::(..)` or a separate + `cipher.nearest(..)` entry, and whether a ranked query can carry further + predicates. The latter is possible only when the server can join candidates + to rows (the `ByRow` similarity mode), not in the per-band pseudonymous mode. +- Whether a match index declares `expected_tokens` so `target_limit` has a + default fill without a stats query. +- `ORDER BY cardinality(bf)` on the EQL side, so low-fill rows — the least + likely false positives — come first. It reveals nothing `bf` does not. +- Whether `Nearest` over Bloom scores by bit overlap or needs token-level + terms. An index-design question, not an API one. +- Packaging the keyset and ranked loops once Stack Similarity exists. From 891d30a312d20b68a34474d7b8d54d3cd1135cca Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 21:07:45 +1100 Subject: [PATCH 02/38] docs(adr): ADR-0009 review round, with sqlx and language-SDK appendices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the first review of #1177. - "probe" becomes "query term" throughout, except where ADR-0004 is quoted. - The central claim is index-specific: HMAC's query term is the stored term; OPE's is only when its order encoding is lossless, and cllw-ore collates text before encrypting it. ADR-0004's note says the same. - `Verifiable` is renamed `Probabilistic`; `fetch_for` becomes `target_limit` on `Ranked` too, with one meaning on both traits. `verify` is infallible, and the ADR says why. `Term::Plaintext` is the decrypted value the term is defined over, which for a similarity index is the embedding or the sketch. - The Bloom bound uses the mean of per-row `f_i^b`, not the mean fill raised to `b`, with the mixed-length example showing the gap. The statistics are a set-bit histogram from one sampled aggregate, or the declared `expected_tokens` prior, which is decided rather than deferred. The 200-token row of the worked table is corrected (λ 7,600, limit want + ~7,800; the `alice` row is want + 3). - Completeness is conditional on the server returning fewer candidates than the limit; the one-shot example checks it, and the ADR says what 99% means. - The cost of verification is λ wasted rows in a batched key retrieval, not λ ZeroKMS calls. - "Keyset paging" is "cursor paging", defined, with a paragraph on LIMIT, OFFSET and cursors. `Between` is spelled with SQL's BETWEEN and two parameters; the planner claim was wrong and is withdrawn. - The range-as-two-terms option says Gte and Lte stay. ADR-0007 is flagged for amendment. A tuning example follows the worked table. - Appendix A: each operator as the Rust call, the sqlx statement with EQL v4 casts, and what the caller does after decrypt. - Appendix B: what crosses the Go and TypeScript bindings under this model, what the generated Go and the TypeScript types look like, and the same query in sqlc, pgx, GORM, Drizzle, Prisma and Kysely. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- ...t-threaded-through-the-declaration-tree.md | 9 +- ...and-a-query-term-is-not-the-stored-term.md | 680 +++++++++++++++--- 2 files changed, 596 insertions(+), 93 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md b/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md index 44ec5e88e..111da4472 100644 --- a/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md +++ b/packages/stack-encrypt/docs/adr/0004-one-context-per-target-threaded-through-the-declaration-tree.md @@ -199,10 +199,11 @@ reach: a probe and a stored term are the same bytes, and the term methods cannot tell which they are producing. > **Superseded by [ADR-0009](0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md) -> (2026-10-13).** The rule stands; the justification does not. A probe and a -> stored term are the same bytes only for HMAC equality. For every other index -> the query term differs from the stored term, and ADR-0009 makes the operator, -> not the index, the thing a caller names. +> (2026-10-13).** The rule stands; the justification does not. A query term +> and a stored term are the same bytes for HMAC equality, and for OPE only +> when its order encoding is lossless; for every other index the query term +> differs from the stored term, and ADR-0009 makes the operator, not the +> index, the thing a caller names. ### 6. Targets declare their sources; `FfiValue` is not one diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 0e0baaa9a..d652690c9 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -19,25 +19,37 @@ what a caller asks for, how the library chooses the index that answers it, what it hands back, and how far that answer can be trusted. It is a design record. No code changes with it. -Scope is the Rust library only. The TypeScript bindings and the ProtectJS FFI -are not covered; ADR-0007 governs how a binding reaches this surface. +Scope is the Rust library. The decisions bind the Rust API; the bindings and +the language SDKs are governed by ADR-0007, which will need an amendment once +this ADR is accepted, because the data grammar a binding lowers has no +operator in it today. [Appendix B](#appendix-b-the-model-in-go-and-typescript) +sketches what that amendment would carry, so the model can be checked against +two more languages before it is built. + +Two words, so they are not re-litigated. The value a caller sends to the +server to be compared against stored values is a **query term**, or a query. +ADR-0004 called it a probe; "probe" is not used below except when quoting +that ADR. The value a write puts in the column is the **stored term**. ## The problem -### A probe and a stored term are not the same bytes +### A query term and a stored term are not the same bytes -Decision 5 of ADR-0004 assumed they were. They are not, for any index except -HMAC equality: +Decision 5 of ADR-0004 assumed they were. In general they are not: | index | stored term | query term | | --- | --- | --- | | HMAC | the HMAC | the same HMAC | | ORE | left and right ciphertext (right only, in future) | the left ciphertext only | -| OPE | the OPE value | the same OPE value | +| OPE | the OPE value of the order key | the same value, when the order encoding is lossless; for text the key is collated first (cllw-ore's `orderize_string` folds `café` and `cafe` together), so equal terms do not mean equal plaintexts | | Bloom match | one filter over every n-gram of the value | a filter over the n-grams of the needle: a subset, or bits the stored filter lacks | | Bloom prefix (planned) | an HMAC per edge n-gram | one HMAC | | JSON path | a selector and, per leaf, that leaf's term | a selector, or a selector plus a leaf query term | -| similarity (research preview) | `[u64; BANDS]` tokens per row | a probe set of `BANDS × PROBES` tokens | +| similarity (research preview) | `[u64; BANDS]` tokens per row | a query set of `BANDS × PROBES` tokens | + +HMAC equality is the one index whose query term is byte-for-byte the stored +term, and whose comparison means plaintext equality. OPE comes close only when +its encoding is lossless, as it is for integers. The others differ in kind. The term is a function of the index *and the operator*, not of the index alone. An ORE index answers `=`, `<` and `BETWEEN` with different query shapes; @@ -64,8 +76,8 @@ index, and the operator is implied. That has three costs. ### The server's answer is sometimes approximate, and the library says nothing A Bloom filter is probabilistic. A row whose filter happens to contain every -bit of the query passes `@@` without containing the needle. The lossy -`orderise` transform behind text ordering collapses case and punctuation, so +bit of the query passes `@@` without containing the needle. The collating +`orderize_string` transform behind text ordering folds case and diacritics, so `<` on text returns false positives at the boundary. A similarity index returns candidates by construction. @@ -127,14 +139,16 @@ trait Supports: Index { } ``` -`Query` is the query term, and decision 3 says what it must be. It is not +`Query` is the query term, and decision 3 says what it must be. `Term` and +the two traits that refine it are defined in decision 4. `Query` is not `Stored`. An ORE index's `Supports::Query` is the left ciphertext; its `Stored` is left and right. A Bloom index's `Supports::Query` is a filter over the needle; its `Stored` is a filter over the value. The bound tightens per operator. `Supports` requires -`Query: Verifiable`; `Supports` requires `Query: Ranked`. An index -cannot claim an inexact operator without saying how its answer is checked. +`Query: Probabilistic`; `Supports` requires `Query: Ranked`. An +index cannot claim an inexact operator without saying how its answer is +checked. ### 3. The domain type chooses the best index for an operator @@ -160,40 +174,67 @@ query is a compile error. ```rust pub trait Term: Serialize { type Plaintext; } -pub trait Verifiable: Term { +pub trait Probabilistic: Term { fn verify(&self, value: &Self::Plaintext) -> bool; fn target_limit(&self, want: u64, stats: &Stats) -> Limit; } pub trait Ranked: Term { fn score(&self, value: &Self::Plaintext) -> Score; - fn fetch_for(&self, want: u64) -> u64; + fn target_limit(&self, want: u64) -> u64; } ``` - A term that implements `Term` only is **exact**. The server's answer is the answer. HMAC `Eq`, ORE and OPE on numbers (`Lt`, `Between`), JSON `AtPath`. -- A `Verifiable` term can return **false positives and no false negatives**. - `verify` is the operator's own meaning, run on a decrypted value: it - decides whether a row the server returned is a real match. Bloom `Matches` - and `StartsWith`, and `Lt` on text through the lossy `orderise` (the ties). +- A `Probabilistic` term can return **false positives and no false + negatives**. `verify` is the operator's own meaning, run on a decrypted + value: it decides whether a row the server returned is a real match. Bloom + `Matches` and `StartsWith`, and `Lt` on text through the collating + `orderize_string` (the ties). - A `Ranked` term returns **candidates in approximate order**. `score` is the exact score on a decrypted value, for re-ranking. `Nearest` over a similarity index, and `Nearest` over a Bloom index if used for ranked text search. +`target_limit` has one meaning on both traits: *fetch this many candidate +rows so that, at the stated confidence, `want` of them survive*. On a +`Probabilistic` term the survivors are the rows `verify` keeps, and the +number depends on the table, so it takes `Stats` and may answer +`Limit::Unbounded`. On a `Ranked` term the survivors are the top `want` after +`score` re-ranks, and the number is the index's configured recall factor +times `want`, so it takes nothing else. + +`verify` is infallible. It runs on a value that is already decrypted and +already the type the index was declared over, so the only question it can +answer is "does this row match", and every other outcome is "no": a JSON +path absent from the document, text the index would have refused to store. +A `bool` is what `Iterator::filter` takes; an error would have no recovery +except dropping the row, which is `false` by another name. The one input the +library refuses, a needle too short to tokenise, is refused at +`cipher.query`, before a term exists. + +`Term::Plaintext` is the decrypted value the term is defined over, and that +is not always the value the user wrote. For `Matches` it is the text. For +`Nearest` over a similarity index it is the stored embedding, or the LSH +sketch when only the sketch is stored; `score` is exact with respect to what +was stored, not to a document the row never held. For `AtPath` it is the +JSON document, and `verify` applies `Op` at the path. A query has to select +whatever `Plaintext` is, which decision 5 and the consequences return to. + There is no `Fidelity` enum and no `verify` on exact terms. A caller cannot -write `term.verify(..)` for an HMAC term, because there is nothing to verify, -and the compiler says so. +write `term.verify(..)` for an HMAC term: verification would pass for every +row the server returned, so the method would say nothing, and the compiler +says so. `AtPath` inherits the fidelity of the leaf: `AtPath` on a text leaf is -`Verifiable`, as `Lt` on a text column is. +`Probabilistic`, as `Lt` on a text column is. ### 5. Verification is the caller's, after decrypt; the library supplies the predicate and the bound The library never runs a query, so it cannot filter the result. What it gives -a caller of a `Verifiable` term is two things. +a caller of a `Probabilistic` term is two things. `verify` is the predicate. For `Matches`, it folds case, tokenises both sides with the index's own tokenizer and checks that the @@ -202,63 +243,134 @@ already carries the operator's input and the index's configuration, so it cannot disagree with what was stored. `target_limit` is how many rows to ask for to expect `want` real matches. -For a Bloom filter it is computable from what the client knows, plus one -number the client cannot know: +For a Bloom filter it is computable from what the client knows, plus what +the server has to tell it: - `m` bits and `k` hashes: the index configuration. - `b`: the distinct bits set in the *query* filter. The client built it, so this is exact, not `k × n-grams`. -- `f`: the fill of a *stored* filter, `1 − e^(−k·n/m)` for `n` stored tokens. - The client cannot see `n` per row; it takes either a declared estimate or a - measured fill. +- `f_i`: the fill of the stored filter in row `i`, `1 − e^(−k·n_i/m)` for + `n_i` stored tokens. It is per row, because values differ in length. - `N`: the rows the predicate scans. -A non-matching row passes when every one of the `b` query bits is set in its -filter: `p ≈ f^b`. False positives are then `Poisson(λ = p·N)`, and the -worst case that still satisfies the caller is exactly `want` true matches, so +A non-matching row `i` passes when every one of the `b` query bits is set in +its filter, with probability about `f_i^b`. Over the table the pass rate is +the *mean of* `f_i^b`, which is not the mean fill raised to `b`: for `b > 1` +the mean of a power exceeds the power of the mean, and a column of mixed +lengths makes the gap large. Half a million rows of 20 tokens and half a +million of 200, with `b = 6`, pass at a mean rate of 3.8e-3 (3,800 false +positives); the mean fill, 0.25, raised to 6 predicts 245. The number the +bound needs is therefore `p = mean(f_i^b)`, and the statistics the client +holds must let it compute that for the `b` of each query. + +False positives are then `Poisson(λ = p·N)`, and the worst case that still +satisfies the caller is exactly `want` true matches preceded by every false +positive, so ``` target_limit(want, N) = want + Poisson_quantile(λ = p·N, confidence) ``` -If fewer than `want` verified rows come back, there were fewer than `want` -matches and the caller has them all. +This is a probabilistic bound, and the ADR does not pretend otherwise. At 99% +confidence one query in a hundred draws more false positives than the +quantile, and if those rows sort before a real match under the query's +`ORDER BY`, the fetch holds fewer than `want` real rows while more exist. The +caller can tell the two apart from one number it already has: if the server +returned *fewer candidates than the limit*, the predicate is exhausted, and +fewer than `want` verified rows means there were fewer than `want` matches. +If it returned exactly the limit, the result is a page, not an answer, and +the cursor loop in decision 6 continues from it. `verify` makes every +returned row correct; only the candidate count makes the result complete. + +#### Declared and measured statistics + +`N` and the fill distribution come from the server, always. The library +takes them as numbers in `Stats` and does not care where they came from. +There are two sources, and a column has the first one from the day it is +declared. + +**Declared.** A match index declares `expected_tokens`, the typical token +count of the values it will hold, beside `m` and `k`: 20 for a name column, +200 for a notes column. From it `Stats::declared(rows)` takes +`f = 1 − e^(−k·expected_tokens/m)` and `p = f^b`. This is a prior, and +`rows` is a planning estimate too. It is enough to warn before a query runs +(a three-letter needle against a notes column) and to size a first fetch; it +knows nothing about the spread of lengths, so it carries the mean-fill error +above. + +**Measured.** The fill of a stored filter is `cardinality(bf) / m`, and `bf` +is the set-bit positions, so a histogram of set-bit counts is a histogram of +fills. One aggregate gives it, at most `m` buckets: + +```sql +SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) +FROM users TABLESAMPLE SYSTEM (1) +GROUP BY 1 +``` -With the EQL v4 defaults (3-grams, `k = 6`, `m = 2048`), a million rows and 99% -confidence: +`Stats::measured(rows, histogram)` keeps the buckets, and `target_limit` +computes `p = Σ count_i · (bits_i / m)^b / Σ count_i` for whatever `b` the +query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a +page sample, which is fine for a distribution and unsuited to an exact count. +`rows` comes from `SELECT reltuples FROM pg_class WHERE oid = +'users'::regclass`, a catalog read. + +Where the numbers live and who keeps them in step was asked on the review, +so it is written down. They are read from the server by the application, at +startup or on a timer, and cached in the client as plain numbers. Nothing +about them is secret: the set-bit count of a stored filter is visible to the +server already, since the server holds the filter, and the row count is in +the catalog. Several clients of one database each read the same table and +need no coordination; two that disagree are both approximately right. A +stale `Stats` moves the confidence, never the correctness, because `verify` +still decides every row. A column that has never been measured uses the +declared prior. + +A third statistic answers a different question. `SELECT count(*) FROM users +WHERE name @@ $1` is the candidate count `C = T + FP`, index-only. `C` is the +true minimum: fetch `C` rows and every real match is among them. It is a +count per query rather than a cached statistic, and `p·N` then says how much +of `C` is noise. + +#### Worked numbers + +With the EQL v4 defaults (3-grams, `k = 6`, `m = 2048`), a million rows, every +row at the stated length and 99% confidence: | stored tokens | needle | `b` | `p` | `λ` | `LIMIT` | | --- | --- | --- | --- | --- | --- | | 20 (a name) | `ali` | 6 | 3.4e-8 | 0.03 | `want + 1` | -| 200 (a paragraph) | `ali` | 6 | 7.3e-3 | 7300 | `want + ~7500` | -| 200 | `alice` | 18 | 3.8e-7 | 0.4 | `want + 2` | +| 200 (a paragraph) | `ali` | 6 | 7.6e-3 | 7,600 | `want + ~7,800` | +| 200 | `alice` | 18 | 4.4e-7 | 0.44 | `want + 3` | The bad case is a short needle against long stored text, and the library can -say so before the query runs. `λ` is also the number of decrypts that will be -wasted, which is a better signal for tuning `m` and `k` than anything the -library reports today. - -`N` and `f` come from the server, always. The library takes them as numbers in -`Stats` and does not care where they came from. The documented sources are: - -- `SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass` — a catalog - read, approximate, cached at startup. Staleness only moves the confidence; - `verify` keeps the result correct. -- `SELECT count(*) FROM users WHERE name @@ $1` — the candidate count - `C = T + FP`, index-only. `C` is the true minimum: fetch `C` rows and every - real match is among them. `p·N` then says how much of `C` is noise. -- `SELECT avg(cardinality(bf)) FROM users` — `bf` stores set-bit positions, - so this is the mean fill directly, and removes the token estimate. +say so before the query runs. `λ` is also the number of decrypts a query will +waste, which gives the Bloom parameters a number to be tuned against. Taking +the bad row: a needle one character longer (`alic`, two 3-grams, `b = 12`) +brings `λ` to 58; doubling `m` to 4,096 brings it to 268 at `b = 6`, and +`m = 8,192` to 6. Halving `k` to 3 at `m = 2,048` also gives 268, since fewer +hashes set fewer bits, but it weakens the filter against longer needles. The +cheapest lever is often a minimum needle length for long-text columns, which +a `Probabilistic` term can report from `target_limit` alone. + +#### Text ordering `Lt` on text has no `p`: the false positives are the rows that collapse onto the boundary key, which depends on the data. Its `target_limit` is -`Limit::Unbounded`, and the caller uses the loop below. A `Ranked` term has no -`target_limit`; `fetch_for` applies the index's configured recall factor. +`Limit::Unbounded`, and the caller uses the cursor loop below. Its use is +ordering and cursor paging on names (`ORDER BY name`, "the page after +Nguyen"), and EQL v4 ships `query_text_ord` for it. Whether the Rust library +produces that term is a catalog question (`TextOrdOre` is unproducible +today, pending cllw-ore); nothing else in this ADR depends on it, and if +text ordering is dropped this paragraph goes with it. + +A `Ranked` term's `target_limit` applies the index's configured recall +factor and needs no `Stats`. ### 6. The loop is a documented pattern, not a library helper -A `Verifiable` term composes with the caller's query library like this. One -shot, with a bound: +A `Probabilistic` term composes with the caller's query library like this. One +shot, with a bound, and the completeness check from decision 5: ```rust let term = cipher.query::("ali").on(&users::name)?; @@ -271,11 +383,17 @@ let rows: Vec = sqlx::query_as( .bind(fetch as i64) .fetch_all(&pool).await?; +let exhausted = (rows.len() as u64) < fetch; let users = cipher.decrypt_all(rows).await?; let hits: Vec = users.into_iter().filter(|u| term.verify(&u.name)).take(10).collect(); +// hits.len() < 10 && exhausted: there are no more matches. +// hits.len() < 10 && !exhausted: page on from the last id. ``` -Keyset paging, which needs no `Stats` and works for `Limit::Unbounded`: +Cursor paging, which needs no `Stats`, works for `Limit::Unbounded`, and is +the form that is complete by construction. A cursor is the last row's value of +an exactly ordered column, carried into the next page as `WHERE id > $cursor` +instead of an `OFFSET`: ```rust let term = cipher.query::("ali").on(&users::name)?; @@ -294,25 +412,48 @@ while hits.len() < 10 { hits.truncate(10); ``` +On paging generally. `LIMIT` and `OFFSET` are not going anywhere: every SQL +library emits them and legacy applications page with them. The one that is +unsound over a `Probabilistic` predicate is `OFFSET`, because the false +positives shift it, and a page of real rows is not a fixed distance into the +candidates. `LIMIT` with the bound is sound for one page; a cursor is sound +for every page. Two further options are on the EQL side and listed under +deferred: ordering candidates by fill, so the least likely false positives +come first, and the candidate count `C`, which turns "how far through" into +a number. + The library ships `verify` and `target_limit`. It does not ship the loop, because the loop runs queries. That may change when Stack Similarity lands and the ranked loop is worth packaging; it is deferred, not refused. -### 7. `Between` is one query +### 7. `Between` is one term, spelled `BETWEEN` `cipher.query::((lo, hi))` produces one term carrying both ORE left -ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. It is -one operand in SQL. Two `Gte`/`Lte` terms would be two round trips through -the term path and two operands the planner cannot see as a range. +ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. One +call derives both bounds, under one context, so a caller cannot pair a bound +from one column with a bound from another. + +In SQL it is the native `BETWEEN`, with the term's two halves bound in order: + +```sql +WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore +``` + +PostgreSQL rewrites `a BETWEEN x AND y` to `a >= x AND a <= y` when it parses +the statement, so there is no range the planner could see and this spelling +could hide; an earlier draft of this decision said otherwise and was wrong. A +btree over the ORE term uses both bounds as one range scan either way. The +term exposes its halves (`let (lo, hi) = term.bounds();`) for drivers that +bind one value per parameter, which is all of them. ### 8. One query operand domain per (family, term) in EQL v4 The EQL v4 side carries one query domain per (type family, term), such as -`eql_v4.query_text_match` and `eql_v4.query_integer_range`, rather than one per -operator or one per stored domain. This is the "option B" of the design +`eql_v4.query_text_match` and `eql_v4.query_integer_ord_ore`, rather than one +per operator or one per stored domain. This is the "option B" of the design discussion and is recorded here so the Rust side's `Supports::Query` has a fixed target: the query term for an (index, operator) serialises to exactly -one of those domains. +one of those domains. `Between` serialises to two values of one domain. ## Considered options @@ -334,40 +475,61 @@ terms**. Considered and dropped: a method that is a no-op on most terms is noise, and a runtime enum says at runtime what the trait says at compile time. +**A fallible `verify`.** Considered and dropped, decision 4: every failure +is a non-match. + **The library returns candidates and verifies them itself** (`cipher.query(..).fetch(&pool)`). Rejected: the library would run queries, own `LIMIT` and paging, and need a driver. That is a query interface, which this library is not. -**Two `Gte` and `Lte` terms for a range.** Rejected, decision 7. +**A range as two terms**, `query::(lo)` and `query::(hi)` bound as +two predicates. Rejected in favour of one `Between` term, decision 7. `Gte` +and `Lte` themselves stay; they are the open-ended comparisons, and a range +is not two of them. + +**`expected_tokens` deferred until the stats query exists.** Dropped: the +declared prior is what makes `target_limit` usable on a column nobody has +measured, and it costs one number in the index declaration. Decision 5. ## Consequences ADR-0004 decision 5 is superseded. Its rule — a stored term goes through a -target, a probe comes from the query path — stands, but its justification was -wrong and is replaced: the two are different bytes for every index but HMAC, -and the query path is `cipher.query::(..)`, which produces a term the -stored side never would. +target, a query term comes from the query path — stands, but its +justification was wrong and is replaced: the two differ for every index but +HMAC and lossless OPE, and the query path is `cipher.query::(..)`, which +produces a term the stored side never would. + +ADR-0007 needs an amendment once this ADR is accepted. The plan a binding +lowers names a field and, for a query, an index; it has no operator and no +operator input, and the generated code of a language SDK has no `verify`. +Appendix B is the sketch to amend from. -A caller who wants a real-match count or offset paging on a `Verifiable` term -cannot have either from the server. `COUNT(*)` is an upper bound, and -`OFFSET` shifts with the false positives. Keyset paging only. This is the most -visible change for users and is the reason to settle it now. +A caller who wants a real-match count or offset paging on a `Probabilistic` +term cannot have either from the server. `COUNT(*)` is an upper bound, and +`OFFSET` shifts with the false positives. Cursor paging only. This is the +most visible change for users and is the reason to settle it now. Only the inexact terms in a `WHERE` need verifying. `age > $2 AND name @@ $1` verifies `name` and not `age`. Verification costs decrypts, not database work, and tells the server nothing -it does not already know. The cost is `λ` ZeroKMS calls per query, so the -Bloom parameters now have a number to be tuned against. - -A verified row needs its ciphertext. EQL v4 values carry `c` beside their -terms. A similarity index stores its vector separately, so a `Nearest` query -must select it too. - -The lossy text-order case falls out without a special case: EQL's ADR-0002 -(the value model, #1139) already refuses equality through `orderise`, and `Lt` -on text is now simply a `Verifiable` term. +it does not already know. The cost is `λ` wasted rows per query: `λ` more +rows on the wire and `λ` more entries in the batched key retrieval the real +rows ride in anyway, one request per 500 values per keyset. The network call +count is set by the batch size and the number of keysets, not by `λ`; the +decrypt work and the wire bytes are. That is the number the Bloom parameters +are tuned against. + +A verified row needs its `Plaintext`, which decision 4 says is not always +the user's value. EQL v4 values carry `c` beside their terms, so a text or +JSON column's query already selects it. A similarity index stores its vector +separately, so a `Nearest` query must select the embedding, or the sketch, +as well. + +The collating text-order case falls out without a special case: EQL's +ADR-0002 (the value model, #1139) already refuses equality through +`orderize`, and `Lt` on text is now simply a `Probabilistic` term. It is a breaking change to the query chain. `QueryUsing::index(..)` goes; `QueryUsing::equality()` becomes `query::`. The write side (`encrypt`, @@ -375,16 +537,356 @@ targets, plans) does not change. ## Deferred -- The SQL spelling of `Between`: `age <@ $1` through an operator on the range - domain, or `eql_v4.between(age, $1)`. Either must stay inlinable. - Whether `Nearest` is `cipher.query::(..)` or a separate `cipher.nearest(..)` entry, and whether a ranked query can carry further predicates. The latter is possible only when the server can join candidates to rows (the `ByRow` similarity mode), not in the per-band pseudonymous mode. -- Whether a match index declares `expected_tokens` so `target_limit` has a - default fill without a stats query. -- `ORDER BY cardinality(bf)` on the EQL side, so low-fill rows — the least - likely false positives — come first. It reveals nothing `bf` does not. +- `ORDER BY cardinality(eql_v4.match_term(col))` on the EQL side, so + low-fill rows — the least likely false positives — come first. It reveals + nothing `bf` does not. - Whether `Nearest` over Bloom scores by bit overlap or needs token-level terms. An index-design question, not an API one. -- Packaging the keyset and ranked loops once Stack Similarity exists. +- Packaging the cursor and ranked loops once Stack Similarity exists. + +## Appendix A: the terms in sqlx + +Each operator below is shown as the Rust call that produces the term, the +SQL it binds into, and what the caller does with the rows. The casts name +EQL v4's query domains (decision 8), as the v4 tests bind them today: +`$1::eql_v4.query_text_eq`, with the JSON text of the term as the parameter. +The JSON operators are spelled as EQL v3 ships them, since v4 does not carry +`json` yet. `UserRow` is the row as the driver returns it, with the EQL +columns as `sqlx::types::Json`, and `cipher.decrypt_all` turns rows +into `User`s. + +**`Eq`, exact.** The term is an HMAC; the operator is `=`. Nothing to verify. + +```rust +let term = cipher.query::("bob@example.com").on(&users::email)?; + +let rows: Vec = sqlx::query_as( + "SELECT id, email, name FROM users WHERE email = $1::eql_v4.query_text_eq", +) +.bind(sqlx::types::Json(&term)) +.fetch_all(&pool).await?; +let users = cipher.decrypt_all(rows).await?; +``` + +**`Lt` on an integer, exact.** The term is the ORE left ciphertext; the +operator is `<`. `Lte`, `Gt` and `Gte` are the same shape. + +```rust +let term = cipher.query::(40).on(&users::age)?; + +let rows: Vec = sqlx::query_as( + "SELECT id, email, name, age FROM users WHERE age < $1::eql_v4.query_integer_ord_ore ORDER BY age", +) +.bind(sqlx::types::Json(&term)) +.fetch_all(&pool).await?; +``` + +**`Between`, exact, one term and two parameters** (decision 7). + +```rust +let term = cipher.query::((18, 65)).on(&users::age)?; +let (lo, hi) = term.bounds(); + +let rows: Vec = sqlx::query_as( + "SELECT id, email, name, age FROM users + WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore", +) +.bind(sqlx::types::Json(lo)) +.bind(sqlx::types::Json(hi)) +.fetch_all(&pool).await?; +``` + +**`Matches`, probabilistic.** The term is a Bloom filter over the needle's +3-grams, as `{"bf": [59, 96, …]}` in the fixture the Go SDK tests against; +the operator is `@@`, column on the left. The query selects the ciphertext +(`name` carries `c`) because `verify` needs the decrypted text, and the +`LIMIT` comes from `target_limit`: + +```rust +let term = cipher.query::("ali").on(&users::name)?; +let fetch = term.target_limit(10, &stats).fetch(); + +let rows: Vec = sqlx::query_as( + "SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2", +) +.bind(sqlx::types::Json(&term)) +.bind(fetch as i64) +.fetch_all(&pool).await?; + +let exhausted = (rows.len() as u64) < fetch; +let hits: Vec = cipher.decrypt_all(rows).await? + .into_iter() + .filter(|u| term.verify(&u.name)) + .take(10) + .collect(); +``` + +The `Stats` it used were read once, at startup, by the application: + +```rust +let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass") + .fetch_one(&pool).await?; +let histogram: Vec<(i16, i64)> = sqlx::query_as( + "SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) + FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1", +) +.fetch_all(&pool).await?; +let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u16, n as u64))); +``` + +or, before anyone has measured, `Stats::declared(1_000_000)`, which uses the +`expected_tokens` the index declares. + +**`Matches` with another predicate.** Only the probabilistic term is +verified; `age` is exact and the server's answer stands: + +```rust +let name = cipher.query::("ali").on(&users::name)?; +let age = cipher.query::(30).on(&users::age)?; + +let rows: Vec = sqlx::query_as( + "SELECT id, name, age FROM users + WHERE name @@ $1::eql_v4.query_text_match AND age > $2::eql_v4.query_integer_ord_ore + ORDER BY id LIMIT $3", +) +.bind(sqlx::types::Json(&name)) +.bind(sqlx::types::Json(&age)) +.bind(name.target_limit(10, &stats).fetch() as i64) +.fetch_all(&pool).await?; +let hits: Vec = cipher.decrypt_all(rows).await? + .into_iter().filter(|u| name.verify(&u.name)).take(10).collect(); +``` + +**`AtPath`, exact.** The term is a value-selector: the MAC of the path +and the leaf's equality term, as one `query_json` operand; the operator is +containment, `@>`, which a GIN index serves. The row is a match when the +document holds that value at that path, and nothing is verified. + +```rust +let term = cipher.query::>(("$.address.city", "Sydney")).on(&users::profile)?; + +let rows: Vec = sqlx::query_as( + "SELECT id, profile FROM users WHERE profile @> $1::eql_v3.query_json", +) +.bind(sqlx::types::Json(&term)) +.fetch_all(&pool).await?; +``` + +**`AtPath` on a number, exact.** Ordering at a path takes two parts, a +selector and a leaf term, because the SQL is `col -> selector >= term`. The +accessor term exposes both: + +```rust +let term = cipher.query::>(("$.score", 80)).on(&users::profile)?; + +let rows: Vec = sqlx::query_as( + "SELECT id, profile FROM users WHERE profile -> $1::text >= $2::eql_v3.query_double_ord", +) +.bind(term.selector()) +.bind(sqlx::types::Json(term.leaf())) +.fetch_all(&pool).await?; +``` + +`AtPath` on a text leaf is `Probabilistic` and verifies like `Lt` on a +text column: `term.verify(&user.profile)` applies the comparison at the path +on the decrypted document. + +**`Nearest`, ranked.** Stack Similarity has no EQL spelling yet, so this is +the shape and not a statement that runs. The term is a set of band tokens; +the server returns candidates by token overlap; `score` re-ranks on the +decrypted embedding, which the query must select: + +```rust +let term = cipher.query::(embedding).on(&docs::vector)?; +let fetch = term.target_limit(10); + +let rows: Vec = sqlx::query_as( + "SELECT id, vector FROM docs WHERE vector ~~ $1 LIMIT $2", // operator not yet defined +) +.bind(sqlx::types::Json(&term)).bind(fetch as i64) +.fetch_all(&pool).await?; + +let mut docs = cipher.decrypt_all(rows).await?; +docs.sort_by_key(|d| std::cmp::Reverse(term.score(&d.vector))); +docs.truncate(10); +``` + +What the appendix shows, read across: an exact term is one `bind` and +nothing after the query; a probabilistic term is one `bind`, a `LIMIT` from +the term, and a `filter` after decrypt; a ranked term is one `bind`, a +`LIMIT` from the term, and a sort after decrypt. No SQL library has to know +which is which, and the Rust types make it impossible to forget. + +## Appendix B: the model in Go and TypeScript + +ADR-0007 says a binding reaches the engine through a plan lowered from data, +and ADR-0008 says a language SDK is what users of that language touch, which +is never the plan. This appendix asks what each of those gains from the +model above, in the two languages that have an SDK today, and what a query +then looks like in the SQL libraries those languages use. It is a check on +the model, not a decision; the decisions for the bindings come with the +ADR-0007 amendment. + +### What crosses the binding + +Today the Go guest has one query export: `Cipher.Query(plan, field, value)` +returns the JSON of the field's query operand, and the engine picks the term +by the field's EQL type. The TypeScript binding (`@cipherstash/protect-ffi`) +has `encryptQuery(value, { table, column, queryType })`, where `queryType` +names an index flavour (`equality`, `freeTextSearch`, `orderAndRange`, the +`steVec*` forms). Both name an index, which is the problem statement. + +Under this ADR four things cross, all as data, and each is a capability the +data grammar does not have yet, added once (ADR-0007, last bullet of the +decision): + +1. **An operator name with its input**, in place of the index: + `{"op": "matches", "input": "ali"}`, `{"op": "between", "input": [18, 65]}`, + `{"op": "at_path", "path": "$.score", "inner": {"op": "gte", "input": 80}}`. + The engine resolves `Answers` for the field's type and refuses an + operator the type does not answer, with the same shape of error a + mistyped field gets today. +2. **The term back**, as the JSON its query domain takes, plus which domain + that is. The `Between` term comes back as its two halves. +3. **`verify`** as an export: `(term, decrypted value) → bool`, batched as + `(term, [values]) → [bool]` so a page costs one crossing. The value has + already come out of the guest as a decrypt, so sending it back in is no + new exposure. The alternative, a tokenizer in each language, is the + "lives twice" cost ADR-0007 accepts only for an encoder, and a tokenizer + that drifts from the one that stored the filter verifies the wrong thing. +4. **`target_limit`** as an export, `(term, want, stats) → limit`, for the + same reason: the arithmetic is small, but `b` is a property of the term the + engine built, and the fill formula belongs beside the index configuration. + +What does *not* cross is the fidelity as a runtime value. The generator asks +the engine, at `go generate` or at build time, which operators each EQL type +answers and whether each is exact, probabilistic or ranked, the way `stashgen` +already asks `eql.Types` whether a type is producible. The generated code then +has `Verify` only where there is something to verify, which is the +compile-time property decision 4 wants, reconstructed in a language without +traits. + +### Go + +The generated field surface grows from one `Query` to one method per +operator the type answers, with the operator's input in the signature: + +```go +q, err := users.Fields.Email.Eq(ctx, cipher, "bob@example.com") // eql.TextEqQuery +q, err := users.Fields.Age.Lt(ctx, cipher, int32(40)) // eql.IntegerOrdOreQuery +q, err := users.Fields.Age.Between(ctx, cipher, int32(18), int32(65)) // eql.IntegerRangeQuery: Lo(), Hi() +q, err := users.Fields.Name.Matches(ctx, cipher, "ali") // eql.TextMatchQuery +``` + +A field whose type does not answer an operator has no such method, so the +call does not compile. `eql.TextMatchQuery` carries `Verify(value string) +bool` and `TargetLimit(want uint64, stats eql.Stats) eql.Limit`; +`eql.TextEqQuery` carries neither. Each query type keeps `driver.Valuer`, so +it binds as a parameter in any library built on `database/sql` or pgx: + +```go +q, _ := users.Fields.Name.Matches(ctx, cipher, "ali") +limit := q.TargetLimit(10, stats).Fetch() + +rows, err := db.QueryContext(ctx, + `SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2`, + q, limit) +// scan into []users.Encrypted, then +people, err := users.Decrypt(ctx, cipher, encrypted) +hits := people[:0] +for _, p := range people { + if q.Verify(p.Name) { + hits = append(hits, p) + } +} +``` + +The same statement in the libraries Go programs use: + +- **sqlc** keeps the SQL in a `.sql` file with the cast written out, and a + type override maps `eql_v4.query_text_match` to `eql.TextMatchQuery`, so + the generated `ListUsersByName(ctx, q, limit)` takes the term as a typed + argument. +- **pgx** binds it the same way as `database/sql`; `pgx.CollectRows` into + the generated struct, then `Decrypt` and `Verify`. +- **GORM**: `db.Where("name @@ ?::eql_v4.query_text_match", q).Order("id").Limit(limit).Find(&encrypted)`. +- **Bun** and **sqlx** (the Go one): the same `?` placeholder with the cast. + +`Verify` runs in the guest, batched. A page of fifty rows is one call after +the one `Decrypt` call, which is in keeping with the batching the Go SDK +already does for seal and open. The cursor loop is the Rust one with +`WHERE id > $2`, and `TargetLimit` is read once per query. The width rules +hold: `Age.Lt` takes an `int32` because `Integer` does, so a widened `int` +is refused at compile time, as it is at generate time for the field. + +### TypeScript + +The binding call that names an operator replaces the one that names a +query type: + +```ts +const term = await client.encryptQuery('ali', { table: 'users', column: 'name', operator: 'matches' }) +// term.verify(value): present on a probabilistic term's type, absent on an exact one +// term.targetLimit(want, stats) +``` + +TypeScript has no traits, but it has discriminated unions: the return type +of `encryptQuery` narrows on `operator`, so `term.verify` type-checks for +`'matches'` and is a type error for `'eq'`. The engine does the verifying +through the FFI, batched per page, for the reason given above. + +In the SQL libraries TypeScript programs use, the operator already exists on +the SDK side and the change is what it emits and what comes back: + +- **Drizzle** (`@cipherstash/stack-drizzle`): `ops.matches(users.name, + 'ali')` already names the operator rather than the index, and emits the + `@@` form with the query-domain cast. It gains `ops.matches(..).limit(10, + stats)` to size the fetch, and the rows come back through + `client.decryptModels` and `term.verify`, which the operator helper can + return beside the SQL so the application never handles the term itself: + + ```ts + const q = await ops.matches(users.name, 'ali') + const rows = await db.select().from(users).where(q.sql).orderBy(users.id).limit(q.limit(10, stats)) + const hits = (await client.bulkDecryptModels(rows)).data.filter((u) => q.verify(u.name)).slice(0, 10) + ``` + + `ops.between(users.age, 18, 65)` is one call that binds two parameters, + which is what it does today. +- **Prisma** (`@cipherstash/stack-prisma`): `u.name.eqlMatch('ali')` is the + operator; the verification step is the same `filter` over the decrypted + result, and `eqlBetween(lo, hi)` is one call and two parameters. +- **Kysely**, **postgres.js**, **pg**: the term is a parameter with the cast + in the SQL text, exactly as in Go. + +One difference from Go. The TypeScript SDK writes EQL v3 today and the Rust +engine writes v4, so the casts a TypeScript query emits are +`eql_v3.query_*` until the SDK moves. The model does not change with the +version; the domain names do. + +### What the two languages say about the model + +Three things survive the translation, and one does not. + +The operator-not-index call survives, and is already how both language SDKs +spell a query: `ops.matches`, `eqlBetween`, `eqlJsonPathGte`. The Rust API is +catching up to its own bindings. + +The split between an exact and a probabilistic term survives as the presence +or absence of a `verify` method, decided by a generator in Go and a union +type in TypeScript, both fed by the engine's catalog. Neither language needs +a `Fidelity` enum either. + +The "one term, two parameters" shape of `Between` survives, because every +driver binds one value per placeholder and both SDKs already do it. + +What does not translate is the compile error for an unsupported operator in +TypeScript when the schema is only known at runtime, which is the +`wasm-inline` path with a declared schema. There the engine's refusal is the +check, and it fires on the first query rather than at build time. That is +the same place the Go SDK was before `stashgen`, and the same fix applies if +it matters: a generator that reads the schema. From 8c4617cbd048a1217bada1e6bf7667a0d8b122ee Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 21:32:11 +1100 Subject: [PATCH 03/38] docs(adr): ADR-0009 appendix B sketches a Stack Encrypt TypeScript SDK The TypeScript section described the existing EQL v3 SDK. It now sketches a binding and an SDK built on Stack Encrypt and EQL v4, which do not exist yet: the schema as the plan, operators as the methods a column's type answers, fidelity as the presence of `verify`, terms bound through pg, postgres.js and Kysely with the v4 casts, and the ORM helpers returning the term beside the SQL. Decision 5 also notes that a page sample can select nothing on a small table. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- ...and-a-query-term-is-not-the-stored-term.md | 252 +++++++++++++----- 1 file changed, 181 insertions(+), 71 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index d652690c9..c58e7c314 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -312,7 +312,9 @@ GROUP BY 1 computes `p = Σ count_i · (bits_i / m)^b / Σ count_i` for whatever `b` the query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a page sample, which is fine for a distribution and unsuited to an exact count. -`rows` comes from `SELECT reltuples FROM pg_class WHERE oid = +It is also a sample of pages, so on a table of a few thousand rows a 1% +sample can select none; below that size the clause is dropped and the whole +table is read, which costs nothing worth saving. `rows` comes from `SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog read. Where the numbers live and who keeps them in step was asked on the review, @@ -726,23 +728,28 @@ which is which, and the Rust types make it impossible to forget. ADR-0007 says a binding reaches the engine through a plan lowered from data, and ADR-0008 says a language SDK is what users of that language touch, which is never the plan. This appendix asks what each of those gains from the -model above, in the two languages that have an SDK today, and what a query -then looks like in the SQL libraries those languages use. It is a check on -the model, not a decision; the decisions for the bindings come with the -ADR-0007 amendment. +model above, in two languages: Go, which has a Stack Encrypt SDK today, and +TypeScript, which does not. The TypeScript half is a sketch of a binding and +an SDK built on Stack Encrypt and EQL v4, written to check that the Rust +model will support them well; it is not the existing `@cipherstash/stack`, +which runs on cipherstash-client through `@cipherstash/protect-ffi` and +writes EQL v3. It is a check on the model, not a decision; the decisions for +the bindings come with the ADR-0007 amendment. ### What crosses the binding Today the Go guest has one query export: `Cipher.Query(plan, field, value)` returns the JSON of the field's query operand, and the engine picks the term -by the field's EQL type. The TypeScript binding (`@cipherstash/protect-ffi`) +by the field's EQL type. The existing TypeScript binding, for comparison, has `encryptQuery(value, { table, column, queryType })`, where `queryType` -names an index flavour (`equality`, `freeTextSearch`, `orderAndRange`, the -`steVec*` forms). Both name an index, which is the problem statement. +names an index flavour (`equality`, `freeTextSearch`, `orderAndRange`). Both +name an index, which is the problem statement. -Under this ADR four things cross, all as data, and each is a capability the -data grammar does not have yet, added once (ADR-0007, last bullet of the -decision): +A Stack Encrypt binding for TypeScript would be the guest the Go SDK embeds, +or a napi build of the same crate, and would take the same plan the Go SDK +lowers (ADR-0007). Under this ADR four more things cross it, all as data, +and each is a capability the data grammar does not have yet, added once +(ADR-0007, last bullet of the decision): 1. **An operator name with its input**, in place of the index: `{"op": "matches", "input": "ali"}`, `{"op": "between", "input": [18, 65]}`, @@ -762,13 +769,14 @@ decision): same reason: the arithmetic is small, but `b` is a property of the term the engine built, and the fill formula belongs beside the index configuration. -What does *not* cross is the fidelity as a runtime value. The generator asks -the engine, at `go generate` or at build time, which operators each EQL type -answers and whether each is exact, probabilistic or ranked, the way `stashgen` -already asks `eql.Types` whether a type is producible. The generated code then -has `Verify` only where there is something to verify, which is the -compile-time property decision 4 wants, reconstructed in a language without -traits. +What does *not* cross is the fidelity as a runtime value. Something on the +language side asks the engine, before the program runs, which operators each +EQL type answers and whether each is exact, probabilistic or ranked, the way +`stashgen` already asks `eql.Types` whether a type is producible. In Go that +something is the generator; in TypeScript it can be the type system, as the +TypeScript section shows. Either way the code a user writes has `verify` +only where there is something to verify, which is the compile-time property +decision 4 wants, reconstructed in a language without traits. ### Go @@ -825,68 +833,170 @@ is refused at compile time, as it is at generate time for the field. ### TypeScript -The binding call that names an operator replaces the one that names a -query type: +Go needs a generator because its type system cannot derive a field's +methods from a declaration. TypeScript can, so the sketch below has no +generate step: the schema is a value, its type is the plan, and the +operators a column answers are read off that type. The names are +placeholders; the shapes are the point. + +**The schema is the plan.** A table is declared with the EQL v4 types, which +is the data grammar of ADR-0007 written as TypeScript, and the match index +carries its `expectedTokens` prior from decision 5: + +```ts +import { table, passthrough, eql } from '@cipherstash/stack-encrypt' // placeholder name + +const users = table('users', { + id: passthrough(), + email: eql.TextEq(), + name: eql.TextMatch({ expectedTokens: 20 }), + age: eql.IntegerOrdOre(), + profile: eql.Json(), +}) + +const client = await StackEncrypt.connect() // the guest, through napi or wasm +const cipher = client.keyset('tenant-42') +``` + +**An operator is a method on the column, and only the answered ones exist.** +`cipher.query(column)` returns an object whose methods are the operators the +column's EQL type answers, each taking that operator's input: + +```ts +const email = await cipher.query(users.email).eq('bob@example.com') // EqTerm +const age = await cipher.query(users.age).lt(40) // LtTerm +const range = await cipher.query(users.age).between(18, 65) // BetweenTerm: bounds() +const name = await cipher.query(users.name).matches('ali') // MatchesTerm +const city = await cipher.query(users.profile).atPath('$.address.city').eq('Sydney') + +cipher.query(users.email).matches('bob') // type error: TextEq answers eq and neq +cipher.query(users.age).lt('40') // type error: IntegerOrdOre takes a number +``` + +The type behind it is the `Answers` table of decision 3, as a mapped +type over the column's EQL type: + +```ts +type Query = { + [Op in AnswersOf]: (input: InputOf) => Promise> +} +``` + +and `TermOf` carries the fidelity of decision 4 as the presence of methods, +not as a flag: ```ts -const term = await client.encryptQuery('ali', { table: 'users', column: 'name', operator: 'matches' }) -// term.verify(value): present on a probabilistic term's type, absent on an exact one -// term.targetLimit(want, stats) +interface Term

{ readonly domain: string; toJSON(): unknown } // exact: nothing else +interface Probabilistic

extends Term

{ verify(value: P): boolean; targetLimit(want: number, stats: Stats): Limit } +interface Ranked

extends Term

{ score(value: P): number; targetLimit(want: number): number } +``` + +`name.verify('Alice')` type-checks; `email.verify(..)` does not exist. The +engine does the verifying through the binding, batched per page, for the +reason given above; the method on the term is a thin call. `verify` returns +`boolean`, never throws, as decision 4 says. + +**The term binds as a parameter, and the cast is in the SQL.** A term +serialises to the JSON its query domain takes (`toJSON`), so every PostgreSQL +driver binds it unchanged. The statement from decision 6, in the drivers +TypeScript programs use: + +```ts +// pg +const limit = name.targetLimit(10, stats).fetch() +const { rows } = await pool.query( + 'SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2', + [name, limit], +) + +// postgres.js +const rows = await sql` + SELECT id, name FROM users + WHERE name @@ ${name}::eql_v4.query_text_match ORDER BY id LIMIT ${limit}` + +// Kysely +const rows = await db.selectFrom('users').select(['id', 'name']) + .where(sql`name @@ ${name}::eql_v4.query_text_match`) + .orderBy('id').limit(limit).execute() ``` -TypeScript has no traits, but it has discriminated unions: the return type -of `encryptQuery` narrows on `operator`, so `term.verify` type-checks for -`'matches'` and is a type error for `'eq'`. The engine does the verifying -through the FFI, batched per page, for the reason given above. - -In the SQL libraries TypeScript programs use, the operator already exists on -the SDK side and the change is what it emits and what comes back: - -- **Drizzle** (`@cipherstash/stack-drizzle`): `ops.matches(users.name, - 'ali')` already names the operator rather than the index, and emits the - `@@` form with the query-domain cast. It gains `ops.matches(..).limit(10, - stats)` to size the fetch, and the rows come back through - `client.decryptModels` and `term.verify`, which the operator helper can - return beside the SQL so the application never handles the term itself: - - ```ts - const q = await ops.matches(users.name, 'ali') - const rows = await db.select().from(users).where(q.sql).orderBy(users.id).limit(q.limit(10, stats)) - const hits = (await client.bulkDecryptModels(rows)).data.filter((u) => q.verify(u.name)).slice(0, 10) - ``` - - `ops.between(users.age, 18, 65)` is one call that binds two parameters, - which is what it does today. -- **Prisma** (`@cipherstash/stack-prisma`): `u.name.eqlMatch('ali')` is the - operator; the verification step is the same `filter` over the decrypted - result, and `eqlBetween(lo, hi)` is one call and two parameters. -- **Kysely**, **postgres.js**, **pg**: the term is a parameter with the cast - in the SQL text, exactly as in Go. - -One difference from Go. The TypeScript SDK writes EQL v3 today and the Rust -engine writes v4, so the casts a TypeScript query emits are -`eql_v3.query_*` until the SDK moves. The model does not change with the -version; the domain names do. +Decrypt and verify are the same two steps in each: + +```ts +const people = await cipher.decrypt(users, rows) // typed by the schema: name is a string +const hits = people.filter((p) => name.verify(p.name)).slice(0, 10) +const exhausted = rows.length < limit +``` + +`Between` is one call and two parameters, because every driver binds one +value per placeholder: + +```ts +const [lo, hi] = range.bounds() +await pool.query( + 'SELECT id, age FROM users WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore', + [lo, hi], +) +``` + +**The ORM integrations are the same terms with the SQL written for them.** +A Drizzle integration maps the schema to `customType` columns and gives each +operator a helper that returns the SQL fragment and the term together, so the +application never handles the term itself: + +```ts +const q = await ops.matches(users.name, 'ali') // { sql, verify, targetLimit } +const rows = await db.select().from(users) + .where(q.sql) // name @@ '…'::eql_v4.query_text_match + .orderBy(users.id) + .limit(q.targetLimit(10, stats).fetch()) +const hits = (await cipher.decrypt(users, rows)).filter((u) => q.verify(u.name)).slice(0, 10) +``` + +`ops.eq` returns `{ sql }` and no `verify`, by the same types. A Prisma +integration spells the operator on the field (`u.name.eqlMatches('ali')`) +and returns the term beside the query the same way. In both, `between` is +one helper that binds two parameters. + +**Stats** come from the two statements of decision 5, run by the +application, or from the declared prior: + +```ts +const stats = Stats.measured( + (await pool.query("SELECT reltuples::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows, + (await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1')).rows, +) +const prior = Stats.declared(1_000_000) // uses users.name's expectedTokens +``` + +**The runtime-schema case.** The `wasm-inline` path today takes a schema +loaded at runtime, with no TypeScript type behind it. There `cipher.query` +cannot narrow, every operator method exists, and the engine's refusal of an +unsupported operator is the check, on the first query rather than at build +time. That is where the Go SDK was before `stashgen`, and the same fix +applies if it matters: a generator that reads the schema and emits the +declaration above. ### What the two languages say about the model -Three things survive the translation, and one does not. +Four things survive the translation, and one does not. -The operator-not-index call survives, and is already how both language SDKs -spell a query: `ops.matches`, `eqlBetween`, `eqlJsonPathGte`. The Rust API is -catching up to its own bindings. +The operator-not-index call survives, and the existing TypeScript SDK +already spells its queries that way (`ops.matches`, `eqlBetween`, +`eqlJsonPathGte`): the Rust API is catching up to its own bindings. The split between an exact and a probabilistic term survives as the presence -or absence of a `verify` method, decided by a generator in Go and a union -type in TypeScript, both fed by the engine's catalog. Neither language needs -a `Fidelity` enum either. +or absence of `verify`, decided by a generator in Go and by a mapped type in +TypeScript, both fed by the engine's catalog. Neither language needs a +`Fidelity` enum either. The "one term, two parameters" shape of `Between` survives, because every -driver binds one value per placeholder and both SDKs already do it. - -What does not translate is the compile error for an unsupported operator in -TypeScript when the schema is only known at runtime, which is the -`wasm-inline` path with a declared schema. There the engine's refusal is the -check, and it fires on the first query rather than at build time. That is -the same place the Go SDK was before `stashgen`, and the same fix applies if -it matters: a generator that reads the schema. +driver binds one value per placeholder. + +The declared prior survives as one field in the schema (`expectedTokens`), +which is where a TypeScript user would expect to write it. + +What does not translate is the compile error for an unsupported operator +when the schema is only known at runtime. There the engine's refusal is the +check, and nothing about the Rust model makes it fire earlier; only a +generator does. From 22e8a4ceb4b44fff18a55feeea077a9fb8262a38 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 21:36:01 +1100 Subject: [PATCH 04/38] docs(adr): ADR-0009 appendix B names the next major of @cipherstash/stack The TypeScript sketch imports from `@cipherstash/stack`, which is where it will ship, rather than a placeholder package name. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- ...-and-a-query-term-is-not-the-stored-term.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index c58e7c314..6fd63e496 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -731,10 +731,11 @@ is never the plan. This appendix asks what each of those gains from the model above, in two languages: Go, which has a Stack Encrypt SDK today, and TypeScript, which does not. The TypeScript half is a sketch of a binding and an SDK built on Stack Encrypt and EQL v4, written to check that the Rust -model will support them well; it is not the existing `@cipherstash/stack`, -which runs on cipherstash-client through `@cipherstash/protect-ffi` and -writes EQL v3. It is a check on the model, not a decision; the decisions for -the bindings come with the ADR-0007 amendment. +model will support them well. That SDK will be the next major version of +`@cipherstash/stack`; the current major runs on cipherstash-client through +`@cipherstash/protect-ffi` and writes EQL v3, and nothing below describes +it. It is a check on the model, not a decision; the decisions for the +bindings come with the ADR-0007 amendment. ### What crosses the binding @@ -836,15 +837,16 @@ is refused at compile time, as it is at generate time for the field. Go needs a generator because its type system cannot derive a field's methods from a declaration. TypeScript can, so the sketch below has no generate step: the schema is a value, its type is the plan, and the -operators a column answers are read off that type. The names are -placeholders; the shapes are the point. +operators a column answers are read off that type. The function and type +names are illustrative; the package is `@cipherstash/stack` at its next +major, and the shapes are the point. **The schema is the plan.** A table is declared with the EQL v4 types, which is the data grammar of ADR-0007 written as TypeScript, and the match index carries its `expectedTokens` prior from decision 5: ```ts -import { table, passthrough, eql } from '@cipherstash/stack-encrypt' // placeholder name +import { table, passthrough, eql } from '@cipherstash/stack' // the next major const users = table('users', { id: passthrough(), @@ -854,7 +856,7 @@ const users = table('users', { profile: eql.Json(), }) -const client = await StackEncrypt.connect() // the guest, through napi or wasm +const client = await Encryption.connect() // the Stack Encrypt guest, through napi or wasm const cipher = client.keyset('tenant-42') ``` From decd2bbed98f59faac00d5d2ac652a0ddd222506 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 15:12:35 +1100 Subject: [PATCH 05/38] docs(adr): ADR-0009 second review round Between is one term and one operand: `eql_v4.between(column, range)` with a range query domain per ordered family, in place of the native BETWEEN with the term's halves as two parameters. The function must inline so a btree over the ORE term still serves it as one range scan; that requirement and the domain are EQL-side work the consequences now ask for. Appendix A and B lose `bounds()`, `Lo()`/`Hi()`, and the two-parameter statements. Order terms are exact. `Lt` on text leaves the `Probabilistic` class, which is for Bloom filters only: an order term compares order keys, and two values with one key tie by definition, so there is nothing to verify. The order key is `orderable-bytes`' encoding after the column's declared transform (ADR-0002); the ADR no longer names cllw-ore's `orderize_string`, which the library does not use. The ORE row drops "right only, in future": PostgreSQL stores the left ciphertext too. Whether `verify` for `Matches` tests n-gram containment or a substring is the implementation's choice, and the ADR says so instead of fixing it. Measured statistics come from the server and the declared prior from the index declaration; the sentence saying both always came from the server is gone. The examples open rows with `cipher.open(rows).using(&plan)`; there is no `decrypt_all` on the cipher. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- ...and-a-query-term-is-not-the-stored-term.md | 169 +++++++++++------- 1 file changed, 101 insertions(+), 68 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 6fd63e496..c5a30c335 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -40,8 +40,8 @@ Decision 5 of ADR-0004 assumed they were. In general they are not: | index | stored term | query term | | --- | --- | --- | | HMAC | the HMAC | the same HMAC | -| ORE | left and right ciphertext (right only, in future) | the left ciphertext only | -| OPE | the OPE value of the order key | the same value, when the order encoding is lossless; for text the key is collated first (cllw-ore's `orderize_string` folds `café` and `cafe` together), so equal terms do not mean equal plaintexts | +| ORE | left and right ciphertext | the left ciphertext only | +| OPE | the OPE value of the order key | the same value; equal terms mean equal order keys, which is plaintext equality only when the key is lossless (it is for integers, and not for text after the column's declared transform) | | Bloom match | one filter over every n-gram of the value | a filter over the n-grams of the needle: a subset, or bits the stored filter lacks | | Bloom prefix (planned) | an HMAC per edge n-gram | one HMAC | | JSON path | a selector and, per leaf, that leaf's term | a selector, or a selector plus a leaf query term | @@ -49,10 +49,15 @@ Decision 5 of ADR-0004 assumed they were. In general they are not: HMAC equality is the one index whose query term is byte-for-byte the stored term, and whose comparison means plaintext equality. OPE comes close only when -its encoding is lossless, as it is for integers. The others differ in kind. +its order key is lossless, as it is for integers. The others differ in kind. +The order key is `orderable-bytes`' order-preserving encoding of the value, +applied after whatever transform the column declares for the kind (ADR-0002 +defines a text fold; the encoding is pluggable, and what the term means +follows it). The Rust library does not use cllw-ore's `orderize_string`, +which an earlier draft of this ADR named. The term is a function of the index *and the operator*, not of the index -alone. An ORE index answers `=`, `<` and `BETWEEN` with different query shapes; +alone. An ORE index answers `=`, `<` and a range with different query shapes; a Bloom index answers `Matches` and (if used for ranked search) `Nearest` with the same stored filter and different queries. @@ -76,10 +81,8 @@ index, and the operator is implied. That has three costs. ### The server's answer is sometimes approximate, and the library says nothing A Bloom filter is probabilistic. A row whose filter happens to contain every -bit of the query passes `@@` without containing the needle. The collating -`orderize_string` transform behind text ordering folds case and diacritics, so -`<` on text returns false positives at the boundary. A similarity index returns -candidates by construction. +bit of the query passes `@@` without containing the needle. A similarity index +returns candidates by construction. The library has never said which terms are exact. A caller who wants ten real matches has no way to know how many rows to fetch, and no help deciding, after @@ -186,13 +189,16 @@ pub trait Ranked: Term { ``` - A term that implements `Term` only is **exact**. The server's answer is the - answer. HMAC `Eq`, ORE and OPE on numbers (`Lt`, `Between`), JSON - `AtPath`. + answer. HMAC `Eq`, ORE and OPE on every kind, text included (`Lt`, + `Between`), JSON `AtPath`. - A `Probabilistic` term can return **false positives and no false negatives**. `verify` is the operator's own meaning, run on a decrypted value: it decides whether a row the server returned is a real match. Bloom - `Matches` and `StartsWith`, and `Lt` on text through the collating - `orderize_string` (the ties). + `Matches` and `StartsWith`. Bloom filters are the only index this applies + to today, and the trait is not a home for an order term: an order term + over text compares order keys, and two values with one key tie by the + term's definition, not by error, so there is nothing a decrypted value + could be checked against. - A `Ranked` term returns **candidates in approximate order**. `score` is the exact score on a decrypted value, for re-ranking. `Nearest` over a similarity index, and `Nearest` over a Bloom index if used for ranked text @@ -228,8 +234,9 @@ write `term.verify(..)` for an HMAC term: verification would pass for every row the server returned, so the method would say nothing, and the compiler says so. -`AtPath` inherits the fidelity of the leaf: `AtPath` on a text leaf is -`Probabilistic`, as `Lt` on a text column is. +`AtPath` inherits the fidelity of `Op` on the leaf: `AtPath` and +`AtPath` are exact, as `Eq` and `Lt` are, and an accessor over a +probabilistic leaf operator, should one be added, is `Probabilistic`. ### 5. Verification is the caller's, after decrypt; the library supplies the predicate and the bound @@ -238,9 +245,12 @@ a caller of a `Probabilistic` term is two things. `verify` is the predicate. For `Matches`, it folds case, tokenises both sides with the index's own tokenizer and checks that the -needle's n-grams occur in the value. It lives on the term because the term -already carries the operator's input and the index's configuration, so it -cannot disagree with what was stored. +needle's n-grams occur in the value. Whether that check is n-gram containment +or a substring test (they differ once the query is longer than one n-gram) +is the implementation's choice; both are sound against a Bloom match, and +the ADR fixes neither. It lives on the term because the term already carries +the operator's input and the index's configuration, so it cannot disagree +with what was stored. `target_limit` is how many rows to ask for to expect `want` real matches. For a Bloom filter it is computable from what the client knows, plus what @@ -284,9 +294,9 @@ returned row correct; only the candidate count makes the result complete. #### Declared and measured statistics -`N` and the fill distribution come from the server, always. The library -takes them as numbers in `Stats` and does not care where they came from. -There are two sources, and a column has the first one from the day it is +Measured statistics come from the server; the declared prior comes from the +index declaration. The library takes either as numbers in `Stats` and does +not care which it was given. A column has the prior from the day it is declared. **Declared.** A match index declares `expected_tokens`, the typical token @@ -357,14 +367,19 @@ a `Probabilistic` term can report from `target_limit` alone. #### Text ordering -`Lt` on text has no `p`: the false positives are the rows that collapse onto -the boundary key, which depends on the data. Its `target_limit` is -`Limit::Unbounded`, and the caller uses the cursor loop below. Its use is -ordering and cursor paging on names (`ORDER BY name`, "the page after -Nguyen"), and EQL v4 ships `query_text_ord` for it. Whether the Rust library -produces that term is a catalog question (`TextOrdOre` is unproducible -today, pending cllw-ore); nothing else in this ADR depends on it, and if -text ordering is dropped this paragraph goes with it. +`Lt` on text is exact, decision 4, so none of the above applies to it: no +`p`, no `verify`, no `target_limit`. The term is an order term over the +column's order key, which is `orderable-bytes`' encoding of the value after +the text transform the column declares (ADR-0002's fold, or another, since +the encoding is pluggable), and the server compares keys. Two values with +one key tie; that is what the column declared, and an earlier draft of this +ADR was wrong to call it a false positive and put the term under +`Probabilistic`. Its use is ordering and cursor paging on names (`ORDER BY +name`, "the page after Nguyen"), and EQL v4 carries `query_text_ord` and +`query_text_ord_ore` for it. Whether the Rust library produces the term is a +catalog question (`TextOrdOre` is unproducible today, pending ADR-0002); +nothing else in this ADR depends on it, and if text ordering is dropped this +paragraph goes with it. A `Ranked` term's `target_limit` applies the index's configured recall factor and needs no `Stats`. @@ -386,7 +401,7 @@ let rows: Vec = sqlx::query_as( .fetch_all(&pool).await?; let exhausted = (rows.len() as u64) < fetch; -let users = cipher.decrypt_all(rows).await?; +let users = cipher.open(rows).using(&users_plan).await?; let hits: Vec = users.into_iter().filter(|u| term.verify(&u.name)).take(10).collect(); // hits.len() < 10 && exhausted: there are no more matches. // hits.len() < 10 && !exhausted: page on from the last id. @@ -409,7 +424,7 @@ while hits.len() < 10 { .fetch_all(&pool).await?; if rows.is_empty() { break; } after = rows.last().unwrap().id; - hits.extend(cipher.decrypt_all(rows).await?.into_iter().filter(|u| term.verify(&u.name))); + hits.extend(cipher.open(rows).using(&users_plan).await?.into_iter().filter(|u| term.verify(&u.name))); } hits.truncate(10); ``` @@ -428,25 +443,38 @@ The library ships `verify` and `target_limit`. It does not ship the loop, because the loop runs queries. That may change when Stack Similarity lands and the ranked loop is worth packaging; it is deferred, not refused. -### 7. `Between` is one term, spelled `BETWEEN` +### 7. `Between` is one term and one operand `cipher.query::((lo, hi))` produces one term carrying both ORE left ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. One call derives both bounds, under one context, so a caller cannot pair a bound from one column with a bound from another. -In SQL it is the native `BETWEEN`, with the term's two halves bound in order: +In SQL the term stays one operand. EQL v4 gains a function that takes the +column and the range, and a range query domain beside each ordered family's +`ord` domain to carry it (shown here as `query_integer_range_ore`; the +catalog names it): ```sql -WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore +WHERE eql_v4.between(age, $1::eql_v4.query_integer_range_ore) ``` -PostgreSQL rewrites `a BETWEEN x AND y` to `a >= x AND a <= y` when it parses -the statement, so there is no range the planner could see and this spelling -could hide; an earlier draft of this decision said otherwise and was wrong. A -btree over the ORE term uses both bounds as one range scan either way. The -term exposes its halves (`let (lo, hi) = term.bounds();`) for drivers that -bind one value per parameter, which is all of them. +The term is as indivisible in SQL as it is in Rust: one placeholder, so no +statement can bind the low bound of one term with the high bound of another +or bind them in the wrong order, and a binding hands back one value for the +term as it does for every other (appendix B). An earlier draft spelled the +range as the native `BETWEEN` with the term's two halves as two parameters; +this one withdraws it, because it split in SQL the one thing the Rust side +had just joined, and every SDK had to grow a `bounds()` to feed it. + +What the one-operand form asks of EQL is that the planner still see a range. +`eql_v4.between` must be a SQL function PostgreSQL inlines at planning time +(`LANGUAGE sql`, one expression, `$1 >= lo($2) AND $1 <= hi($2)` with +immutable accessors), so that a btree over the ORE term serves it as one +range scan. A function the planner cannot inline is a call per row on a +sequential scan, which no index can rescue. That is an EQL-side requirement +with an EQL-side test, and it is why the function and the domain arrive +together. ### 8. One query operand domain per (family, term) in EQL v4 @@ -455,7 +483,8 @@ The EQL v4 side carries one query domain per (type family, term), such as per operator or one per stored domain. This is the "option B" of the design discussion and is recorded here so the Rust side's `Supports::Query` has a fixed target: the query term for an (index, operator) serialises to exactly -one of those domains. `Between` serialises to two values of one domain. +one of those domains. `Between` serialises to one value of the family's range +domain, which carries both bounds (decision 7). ## Considered options @@ -490,6 +519,10 @@ two predicates. Rejected in favour of one `Between` term, decision 7. `Gte` and `Lte` themselves stay; they are the open-ended comparisons, and a range is not two of them. +**Native `BETWEEN` with the term's halves as two parameters.** The previous +draft of decision 7. Dropped: it splits one operand across two placeholders, +and every binding has to hand back halves to feed it. + **`expected_tokens` deferred until the stats query exists.** Dropped: the declared prior is what makes `target_limit` usable on a column nobody has measured, and it costs one number in the index declaration. Decision 5. @@ -529,9 +562,13 @@ JSON column's query already selects it. A similarity index stores its vector separately, so a `Nearest` query must select the embedding, or the sketch, as well. -The collating text-order case falls out without a special case: EQL's -ADR-0002 (the value model, #1139) already refuses equality through -`orderize`, and `Lt` on text is now simply a `Probabilistic` term. +Text ordering needs no special case: `Lt` on text is an exact term over the +order key, and what that key folds is ADR-0002's decision (the value model, +#1139), which keeps equality off the order term for the same reason. + +EQL v4 gains `eql_v4.between(column, range)` and a range query domain per +ordered family, under the inlining requirement of decision 7. That is +EQL-side work this ADR asks for, not a deferral. It is a breaking change to the query chain. `QueryUsing::index(..)` goes; `QueryUsing::equality()` becomes `query::`. The write side (`encrypt`, @@ -558,8 +595,9 @@ EQL v4's query domains (decision 8), as the v4 tests bind them today: `$1::eql_v4.query_text_eq`, with the JSON text of the term as the parameter. The JSON operators are spelled as EQL v3 ships them, since v4 does not carry `json` yet. `UserRow` is the row as the driver returns it, with the EQL -columns as `sqlx::types::Json`, and `cipher.decrypt_all` turns rows -into `User`s. +columns as `sqlx::types::Json`, and `cipher.open(rows).using(&users_plan)` +opens every row under the plan the table was written with, in one key +request, and yields `User`s. **`Eq`, exact.** The term is an HMAC; the operator is `=`. Nothing to verify. @@ -571,7 +609,7 @@ let rows: Vec = sqlx::query_as( ) .bind(sqlx::types::Json(&term)) .fetch_all(&pool).await?; -let users = cipher.decrypt_all(rows).await?; +let users = cipher.open(rows).using(&users_plan).await?; ``` **`Lt` on an integer, exact.** The term is the ORE left ciphertext; the @@ -587,18 +625,16 @@ let rows: Vec = sqlx::query_as( .fetch_all(&pool).await?; ``` -**`Between`, exact, one term and two parameters** (decision 7). +**`Between`, exact, one term and one parameter** (decision 7). ```rust let term = cipher.query::((18, 65)).on(&users::age)?; -let (lo, hi) = term.bounds(); let rows: Vec = sqlx::query_as( "SELECT id, email, name, age FROM users - WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore", + WHERE eql_v4.between(age, $1::eql_v4.query_integer_range_ore)", ) -.bind(sqlx::types::Json(lo)) -.bind(sqlx::types::Json(hi)) +.bind(sqlx::types::Json(&term)) .fetch_all(&pool).await?; ``` @@ -620,7 +656,7 @@ let rows: Vec = sqlx::query_as( .fetch_all(&pool).await?; let exhausted = (rows.len() as u64) < fetch; -let hits: Vec = cipher.decrypt_all(rows).await? +let hits: Vec = cipher.open(rows).using(&users_plan).await? .into_iter() .filter(|u| term.verify(&u.name)) .take(10) @@ -659,7 +695,7 @@ let rows: Vec = sqlx::query_as( .bind(sqlx::types::Json(&age)) .bind(name.target_limit(10, &stats).fetch() as i64) .fetch_all(&pool).await?; -let hits: Vec = cipher.decrypt_all(rows).await? +let hits: Vec = cipher.open(rows).using(&users_plan).await? .into_iter().filter(|u| name.verify(&u.name)).take(10).collect(); ``` @@ -693,9 +729,8 @@ let rows: Vec = sqlx::query_as( .fetch_all(&pool).await?; ``` -`AtPath` on a text leaf is `Probabilistic` and verifies like `Lt` on a -text column: `term.verify(&user.profile)` applies the comparison at the path -on the decrypted document. +`AtPath` on a text leaf is exact, as `Lt` on a text column is: the +leaf's order term compares order keys, and there is nothing to verify. **`Nearest`, ranked.** Stack Similarity has no EQL spelling yet, so this is the shape and not a statement that runs. The term is a set of band tokens; @@ -712,7 +747,7 @@ let rows: Vec = sqlx::query_as( .bind(sqlx::types::Json(&term)).bind(fetch as i64) .fetch_all(&pool).await?; -let mut docs = cipher.decrypt_all(rows).await?; +let mut docs = cipher.open(rows).using(&docs_plan).await?; docs.sort_by_key(|d| std::cmp::Reverse(term.score(&d.vector))); docs.truncate(10); ``` @@ -759,7 +794,7 @@ and each is a capability the data grammar does not have yet, added once operator the type does not answer, with the same shape of error a mistyped field gets today. 2. **The term back**, as the JSON its query domain takes, plus which domain - that is. The `Between` term comes back as its two halves. + that is. A `Between` term is one value like any other. 3. **`verify`** as an export: `(term, decrypted value) → bool`, batched as `(term, [values]) → [bool]` so a page costs one crossing. The value has already come out of the guest as a decrypt, so sending it back in is no @@ -787,7 +822,7 @@ operator the type answers, with the operator's input in the signature: ```go q, err := users.Fields.Email.Eq(ctx, cipher, "bob@example.com") // eql.TextEqQuery q, err := users.Fields.Age.Lt(ctx, cipher, int32(40)) // eql.IntegerOrdOreQuery -q, err := users.Fields.Age.Between(ctx, cipher, int32(18), int32(65)) // eql.IntegerRangeQuery: Lo(), Hi() +q, err := users.Fields.Age.Between(ctx, cipher, int32(18), int32(65)) // eql.IntegerRangeOreQuery, one value q, err := users.Fields.Name.Matches(ctx, cipher, "ali") // eql.TextMatchQuery ``` @@ -867,7 +902,7 @@ column's EQL type answers, each taking that operator's input: ```ts const email = await cipher.query(users.email).eq('bob@example.com') // EqTerm const age = await cipher.query(users.age).lt(40) // LtTerm -const range = await cipher.query(users.age).between(18, 65) // BetweenTerm: bounds() +const range = await cipher.query(users.age).between(18, 65) // BetweenTerm, one value const name = await cipher.query(users.name).matches('ali') // MatchesTerm const city = await cipher.query(users.profile).atPath('$.address.city').eq('Sydney') @@ -930,14 +965,12 @@ const hits = people.filter((p) => name.verify(p.name)).slice(0, 10) const exhausted = rows.length < limit ``` -`Between` is one call and two parameters, because every driver binds one -value per placeholder: +`Between` is one call and one parameter, like every other term: ```ts -const [lo, hi] = range.bounds() await pool.query( - 'SELECT id, age FROM users WHERE age BETWEEN $1::eql_v4.query_integer_ord_ore AND $2::eql_v4.query_integer_ord_ore', - [lo, hi], + 'SELECT id, age FROM users WHERE eql_v4.between(age, $1::eql_v4.query_integer_range_ore)', + [range], ) ``` @@ -958,7 +991,7 @@ const hits = (await cipher.decrypt(users, rows)).filter((u) => q.verify(u.name)) `ops.eq` returns `{ sql }` and no `verify`, by the same types. A Prisma integration spells the operator on the field (`u.name.eqlMatches('ali')`) and returns the term beside the query the same way. In both, `between` is -one helper that binds two parameters. +one helper that binds one parameter. **Stats** come from the two statements of decision 5, run by the application, or from the declared prior: @@ -992,8 +1025,8 @@ or absence of `verify`, decided by a generator in Go and by a mapped type in TypeScript, both fed by the engine's catalog. Neither language needs a `Fidelity` enum either. -The "one term, two parameters" shape of `Between` survives, because every -driver binds one value per placeholder. +The one-operand `Between` survives unchanged: one term, one placeholder, in +every driver and every ORM. The declared prior survives as one field in the schema (`expectedTokens`), which is where a TypeScript user would expect to write it. From 96f45915cf85a5bdaa4bcdd09c28b6f8d16e4f59 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 15:20:38 +1100 Subject: [PATCH 06/38] docs(adr): ADR-0009 carries #1190's signatures, and a smaller k is worse - Decisions 1 to 4 carry the signatures from the implementation in #1190: operators passed by value and used by path (op::Eq), the query awaited, Supports with check and query (behind a QueryToken), Answers with a Default index, DefaultIndex for plans that declare their indexes, and QueryTerm as the base trait that Probabilistic and Ranked refine. Case is CaseInsensitive / CaseSensitive. Every example uses the same call shape. - The Bloom tuning paragraph said k = 3 also gives 268. It keeps b = 6; with k = 3 the needle sets 3 bits, and lambda is about 16,400, twice the k = 6 figure. A smaller k makes short needles worse. - ADR-0002 is the system ADR under docs/adr, linked, not this directory's ADR-0002. --- ...and-a-query-term-is-not-the-stored-term.md | 210 +++++++++++------- 1 file changed, 125 insertions(+), 85 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index c5a30c335..4901477ff 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -51,10 +51,12 @@ HMAC equality is the one index whose query term is byte-for-byte the stored term, and whose comparison means plaintext equality. OPE comes close only when its order key is lossless, as it is for integers. The others differ in kind. The order key is `orderable-bytes`' order-preserving encoding of the value, -applied after whatever transform the column declares for the kind (ADR-0002 -defines a text fold; the encoding is pluggable, and what the term means -follows it). The Rust library does not use cllw-ore's `orderize_string`, -which an earlier draft of this ADR named. +applied after whatever transform the column declares for the kind (the +system +[ADR-0002](../../../../docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md), +not this directory's, defines a text fold; the encoding is pluggable, and +what the term means follows it). The Rust library does not use cllw-ore's +`orderize_string`, which an earlier draft of this ADR named. The term is a function of the index *and the operator*, not of the index alone. An ORE index answers `=`, `<` and a range with different query shapes; @@ -103,16 +105,21 @@ loop are yours. ### 1. The caller names an operator; the operator is a type ```rust -let term = cipher.query::("bob@example.com").on(&users::email)?; -let term = cipher.query::(40).on(&users::age)?; -let term = cipher.query::((18, 65)).on(&users::age)?; -let term = cipher.query::("ali").on(&users::name)?; -let term = cipher.query::>(("$.address.city", "Sydney")).on(&users::profile)?; +use stack_encrypt::op; + +let term = cipher.query(op::Eq, "bob@example.com").on(&users::email).await?; +let term = cipher.query(op::Lt, 40).on(&users::age).await?; +let term = cipher.query(op::Between, (18, 65)).on(&users::age).await?; +let term = cipher.query(op::Matches, "ali").on(&users::name).await?; +let term = cipher.query(op::AtPath(op::Eq), ("$.address.city", "Sydney")).on(&users::profile).await?; ``` -Operators are unit types, as in the original client, not a runtime enum. An -operator carries its own input type: `Eq` takes one value, `Between` a pair, -`Nearest` an embedding, `AtPath` a path and `Op`'s input. +Operators are unit types, as in the original client, not a runtime enum, and +a query passes one by value, as a plan passes an index (`.via(Ore)`). They are +used by path (`op::Eq`), because an imported `Eq` would shadow the prelude's +`Eq` trait. An operator carries its own input type: `Eq` takes one value, +`Between` a pair, `Nearest` an embedding, `AtPath` a path and `Op`'s +input. The query is awaited because deriving the term can retrieve a key. Operators come in three kinds. A **predicate** (`Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between`, `Matches`, `StartsWith`, JSON `Contains`, `ContainedBy`) @@ -126,41 +133,52 @@ match is not a containment. The accessor is `AtPath`, not `At`, because `At` alone does not say what it is at. Case is a parameter of the text operators, defaulting to the common case: -`Matches`, `StartsWith`. A Bloom index -implements the variant its configuration fixes (`downcase` on or off), so a -case-sensitive query against a case-folding index is a type error, not an -empty result. +`Matches`, `StartsWith`, with +`op::Matches.case_sensitive()` for the other. A Bloom index implements the +variant its configuration fixes (`downcase` on or off), so a case-sensitive +query against a case-folding index is a type error, not an empty result. ### 2. An index declares the operators it supports, and the query term for each ```rust -trait Index { type Stored; } +pub trait Operator: Copy + 'static {} -trait Supports: Index { - type Input; // what the operator takes: T, (T, T), an embedding … - type Query: Term; // what this index sends for this operator +pub trait Supports: Index { + type Query: QueryTerm; // what this index sends for this operator + fn check(&self) -> Result<(), PlanError>; + fn query(term: Self::Term, token: QueryToken) -> Self::Query; } ``` -`Query` is the query term, and decision 3 says what it must be. `Term` and -the two traits that refine it are defined in decision 4. `Query` is not -`Stored`. An ORE index's `Supports::Query` is the left ciphertext; its -`Stored` is left and right. A Bloom index's `Supports::Query` is a -filter over the needle; its `Stored` is a filter over the value. - -The bound tightens per operator. `Supports` requires -`Query: Probabilistic`; `Supports` requires `Query: Ranked`. An +`S` is the plaintext type the index is over, and `Index::Term` is the +stored term it derives. `Query` is the query term, and `query` makes it from +the term the index derived for the query's value. `QueryToken` can be made +only inside the crate, so only a query chain calls `query`, and no caller can +turn a stored term into a query term by hand. `check` refuses a configuration +that cannot answer `Op` although its type says it can: a match index whose +declared case disagrees with its `downcase` option. + +`QueryTerm` and the two traits that refine it are defined in decision 4. +`Query` is not `Term`. An ORE index's `Supports::Query` is the left +ciphertext; its `Term` is left and right. A Bloom index's +`Supports::Query` is a filter over the needle (`MatchQuery`); its +`Term` is a filter over the value (`MatchTerms`). + +The bound tightens per operator. `Supports` requires +`Query: Probabilistic`; `Supports` requires `Query: Ranked`. An index cannot claim an inexact operator without saying how its answer is checked. ### 3. The domain type chooses the best index for an operator ```rust -trait Answers { type Via: Supports; } +pub trait Answers { + type Via: Supports + Default; // an index is a unit type, so its Default is the index +} -impl Answers for TextSearch { type Via = Hmac; } -impl Answers for TextSearch { type Via = Ore; } -impl Answers for TextSearch { type Via = Bloom; } +impl Answers for TextSearch { type Via = Equality; } +impl Answers for TextSearch { type Via = Ore; } +impl Answers for TextSearch { type Via = Match; } ``` "Best" is defined per domain type, by hand, and means the most efficient or @@ -169,28 +187,41 @@ dictate. A domain with both HMAC and ORE answers `Eq` with HMAC: cheaper to compare, and the ORE term reveals order. This is the decision the original client made by accident (first after a sort) and this ADR makes on purpose. -A domain that does not answer an operator has no `Answers` impl, and the -query is a compile error. +A domain that does not answer an operator has no `Answers` impl, and +the query is a compile error. + +A plan that declares its indexes rather than a domain type (a field of a +fields plan, or a one-value plan built `with` indexes) has no type to hold +`Answers`, so the operator names its own default, and a plan that declared +another index for it names that one with `.via(index)`: + +```rust +pub trait DefaultIndex: Operator { + type Via: Supports + Default; +} +``` ### 4. A query term is one of three things, and the trait it implements is its fidelity ```rust -pub trait Term: Serialize { type Plaintext; } +pub trait QueryTerm {} -pub trait Probabilistic: Term { +pub trait Probabilistic: QueryTerm { + type Plaintext; fn verify(&self, value: &Self::Plaintext) -> bool; fn target_limit(&self, want: u64, stats: &Stats) -> Limit; } -pub trait Ranked: Term { +pub trait Ranked: QueryTerm { + type Plaintext; fn score(&self, value: &Self::Plaintext) -> Score; fn target_limit(&self, want: u64) -> u64; } ``` -- A term that implements `Term` only is **exact**. The server's answer is the - answer. HMAC `Eq`, ORE and OPE on every kind, text included (`Lt`, - `Between`), JSON `AtPath`. +- A term that implements `QueryTerm` only is **exact**. The server's answer + is the answer. HMAC `Eq`, ORE and OPE (`Lt`, `Between`) on numbers and on + text, JSON `AtPath`. - A `Probabilistic` term can return **false positives and no false negatives**. `verify` is the operator's own meaning, run on a decrypted value: it decides whether a row the server returned is a real match. Bloom @@ -221,7 +252,7 @@ except dropping the row, which is `false` by another name. The one input the library refuses, a needle too short to tokenise, is refused at `cipher.query`, before a term exists. -`Term::Plaintext` is the decrypted value the term is defined over, and that +`Plaintext` is the decrypted value the term is defined over, and that is not always the value the user wrote. For `Matches` it is the text. For `Nearest` over a similarity index it is the stored embedding, or the LSH sketch when only the sketch is stored; `score` is exact with respect to what @@ -238,12 +269,17 @@ says so. `AtPath` are exact, as `Eq` and `Lt` are, and an accessor over a probabilistic leaf operator, should one be added, is `Probabilistic`. +`QueryTerm` is a marker today, and the two refinements are added to it as +the indexes that need them land. A term serialises to the JSON its query +domain takes (decision 8), which is a property of each term type rather than +of the trait. + ### 5. Verification is the caller's, after decrypt; the library supplies the predicate and the bound The library never runs a query, so it cannot filter the result. What it gives a caller of a `Probabilistic` term is two things. -`verify` is the predicate. For `Matches`, it folds case, +`verify` is the predicate. For `Matches`, it folds case, tokenises both sides with the index's own tokenizer and checks that the needle's n-grams occur in the value. Whether that check is n-gram containment or a substring test (they differ once the query is longer than one n-gram) @@ -360,26 +396,29 @@ say so before the query runs. `λ` is also the number of decrypts a query will waste, which gives the Bloom parameters a number to be tuned against. Taking the bad row: a needle one character longer (`alic`, two 3-grams, `b = 12`) brings `λ` to 58; doubling `m` to 4,096 brings it to 268 at `b = 6`, and -`m = 8,192` to 6. Halving `k` to 3 at `m = 2,048` also gives 268, since fewer -hashes set fewer bits, but it weakens the filter against longer needles. The -cheapest lever is often a minimum needle length for long-text columns, which -a `Probabilistic` term can report from `target_limit` alone. +`m = 8,192` to 6. A smaller `k` makes it worse, not better: at `k = 3` the +needle `ali` sets 3 bits, so `b = 3`, and although the fill falls to +`1 − e^(−3·200/2048) = 0.254`, `p = 0.254³ = 0.016` and `λ` is about 16,400, +twice the `k = 6` figure. For one-gram needles the best `k` is near +`(m/n)·ln 2 ≈ 7`, so `k = 6` is close to it already. The cheapest lever is +often a minimum needle length for long-text columns, which a `Probabilistic` +term can report from `target_limit` alone. #### Text ordering -`Lt` on text is exact, decision 4, so none of the above applies to it: no -`p`, no `verify`, no `target_limit`. The term is an order term over the -column's order key, which is `orderable-bytes`' encoding of the value after -the text transform the column declares (ADR-0002's fold, or another, since -the encoding is pluggable), and the server compares keys. Two values with -one key tie; that is what the column declared, and an earlier draft of this -ADR was wrong to call it a false positive and put the term under -`Probabilistic`. Its use is ordering and cursor paging on names (`ORDER BY -name`, "the page after Nguyen"), and EQL v4 carries `query_text_ord` and -`query_text_ord_ore` for it. Whether the Rust library produces the term is a -catalog question (`TextOrdOre` is unproducible today, pending ADR-0002); -nothing else in this ADR depends on it, and if text ordering is dropped this -paragraph goes with it. +`Lt` on text is exact, decision 4, so none of the above applies to it: no `p`, +no `verify`, no `target_limit`. The term is an order term over the column's +order key, which is `orderable-bytes`' encoding of the value after the text +transform the column declares (the system ADR-0002's fold, or another, since +the encoding is pluggable), and the server compares keys. Two values with one +key tie; that is what the column declared, and an earlier draft of this ADR +was wrong to call it a false positive and put the term under `Probabilistic`. +Its use is ordering and cursor paging on names (`ORDER BY name`, "the page +after Nguyen"), and EQL v4 carries `query_text_ord` and `query_text_ord_ore` +for it. Whether the Rust library produces the term is a catalog question +(`TextOrdOre` is unproducible today, pending the system ADR-0002); nothing +else in this ADR depends on it, and if text ordering is dropped this paragraph +goes with it. A `Ranked` term's `target_limit` applies the index's configured recall factor and needs no `Stats`. @@ -390,7 +429,7 @@ A `Probabilistic` term composes with the caller's query library like this. One shot, with a bound, and the completeness check from decision 5: ```rust -let term = cipher.query::("ali").on(&users::name)?; +let term = cipher.query(op::Matches, "ali").on(&users::name).await?; let fetch = term.target_limit(10, &stats).fetch(); let rows: Vec = sqlx::query_as( @@ -413,7 +452,7 @@ an exactly ordered column, carried into the next page as `WHERE id > $cursor` instead of an `OFFSET`: ```rust -let term = cipher.query::("ali").on(&users::name)?; +let term = cipher.query(op::Matches, "ali").on(&users::name).await?; let (mut hits, mut after, page) = (Vec::new(), 0_i64, 50_i64); while hits.len() < 10 { @@ -445,7 +484,7 @@ the ranked loop is worth packaging; it is deferred, not refused. ### 7. `Between` is one term and one operand -`cipher.query::((lo, hi))` produces one term carrying both ORE left +`cipher.query(op::Between, (lo, hi))` produces one term carrying both ORE left ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. One call derives both bounds, under one context, so a caller cannot pair a bound from one column with a bound from another. @@ -481,10 +520,10 @@ together. The EQL v4 side carries one query domain per (type family, term), such as `eql_v4.query_text_match` and `eql_v4.query_integer_ord_ore`, rather than one per operator or one per stored domain. This is the "option B" of the design -discussion and is recorded here so the Rust side's `Supports::Query` has a -fixed target: the query term for an (index, operator) serialises to exactly -one of those domains. `Between` serialises to one value of the family's range -domain, which carries both bounds (decision 7). +discussion and is recorded here so the Rust side's `Supports::Query` +has a fixed target: the query term for an (index, operator) serialises to +exactly one of those domains. `Between` serialises to one value of the +family's range domain, which carries both bounds (decision 7). ## Considered options @@ -514,8 +553,8 @@ is a non-match. own `LIMIT` and paging, and need a driver. That is a query interface, which this library is not. -**A range as two terms**, `query::(lo)` and `query::(hi)` bound as -two predicates. Rejected in favour of one `Between` term, decision 7. `Gte` +**A range as two terms**, `query(op::Gte, lo)` and `query(op::Lte, hi)` bound +as two predicates. Rejected in favour of one `Between` term, decision 7. `Gte` and `Lte` themselves stay; they are the open-ended comparisons, and a range is not two of them. @@ -532,7 +571,7 @@ measured, and it costs one number in the index declaration. Decision 5. ADR-0004 decision 5 is superseded. Its rule — a stored term goes through a target, a query term comes from the query path — stands, but its justification was wrong and is replaced: the two differ for every index but -HMAC and lossless OPE, and the query path is `cipher.query::(..)`, which +HMAC and lossless OPE, and the query path is `cipher.query(op, ..)`, which produces a term the stored side never would. ADR-0007 needs an amendment once this ADR is accepted. The plan a binding @@ -563,20 +602,21 @@ separately, so a `Nearest` query must select the embedding, or the sketch, as well. Text ordering needs no special case: `Lt` on text is an exact term over the -order key, and what that key folds is ADR-0002's decision (the value model, -#1139), which keeps equality off the order term for the same reason. +order key, and what that key folds is the system ADR-0002's decision (the +value model, linked above), which keeps equality off the order term for the +same reason. EQL v4 gains `eql_v4.between(column, range)` and a range query domain per ordered family, under the inlining requirement of decision 7. That is EQL-side work this ADR asks for, not a deferral. It is a breaking change to the query chain. `QueryUsing::index(..)` goes; -`QueryUsing::equality()` becomes `query::`. The write side (`encrypt`, -targets, plans) does not change. +`QueryUsing::equality()` becomes `query(op::Eq, ..)`. The write side +(`encrypt`, targets, plans) does not change. ## Deferred -- Whether `Nearest` is `cipher.query::(..)` or a separate +- Whether `Nearest` is `cipher.query(op::Nearest, ..)` or a separate `cipher.nearest(..)` entry, and whether a ranked query can carry further predicates. The latter is possible only when the server can join candidates to rows (the `ByRow` similarity mode), not in the per-band pseudonymous mode. @@ -602,7 +642,7 @@ request, and yields `User`s. **`Eq`, exact.** The term is an HMAC; the operator is `=`. Nothing to verify. ```rust -let term = cipher.query::("bob@example.com").on(&users::email)?; +let term = cipher.query(op::Eq, "bob@example.com").on(&users::email).await?; let rows: Vec = sqlx::query_as( "SELECT id, email, name FROM users WHERE email = $1::eql_v4.query_text_eq", @@ -616,7 +656,7 @@ let users = cipher.open(rows).using(&users_plan).await?; operator is `<`. `Lte`, `Gt` and `Gte` are the same shape. ```rust -let term = cipher.query::(40).on(&users::age)?; +let term = cipher.query(op::Lt, 40).on(&users::age).await?; let rows: Vec = sqlx::query_as( "SELECT id, email, name, age FROM users WHERE age < $1::eql_v4.query_integer_ord_ore ORDER BY age", @@ -628,7 +668,7 @@ let rows: Vec = sqlx::query_as( **`Between`, exact, one term and one parameter** (decision 7). ```rust -let term = cipher.query::((18, 65)).on(&users::age)?; +let term = cipher.query(op::Between, (18, 65)).on(&users::age).await?; let rows: Vec = sqlx::query_as( "SELECT id, email, name, age FROM users @@ -645,7 +685,7 @@ the operator is `@@`, column on the left. The query selects the ciphertext `LIMIT` comes from `target_limit`: ```rust -let term = cipher.query::("ali").on(&users::name)?; +let term = cipher.query(op::Matches, "ali").on(&users::name).await?; let fetch = term.target_limit(10, &stats).fetch(); let rows: Vec = sqlx::query_as( @@ -683,8 +723,8 @@ or, before anyone has measured, `Stats::declared(1_000_000)`, which uses the verified; `age` is exact and the server's answer stands: ```rust -let name = cipher.query::("ali").on(&users::name)?; -let age = cipher.query::(30).on(&users::age)?; +let name = cipher.query(op::Matches, "ali").on(&users::name).await?; +let age = cipher.query(op::Gt, 30).on(&users::age).await?; let rows: Vec = sqlx::query_as( "SELECT id, name, age FROM users @@ -705,7 +745,7 @@ containment, `@>`, which a GIN index serves. The row is a match when the document holds that value at that path, and nothing is verified. ```rust -let term = cipher.query::>(("$.address.city", "Sydney")).on(&users::profile)?; +let term = cipher.query(op::AtPath(op::Eq), ("$.address.city", "Sydney")).on(&users::profile).await?; let rows: Vec = sqlx::query_as( "SELECT id, profile FROM users WHERE profile @> $1::eql_v3.query_json", @@ -719,7 +759,7 @@ selector and a leaf term, because the SQL is `col -> selector >= term`. The accessor term exposes both: ```rust -let term = cipher.query::>(("$.score", 80)).on(&users::profile)?; +let term = cipher.query(op::AtPath(op::Gte), ("$.score", 80)).on(&users::profile).await?; let rows: Vec = sqlx::query_as( "SELECT id, profile FROM users WHERE profile -> $1::text >= $2::eql_v3.query_double_ord", @@ -738,7 +778,7 @@ the server returns candidates by token overlap; `score` re-ranks on the decrypted embedding, which the query must select: ```rust -let term = cipher.query::(embedding).on(&docs::vector)?; +let term = cipher.query(op::Nearest, embedding).on(&docs::vector).await?; let fetch = term.target_limit(10); let rows: Vec = sqlx::query_as( @@ -790,7 +830,7 @@ and each is a capability the data grammar does not have yet, added once 1. **An operator name with its input**, in place of the index: `{"op": "matches", "input": "ali"}`, `{"op": "between", "input": [18, 65]}`, `{"op": "at_path", "path": "$.score", "inner": {"op": "gte", "input": 80}}`. - The engine resolves `Answers` for the field's type and refuses an + The engine resolves `Answers` for the field's type and refuses an operator the type does not answer, with the same shape of error a mistyped field gets today. 2. **The term back**, as the JSON its query domain takes, plus which domain @@ -910,7 +950,7 @@ cipher.query(users.email).matches('bob') // type error: TextEq answers eq and cipher.query(users.age).lt('40') // type error: IntegerOrdOre takes a number ``` -The type behind it is the `Answers` table of decision 3, as a mapped +The type behind it is the `Answers` table of decision 3, as a mapped type over the column's EQL type: ```ts From 266f4365f48048d0c7c821a1bc73c914e0f95d8a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 22:32:58 +1100 Subject: [PATCH 07/38] docs(adr): ADR-0009 third review round Answers the follow-ups from the second review of ADR-0009 and the one it left for the ordering operators. eql_v4.between must not be STRICT: PostgreSQL inlines a STRICT SQL function only when its body is strict, and a >= x AND b <= y is not, so a STRICT declaration passes a test that calls the function and is then not inlined. Decision 7 says so beside the inlining requirement. The range query domain is per ordered query domain, OPE as well as ORE, one per (family, term) as decision 8 has it, so the OPE ordering that the operators reach through .via(Ope) has a Between operand too. Text ordering stays a catalog question. The ordering operators answer the integer types only, and an ordering on a text plan does not compile, so the exact-term list, the consequences and AtPath on a text leaf say "if the catalog keeps text ordering" rather than stating it as settled. --- ...and-a-query-term-is-not-the-stored-term.md | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 4901477ff..0cd8e8898 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -220,8 +220,9 @@ pub trait Ranked: QueryTerm { ``` - A term that implements `QueryTerm` only is **exact**. The server's answer - is the answer. HMAC `Eq`, ORE and OPE (`Lt`, `Between`) on numbers and on - text, JSON `AtPath`. + is the answer. HMAC `Eq`, ORE and OPE (`Lt`, `Between`) on numbers, JSON + `AtPath`, and ORE and OPE on text if the catalog keeps text ordering + (see "Text ordering" below). - A `Probabilistic` term can return **false positives and no false negatives**. `verify` is the operator's own meaning, run on a decrypted value: it decides whether a row the server returned is a real match. Bloom @@ -490,9 +491,11 @@ call derives both bounds, under one context, so a caller cannot pair a bound from one column with a bound from another. In SQL the term stays one operand. EQL v4 gains a function that takes the -column and the range, and a range query domain beside each ordered family's -`ord` domain to carry it (shown here as `query_integer_range_ore`; the -catalog names it): +column and the range, and a range query domain beside each ordered query +domain to carry it, for OPE as well as ORE: `query_integer_ord_ore` gains +`query_integer_range_ore` and `query_integer_ord_ope` gains +`query_integer_range_ope`, one per (family, term) as decision 8 has it. The +example uses the ORE one; the catalog names them: ```sql WHERE eql_v4.between(age, $1::eql_v4.query_integer_range_ore) @@ -509,8 +512,11 @@ had just joined, and every SDK had to grow a `bounds()` to feed it. What the one-operand form asks of EQL is that the planner still see a range. `eql_v4.between` must be a SQL function PostgreSQL inlines at planning time (`LANGUAGE sql`, one expression, `$1 >= lo($2) AND $1 <= hi($2)` with -immutable accessors), so that a btree over the ORE term serves it as one -range scan. A function the planner cannot inline is a call per row on a +immutable accessors, and not `STRICT`), so that a btree over the ORE term +serves it as one range scan. `STRICT` matters: PostgreSQL inlines a `STRICT` +SQL function only when its body is strict too, and `a >= x AND b <= y` is not +(`NULL AND false` is `false`), so a `STRICT` declaration would pass a test +that only calls the function and then not be inlined. A function the planner cannot inline is a call per row on a sequential scan, which no index can rescue. That is an EQL-side requirement with an EQL-side test, and it is why the function and the domain arrive together. @@ -601,13 +607,13 @@ JSON column's query already selects it. A similarity index stores its vector separately, so a `Nearest` query must select the embedding, or the sketch, as well. -Text ordering needs no special case: `Lt` on text is an exact term over the -order key, and what that key folds is the system ADR-0002's decision (the -value model, linked above), which keeps equality off the order term for the -same reason. +Text ordering, if the catalog keeps it, needs no special case: `Lt` on text +is an exact term over the order key, and what that key folds is the system +ADR-0002's decision (the value model, linked above), which keeps equality off +the order term for the same reason. EQL v4 gains `eql_v4.between(column, range)` and a range query domain per -ordered family, under the inlining requirement of decision 7. That is +ordered query domain, ORE and OPE, under the inlining requirement of decision 7. That is EQL-side work this ADR asks for, not a deferral. It is a breaking change to the query chain. `QueryUsing::index(..)` goes; @@ -769,8 +775,9 @@ let rows: Vec = sqlx::query_as( .fetch_all(&pool).await?; ``` -`AtPath` on a text leaf is exact, as `Lt` on a text column is: the -leaf's order term compares order keys, and there is nothing to verify. +`AtPath` on a text leaf, if the catalog keeps text ordering, is exact, +as `Lt` on a text column would be: the leaf's order term compares order +keys, and there is nothing to verify. **`Nearest`, ranked.** Stack Similarity has no EQL spelling yet, so this is the shape and not a statement that runs. The term is a set of band tokens; From 7b9ed2c187f66b4b94ae5eed4a7a05ff1af44445 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 22:10:22 +1100 Subject: [PATCH 08/38] feat(stack-encrypt)!: a query names an operator, not an index `cipher.query(v).using(&plan)` now finishes with `.op(op::Eq)` or `.op(op::Matches::new())`, and the plan finds the index that answers the operator (ADR-0009, decisions 1 to 4). - `stack_encrypt::op` holds the operators `Eq` and `Matches`, the case markers, and the traits `Operator`, `QueryTerm`, `Supports` (on an index), `Answers` (on a target type, by hand) and `DefaultIndex` (the index a plan that declares indexes answers an operator with). - `.op_via(op, index)` names another index, such as a match index with its own configuration, or one of a target's several terms. - A match query is its own type, `sem::MatchQuery`, the needle's filter, so the query term is not typed as the stored `MatchTerms`. - A case-sensitive `Matches` against a case-folding index, an operator a target does not answer, and a text operator on an integer are compile errors (tests/ui). BREAKING CHANGE: `QueryUsing::equality()` is removed; write `.op(op::Eq)`. `QueryUsing::index(..)` stays: it is the only way to query ORE and OPE until the ordering operators exist (#1186). The ADR's code now uses the spelling that ships, and its base trait is `QueryTerm`, since `sem::Term` already names the derivation trait. Closes #1182 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 18 + packages/stack-encrypt/CONTEXT.md | 19 +- packages/stack-encrypt/src/lib.rs | 1 + packages/stack-encrypt/src/op.rs | 328 +++++++++++++++ packages/stack-encrypt/src/plan/build.rs | 8 +- packages/stack-encrypt/src/plan/chain.rs | 145 ++++++- packages/stack-encrypt/src/plan/mod.rs | 12 +- packages/stack-encrypt/src/plan/value.rs | 4 +- packages/stack-encrypt/src/sem/mod.rs | 62 +++ packages/stack-encrypt/tests/derive_plan.rs | 3 +- packages/stack-encrypt/tests/op.rs | 389 ++++++++++++++++++ packages/stack-encrypt/tests/plan.rs | 27 +- packages/stack-encrypt/tests/plan_grammar.rs | 29 +- .../tests/ui/op_case_mismatch.rs | 18 + .../tests/ui/op_case_mismatch.stderr | 26 ++ .../tests/ui/op_matches_on_integer.rs | 11 + .../tests/ui/op_matches_on_integer.stderr | 21 + .../tests/ui/op_target_does_not_answer.rs | 16 + .../tests/ui/op_target_does_not_answer.stderr | 22 + .../stack-encrypt/tests/ui/pass/plan_chain.rs | 9 +- .../tests/ui/plan_query_undeclared_index.rs | 3 +- .../ui/plan_query_undeclared_index.stderr | 27 +- 22 files changed, 1112 insertions(+), 86 deletions(-) create mode 100644 packages/stack-encrypt/src/op.rs create mode 100644 packages/stack-encrypt/tests/op.rs create mode 100644 packages/stack-encrypt/tests/ui/op_case_mismatch.rs create mode 100644 packages/stack-encrypt/tests/ui/op_case_mismatch.stderr create mode 100644 packages/stack-encrypt/tests/ui/op_matches_on_integer.rs create mode 100644 packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr create mode 100644 packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs create mode 100644 packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index e3396812a..315d7bd70 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -9,6 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **A query names an operator, not an index.** `cipher.query(v).using(&plan)` + finishes with `.op(op::Eq)` or `.op(op::Matches::new())`, and the plan + finds the index that answers the operator (ADR-0009). An index declares the + operators it answers (`op::Supports`), a target type names the index that + answers each one for it (`op::Answers`), and a plan that declares indexes + uses the operator's default index (`op::DefaultIndex`). `.op_via(op, index)` + names another index, such as a match index with its own configuration. An + operator the plan cannot answer is a compile error; a case-sensitive + `Matches` against a case-folding index is one too. +- **A match query has its own type, `sem::MatchQuery`.** It is the filter over + the needle that the server tests a stored `MatchTerms` for, and it comes + only from a query. + - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as `stack_encrypt::foreign_keyset`. Help text explains how to resolve @@ -47,6 +60,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Breaking +- **`QueryUsing::equality()` is removed.** Write `.op(op::Eq)`. A plan that + declares the equality index answers it the same way, and a target answers + it through `op::Answers`. `QueryUsing::index(..)` stays until the ordering + operators exist. + - **Update pattern matches for `dynamic::Error`.** `Context`, `Plan`, `Source`, and `Record` now carry `{ field: Option, reason: Reason }`; `Term` also gains diff --git a/packages/stack-encrypt/CONTEXT.md b/packages/stack-encrypt/CONTEXT.md index 06e677f72..9a8b342c8 100644 --- a/packages/stack-encrypt/CONTEXT.md +++ b/packages/stack-encrypt/CONTEXT.md @@ -223,12 +223,27 @@ _Avoid_: index (that is the declaration; a term is what it produces), token, hash **Query**: -Deriving the terms for a plaintext under a field's plan, with no ciphertext, -to match terms already stored. A query against an index the plan never +Deriving the query term for a plaintext under a field's plan, with no +ciphertext, to match terms already stored. A query names an operator, and the +plan finds the index that answers it. A query against an index the plan never declared is refused, so a query cannot silently match nothing. _Avoid_: probe, search term, lookup, encrypt query (that is the TypeScript package's method name, not the concept) +**Operator**: +What a query asks of the server — equality, text match — as a unit type +(`op::Eq`, `op::Matches`). An index declares the operators it answers and the +query term it sends for each; a target type names the index that answers each +operator for it. An operator nothing answers is a compile error (ADR-0009). +_Avoid_: query type, index (that is what answers the operator) + +**Query term**: +The value a query sends to the server. It is not the stored term: equality +sends the same HMAC it stores, but a match query is a filter over the needle, +where the stored term is a filter over the whole value. A query term is never +stored, and a stored term is never sent as a query. +_Avoid_: probe, stored term (that is what a write keeps) + **Open**: Decrypting through a plan: reading back every field of a record that can come back (sealed and passthrough fields, not index-only ones), each under diff --git a/packages/stack-encrypt/src/lib.rs b/packages/stack-encrypt/src/lib.rs index 37b36ba4f..65059a6ae 100644 --- a/packages/stack-encrypt/src/lib.rs +++ b/packages/stack-encrypt/src/lib.rs @@ -277,6 +277,7 @@ pub mod descriptor; #[cfg(feature = "dynamic")] pub mod dynamic; pub mod keyset; +pub mod op; pub mod plan; pub mod sem; pub mod target; diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op.rs new file mode 100644 index 000000000..9b083ac1b --- /dev/null +++ b/packages/stack-encrypt/src/op.rs @@ -0,0 +1,328 @@ +//! Operators: what a query asks of the server, as a type. +//! +//! A query names an *operator* ([`Eq`](struct@Eq), [`Matches`]), and the plan it runs +//! through finds the index that answers it. The operator is a unit type +//! passed by value, as an index is (`.index(Ore)`): +//! +//! ``` +//! # async fn example() -> Result<(), stack_encrypt::Error> { +//! use stack_encrypt::kms::FakeDataKeySource; +//! use stack_encrypt::{op, Equality, Match, Plan, StackCipher}; +//! +//! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; +//! let email_plan = Plan::context("users/email") +//! .with::((Equality, Match::default())) +//! .build()?; +//! +//! let exact = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; +//! let search = cipher.query("bob").using(&email_plan).op(op::Matches::new()).await?; +//! # let _ = (exact, search); +//! # Ok(()) +//! # } +//! # tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(example()).unwrap(); +//! ``` +//! +//! Three traits connect an operator to an index, and ADR-0009 records why: +//! +//! - [`Supports`] is on an index, for each operator it can answer +//! over plaintexts `S`, and names the [`QueryTerm`] it sends. That term is +//! not the stored term: an equality index sends the same HMAC it stores, +//! but a match index sends a filter over the needle ([`MatchQuery`]), +//! where it stores a filter over the whole value ([`MatchTerms`]). +//! - [`Answers`] is on a target type, a domain type such as an EQL +//! type, and names the index that answers `Op` for it. It is written by +//! hand per type, because the choice decides what the query term reveals. +//! - [`DefaultIndex`] is on an operator, and names the index a plan that +//! declares its indexes, rather than a target type, answers it with. +//! +//! An operator a plan cannot answer is a compile error: a match query on an +//! integer, a case-sensitive match against a case-folding index, or an +//! operator a target type does not answer. An index the plan does not +//! declare is refused when the query is awaited, as with +//! [`index`](crate::plan::QueryUsing::index), before any key is requested. +//! +//! The module is meant to be used by path (`op::Eq`): an imported `Eq` +//! would shadow the prelude's `Eq` trait. +//! +//! [`MatchQuery`]: crate::sem::MatchQuery +//! [`MatchTerms`]: crate::sem::MatchTerms +use std::fmt; +use std::marker::PhantomData; + +use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms}; +use crate::target::{Equality, Index, Match}; + +/// A query operator: a unit type naming what a query asks of the server. +/// +/// The trait is open, as [`Index`] is: an operator defined in another crate +/// is answered by the indexes that implement [`Supports`] for it. +pub trait Operator: Copy + 'static { + /// The operator's name, as a binding or an error message spells it. + const NAME: &'static str; +} + +/// Equality: rows whose value equals the query's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Eq; + +impl Operator for Eq { + const NAME: &'static str = "eq"; +} + +/// Text search: rows whose text contains the needle's tokens, under the +/// match index's tokenizer. Not containment: a Bloom match is order- and +/// multiplicity-insensitive. +/// +/// `C` is the case the query is asked in, [`Insensitive`] unless named +/// otherwise. A match index answers only the case its configuration fixes +/// ([`MatchCase`]), so asking a case-folding index for a case-sensitive +/// match is a compile error rather than a query that quietly folds case. +pub struct Matches(PhantomData C>); + +impl Matches { + /// The operator. `Matches::new()` takes its case from the index it is + /// asked of; `Matches::::new()` names it. + pub const fn new() -> Self { + Self(PhantomData) + } +} + +// By hand, so `C` (a marker type) need not implement them. +impl Clone for Matches { + fn clone(&self) -> Self { + *self + } +} +impl Copy for Matches {} +impl Default for Matches { + fn default() -> Self { + Self::new() + } +} +impl PartialEq for Matches { + fn eq(&self, _: &Self) -> bool { + true + } +} +impl std::cmp::Eq for Matches {} +impl fmt::Debug for Matches { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Matches<{}>", C::NAME) + } +} + +impl Operator for Matches { + const NAME: &'static str = "matches"; +} + +mod sealed { + pub trait Sealed {} + impl Sealed for super::Insensitive {} + impl Sealed for super::Sensitive {} +} + +/// The case a text operator is asked in: [`Insensitive`] or [`Sensitive`]. +/// Sealed: there are exactly two. +pub trait Case: sealed::Sealed + 'static { + /// Whether text is lower-cased before it is tokenised. + const FOLDS: bool; + /// The case's name, for `Debug`. + const NAME: &'static str; +} + +/// Case-insensitive: text is lower-cased before it is tokenised. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Insensitive; + +/// Case-sensitive: text is tokenised as written. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Sensitive; + +impl Case for Insensitive { + const FOLDS: bool = true; + const NAME: &'static str = "Insensitive"; +} +impl Case for Sensitive { + const FOLDS: bool = false; + const NAME: &'static str = "Sensitive"; +} + +/// The case a match configuration fixes, as a type, so the operators its +/// index answers are known at compile time. +/// +/// `Case::FOLDS` must equal the configuration's +/// [`MatchOptions::downcase`](crate::sem::MatchOptions::downcase). A +/// configuration that does not implement this can still store and query +/// match terms; it cannot be queried by [`Matches`]. +pub trait MatchCase: MatchConfig { + /// The case the configuration's index answers. + type Case: Case; +} + +impl MatchCase for DefaultMatch { + type Case = Insensitive; +} + +/// A value a query sends to the server to compare against stored terms. +/// +/// A term that implements this and nothing more is *exact*: the server's +/// answer is the answer. +pub trait QueryTerm {} + +impl QueryTerm for EqualityTerm {} +impl QueryTerm for MatchQuery {} + +/// An index that answers the operator `Op` over plaintexts `S`, and the +/// query term it sends for it. +/// +/// [`Index::Term`] is what the index derives; [`query`](Self::query) makes +/// the query term from it. For an equality index the two are the same +/// HMAC. For a match index they are not: the query is a filter over the +/// needle, and its type says so. +#[diagnostic::on_unimplemented( + message = "`{Self}` does not answer `{Op}` over `{S}`", + label = "this index cannot answer the operator", + note = "an equality index answers `op::Eq`; a match index answers `op::Matches` over text, in the case its configuration fixes" +)] +pub trait Supports: Index { + /// What this index sends for `Op`. + type Query: QueryTerm; + /// The query term, from the term the index derived for the query's + /// value. + fn query(term: Self::Term) -> Self::Query; +} + +impl Supports for Equality { + type Query = EqualityTerm; + fn query(term: EqualityTerm) -> EqualityTerm { + term + } +} + +impl Supports, S> for Match +where + S: AsRef, + O: MatchCase, +{ + type Query = MatchQuery; + fn query(term: MatchTerms) -> MatchQuery { + MatchQuery::from_terms(term) + } +} + +/// A target type that answers the operator `Op` over plaintexts `S`, and +/// the index it answers it with. +/// +/// Written by hand per type: a type with two indexes that both answer an +/// operator chooses the cheaper or the more private, as the indexes in play +/// dictate, and that choice decides what the query term reveals. A type +/// that does not answer an operator has no impl, and the query does not +/// compile. +#[diagnostic::on_unimplemented( + message = "`{Self}` does not answer `{Op}` over `{S}`", + label = "this target type does not answer the operator", + note = "a target answers the operators its type implements `Answers` for; `.op_via(op, index)` names an index the target declares" +)] +pub trait Answers { + /// The index that answers `Op` for this type. + type Via: Supports; + /// That index, as a value. + fn via() -> Self::Via; +} + +impl Answers for EqualityTerm { + type Via = Equality; + fn via() -> Equality { + Equality + } +} + +impl Answers, S> for MatchTerms +where + S: AsRef, + O: MatchCase, +{ + type Via = Match; + fn via() -> Match { + Match::new() + } +} + +/// A ciphertext with one term beside it, the shape of a `plaintext = ..` +/// record without the struct, answers what its term answers. +impl> Answers for (crate::StackCipherText, T) { + type Via = T::Via; + fn via() -> T::Via { + T::via() + } +} + +/// A ciphertext with one term, as [`indexed`](crate::target::indexed) +/// produces it, answers what its term answers. One with several terms +/// names the index with `.op_via(op, index)`. +impl> Answers for crate::target::Encrypted { + type Via = T::Via; + fn via() -> T::Via { + T::via() + } +} + +/// The index a plan that declares its indexes answers an operator with: +/// a field of a fields plan, or a one-value plan built `with` indexes. +/// +/// A plan that declares a different index for the operator, such as a +/// match index with its own configuration, names it with +/// `.op_via(op, index)`. +#[diagnostic::on_unimplemented( + message = "`{Self}` has no default index over `{S}`", + label = "no default index answers this operator here", + note = "`op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; name another index with `.op_via(op, index)`" +)] +pub trait DefaultIndex: Operator { + /// The index. + type Via: Supports; + /// That index, as a value. + fn via() -> Self::Via; +} + +impl DefaultIndex for Eq { + type Via = Equality; + fn via() -> Equality { + Equality + } +} + +impl> DefaultIndex for Matches<::Case> { + type Via = Match; + fn via() -> Match { + Match::default() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_default_match_configuration_folds_case_as_its_type_says() { + assert_eq!( + ::Case::FOLDS, + DefaultMatch::options().downcase + ); + } + + #[test] + fn an_operator_is_equal_to_itself() { + assert_eq!(Matches::::new(), Matches::default()); + assert_eq!(Eq, Eq); + } + + #[test] + fn an_operator_debugs_with_its_case() { + assert_eq!( + format!("{:?}", Matches::::new()), + "Matches" + ); + assert_eq!(format!("{:?}", Eq), "Eq"); + } +} diff --git a/packages/stack-encrypt/src/plan/build.rs b/packages/stack-encrypt/src/plan/build.rs index d8537f64a..47825db6c 100644 --- a/packages/stack-encrypt/src/plan/build.rs +++ b/packages/stack-encrypt/src/plan/build.rs @@ -244,7 +244,7 @@ pub(crate) fn check_declared( /// # struct User { email: String, id: u64 } /// # async fn example() -> Result<(), stack_encrypt::Error> { /// use stack_encrypt::kms::FakeDataKeySource; -/// use stack_encrypt::{Equality, Plan, StackCipher}; +/// use stack_encrypt::{op, Equality, Plan, StackCipher}; /// /// let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; /// let user = User { email: "bob@example.com".into(), id: 42 }; @@ -262,7 +262,7 @@ pub(crate) fn check_declared( /// .query("bob@example.com") /// .context("tenants/acme/users") /// .using(&email_plan) -/// .equality() +/// .op(op::Eq) /// .await?; /// # let _ = query_value; /// let back = cipher.open(record).context("tenants/acme/users").using(&users_plan).await?; @@ -915,7 +915,7 @@ impl FieldsBuilder { /// # async fn example() -> Result<(), stack_encrypt::Error> { /// use stack_encrypt::kms::FakeDataKeySource; /// use stack_encrypt::sem::{EqualityTerm, MatchTerms}; - /// use stack_encrypt::{Encrypted, Plan, StackCipher}; + /// use stack_encrypt::{op, Encrypted, Plan, StackCipher}; /// /// let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; /// let users_plan = Plan::context("users") @@ -928,7 +928,7 @@ impl FieldsBuilder { /// let email: Encrypted<(EqualityTerm, MatchTerms)> = record.take("email")?; /// /// // The target declares an equality index, so the field answers it. - /// let query_value = cipher.query("bob@example.com").using(&users_plan.field("email")?).equality().await?; + /// let query_value = cipher.query("bob@example.com").using(&users_plan.field("email")?).op(op::Eq).await?; /// assert_eq!(query_value, email.terms.0); /// # Ok(()) /// # } diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index abe9b4622..51c2cccca 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -11,9 +11,10 @@ use super::build::{FieldPlan, FieldsBuilder, IntoLabel, Opens, PlanContext, Runs use super::field_ref::FieldRef; use super::value::{Indexed, Typed, ValuePlan, ValueShape}; use super::PlanError; +use crate::op::{Answers, DefaultIndex, Operator, Supports}; use crate::target::{ ciphertext, Borrowed, CallerContext, DeclaredContext, DecryptField, Decryptable, EncryptFrom, - Equality, Index, Indexes, Owned, Pending, PendingFuture, Select, + Index, Indexes, Owned, Pending, PendingFuture, Select, }; use crate::{Error, KeysetCipher, Label, LabelError, NonEmpty, StackCipher, StackCipherText}; @@ -280,7 +281,7 @@ impl StackCipher { /// Start a chain that derives a query term for `value`, to match terms /// a plan wrote. Name the plan with [`using`](QueryBuilder::using) and - /// the index with `equality()` or `index(..)`. + /// the operator with [`op`](QueryUsing::op): `.op(op::Eq)`. /// /// The value is copied once (`to_owned`), so `query("bob@example.com")` /// queries a `String` field. @@ -625,12 +626,27 @@ pub struct QueryUsing<'a, F, P, K> { } impl<'a, F: 'static, K: 'static> QueryUsing<'a, F, FieldPlan, K> { - /// The equality term: sugar for `index(Equality)`. - pub fn equality(self) -> QueryIndex<'a, F, Equality, K> + /// The query term for the operator `op`, through the index a plan that + /// declares its indexes answers it with ([`DefaultIndex`]): + /// `.op(op::Eq)` is the equality term. The field must have declared + /// that index, with the same options; otherwise awaiting is + /// [`PlanError::IndexNotDeclared`] or [`PlanError::IndexOptions`]. + pub fn op>(self, op: Op) -> QueryOp<'a, F, Op, Op::Via, K> { + self.op_via(op, Op::via()) + } + + /// The query term for `op` through `index`, for a field that declares + /// an index other than the operator's default, such as a match index + /// with its own configuration. + pub fn op_via(self, _op: Op, index: I) -> QueryOp<'a, F, Op, I, K> where - Equality: Index, + Op: Operator, + I: Supports, { - self.index(Equality) + QueryOp { + inner: self.index(index), + op: PhantomData, + } } /// The term of `index`. The field must have declared it, and the query's @@ -652,14 +668,32 @@ impl<'a, F: 'static, K: 'static> QueryUsing<'a, F, FieldPlan, K> { } impl<'a, F: 'static, X: Indexes, K: 'static> QueryUsing<'a, F, ValuePlan>, K> { - /// The equality term, selected from the plan's indexes by type: a plan - /// with no `Equality` index does not compile here. - pub fn equality(self) -> QueryIndex<'a, F, Equality, K> + /// The query term for the operator `op`, through the index a plan that + /// declares its indexes answers it with ([`DefaultIndex`]), selected + /// from the plan's indexes by type: a plan that does not hold that + /// index does not compile here. + pub fn op(self, op: Op) -> QueryOp<'a, F, Op, Op::Via, K> where - Equality: Index, - X: Select, + Op: DefaultIndex, + Op::Via: Clone, + X: Select, { - self.index::() + self.op_via(op, Op::via()) + } + + /// The query term for `op` through the plan's index of the type of + /// `index`, for a plan that holds an index other than the operator's + /// default. The plan's own index is used; `index` names its type. + pub fn op_via(self, _op: Op, _index: I) -> QueryOp<'a, F, Op, I, K> + where + Op: Operator, + I: Supports + Clone, + X: Select, + { + QueryOp { + inner: self.index::(), + op: PhantomData, + } } /// The term of the plan's index of type `I`, selected by type @@ -687,12 +721,30 @@ impl<'a, F: 'static, T, K: 'static> QueryUsing<'a, F, ValuePlan>, K> where Typed: ValueShape, { - /// The equality term: sugar for `index(Equality)`. - pub fn equality(self) -> QueryIndex<'a, F, Equality, K> + /// The query term for the operator `op`, through the index the target + /// type answers it with ([`Answers`]). A target that does not answer + /// `op` does not compile here; one that answers it through an index it + /// does not declare is [`PlanError::IndexNotDeclared`] when awaited. + pub fn op(self, op: Op) -> QueryOp<'a, F, Op, T::Via, K> + where + Op: Operator, + T: Answers, + { + self.op_via(op, T::via()) + } + + /// The query term for `op` through `index`, which the target must + /// declare: for a target with no [`Answers`] impl for `op`, such as + /// an [`Encrypted`](crate::target::Encrypted) tuple of terms. + pub fn op_via(self, _op: Op, index: I) -> QueryOp<'a, F, Op, I, K> where - Equality: Index, + Op: Operator, + I: Supports, { - self.index(Equality) + QueryOp { + inner: self.index(index), + op: PhantomData, + } } /// The term of `index`. The target must declare it @@ -754,6 +806,61 @@ where } } +/// A chain deriving one query term for an operator, ready to await. It +/// resolves to the [`QueryTerm`](crate::op::QueryTerm) the index sends for +/// the operator ([`Supports::Query`]). +pub struct QueryOp<'a, F, Op, I, K> { + inner: QueryIndex<'a, F, I, K>, + op: PhantomData Op>, +} + +impl<'a, F, Op, I, K: 'static> QueryOp<'a, F, Op, I, K> { + /// Run under this keyset, as [`QueryBuilder::keyset`]. + pub fn keyset(mut self, keyset: impl Into>) -> Self { + self.inner = self.inner.keyset(keyset); + self + } + + /// Extend the plan's every context by `parts` for this call, as + /// [`QueryBuilder::extend`]: the same parts the write was extended by. + pub fn extend(mut self, parts: impl Into) -> Self { + self.inner = self.inner.extend(parts); + self + } +} + +impl sealed::Sealed for QueryOp<'_, F, Op, I, K> {} + +impl<'a, F: 'static, Op: Operator, I: Supports + 'static, K: 'static> Operation<'a, K> + for QueryOp<'a, F, Op, I, K> +where + I::Term: MaybeSend, + I::Query: MaybeSend + 'static, +{ + type Output = I::Query; + fn cipher(&self) -> &'a StackCipher { + self.inner.cipher() + } + fn take_keyset(&mut self) -> Option> { + self.inner.take_keyset() + } + fn check(&self) -> Result<(), Error> { + self.inner.check() + } + fn prepare<'p>( + self, + keyset: &'p KeysetCipher<'a, K>, + scoped: bool, + ) -> Pending<'p, Self::Output, K> + where + 'a: 'p, + { + self.inner + .prepare(keyset, scoped) + .map(>::query) + } +} + /// A chain decrypting a stored record, before its plan is named. pub struct OpenBuilder<'a, R, K> { common: Common<'a, K>, @@ -846,6 +953,7 @@ settles! { ['a, S: 'static, K: 'static] EncryptFields<'a, S, K>; ['a, S: ?Sized, P, K] EncryptUsing<'a, S, P, K>; ['a, F, I, K] QueryIndex<'a, F, I, K>; + ['a, F, Op, I, K] QueryOp<'a, F, Op, I, K>; ['a, R, P, K] OpenUsing<'a, R, P, K>; } @@ -869,6 +977,7 @@ opaque_debug! { ['a, F, K] QueryBuilder<'a, F, K>, "QueryBuilder"; ['a, F, P, K] QueryUsing<'a, F, P, K>, "QueryUsing"; ['a, F, I, K] QueryIndex<'a, F, I, K>, "QueryIndex"; + ['a, F, Op, I, K] QueryOp<'a, F, Op, I, K>, "QueryOp"; ['a, R, K] OpenBuilder<'a, R, K>, "OpenBuilder"; ['a, R, P, K] OpenUsing<'a, R, P, K>, "OpenUsing"; } @@ -911,7 +1020,7 @@ impl std::fmt::Debug for All<'_, K, T> { /// ``` /// # async fn example() -> Result<(), stack_encrypt::Error> { /// use stack_encrypt::kms::FakeDataKeySource; -/// use stack_encrypt::{Equality, Plan, StackCipher}; +/// use stack_encrypt::{op, Equality, Plan, StackCipher}; /// /// let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; /// let email_plan = Plan::context("users/email").with(Equality).build()?; @@ -919,7 +1028,7 @@ impl std::fmt::Debug for All<'_, K, T> { /// let email = String::from("bob@example.com"); /// let (written, query_value) = stack_encrypt::all(( /// cipher.encrypt(&email).using(&email_plan), -/// cipher.query("bob@example.com").using(&email_plan).equality(), +/// cipher.query("bob@example.com").using(&email_plan).op(op::Eq), /// )) /// .await?; /// assert_eq!(written.terms, query_value); diff --git a/packages/stack-encrypt/src/plan/mod.rs b/packages/stack-encrypt/src/plan/mod.rs index 70ffd9f2a..d7fc1512d 100644 --- a/packages/stack-encrypt/src/plan/mod.rs +++ b/packages/stack-encrypt/src/plan/mod.rs @@ -151,7 +151,7 @@ //! # async fn example() -> Result<(), stack_encrypt::Error> { //! use stack_encrypt::kms::FakeDataKeySource; //! use stack_encrypt::sem::{EqualityTerm, MatchTerms}; -//! use stack_encrypt::{Encrypted, Equality, Match, Ore, Plan, StackCipher}; +//! use stack_encrypt::{op, Encrypted, Equality, Match, Ore, Plan, StackCipher}; //! //! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; //! let user = User { email: "bob@example.com".into(), age: 34, notes: "hi".into(), id: 42 }; @@ -170,7 +170,7 @@ //! //! // Query: derived under the label the write used, so it matches. //! let email_plan = users_plan.field("email")?; -//! let query_value = cipher.query("bob@example.com").using(&email_plan).equality().await?; +//! let query_value = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; //! assert_eq!(query_value, stored.terms.0); //! //! // Read: every field that can come back. @@ -222,7 +222,7 @@ //! use stack_encrypt::kms::FakeDataKeySource; //! use stack_encrypt::plan::pick; //! use stack_encrypt::sem::EqualityTerm; -//! use stack_encrypt::{Encrypted, EncryptFrom, Plan, StackCipher, StackCipherText}; +//! use stack_encrypt::{op, Encrypted, EncryptFrom, Plan, StackCipher, StackCipherText}; //! //! struct User { //! email: String, @@ -252,7 +252,7 @@ //! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; //! let user = User { email: "bob@example.com".into(), nickname: "bob".into() }; //! let record: EncryptedUser = cipher.default_keyset().encrypt_as(&user, ().into()).await?; -//! let query_value = cipher.query("bob@example.com").using(&by_hand.field("email")?).equality().await?; +//! let query_value = cipher.query("bob@example.com").using(&by_hand.field("email")?).op(op::Eq).await?; //! assert_eq!(record.email.terms, query_value); //! # Ok(()) //! # } @@ -301,7 +301,7 @@ //! | `fields()` | each field picked out of the value (by name, or by a picker's accessor as `project` does), `zip`ped | //! | `extend(parts)` | the [`DeclaredContext`](crate::target::DeclaredContext) the description runs under | //! | `using(&plan)` | [`KeysetCipher::run`](crate::KeysetCipher::run) of [`Plan::encryption`] | -//! | `query(v).using(..).equality()` | the selected index's operation alone, `.under(label)` | +//! | `query(v).using(..).op(op)`, `.index(x)` | the answering index's operation alone, `.under(label)`; for `op`, mapped to its query term ([`Supports::query`](crate::op::Supports::query)) | //! | `open(record).using(&plan)` | [`open`](crate::target::open) per field, run by `run_decryption` | //! | `.await`, [`all`] | `Pending::zip`, then one settle | //! @@ -321,7 +321,7 @@ pub use build::{FieldKind, FieldPlan, FieldsBuilder, IntoLabel, Opens, Plan, Pla pub use chain::{ all, All, Batch, EncryptBuilder, EncryptFields, EncryptIndexed, EncryptUsing, EncryptWithContext, KeysetChoice, OpenBuilder, OpenUsing, Operation, PlanKms, QueryBuilder, - QueryIndex, QueryUsing, + QueryIndex, QueryOp, QueryUsing, }; pub use error::PlanError; pub use field_ref::{pick, FieldRef}; diff --git a/packages/stack-encrypt/src/plan/value.rs b/packages/stack-encrypt/src/plan/value.rs index c4dfe4dad..76ae45f23 100644 --- a/packages/stack-encrypt/src/plan/value.rs +++ b/packages/stack-encrypt/src/plan/value.rs @@ -34,7 +34,7 @@ impl Plan<(), ()> { /// # async fn example() -> Result<(), stack_encrypt::Error> { /// use stack_encrypt::kms::FakeDataKeySource; /// use stack_encrypt::sem::EqualityTerm; - /// use stack_encrypt::{Equality, Ore, Plan, StackCipher, StackCipherText}; + /// use stack_encrypt::{op, Equality, Ore, Plan, StackCipher, StackCipherText}; /// /// let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; /// @@ -55,7 +55,7 @@ impl Plan<(), ()> { /// .query("bob@example.com") /// .context("users/email") /// .using(&email_plan) - /// .equality() + /// .op(op::Eq) /// .await?; /// assert_eq!(query_value, eq); /// let back = cipher.open((c, eq)).context("users/email").using(&email_plan).await?; diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index e0d383d9e..64f8b103f 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -633,6 +633,68 @@ impl PartialEq for MatchTerms { impl Eq for MatchTerms {} +/// The query term of a match index: a Bloom filter over the *needle's* +/// tokens, where the stored [`MatchTerms`] is a filter over the whole +/// value's. A row matches when its stored filter holds every position of +/// the query's ([`MatchTerms::contains`]). +/// +/// The two are different types because they are different things, even +/// where their bytes share an encoding: a query is never stored, and a +/// stored filter is never sent as a query. ADR-0009 records why. A +/// `MatchQuery` comes from a query chain ([`op::Matches`](crate::op::Matches)), +/// never from a target. +pub struct MatchQuery { + terms: MatchTerms, +} + +impl MatchQuery { + /// The query term of the filter `terms`, derived over a needle. + pub(crate) fn from_terms(terms: MatchTerms) -> Self { + Self { terms } + } + + /// The set positions of the needle's filter, sorted ascending, no + /// duplicates: what the server tests a stored filter for. + pub fn positions(&self) -> &[u16] { + self.terms.positions() + } + + /// The transport byte encoding, as [`MatchTerms::to_bytes`]. + pub fn to_bytes(&self) -> Vec { + self.terms.to_bytes() + } + + /// The needle's filter, for a caller that tests stored filters itself + /// ([`MatchTerms::contains`]). + pub fn filter(&self) -> &MatchTerms { + &self.terms + } +} + +impl fmt::Debug for MatchQuery { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("MatchQuery") + .field("positions", &self.terms.positions) + .finish() + } +} + +impl Clone for MatchQuery { + fn clone(&self) -> Self { + Self { + terms: self.terms.clone(), + } + } +} + +impl PartialEq for MatchQuery { + fn eq(&self, other: &Self) -> bool { + self.terms == other.terms + } +} + +impl Eq for MatchQuery {} + /// A [`PrfVisitor`] that folds a sequence of per-token PRF blocks into /// Bloom-filter bit positions: `k` little-endian 2-byte slices of each block, /// masked to the filter size. diff --git a/packages/stack-encrypt/tests/derive_plan.rs b/packages/stack-encrypt/tests/derive_plan.rs index 408543f82..018f35cda 100644 --- a/packages/stack-encrypt/tests/derive_plan.rs +++ b/packages/stack-encrypt/tests/derive_plan.rs @@ -11,6 +11,7 @@ use std::sync::atomic::Ordering as AtomicOrdering; use common::{counting_cipher, recording_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::op; use stack_encrypt::plan::{pick, FieldValues}; use stack_encrypt::sem::{EqualityTerm, MatchTerms, OpeTerm, OreTerm}; use stack_encrypt::target::{ @@ -131,7 +132,7 @@ async fn a_derived_record_as_a_typed_field_answers_its_queries() { let query_value = cipher .query("bob@example.com") .using(&users_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, email.hm); diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs new file mode 100644 index 000000000..e58130d02 --- /dev/null +++ b/packages/stack-encrypt/tests/op.rs @@ -0,0 +1,389 @@ +//! Operator-typed queries (`stack_encrypt::op`, ADR-0009): a query names an +//! operator, the plan finds the index that answers it, and the term that +//! comes back is the query term for that operator, which for a match index +//! is not the stored term. +mod common; + +use common::stack_cipher; +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::op::{self, MatchCase}; +use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; +use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms}; +use stack_encrypt::target::{Encrypted, IndexSpec}; +use stack_encrypt::{Equality, Error, Match, Plan}; + +/// A match configuration that keeps case, so its index answers only a +/// case-sensitive `Matches`. +struct Exact; +impl MatchConfig for Exact { + fn options() -> MatchOptions { + MatchOptions { + downcase: false, + ..Default::default() + } + } +} +impl MatchCase for Exact { + type Case = op::Sensitive; +} + +#[derive(Clone)] +struct Person { + email: String, + name: String, +} + +impl Fields for Person { + fn field_names(&self) -> Vec<&str> { + vec!["email", "name"] + } + fn schema() -> Option> { + Some(vec![ + FieldSchema::of::("email"), + FieldSchema::of::("name"), + ]) + } +} +impl Field for Person { + fn field(&self, name: &str) -> Option<&String> { + match name { + "email" => Some(&self.email), + "name" => Some(&self.name), + _ => None, + } + } +} + +fn person() -> Person { + Person { + email: "bob@example.com".into(), + name: "Alice Nguyen".into(), + } +} + +fn people_plan() -> Plan { + Plan::context("people") + .fields() + .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("name", Match::::new()) + .build() + .expect("a valid plan") +} + +fn plan_error(result: Result) -> PlanError { + match result { + Err(Error::Plan(error)) => error, + Err(other) => panic!("expected a plan error, got {other:?}"), + Ok(_) => panic!("expected a plan error, got a value"), + } +} + +#[tokio::test] +async fn an_eq_query_is_the_stored_equality_term() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap(); + let email: Encrypted<(EqualityTerm, MatchTerms)> = row.take("email").unwrap(); + + let query: EqualityTerm = cipher + .query("bob@example.com") + .using(&plan.field("email").unwrap()) + .op(op::Eq) + .await + .unwrap(); + assert_eq!(query, email.terms.0, "HMAC equality sends what it stores"); +} + +#[tokio::test] +async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_holds() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap(); + let email: Encrypted<(EqualityTerm, MatchTerms)> = row.take("email").unwrap(); + let stored = email.terms.1; + let email_plan = plan.field("email").unwrap(); + + let query: MatchQuery = cipher + .query("BOB") + .using(&email_plan) + .op(op::Matches::new()) + .await + .unwrap(); + assert!( + stored.contains(query.filter()), + "a case-folding index matches BOB" + ); + assert!( + query.positions().len() < stored.positions().len(), + "the query is the needle's filter, not the value's" + ); + + // The operator derives the term the index does, under the same label. + let by_index: MatchTerms = cipher + .query("BOB") + .using(&email_plan) + .index(Match::default()) + .await + .unwrap(); + assert_eq!(query.to_bytes(), by_index.to_bytes()); + assert_eq!(query.positions(), by_index.positions()); + assert_eq!(query.filter(), &by_index); + assert_eq!(query.clone(), query); + + let elsewhere: MatchQuery = cipher + .query("zzz") + .using(&email_plan) + .op(op::Matches::new()) + .await + .unwrap(); + assert!(!stored.contains(elsewhere.filter())); + assert_ne!(elsewhere, query); +} + +#[tokio::test] +async fn a_match_index_with_its_own_configuration_is_named_with_op_via() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap(); + let name: Encrypted> = row.take("name").unwrap(); + let name_plan = plan.field("name").unwrap(); + + let exact = |needle: &'static str| { + cipher + .query(needle) + .using(&name_plan) + .op_via(op::Matches::::new(), Match::::new()) + }; + let hit: MatchQuery = exact("Nguyen").await.unwrap(); + assert!(name.terms.contains(hit.filter())); + let miss: MatchQuery = exact("nguyen").await.unwrap(); + assert!( + !name.terms.contains(miss.filter()), + "a case-sensitive index does not fold the needle" + ); + + // The operator's default index is the default configuration, which + // this field does not declare. + assert_eq!( + plan_error( + cipher + .query("Nguyen") + .using(&name_plan) + .op(op::Matches::new()) + .await + ), + PlanError::IndexOptions { + field: "name".into(), + declared: IndexSpec::Match(Exact::options()), + asked: IndexSpec::Match(MatchOptions::default()), + } + ); +} + +#[tokio::test] +async fn an_operator_the_field_has_no_index_for_is_refused_when_awaited() { + let cipher = stack_cipher().await; + let plan = people_plan(); + assert_eq!( + plan_error( + cipher + .query("Alice") + .using(&plan.field("name").unwrap()) + .op(op::Eq) + .await + ), + PlanError::IndexNotDeclared { + field: "name".into(), + index: "eq" + } + ); +} + +#[tokio::test] +async fn a_one_value_plan_answers_from_its_indexes_by_type() { + let cipher = stack_cipher().await; + let plan = Plan::context("users/email") + .with::((Equality, Match::default())) + .build() + .unwrap(); + let stored: Encrypted<(EqualityTerm, MatchTerms)> = cipher + .encrypt(&"bob@example.com".to_owned()) + .using(&plan) + .await + .unwrap(); + + let eq = cipher + .query("bob@example.com") + .using(&plan) + .op(op::Eq) + .await + .unwrap(); + assert_eq!(eq, stored.terms.0); + let matches = cipher + .query("example") + .using(&plan) + .op(op::Matches::new()) + .await + .unwrap(); + assert!(stored.terms.1.contains(matches.filter())); + + let exact_plan = Plan::context("users/name") + .with::(Match::::new()) + .build() + .unwrap(); + let name: Encrypted> = cipher + .encrypt(&"Alice".to_owned()) + .using(&exact_plan) + .await + .unwrap(); + let query = cipher + .query("Ali") + .using(&exact_plan) + .op_via(op::Matches::::new(), Match::::new()) + .await + .unwrap(); + assert!(name.terms.contains(query.filter())); +} + +#[tokio::test] +async fn a_target_answers_an_operator_through_the_index_it_names() { + let cipher = stack_cipher().await; + let eq_plan = Plan::value::() + .context("users/email") + .encrypt_into::() + .build() + .unwrap(); + let stored: EqualityTerm = cipher + .encrypt(&"bob@example.com".to_owned()) + .using(&eq_plan) + .await + .unwrap(); + let query = cipher + .query("bob@example.com") + .using(&eq_plan) + .op(op::Eq) + .await + .unwrap(); + assert_eq!(query, stored); + + let match_plan = Plan::value::() + .context("users/email") + .encrypt_into::() + .build() + .unwrap(); + let stored: MatchTerms = cipher + .encrypt(&"bob@example.com".to_owned()) + .using(&match_plan) + .await + .unwrap(); + let query = cipher + .query("bob") + .using(&match_plan) + .op(op::Matches::new()) + .await + .unwrap(); + assert!(stored.contains(query.filter())); +} + +#[tokio::test] +async fn a_target_with_several_terms_names_the_index_with_op_via() { + let cipher = stack_cipher().await; + let plan = Plan::value::() + .context("users/email") + .encrypt_into::>() + .build() + .unwrap(); + let stored = cipher + .encrypt(&"bob@example.com".to_owned()) + .using(&plan) + .await + .unwrap(); + let eq = cipher + .query("bob@example.com") + .using(&plan) + .op_via(op::Eq, Equality) + .await + .unwrap(); + assert_eq!(eq, stored.terms.0); + let matches = cipher + .query("bob") + .using(&plan) + .op_via(op::Matches::new(), Match::default()) + .await + .unwrap(); + assert!(stored.terms.1.contains(matches.filter())); + + // An index the target does not declare is refused when awaited. + let one_term = Plan::value::() + .context("users/email") + .encrypt_into::>() + .build() + .unwrap(); + assert_eq!( + plan_error( + cipher + .query("bob") + .using(&one_term) + .op_via(op::Matches::new(), Match::default()) + .await + ), + PlanError::IndexNotDeclared { + field: "the value".into(), + index: "match" + } + ); + let eq = cipher.query("bob").using(&one_term).op(op::Eq).await; + assert!(eq.is_ok(), "one term answers what the term answers"); +} + +#[tokio::test] +async fn operator_queries_settle_together_and_extend_like_the_write() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let mut row = cipher + .encrypt(&person()) + .using(&plan) + .extend(7u64) + .await + .unwrap(); + let email: Encrypted<(EqualityTerm, MatchTerms)> = row.take("email").unwrap(); + let email_plan = plan.field("email").unwrap(); + + let (eq, matches) = stack_encrypt::all(( + cipher + .query("bob@example.com") + .using(&email_plan) + .op(op::Eq) + .extend(7u64), + cipher + .query("bob") + .using(&email_plan) + .op(op::Matches::new()) + .extend(7u64), + )) + .await + .unwrap(); + assert_eq!(eq, email.terms.0); + assert!(email.terms.1.contains(matches.filter())); + + let unextended = cipher + .query("bob@example.com") + .using(&email_plan) + .op(op::Eq) + .await + .unwrap(); + assert_ne!( + unextended, email.terms.0, + "the query is derived under the extended label" + ); +} + +#[tokio::test] +async fn a_query_chain_debugs_without_its_value() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let email_plan = plan.field("email").unwrap(); + let secret = "bob@example.com"; + let shown = format!("{:?}", cipher.query(secret).using(&email_plan).op(op::Eq)); + assert_eq!(shown, "QueryOp { .. }"); +} diff --git a/packages/stack-encrypt/tests/plan.rs b/packages/stack-encrypt/tests/plan.rs index 48008f65f..76133e69d 100644 --- a/packages/stack-encrypt/tests/plan.rs +++ b/packages/stack-encrypt/tests/plan.rs @@ -8,6 +8,7 @@ use std::sync::atomic::Ordering; use common::{counting_cipher, loads_counting_cipher, recording_cipher, stack_cipher}; use stack_encrypt::kms::{FakeDataKeySource, IdentifiedBy}; +use stack_encrypt::op; use stack_encrypt::plan::{ Field, FieldKind, FieldSchema, FieldValues, Fields, Operation, PlanError, }; @@ -291,7 +292,7 @@ async fn a_query_derives_under_the_label_the_write_used() { let eq: EqualityTerm = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!( @@ -317,7 +318,7 @@ async fn a_query_derives_under_the_label_the_write_used() { let other = cipher .query("eve@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_ne!(other, email.terms.0); @@ -341,7 +342,7 @@ async fn a_query_extended_like_the_write_matches_it() { Some(parts) => chain.extend(parts), None => chain, }; - chain.equality() + chain.op(op::Eq) }; assert_eq!(query(Some(7)).await.unwrap(), email.terms.0); assert_ne!(query(None).await.unwrap(), email.terms.0); @@ -363,14 +364,14 @@ async fn a_query_against_an_undeclared_index_or_type_is_refused() { } ); assert_eq!( - plan_error(cipher.query("x").using(¬es_plan).equality().await), + plan_error(cipher.query("x").using(¬es_plan).op(op::Eq).await), PlanError::IndexNotDeclared { field: "notes".into(), index: "eq" } ); assert_eq!( - plan_error(cipher.query(&34u64).using(&age_plan).equality().await), + plan_error(cipher.query(&34u64).using(&age_plan).op(op::Eq).await), PlanError::FieldType { field: "age".into(), expected: "u32" @@ -405,7 +406,7 @@ async fn a_one_value_plan_writes_queries_by_type_and_reads_back() { let eq = cipher .query(&34u32) .using(&age_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(eq, out.terms.0); @@ -510,7 +511,7 @@ async fn a_one_value_plan_extended_on_write_needs_the_extension_to_query_and_ope assert_eq!(sent.lock().unwrap().generated(), ["(users/age)/7u64"]); let query = |parts: Option| { - let chain = cipher.query(&34u32).using(&age_plan).equality(); + let chain = cipher.query(&34u32).using(&age_plan).op(op::Eq); match parts { Some(parts) => chain.extend(parts), None => chain, @@ -563,7 +564,7 @@ async fn an_index_only_field_matches_its_query_and_mints_no_data_key() { let query = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query, eq, "the query derives under the write's label"); @@ -967,7 +968,7 @@ async fn a_passthrough_name_need_not_be_a_label_segment() { cipher .query(&1u64) .using(&passthrough_plan) - .equality() + .op(op::Eq) .await ), PlanError::IndexNotDeclared { @@ -1262,7 +1263,7 @@ async fn all_settles_several_chains_in_one_request() { cipher .query("bob@example.com") .using(&email_plan) - .equality(), + .op(op::Eq), )) .await .unwrap(); @@ -1749,7 +1750,7 @@ async fn a_chain_shows_no_value() { format!("{:?}", cipher.encrypt(&user()).using(&users_plan)), format!("{:?}", cipher.query(&secret)), format!("{:?}", cipher.query(&secret).using(&email_plan)), - format!("{:?}", cipher.query(&secret).using(&email_plan).equality()), + format!("{:?}", cipher.query(&secret).using(&email_plan).op(op::Eq)), format!("{:?}", cipher.open(secret.clone())), format!("{:?}", cipher.open(FieldValues::new()).using(&users_plan)), format!( @@ -1874,14 +1875,14 @@ async fn every_chain_honours_its_named_keyset() { let under_tenant = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .keyset("tenant") .await .unwrap(); let under_default = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(under_tenant, email.terms.0); diff --git a/packages/stack-encrypt/tests/plan_grammar.rs b/packages/stack-encrypt/tests/plan_grammar.rs index e88397b63..100c20f72 100644 --- a/packages/stack-encrypt/tests/plan_grammar.rs +++ b/packages/stack-encrypt/tests/plan_grammar.rs @@ -12,6 +12,7 @@ use std::sync::atomic::Ordering; use common::{counting_cipher, loads_counting_cipher, recording_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::op; use stack_encrypt::plan::{pick, Field, FieldKind, FieldValues, Fields, Opens, PlanError, Runs}; use stack_encrypt::sem::{ EqualityTerm, MatchConfig, MatchOptions, MatchTerms, OpeTerm, OreTerm, Tokenizer, @@ -144,7 +145,7 @@ async fn a_plan_built_without_a_context_takes_the_calls() { .query("bob@example.com") .context("tenants/acme") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!( @@ -155,7 +156,7 @@ async fn a_plan_built_without_a_context_takes_the_calls() { .query("bob@example.com") .context("tenants/globex") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_ne!(elsewhere, stored.terms); @@ -187,7 +188,7 @@ async fn a_plan_run_with_no_context_is_refused_before_any_key_request() { cipher .query("bob@example.com") .using(&users_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await ), PlanError::NoContext @@ -210,7 +211,7 @@ async fn a_plan_run_with_no_context_is_refused_before_any_key_request() { PlanError::NoContext ); assert_eq!( - plan_error(cipher.query(&34u32).using(&age_plan).equality().await), + plan_error(cipher.query(&34u32).using(&age_plan).op(op::Eq).await), PlanError::NoContext ); let age = cipher @@ -292,7 +293,7 @@ async fn a_context_given_twice_is_refused_at_the_call_before_any_key_request() { .query("bob@example.com") .context("users") .using(&users_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await ), two("the plan", "the call") @@ -342,7 +343,7 @@ async fn a_context_given_twice_is_refused_at_the_call_before_any_key_request() { .query(&34u32) .context("users/age") .using(&age_plan) - .equality() + .op(op::Eq) .await ), two("the plan", "the call") @@ -368,7 +369,7 @@ async fn a_call_context_that_is_not_a_label_is_refused() { .query("bob@example.com") .context("a//b") .using(&users_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await ))); assert!(bad(plan_error( @@ -839,7 +840,7 @@ async fn a_context_field_is_the_context_of_every_other_field() { .query("bob@example.com") .context("tenants/acme") .using(&records_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, stored.terms); @@ -1130,7 +1131,7 @@ async fn a_typed_field_answers_the_queries_its_target_declares_and_no_other() { let query_value = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, stored.terms.0); @@ -1159,7 +1160,7 @@ async fn a_typed_field_answers_the_queries_its_target_declares_and_no_other() { cipher .query(&34u32) .using(&users_plan.field("age").unwrap()) - .equality() + .op(op::Eq) .await ), PlanError::IndexNotDeclared { @@ -1185,7 +1186,7 @@ async fn a_typed_field_of_terms_alone_does_not_come_back() { let query_value = cipher .query("bob@example.com") .using(&users_plan.field("email").unwrap()) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, term); @@ -1334,7 +1335,7 @@ async fn a_one_value_typed_plan_answers_its_targets_queries_and_no_other() { let query_value = cipher .query("bob@example.com") .using(&email_plan) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, hm); @@ -1632,7 +1633,7 @@ async fn a_pinned_identity_rescues_a_field_name_that_is_not_plain() { let query_value = cipher .query(&21u32) .using(&readings_plan.field("0").unwrap()) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, value.terms); @@ -1686,7 +1687,7 @@ async fn a_pinned_identity_keys_a_call_context_plan_for_query_and_open() { .query(&21u32) .context("reading") .using(&readings_plan.field("0").unwrap()) - .equality() + .op(op::Eq) .await .unwrap(); assert_eq!(query_value, stored.terms); diff --git a/packages/stack-encrypt/tests/ui/op_case_mismatch.rs b/packages/stack-encrypt/tests/ui/op_case_mismatch.rs new file mode 100644 index 000000000..ab1c0c2e4 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_case_mismatch.rs @@ -0,0 +1,18 @@ +//! The default match index folds case, so it answers only a +//! case-insensitive `Matches`: asking it for a case-sensitive one does not +//! compile, rather than quietly folding the needle. +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::{op, Match, Plan, StackCipher}; + +async fn query(cipher: &StackCipher) { + let name_plan = Plan::context("users/name") + .with::(Match::default()) + .build() + .unwrap(); + let _ = cipher + .query("Ali") + .using(&name_plan) + .op_via(op::Matches::::new(), Match::default()); +} + +fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr b/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr new file mode 100644 index 000000000..af453429e --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr @@ -0,0 +1,26 @@ +error[E0308]: mismatched types + --> tests/ui/op_case_mismatch.rs:15:17 + | +15 | .op_via(op::Matches::::new(), Match::default()); + | ------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Matches`, found `Matches` + | | + | arguments to this method are incorrect + | + = note: expected struct `stack_encrypt::op::Matches` + found struct `stack_encrypt::op::Matches` +help: the return type of this call is `stack_encrypt::op::Matches` due to the type of the argument passed + --> tests/ui/op_case_mismatch.rs:12:13 + | +12 | let _ = cipher + | _____________^ +13 | | .query("Ali") +14 | | .using(&name_plan) +15 | | .op_via(op::Matches::::new(), Match::default()); + | |_________________-----------------------------------__________________^ + | | + | this argument influences the return type of `op_via` +note: method defined here + --> src/plan/chain.rs + | + | pub fn op_via(self, _op: Op, _index: I) -> QueryOp<'a, F, Op, I, K> + | ^^^^^^ diff --git a/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs b/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs new file mode 100644 index 000000000..b92c7917c --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs @@ -0,0 +1,11 @@ +//! `Matches` is text search: a field of integers has no index that answers +//! it, so the query does not compile. +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::{op, Equality, Plan, StackCipher}; + +async fn query(cipher: &StackCipher) { + let age_plan = Plan::context("users/age").with::(Equality).build().unwrap(); + let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); +} + +fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr b/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr new file mode 100644 index 000000000..966ebf8f9 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr @@ -0,0 +1,21 @@ +error[E0271]: type mismatch resolving `>::Via == Equality` + --> tests/ui/op_matches_on_integer.rs:8:51 + | +8 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); + | ^^ expected `Equality`, found `Match` + | + = note: expected struct `stack_encrypt::Equality` + found struct `stack_encrypt::Match` + +error[E0277]: the trait bound `u32: AsRef` is not satisfied + --> tests/ui/op_matches_on_integer.rs:8:51 + | +8 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); + | ^^ the trait `AsRef` is not implemented for `u32` + | +help: the trait `DefaultIndex` is implemented for `stack_encrypt::op::Matches` + --> src/op.rs + | + | impl> DefaultIndex for Matches<::Case> { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + = note: required for `stack_encrypt::op::Matches` to implement `DefaultIndex` diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs new file mode 100644 index 000000000..0b0bdc6e4 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs @@ -0,0 +1,16 @@ +//! A target answers only the operators its type implements `Answers` for: +//! an equality term is not searchable text. +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::{op, Plan, StackCipher}; + +async fn query(cipher: &StackCipher) { + let email_plan = Plan::value::() + .context("users/email") + .encrypt_into::() + .build() + .unwrap(); + let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); +} + +fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr new file mode 100644 index 000000000..3a2dacda9 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr @@ -0,0 +1,22 @@ +error[E0308]: mismatched types + --> tests/ui/op_target_does_not_answer.rs:13:55 + | +13 | let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); + | -- ^^^^^^^^^^^^^^^^^^ expected `Eq`, found `Matches<_>` + | | + | arguments to this method are incorrect + | + = note: expected struct `stack_encrypt::op::Eq` + found struct `stack_encrypt::op::Matches<_>` +help: the return type of this call is `stack_encrypt::op::Matches<_>` due to the type of the argument passed + --> tests/ui/op_target_does_not_answer.rs:13:13 + | +13 | let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^------------------^ + | | + | this argument influences the return type of `op` +note: method defined here + --> src/plan/chain.rs + | + | pub fn op(self, op: Op) -> QueryOp<'a, F, Op, T::Via, K> + | ^^ diff --git a/packages/stack-encrypt/tests/ui/pass/plan_chain.rs b/packages/stack-encrypt/tests/ui/pass/plan_chain.rs index 16671c333..4a51aa731 100644 --- a/packages/stack-encrypt/tests/ui/pass/plan_chain.rs +++ b/packages/stack-encrypt/tests/ui/pass/plan_chain.rs @@ -1,6 +1,7 @@ //! The chain shapes that must compile: a value with a context, indexed, //! field by field, a saved plan over a value and a collection, a query and //! an opening, and several chains batched. +use stack_encrypt::op; use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::plan::{Field, Fields}; use stack_encrypt::{Equality, Error, Match, Ore, Plan, StackCipher}; @@ -46,14 +47,18 @@ async fn chains(cipher: &StackCipher, person: &Person) -> Res let row = cipher.encrypt(person).using(&people_plan).await?; let _ = cipher.open(row).using(&people_plan).await?; let email_plan = people_plan.field("email")?; - let _ = cipher.query("bob@example.com").using(&email_plan).equality().await?; + let _ = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; + let name_plan = Plan::context("people/name") + .with::(stack_encrypt::Match::default()) + .build()?; + let _ = cipher.query("ali").using(&name_plan).op(op::Matches::new()).await?; let age_plan = Plan::context("people/age").with((Equality, Ore)).build()?; let ages = vec![1u32, 2]; let _ = cipher.encrypt(&ages).using(&age_plan).await?; let _ = cipher.query(&3u32).using(&age_plan).index::().await?; let _ = stack_encrypt::all(( cipher.encrypt(person).using(&people_plan), - cipher.query(&3u32).using(&age_plan).equality(), + cipher.query(&3u32).using(&age_plan).op(op::Eq), )) .await?; Ok(()) diff --git a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs index 9a53fb472..ce1440102 100644 --- a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs +++ b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs @@ -1,11 +1,12 @@ //! A one-value plan selects a query's index by type, so asking it for an //! index it does not hold does not compile. +use stack_encrypt::op; use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::{Ore, Plan, StackCipher}; async fn query(cipher: &StackCipher) { let age_plan = Plan::context("users/age").with::(Ore).build().unwrap(); - let _ = cipher.query(&34u32).using(&age_plan).equality(); + let _ = cipher.query(&34u32).using(&age_plan).op(op::Eq); } fn main() {} diff --git a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr index 8cb4e2b22..3a5c76063 100644 --- a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr +++ b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr @@ -1,24 +1,5 @@ -error[E0277]: the trait bound `stack_encrypt::Ore: Select` is not satisfied - --> tests/ui/plan_query_undeclared_index.rs:8:51 +error[E0271]: type mismatch resolving `>::Via == Ore` + --> tests/ui/plan_query_undeclared_index.rs:9:51 | -8 | let _ = cipher.query(&34u32).using(&age_plan).equality(); - | ^^^^^^^^ the trait `Select` is not implemented for `stack_encrypt::Ore` - | - = help: the following other types implement trait `Select`: - `(A, B)` implements `Select>` - `(A, B)` implements `Select>` - `(A, B, C)` implements `Select>` - `(A, B, C)` implements `Select>` - `(A, B, C)` implements `Select>` - `(A, B, C, D)` implements `Select>` - `(A, B, C, D)` implements `Select>` - `(A, B, C, D)` implements `Select>` - and $N others -note: required by a bound in `QueryUsing::<'a, F, ValuePlan>, K>::equality` - --> src/plan/chain.rs - | - | pub fn equality(self) -> QueryIndex<'a, F, Equality, K> - | -------- required by a bound in this associated function -... - | X: Select, - | ^^^^^^^^^^^^^^^^^^^^ required by this bound in `QueryUsing::<'a, F, ValuePlan>, K>::equality` +9 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Eq); + | ^^ expected `Ore`, found `Equality` From 31328937e4f76a8848bcafdc9a5cfe94c01fdfb4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 22:58:14 +1100 Subject: [PATCH 09/38] fix(stack-encrypt): address the review of the operator-typed query - A match configuration whose `MatchCase` disagrees with its options is refused when the query is awaited, with the new `PlanError::MatchCase`, instead of answering in a case its index does not fold. `Supports` gains `check`. - `Supports::query` takes an `op::QueryToken` only this crate can make, so no caller can turn a stored term into a query term by hand. - `MatchTerms::matches(&MatchQuery)` tests a stored filter against a query; `MatchQuery::filter` is crate-only. - Docs say what is a compile error and what is refused when awaited: on a field of a fields plan, which is data, an undeclared index, other match options and a mislabelled case are runtime refusals. A typed field of a fields plan answers through the operator's default index, not its type's `Answers`; the ADR, `DefaultIndex` and the CHANGELOG say so. - `QueryTerm`'s doc no longer calls every term that refines nothing exact before `Probabilistic` exists. A broken ADR sentence is fixed. - `Operator::NAME` is gone (nothing read it); `Matches` is `Hash`. - Tests: the case mismatch, a refused query loading no named keyset (kills the `QueryOp::check` mutant), a query under a named keyset, and `MatchQuery`'s `Debug`. The three `via` mutants and `Matches::clone`'s are equivalent (unit structs) and are excluded in .cargo/mutants.toml. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- .cargo/mutants.toml | 4 + packages/stack-encrypt/CHANGELOG.md | 16 ++- packages/stack-encrypt/CONTEXT.md | 6 +- packages/stack-encrypt/src/codes.rs | 4 + packages/stack-encrypt/src/dynamic/record.rs | 17 ++- packages/stack-encrypt/src/op.rs | 90 ++++++++---- packages/stack-encrypt/src/plan/chain.rs | 33 +++-- packages/stack-encrypt/src/plan/error.rs | 20 +++ packages/stack-encrypt/src/sem/mod.rs | 16 ++- packages/stack-encrypt/tests/op.rs | 141 +++++++++++++++++-- 10 files changed, 280 insertions(+), 67 deletions(-) diff --git a/.cargo/mutants.toml b/.cargo/mutants.toml index 1fae9ba39..a1d1e4eba 100644 --- a/.cargo/mutants.toml +++ b/.cargo/mutants.toml @@ -72,6 +72,10 @@ exclude_re = [ 'stack-encrypt/src/cipher\.rs:\d+:\d+: replace client_key_provider -> EnvKeyProvider with Default::default\(\)$', 'stack-encrypt/src/target/pending\.rs:443:9: replace >::into_future -> Self::IntoFuture with Default::default\(\)$', # stack-encrypt — equivalent. + # An operator and an operator's index are unit structs, whose `Default` + # is the value `clone` and `via` already return. + 'stack-encrypt/src/op(/mod)?\.rs:\d+:\d+: replace >::clone -> Self with Default::default\(\)$', + 'stack-encrypt/src/op(/mod)?\.rs:\d+:\d+: replace for EqualityTerm|DefaultIndex for Eq|DefaultIndex for Matches<::Case>)>::via -> (Equality|Match) with Default::default\(\)$', 'stack-encrypt/src/cipher\.rs:\d+:\d+: replace StackCipher::builder -> StackCipherBuilder with Default::default\(\)$', 'stack-encrypt/src/sem/mod\.rs:\d+:\d+: replace ::options -> MatchOptions with Default::default\(\)$', # Both unit-context conversions explicitly return Self::default(). diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 315d7bd70..81719e35d 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -15,12 +15,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 operators it answers (`op::Supports`), a target type names the index that answers each one for it (`op::Answers`), and a plan that declares indexes uses the operator's default index (`op::DefaultIndex`). `.op_via(op, index)` - names another index, such as a match index with its own configuration. An - operator the plan cannot answer is a compile error; a case-sensitive - `Matches` against a case-folding index is one too. + names another index, such as a match index with its own configuration. What + the plan's type shows is checked at compile time: a match query on an + integer, an operator a target does not answer, and on a one-value plan an + index it does not hold or a case-sensitive `Matches` against a + case-folding index. On a field of a fields plan those are refused when the + query is awaited, before any key is requested, as is a match configuration + whose `op::MatchCase` disagrees with its options (`PlanError::MatchCase`). - **A match query has its own type, `sem::MatchQuery`.** It is the filter over - the needle that the server tests a stored `MatchTerms` for, and it comes - only from a query. + the needle; test a stored filter against it with + `MatchTerms::matches(&query)`. It comes only from a query chain: + `op::Supports::query` takes an `op::QueryToken` that only this crate can + make. - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as diff --git a/packages/stack-encrypt/CONTEXT.md b/packages/stack-encrypt/CONTEXT.md index 9a8b342c8..3cfab23a8 100644 --- a/packages/stack-encrypt/CONTEXT.md +++ b/packages/stack-encrypt/CONTEXT.md @@ -234,14 +234,16 @@ package's method name, not the concept) What a query asks of the server — equality, text match — as a unit type (`op::Eq`, `op::Matches`). An index declares the operators it answers and the query term it sends for each; a target type names the index that answers each -operator for it. An operator nothing answers is a compile error (ADR-0009). +operator for it. An operator nothing answers is refused before any key is +requested: at compile time where the plan's type shows it, when the query is +awaited where the plan is data (ADR-0009). _Avoid_: query type, index (that is what answers the operator) **Query term**: The value a query sends to the server. It is not the stored term: equality sends the same HMAC it stores, but a match query is a filter over the needle, where the stored term is a filter over the whole value. A query term is never -stored, and a stored term is never sent as a query. +stored, and a query chain never sends a stored term as a query. _Avoid_: probe, stored term (that is what a write keeps) **Open**: diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index d2f752f25..b1f8dd5cb 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -149,6 +149,10 @@ mod tests { ..MatchOptions::default() }), }, + PlanError::MatchCase { .. } => PlanError::MatchCase { + declared: "Insensitive", + folds: false, + }, PlanError::TwoContextSources { .. } => PlanError::TwoContextSources { first: "the plan", second: "the call", diff --git a/packages/stack-encrypt/src/dynamic/record.rs b/packages/stack-encrypt/src/dynamic/record.rs index 0093b9f53..c7c89f9c2 100644 --- a/packages/stack-encrypt/src/dynamic/record.rs +++ b/packages/stack-encrypt/src/dynamic/record.rs @@ -556,9 +556,11 @@ fn refusal(error: crate::Error) -> (Option, Reason) { P::IndexNotDeclared { field, .. } | P::IndexOptions { field, .. } => { (Some(field), Reason::Refused) } - P::IdentityWithoutField | P::MixedCiphers | P::TwoContextSources { .. } | P::NoContext => { - (None, Reason::Refused) - } + P::IdentityWithoutField + | P::MixedCiphers + | P::MatchCase { .. } + | P::TwoContextSources { .. } + | P::NoContext => (None, Reason::Refused), } } @@ -6795,6 +6797,15 @@ mod tests { Refused, ), (P::NoContext.into(), None, Refused), + ( + P::MatchCase { + declared: "Sensitive", + folds: true, + } + .into(), + None, + Refused, + ), ]; for (error, field, reason) in cases { let shown = format!("{error:?}"); diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op.rs index 9b083ac1b..3a70dc46a 100644 --- a/packages/stack-encrypt/src/op.rs +++ b/packages/stack-encrypt/src/op.rs @@ -35,11 +35,15 @@ //! - [`DefaultIndex`] is on an operator, and names the index a plan that //! declares its indexes, rather than a target type, answers it with. //! -//! An operator a plan cannot answer is a compile error: a match query on an -//! integer, a case-sensitive match against a case-folding index, or an -//! operator a target type does not answer. An index the plan does not -//! declare is refused when the query is awaited, as with +//! What the types can see is checked at compile time: a match query on an +//! integer, an operator a target type does not answer, and, on a one-value +//! plan, an index the plan does not hold or a case-sensitive match against +//! a case-folding index. A field of a fields plan is data, so there an +//! index the field does not declare, or declares with other options, is +//! refused when the query is awaited, as with //! [`index`](crate::plan::QueryUsing::index), before any key is requested. +//! So is a match configuration whose [`MatchCase`] disagrees with its +//! options. //! //! The module is meant to be used by path (`op::Eq`): an imported `Eq` //! would shadow the prelude's `Eq` trait. @@ -51,23 +55,19 @@ use std::marker::PhantomData; use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms}; use crate::target::{Equality, Index, Match}; +use crate::PlanError; /// A query operator: a unit type naming what a query asks of the server. /// /// The trait is open, as [`Index`] is: an operator defined in another crate /// is answered by the indexes that implement [`Supports`] for it. -pub trait Operator: Copy + 'static { - /// The operator's name, as a binding or an error message spells it. - const NAME: &'static str; -} +pub trait Operator: Copy + 'static {} /// Equality: rows whose value equals the query's. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] pub struct Eq; -impl Operator for Eq { - const NAME: &'static str = "eq"; -} +impl Operator for Eq {} /// Text search: rows whose text contains the needle's tokens, under the /// match index's tokenizer. Not containment: a Bloom match is order- and @@ -105,15 +105,16 @@ impl PartialEq for Matches { } } impl std::cmp::Eq for Matches {} +impl std::hash::Hash for Matches { + fn hash(&self, _: &mut H) {} +} impl fmt::Debug for Matches { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "Matches<{}>", C::NAME) } } -impl Operator for Matches { - const NAME: &'static str = "matches"; -} +impl Operator for Matches {} mod sealed { pub trait Sealed {} @@ -151,9 +152,12 @@ impl Case for Sensitive { /// index answers are known at compile time. /// /// `Case::FOLDS` must equal the configuration's -/// [`MatchOptions::downcase`](crate::sem::MatchOptions::downcase). A -/// configuration that does not implement this can still store and query -/// match terms; it cannot be queried by [`Matches`]. +/// [`MatchOptions::downcase`](crate::sem::MatchOptions::downcase): the type +/// cannot see the options, so a query through a configuration whose two +/// disagree is refused when it is awaited, with +/// [`PlanError::MatchCase`]. A configuration +/// that does not implement this can still store match terms and query them +/// by index; it cannot be queried by [`Matches`]. pub trait MatchCase: MatchConfig { /// The case the configuration's index answers. type Case: Case; @@ -165,8 +169,10 @@ impl MatchCase for DefaultMatch { /// A value a query sends to the server to compare against stored terms. /// -/// A term that implements this and nothing more is *exact*: the server's -/// answer is the answer. +/// This trait says only that a value is a query term. How far the server's +/// answer can be trusted is said by the traits that refine it: a term that +/// can return false positives says how a row is checked (ADR-0009, +/// decision 4), and one that refines nothing is exact. pub trait QueryTerm {} impl QueryTerm for EqualityTerm {} @@ -187,14 +193,34 @@ impl QueryTerm for MatchQuery {} pub trait Supports: Index { /// What this index sends for `Op`. type Query: QueryTerm; + /// Why this index cannot answer `Op` as configured, if it cannot: a + /// match index whose [`MatchCase`] disagrees with its options. + fn check(&self) -> Result<(), PlanError>; /// The query term, from the term the index derived for the query's - /// value. - fn query(term: Self::Term) -> Self::Query; + /// value. Only a query chain calls this: `QueryToken` cannot be made + /// outside this crate, so a stored term cannot be passed off as a + /// query. + fn query(term: Self::Term, token: QueryToken) -> Self::Query; +} + +/// Proof that [`Supports::query`] is called by a query chain. Only this +/// crate can make one, so an index in another crate can implement +/// [`Supports`] but no caller can turn a term into a query term by hand. +#[derive(Debug)] +pub struct QueryToken(()); + +impl QueryToken { + pub(crate) fn new() -> Self { + Self(()) + } } impl Supports for Equality { type Query = EqualityTerm; - fn query(term: EqualityTerm) -> EqualityTerm { + fn check(&self) -> Result<(), PlanError> { + Ok(()) + } + fn query(term: EqualityTerm, _: QueryToken) -> EqualityTerm { term } } @@ -205,7 +231,18 @@ where O: MatchCase, { type Query = MatchQuery; - fn query(term: MatchTerms) -> MatchQuery { + fn check(&self) -> Result<(), PlanError> { + let folds = O::options().downcase; + if folds == O::Case::FOLDS { + Ok(()) + } else { + Err(PlanError::MatchCase { + declared: O::Case::NAME, + folds, + }) + } + } + fn query(term: MatchTerms, _: QueryToken) -> MatchQuery { MatchQuery::from_terms(term) } } @@ -272,7 +309,10 @@ impl> Answers for crate::target::Encry /// /// A plan that declares a different index for the operator, such as a /// match index with its own configuration, names it with -/// `.op_via(op, index)`. +/// `.op_via(op, index)`. So does a typed field of a fields plan +/// (`encrypt_into::(name)`): the field's type is not known where the +/// query is written, so its [`Answers`] impl is not consulted, and a field +/// whose type answers through another index needs `op_via`. #[diagnostic::on_unimplemented( message = "`{Self}` has no default index over `{S}`", label = "no default index answers this operator here", @@ -313,7 +353,7 @@ mod tests { #[test] fn an_operator_is_equal_to_itself() { - assert_eq!(Matches::::new(), Matches::default()); + assert_eq!(Matches::::new(), Matches::new()); assert_eq!(Eq, Eq); } diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 51c2cccca..6284f3b97 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -11,7 +11,7 @@ use super::build::{FieldPlan, FieldsBuilder, IntoLabel, Opens, PlanContext, Runs use super::field_ref::FieldRef; use super::value::{Indexed, Typed, ValuePlan, ValueShape}; use super::PlanError; -use crate::op::{Answers, DefaultIndex, Operator, Supports}; +use crate::op::{Answers, DefaultIndex, Operator, QueryToken, Supports}; use crate::target::{ ciphertext, Borrowed, CallerContext, DeclaredContext, DecryptField, Decryptable, EncryptFrom, Index, Indexes, Owned, Pending, PendingFuture, Select, @@ -643,10 +643,7 @@ impl<'a, F: 'static, K: 'static> QueryUsing<'a, F, FieldPlan, K> { Op: Operator, I: Supports, { - QueryOp { - inner: self.index(index), - op: PhantomData, - } + QueryOp::new(self.index(index)) } /// The term of `index`. The field must have declared it, and the query's @@ -690,10 +687,7 @@ impl<'a, F: 'static, X: Indexes, K: 'static> QueryUsing<'a, F, ValuePlan + Clone, X: Select, { - QueryOp { - inner: self.index::(), - op: PhantomData, - } + QueryOp::new(self.index::()) } /// The term of the plan's index of type `I`, selected by type @@ -741,10 +735,7 @@ where Op: Operator, I: Supports, { - QueryOp { - inner: self.index(index), - op: PhantomData, - } + QueryOp::new(self.index(index)) } /// The term of `index`. The target must declare it @@ -814,6 +805,20 @@ pub struct QueryOp<'a, F, Op, I, K> { op: PhantomData Op>, } +impl<'a, F, Op: Operator, I: Supports, K: 'static> QueryOp<'a, F, Op, I, K> { + /// The chain for `inner`'s index, refused when awaited if the index + /// cannot answer `Op` as it is configured. + fn new(mut inner: QueryIndex<'a, F, I, K>) -> Self { + if let Err(error) = inner.index.check() { + inner.label = Err(error); + } + Self { + inner, + op: PhantomData, + } + } +} + impl<'a, F, Op, I, K: 'static> QueryOp<'a, F, Op, I, K> { /// Run under this keyset, as [`QueryBuilder::keyset`]. pub fn keyset(mut self, keyset: impl Into>) -> Self { @@ -857,7 +862,7 @@ where { self.inner .prepare(keyset, scoped) - .map(>::query) + .map(|term| I::query(term, QueryToken::new())) } } diff --git a/packages/stack-encrypt/src/plan/error.rs b/packages/stack-encrypt/src/plan/error.rs index 43a3da260..85932f299 100644 --- a/packages/stack-encrypt/src/plan/error.rs +++ b/packages/stack-encrypt/src/plan/error.rs @@ -183,6 +183,23 @@ pub enum PlanError { /// The index as the query asked for it. asked: IndexSpec, }, + /// A match configuration's [`MatchCase`](crate::op::MatchCase) names one + /// case and its options fold the other, so a `Matches` query would be + /// asked in a case its index does not answer. + #[error( + "a match configuration declares the case {declared} but its options {}", + if *folds { "fold case" } else { "keep case" } + )] + #[diagnostic( + code(stack_encrypt::plan_match_case), + help("Make `MatchCase::Case` agree with `MatchConfig::options().downcase`: `Insensitive` when it is true, `Sensitive` when it is false.") + )] + MatchCase { + /// The case `MatchCase::Case` names: `Insensitive` or `Sensitive`. + declared: &'static str, + /// Whether the configuration's options fold case. + folds: bool, + }, /// The plan's context was given twice. A plan takes its context from /// exactly one place: the plan, the call that runs it, or a context /// field. @@ -263,6 +280,9 @@ impl crate::ErrorPayload for PlanError { ("declared", index_value(declared)), ("asked", index_value(asked)), ]), + Self::MatchCase { declared, folds } => { + payload([("declared", (*declared).into()), ("folds", (*folds).into())]) + } Self::TwoContextSources { first, second } => { payload([("first", (*first).into()), ("second", (*second).into())]) } diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 64f8b103f..473d21775 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -544,10 +544,19 @@ impl MatchTerms { /// false negative). /// /// An empty `query` returns `false`: containment of zero positions is - /// vacuously true, which would turn an empty probe into a match-every-row + /// vacuously true, which would turn an empty query into a match-every-row /// query. Term generation already refuses to build such a term /// ([`TermError::EmptyTermText`]); this guards any other /// (e.g. deserialized) source of an empty term. + pub fn matches(&self, query: &MatchQuery) -> bool { + self.contains(query.filter()) + } + + /// Whether this term's positions are a superset of `other`'s, for two + /// terms of the same kind. [`matches`](Self::matches) is the check for + /// a query. + /// + /// An empty `other` returns `false`, as an empty query does. pub fn contains(&self, query: &MatchTerms) -> bool { !query.positions.is_empty() && query @@ -664,9 +673,8 @@ impl MatchQuery { self.terms.to_bytes() } - /// The needle's filter, for a caller that tests stored filters itself - /// ([`MatchTerms::contains`]). - pub fn filter(&self) -> &MatchTerms { + /// The needle's filter. + pub(crate) fn filter(&self) -> &MatchTerms { &self.terms } } diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index e58130d02..03854af9a 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -4,7 +4,7 @@ //! is not the stored term. mod common; -use common::stack_cipher; +use common::{loads_counting_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::op::{self, MatchCase}; use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; @@ -109,10 +109,7 @@ async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_hold .op(op::Matches::new()) .await .unwrap(); - assert!( - stored.contains(query.filter()), - "a case-folding index matches BOB" - ); + assert!(stored.matches(&query), "a case-folding index matches BOB"); assert!( query.positions().len() < stored.positions().len(), "the query is the needle's filter, not the value's" @@ -127,7 +124,6 @@ async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_hold .unwrap(); assert_eq!(query.to_bytes(), by_index.to_bytes()); assert_eq!(query.positions(), by_index.positions()); - assert_eq!(query.filter(), &by_index); assert_eq!(query.clone(), query); let elsewhere: MatchQuery = cipher @@ -136,7 +132,7 @@ async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_hold .op(op::Matches::new()) .await .unwrap(); - assert!(!stored.contains(elsewhere.filter())); + assert!(!stored.matches(&elsewhere)); assert_ne!(elsewhere, query); } @@ -155,10 +151,10 @@ async fn a_match_index_with_its_own_configuration_is_named_with_op_via() { .op_via(op::Matches::::new(), Match::::new()) }; let hit: MatchQuery = exact("Nguyen").await.unwrap(); - assert!(name.terms.contains(hit.filter())); + assert!(name.terms.matches(&hit)); let miss: MatchQuery = exact("nguyen").await.unwrap(); assert!( - !name.terms.contains(miss.filter()), + !name.terms.matches(&miss), "a case-sensitive index does not fold the needle" ); @@ -225,7 +221,7 @@ async fn a_one_value_plan_answers_from_its_indexes_by_type() { .op(op::Matches::new()) .await .unwrap(); - assert!(stored.terms.1.contains(matches.filter())); + assert!(stored.terms.1.matches(&matches)); let exact_plan = Plan::context("users/name") .with::(Match::::new()) @@ -242,7 +238,7 @@ async fn a_one_value_plan_answers_from_its_indexes_by_type() { .op_via(op::Matches::::new(), Match::::new()) .await .unwrap(); - assert!(name.terms.contains(query.filter())); + assert!(name.terms.matches(&query)); } #[tokio::test] @@ -282,7 +278,7 @@ async fn a_target_answers_an_operator_through_the_index_it_names() { .op(op::Matches::new()) .await .unwrap(); - assert!(stored.contains(query.filter())); + assert!(stored.matches(&query)); } #[tokio::test] @@ -311,7 +307,7 @@ async fn a_target_with_several_terms_names_the_index_with_op_via() { .op_via(op::Matches::new(), Match::default()) .await .unwrap(); - assert!(stored.terms.1.contains(matches.filter())); + assert!(stored.terms.1.matches(&matches)); // An index the target does not declare is refused when awaited. let one_term = Plan::value::() @@ -364,7 +360,7 @@ async fn operator_queries_settle_together_and_extend_like_the_write() { .await .unwrap(); assert_eq!(eq, email.terms.0); - assert!(email.terms.1.contains(matches.filter())); + assert!(email.terms.1.matches(&matches)); let unextended = cipher .query("bob@example.com") @@ -387,3 +383,120 @@ async fn a_query_chain_debugs_without_its_value() { let shown = format!("{:?}", cipher.query(secret).using(&email_plan).op(op::Eq)); assert_eq!(shown, "QueryOp { .. }"); } + +/// A configuration whose `MatchCase` says it folds case while its options +/// keep it: the type cannot see the options, so the query is refused. +struct Mislabelled; +impl MatchConfig for Mislabelled { + fn options() -> MatchOptions { + MatchOptions { + downcase: false, + ..Default::default() + } + } +} +impl MatchCase for Mislabelled { + type Case = op::Insensitive; +} + +#[tokio::test] +async fn a_match_case_that_disagrees_with_its_options_is_refused() { + let cipher = stack_cipher().await; + let plan = Plan::context("users/name") + .with::(Match::::new()) + .build() + .unwrap(); + let refused = plan_error( + cipher + .query("Bob") + .using(&plan) + .op_via(op::Matches::new(), Match::::new()) + .await, + ); + assert_eq!( + refused, + PlanError::MatchCase { + declared: "Insensitive", + folds: false + } + ); + assert_eq!( + refused.to_string(), + "a match configuration declares the case Insensitive but its options keep case" + ); +} + +#[tokio::test] +async fn a_refused_operator_query_loads_no_keyset() { + let (cipher, loads) = loads_counting_cipher().await; + let plan: Plan = Plan::context("people") + .fields() + .encrypt::("email") + .encrypt_index::("name", Match::::new()) + .build() + .unwrap(); + let refused = cipher + .query("Alice") + .using(&plan.field("name").unwrap()) + .op(op::Eq) + .keyset("tenant") + .await; + assert_eq!( + plan_error(refused), + PlanError::IndexNotDeclared { + field: "name".into(), + index: "eq" + } + ); + assert_eq!( + loads.load(std::sync::atomic::Ordering::SeqCst), + 0, + "refused before the named keyset is loaded" + ); +} + +#[tokio::test] +async fn a_match_query_debugs_as_its_positions() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let query = cipher + .query("bob") + .using(&plan.field("email").unwrap()) + .op(op::Matches::new()) + .await + .unwrap(); + assert_eq!( + format!("{query:?}"), + format!("MatchQuery {{ positions: {:?} }}", query.positions()) + ); +} + +#[tokio::test] +async fn an_operator_query_runs_under_the_keyset_it_names() { + let cipher = stack_cipher().await; + let plan = Plan::context("users/email") + .with::(Equality) + .build() + .unwrap(); + let stored: Encrypted = cipher + .encrypt(&"bob@example.com".to_owned()) + .using(&plan) + .keyset("tenant") + .await + .unwrap(); + let under_tenant = cipher + .query("bob@example.com") + .using(&plan) + .op(op::Eq) + .keyset("tenant") + .await + .unwrap(); + assert_eq!(under_tenant, stored.terms); + let under_default = cipher + .query("bob@example.com") + .using(&plan) + .op(op::Eq) + .await + .unwrap(); + assert_ne!(under_default, stored.terms, "another keyset, another key"); +} From b6b8d9dc0f1a89ea45b35477c2a920d4cb267e26 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 10:51:25 +1100 Subject: [PATCH 10/38] refactor(stack-encrypt): the query chain starts with the operator and ends on the plan `cipher.query(op::Eq, "bob@example.com").on(&plan)` replaces `cipher.query("bob@example.com").using(&plan).op(op::Eq)`. The operator comes first, by value, and `on` names the plan, which chooses the index that answers the operator (`plan::Chooses`) and resolves the label the write used (`plan::Resolves`). An index the plan declares beside the default is named before the plan: `.via(index).on(&plan)`. An index's own term, for ORE and OPE until the ordering operators arrive, is `cipher.query_index(Ore, &40).on(&plan)`. `op::Matches` is now a value as well as a type, as `PhantomData` is, so `op::Matches::new()` is `op::Matches`; `op::Matches.case_sensitive()` is the other case. The case types are `CaseInsensitive` and `CaseSensitive`: `Sensitive` on its own read as data sensitivity. `Match` is a value the same way, so `Match::default()` is `Match`, and `Default` is implemented for every `Match`. `Answers::via()` and `DefaultIndex::via()` are gone: an index is a unit type, so `Via: Default` says the same thing without a method per impl, and the three equivalent-mutant exclusions for those bodies go with them. "Needle" is gone from the prose: a match query is a filter over the query's value, where the stored term is a filter over the stored value. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- .cargo/mutants.toml | 7 +- packages/stack-encrypt/CHANGELOG.md | 54 ++- packages/stack-encrypt/CONTEXT.md | 4 +- packages/stack-encrypt/src/codes.rs | 2 +- packages/stack-encrypt/src/dynamic/record.rs | 2 +- packages/stack-encrypt/src/dynamic/term.rs | 4 +- packages/stack-encrypt/src/op.rs | 130 +++---- packages/stack-encrypt/src/plan/build.rs | 7 +- packages/stack-encrypt/src/plan/chain.rs | 368 ++++++++++++------ packages/stack-encrypt/src/plan/error.rs | 4 +- packages/stack-encrypt/src/plan/mod.rs | 14 +- packages/stack-encrypt/src/plan/value.rs | 7 +- packages/stack-encrypt/src/sem/mod.rs | 12 +- packages/stack-encrypt/src/target/index.rs | 25 +- packages/stack-encrypt/tests/derive_plan.rs | 5 +- packages/stack-encrypt/tests/index.rs | 10 +- packages/stack-encrypt/tests/op.rs | 148 +++---- packages/stack-encrypt/tests/plan.rs | 107 ++--- packages/stack-encrypt/tests/plan_grammar.rs | 89 ++--- .../tests/ui/match_in_a_tuple_on_integer.rs | 2 +- .../ui/match_in_a_tuple_on_integer.stderr | 4 +- .../tests/ui/match_on_integer.rs | 2 +- .../tests/ui/match_on_integer.stderr | 4 +- .../tests/ui/op_case_mismatch.rs | 10 +- .../tests/ui/op_case_mismatch.stderr | 51 ++- .../tests/ui/op_matches_on_integer.rs | 2 +- .../tests/ui/op_matches_on_integer.stderr | 82 +++- .../tests/ui/op_target_does_not_answer.rs | 2 +- .../tests/ui/op_target_does_not_answer.stderr | 31 +- .../stack-encrypt/tests/ui/pass/indexes.rs | 4 +- .../stack-encrypt/tests/ui/pass/plan_chain.rs | 12 +- .../tests/ui/plan_match_on_integer.rs | 2 +- .../tests/ui/plan_match_on_integer.stderr | 2 +- .../tests/ui/plan_query_undeclared_index.rs | 2 +- .../ui/plan_query_undeclared_index.stderr | 48 ++- 35 files changed, 719 insertions(+), 540 deletions(-) diff --git a/.cargo/mutants.toml b/.cargo/mutants.toml index a1d1e4eba..289dfad3f 100644 --- a/.cargo/mutants.toml +++ b/.cargo/mutants.toml @@ -72,10 +72,11 @@ exclude_re = [ 'stack-encrypt/src/cipher\.rs:\d+:\d+: replace client_key_provider -> EnvKeyProvider with Default::default\(\)$', 'stack-encrypt/src/target/pending\.rs:443:9: replace >::into_future -> Self::IntoFuture with Default::default\(\)$', # stack-encrypt — equivalent. - # An operator and an operator's index are unit structs, whose `Default` - # is the value `clone` and `via` already return. + # An operator and an index are unit structs, whose `Default` is the value + # `clone` and `case_sensitive` already return. 'stack-encrypt/src/op(/mod)?\.rs:\d+:\d+: replace >::clone -> Self with Default::default\(\)$', - 'stack-encrypt/src/op(/mod)?\.rs:\d+:\d+: replace for EqualityTerm|DefaultIndex for Eq|DefaultIndex for Matches<::Case>)>::via -> (Equality|Match) with Default::default\(\)$', + 'stack-encrypt/src/op(/mod)?\.rs:\d+:\d+: replace Matches::case_sensitive -> Matches with Default::default\(\)$', + 'stack-encrypt/src/target/index\.rs:\d+:\d+: replace >::clone -> Self with Default::default\(\)$', 'stack-encrypt/src/cipher\.rs:\d+:\d+: replace StackCipher::builder -> StackCipherBuilder with Default::default\(\)$', 'stack-encrypt/src/sem/mod\.rs:\d+:\d+: replace ::options -> MatchOptions with Default::default\(\)$', # Both unit-context conversions explicitly return Self::default(). diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 81719e35d..d02d7bc31 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -9,24 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **A query names an operator, not an index.** `cipher.query(v).using(&plan)` - finishes with `.op(op::Eq)` or `.op(op::Matches::new())`, and the plan - finds the index that answers the operator (ADR-0009). An index declares the - operators it answers (`op::Supports`), a target type names the index that - answers each one for it (`op::Answers`), and a plan that declares indexes - uses the operator's default index (`op::DefaultIndex`). `.op_via(op, index)` - names another index, such as a match index with its own configuration. What - the plan's type shows is checked at compile time: a match query on an - integer, an operator a target does not answer, and on a one-value plan an - index it does not hold or a case-sensitive `Matches` against a - case-folding index. On a field of a fields plan those are refused when the - query is awaited, before any key is requested, as is a match configuration - whose `op::MatchCase` disagrees with its options (`PlanError::MatchCase`). +- **A query names an operator, not an index.** + `cipher.query(op::Eq, "bob@example.com").on(&plan)` and + `cipher.query(op::Matches, "ali").on(&plan)`: the operator comes first, as + a value, and the plan named by `on` finds the index that answers it + (ADR-0009). An index declares the operators it answers (`op::Supports`), a + target type names the index that answers each one for it (`op::Answers`), + and a plan that declares indexes uses the operator's default index + (`op::DefaultIndex`); both name the index by its type, whose `Default` is + the index. `.via(index)` before `.on(&plan)` names another index the plan + declares, such as a match index with its own configuration. + `op::Matches` is the case-insensitive operator, as a value; + `op::Matches.case_sensitive()` is the other, and the case types are + `op::CaseInsensitive` and `op::CaseSensitive`. What the plan's type shows + is checked at compile time: a match query on an integer, an operator a + target does not answer, and on a one-value plan an index it does not hold + or a case-sensitive `Matches` against a case-folding index. On a field of + a fields plan those are refused when the query is awaited, before any key + is requested, as is a match configuration whose `op::MatchCase` disagrees + with its options (`PlanError::MatchCase`). - **A match query has its own type, `sem::MatchQuery`.** It is the filter over - the needle; test a stored filter against it with + the query's value; test a stored filter against it with `MatchTerms::matches(&query)`. It comes only from a query chain: `op::Supports::query` takes an `op::QueryToken` that only this crate can make. +- **`Match` is a value.** `.with((Equality, Match))` declares the match index + under the default configuration, as `Equality` declares equality; + `Match::::new()` is another configuration. - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as @@ -66,10 +75,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Breaking -- **`QueryUsing::equality()` is removed.** Write `.op(op::Eq)`. A plan that - declares the equality index answers it the same way, and a target answers - it through `op::Answers`. `QueryUsing::index(..)` stays until the ordering - operators exist. +- **The query chain starts with the operator.** `cipher.query(v)` takes the + operator first: `cipher.query(op::Eq, v).on(&plan)` replaces + `cipher.query(v).using(&plan).equality()`, and `on` replaces `using` for + queries only (a write still runs `using` a plan). An index's own term, for + an index no operator answers yet (`Ore`, `Ope`), is + `cipher.query_index(Ore, v).on(&plan)`, which goes when the ordering + operators arrive. `QueryUsing` is gone; the chain's types are + `QueryBuilder`, `QueryVia`, `QueryIndexBuilder`, `QueryIndex` and + `QueryOp`, and the plan shapes implement `plan::Chooses` and + `plan::Resolves`. +- **`Match::default()` is `Match`.** The match index under the default + configuration is the value `Match`, and `Default` is now implemented for + every `Match`, so `Match::default()` no longer infers the configuration. - **Update pattern matches for `dynamic::Error`.** `Context`, `Plan`, `Source`, and `Record` now carry diff --git a/packages/stack-encrypt/CONTEXT.md b/packages/stack-encrypt/CONTEXT.md index 3cfab23a8..c9a9b678d 100644 --- a/packages/stack-encrypt/CONTEXT.md +++ b/packages/stack-encrypt/CONTEXT.md @@ -241,8 +241,8 @@ _Avoid_: query type, index (that is what answers the operator) **Query term**: The value a query sends to the server. It is not the stored term: equality -sends the same HMAC it stores, but a match query is a filter over the needle, -where the stored term is a filter over the whole value. A query term is never +sends the same HMAC it stores, but a match query is a filter over the query's +value, where the stored term is a filter over the whole stored value. A query term is never stored, and a query chain never sends a stored term as a query. _Avoid_: probe, stored term (that is what a write keeps) diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index b1f8dd5cb..7d6929fc8 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -150,7 +150,7 @@ mod tests { }), }, PlanError::MatchCase { .. } => PlanError::MatchCase { - declared: "Insensitive", + declared: "CaseInsensitive", folds: false, }, PlanError::TwoContextSources { .. } => PlanError::TwoContextSources { diff --git a/packages/stack-encrypt/src/dynamic/record.rs b/packages/stack-encrypt/src/dynamic/record.rs index c7c89f9c2..402850985 100644 --- a/packages/stack-encrypt/src/dynamic/record.rs +++ b/packages/stack-encrypt/src/dynamic/record.rs @@ -6799,7 +6799,7 @@ mod tests { (P::NoContext.into(), None, Refused), ( P::MatchCase { - declared: "Sensitive", + declared: "CaseSensitive", folds: true, } .into(), diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index edeebe492..d02b272c0 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -1206,7 +1206,7 @@ mod wire_form { fn every_typed_index_round_trips_through_the_wire_form() { let specs = [ Index::::spec(&Equality), - Index::::spec(&Match::default()), + Index::::spec(&Match), Index::::spec(&Match::::new()), Index::::spec(&Ore), Index::::spec(&Ope), @@ -1222,7 +1222,7 @@ mod wire_form { #[test] fn a_default_match_index_is_the_bare_match_key() { - let spec = Index::::spec(&Match::default()); + let spec = Index::::spec(&Match); assert!(matches!( spec.to_value(), FfiValue::String(key) if super::utf8(&key) == Some("match") diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op.rs index 3a70dc46a..5f1d26d2e 100644 --- a/packages/stack-encrypt/src/op.rs +++ b/packages/stack-encrypt/src/op.rs @@ -1,8 +1,8 @@ //! Operators: what a query asks of the server, as a type. //! -//! A query names an *operator* ([`Eq`](struct@Eq), [`Matches`]), and the plan it runs -//! through finds the index that answers it. The operator is a unit type -//! passed by value, as an index is (`.index(Ore)`): +//! A query names an *operator* ([`Eq`](struct@Eq), [`Matches`](struct@Matches)), and the plan it +//! runs on finds the index that answers it. The operator is a unit type +//! passed by value, as an index is (`.via(Ore)`): //! //! ``` //! # async fn example() -> Result<(), stack_encrypt::Error> { @@ -11,11 +11,11 @@ //! //! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; //! let email_plan = Plan::context("users/email") -//! .with::((Equality, Match::default())) +//! .with::((Equality, Match)) //! .build()?; //! -//! let exact = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; -//! let search = cipher.query("bob").using(&email_plan).op(op::Matches::new()).await?; +//! let exact = cipher.query(op::Eq, "bob@example.com").on(&email_plan).await?; +//! let search = cipher.query(op::Matches, "bob").on(&email_plan).await?; //! # let _ = (exact, search); //! # Ok(()) //! # } @@ -27,8 +27,9 @@ //! - [`Supports`] is on an index, for each operator it can answer //! over plaintexts `S`, and names the [`QueryTerm`] it sends. That term is //! not the stored term: an equality index sends the same HMAC it stores, -//! but a match index sends a filter over the needle ([`MatchQuery`]), -//! where it stores a filter over the whole value ([`MatchTerms`]). +//! but a match index sends a filter over the query's value +//! ([`MatchQuery`]), where it stores a filter over the stored value +//! ([`MatchTerms`]). //! - [`Answers`] is on a target type, a domain type such as an EQL //! type, and names the index that answers `Op` for it. It is written by //! hand per type, because the choice decides what the query term reveals. @@ -41,7 +42,8 @@ //! a case-folding index. A field of a fields plan is data, so there an //! index the field does not declare, or declares with other options, is //! refused when the query is awaited, as with -//! [`index`](crate::plan::QueryUsing::index), before any key is requested. +//! [`query_index`](crate::StackCipher::query_index), before any key is +//! requested. //! So is a match configuration whose [`MatchCase`] disagrees with its //! options. //! @@ -69,21 +71,40 @@ pub struct Eq; impl Operator for Eq {} -/// Text search: rows whose text contains the needle's tokens, under the +/// Text search: rows whose text contains the query's tokens, under the /// match index's tokenizer. Not containment: a Bloom match is order- and /// multiplicity-insensitive. /// -/// `C` is the case the query is asked in, [`Insensitive`] unless named +/// `C` is the case the query is asked in, [`CaseInsensitive`] unless named /// otherwise. A match index answers only the case its configuration fixes /// ([`MatchCase`]), so asking a case-folding index for a case-sensitive /// match is a compile error rather than a query that quietly folds case. -pub struct Matches(PhantomData C>); +/// +/// The value [`Matches`](constant@Matches) is the case-insensitive operator, +/// as `PhantomData` is a value of its type; [`case_sensitive`](Matches::case_sensitive) +/// is the other. +pub struct Matches { + case: PhantomData C>, +} + +/// The case-insensitive text search operator, as a value: +/// `cipher.query(op::Matches, "ali")`. +#[allow(non_upper_case_globals)] +pub const Matches: Matches = Matches::new(); impl Matches { - /// The operator. `Matches::new()` takes its case from the index it is - /// asked of; `Matches::::new()` names it. + /// The operator in the case `C`: `Matches::::new()`. pub const fn new() -> Self { - Self(PhantomData) + Self { case: PhantomData } + } +} + +impl Matches { + /// The case-sensitive text search operator: + /// `cipher.query(op::Matches.case_sensitive(), "Ali")`. Only a match + /// index whose configuration does not fold case answers it. + pub const fn case_sensitive(self) -> Matches { + Matches::new() } } @@ -118,12 +139,12 @@ impl Operator for Matches {} mod sealed { pub trait Sealed {} - impl Sealed for super::Insensitive {} - impl Sealed for super::Sensitive {} + impl Sealed for super::CaseInsensitive {} + impl Sealed for super::CaseSensitive {} } -/// The case a text operator is asked in: [`Insensitive`] or [`Sensitive`]. -/// Sealed: there are exactly two. +/// The case a text operator is asked in: [`CaseInsensitive`] or +/// [`CaseSensitive`]. Sealed: there are exactly two. pub trait Case: sealed::Sealed + 'static { /// Whether text is lower-cased before it is tokenised. const FOLDS: bool; @@ -133,19 +154,19 @@ pub trait Case: sealed::Sealed + 'static { /// Case-insensitive: text is lower-cased before it is tokenised. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -pub struct Insensitive; +pub struct CaseInsensitive; /// Case-sensitive: text is tokenised as written. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -pub struct Sensitive; +pub struct CaseSensitive; -impl Case for Insensitive { +impl Case for CaseInsensitive { const FOLDS: bool = true; - const NAME: &'static str = "Insensitive"; + const NAME: &'static str = "CaseInsensitive"; } -impl Case for Sensitive { +impl Case for CaseSensitive { const FOLDS: bool = false; - const NAME: &'static str = "Sensitive"; + const NAME: &'static str = "CaseSensitive"; } /// The case a match configuration fixes, as a type, so the operators its @@ -157,14 +178,14 @@ impl Case for Sensitive { /// disagree is refused when it is awaited, with /// [`PlanError::MatchCase`]. A configuration /// that does not implement this can still store match terms and query them -/// by index; it cannot be queried by [`Matches`]. +/// by index; it cannot be queried by [`Matches`](struct@Matches). pub trait MatchCase: MatchConfig { /// The case the configuration's index answers. type Case: Case; } impl MatchCase for DefaultMatch { - type Case = Insensitive; + type Case = CaseInsensitive; } /// A value a query sends to the server to compare against stored terms. @@ -184,7 +205,7 @@ impl QueryTerm for MatchQuery {} /// [`Index::Term`] is what the index derives; [`query`](Self::query) makes /// the query term from it. For an equality index the two are the same /// HMAC. For a match index they are not: the query is a filter over the -/// needle, and its type says so. +/// query's value, and its type says so. #[diagnostic::on_unimplemented( message = "`{Self}` does not answer `{Op}` over `{S}`", label = "this index cannot answer the operator", @@ -258,20 +279,16 @@ where #[diagnostic::on_unimplemented( message = "`{Self}` does not answer `{Op}` over `{S}`", label = "this target type does not answer the operator", - note = "a target answers the operators its type implements `Answers` for; `.op_via(op, index)` names an index the target declares" + note = "a target answers the operators its type implements `Answers` for; `.via(index)` names an index the target declares" )] pub trait Answers { - /// The index that answers `Op` for this type. - type Via: Supports; - /// That index, as a value. - fn via() -> Self::Via; + /// The index that answers `Op` for this type. An index is a unit type, + /// so its `Default` is the index. + type Via: Supports + Default; } impl Answers for EqualityTerm { type Via = Equality; - fn via() -> Equality { - Equality - } } impl Answers, S> for MatchTerms @@ -280,63 +297,46 @@ where O: MatchCase, { type Via = Match; - fn via() -> Match { - Match::new() - } } /// A ciphertext with one term beside it, the shape of a `plaintext = ..` /// record without the struct, answers what its term answers. impl> Answers for (crate::StackCipherText, T) { type Via = T::Via; - fn via() -> T::Via { - T::via() - } } /// A ciphertext with one term, as [`indexed`](crate::target::indexed) /// produces it, answers what its term answers. One with several terms -/// names the index with `.op_via(op, index)`. +/// names the index with `.via(index)`. impl> Answers for crate::target::Encrypted { type Via = T::Via; - fn via() -> T::Via { - T::via() - } } /// The index a plan that declares its indexes answers an operator with: /// a field of a fields plan, or a one-value plan built `with` indexes. /// /// A plan that declares a different index for the operator, such as a -/// match index with its own configuration, names it with -/// `.op_via(op, index)`. So does a typed field of a fields plan -/// (`encrypt_into::(name)`): the field's type is not known where the -/// query is written, so its [`Answers`] impl is not consulted, and a field -/// whose type answers through another index needs `op_via`. +/// match index with its own configuration, names it with `.via(index)`. +/// So does a typed field of a fields plan (`encrypt_into::(name)`): +/// the field's type is not known where the query is written, so its +/// [`Answers`] impl is not consulted, and a field whose type answers +/// through another index needs `via`. #[diagnostic::on_unimplemented( message = "`{Self}` has no default index over `{S}`", label = "no default index answers this operator here", - note = "`op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; name another index with `.op_via(op, index)`" + note = "`op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; name another index with `.via(index)`" )] pub trait DefaultIndex: Operator { - /// The index. - type Via: Supports; - /// That index, as a value. - fn via() -> Self::Via; + /// The index. An index is a unit type, so its `Default` is the index. + type Via: Supports + Default; } impl DefaultIndex for Eq { type Via = Equality; - fn via() -> Equality { - Equality - } } impl> DefaultIndex for Matches<::Case> { type Via = Match; - fn via() -> Match { - Match::default() - } } #[cfg(test)] @@ -353,15 +353,15 @@ mod tests { #[test] fn an_operator_is_equal_to_itself() { - assert_eq!(Matches::::new(), Matches::new()); + assert_eq!(Matches::::new(), Matches); assert_eq!(Eq, Eq); } #[test] fn an_operator_debugs_with_its_case() { assert_eq!( - format!("{:?}", Matches::::new()), - "Matches" + format!("{:?}", Matches.case_sensitive()), + "Matches" ); assert_eq!(format!("{:?}", Eq), "Eq"); } diff --git a/packages/stack-encrypt/src/plan/build.rs b/packages/stack-encrypt/src/plan/build.rs index 47825db6c..5ecc20d1e 100644 --- a/packages/stack-encrypt/src/plan/build.rs +++ b/packages/stack-encrypt/src/plan/build.rs @@ -259,10 +259,9 @@ pub(crate) fn check_declared( /// // A query and a read name the same context. /// let email_plan = users_plan.field("email")?; /// let query_value = cipher -/// .query("bob@example.com") +/// .query(op::Eq, "bob@example.com") /// .context("tenants/acme/users") -/// .using(&email_plan) -/// .op(op::Eq) +/// .on(&email_plan) /// .await?; /// # let _ = query_value; /// let back = cipher.open(record).context("tenants/acme/users").using(&users_plan).await?; @@ -928,7 +927,7 @@ impl FieldsBuilder { /// let email: Encrypted<(EqualityTerm, MatchTerms)> = record.take("email")?; /// /// // The target declares an equality index, so the field answers it. - /// let query_value = cipher.query("bob@example.com").using(&users_plan.field("email")?).op(op::Eq).await?; + /// let query_value = cipher.query(op::Eq, "bob@example.com").on(&users_plan.field("email")?).await?; /// assert_eq!(query_value, email.terms.0); /// # Ok(()) /// # } diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 6284f3b97..1c7181829 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -14,7 +14,7 @@ use super::PlanError; use crate::op::{Answers, DefaultIndex, Operator, QueryToken, Supports}; use crate::target::{ ciphertext, Borrowed, CallerContext, DeclaredContext, DecryptField, Decryptable, EncryptFrom, - Index, Indexes, Owned, Pending, PendingFuture, Select, + Index, Indexes, Owned, Pending, PendingFuture, Select, Whole, }; use crate::{Error, KeysetCipher, Label, LabelError, NonEmpty, StackCipher, StackCipherText}; @@ -279,19 +279,43 @@ impl StackCipher { } } - /// Start a chain that derives a query term for `value`, to match terms - /// a plan wrote. Name the plan with [`using`](QueryBuilder::using) and - /// the operator with [`op`](QueryUsing::op): `.op(op::Eq)`. + /// Start a chain that derives the query term for the operator `op` over + /// `value`, to match terms a plan wrote. Name the plan with + /// [`on`](QueryBuilder::on): `cipher.query(op::Eq, "bob@example.com").on(&plan)`. + /// The plan chooses the index that answers the operator there; an + /// index the plan declares beside that one is named first, with + /// [`via`](QueryBuilder::via). /// - /// The value is copied once (`to_owned`), so `query("bob@example.com")` + /// The value is copied once (`to_owned`), so `query(op::Eq, "bob@example.com")` /// queries a `String` field. - pub fn query<'a, Q>(&'a self, value: &Q) -> QueryBuilder<'a, Q::Owned, K> + pub fn query<'a, Op, Q>(&'a self, op: Op, value: &Q) -> QueryBuilder<'a, Op, F, K> where + Op: Operator, Q: ToOwned + ?Sized, { QueryBuilder { common: Common::new(self), value: value.to_owned(), + op, + } + } + + /// Start a chain that derives the term of `index` over `value`: the + /// index's own term, not a query term, so nothing says what it may be + /// compared with. For an index no operator answers yet ([`Ore`] and + /// [`Ope`]); a query for an operator is [`query`](Self::query), and this + /// goes when the ordering operators arrive. + /// + /// [`Ore`]: crate::target::Ore + /// [`Ope`]: crate::target::Ope + pub fn query_index<'a, I, Q>(&'a self, index: I, value: &Q) -> QueryIndexBuilder<'a, I, F, K> + where + Q: ToOwned + ?Sized, + { + QueryIndexBuilder { + common: Common::new(self), + value: value.to_owned(), + index, } } @@ -589,13 +613,18 @@ where } } -/// A chain deriving a query term, before its plan is named. -pub struct QueryBuilder<'a, F, K> { +/// The owned form of a queried value: `String` for a `str` query. +type F = ::Owned; + +/// A chain deriving the query term for an operator, before its plan is +/// named. +pub struct QueryBuilder<'a, Op, F, K> { common: Common<'a, K>, value: F, + op: Op, } -impl<'a, F, K: 'static> QueryBuilder<'a, F, K> { +impl<'a, Op: Operator, F: 'static, K: 'static> QueryBuilder<'a, Op, F, K> { /// Name the context the term is derived under, for a plan built /// without one: the context the write's call named, or for a plan with /// a context field, the value that field held. A plan built with its @@ -605,150 +634,241 @@ impl<'a, F, K: 'static> QueryBuilder<'a, F, K> { self } - /// Query through `plan`: a field of a fields plan - /// ([`Plan::field`](super::Plan::field)), or a one-value plan ([`ValuePlan`]). The term is - /// derived under exactly the label the write used. - pub fn using

(self, plan: &'a P) -> QueryUsing<'a, F, P, K> { - QueryUsing { + /// Answer the operator through `index` rather than the index the plan + /// chooses for it ([`Chooses`]): a match index with its own + /// configuration, or an index of a target type that has no + /// [`Answers`] impl for the operator. The plan named next must declare + /// it, as [`on`](Self::on) says. + pub fn via>(self, index: I) -> QueryVia<'a, Op, F, I, K> { + QueryVia { common: self.common, value: self.value, - plan, + op: self.op, + index, } } + + /// Query on `plan`: a field of a fields plan + /// ([`Plan::field`](super::Plan::field)), or a one-value plan + /// ([`ValuePlan`]). The plan chooses the index that answers the + /// operator ([`Chooses`]), and the term is derived under exactly the + /// label the write used. + /// + /// What the plan's type can see is checked here: an operator it does + /// not answer, or a one-value plan without the index, does not compile. + /// A field of a fields plan is data, so there an index the field does + /// not declare, or declares with other options, is refused when the + /// query is awaited ([`PlanError::IndexNotDeclared`], + /// [`PlanError::IndexOptions`]), before any key is requested. + pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, P::Via, K> + where + P: Chooses, + { + self.via(P::Via::default()).on(plan) + } } -/// A chain deriving a query term through a plan, before its index is -/// chosen. -pub struct QueryUsing<'a, F, P, K> { +/// A chain deriving the query term for an operator through a named index, +/// before its plan is named. +pub struct QueryVia<'a, Op, F, I, K> { common: Common<'a, K>, value: F, - plan: &'a P, + op: Op, + index: I, } -impl<'a, F: 'static, K: 'static> QueryUsing<'a, F, FieldPlan, K> { - /// The query term for the operator `op`, through the index a plan that - /// declares its indexes answers it with ([`DefaultIndex`]): - /// `.op(op::Eq)` is the equality term. The field must have declared - /// that index, with the same options; otherwise awaiting is - /// [`PlanError::IndexNotDeclared`] or [`PlanError::IndexOptions`]. - pub fn op>(self, op: Op) -> QueryOp<'a, F, Op, Op::Via, K> { - self.op_via(op, Op::via()) - } - - /// The query term for `op` through `index`, for a field that declares - /// an index other than the operator's default, such as a match index - /// with its own configuration. - pub fn op_via(self, _op: Op, index: I) -> QueryOp<'a, F, Op, I, K> +impl<'a, Op: Operator, F: 'static, I: Supports, K: 'static> QueryVia<'a, Op, F, I, K> { + /// Query on `plan` through the named index, which the plan must + /// declare ([`Resolves`]): a one-value plan without it does not + /// compile; a field of a fields plan without it is refused when the + /// query is awaited. A one-value plan built `with` indexes uses its + /// own index of that type, so the named index says only which. + pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, I, K> where - Op: Operator, - I: Supports, + P: Resolves, { - QueryOp::new(self.index(index)) - } - - /// The term of `index`. The field must have declared it, and the query's - /// plaintext must be the field's type; otherwise awaiting is - /// [`PlanError::IndexNotDeclared`] or [`PlanError::FieldType`], never a - /// term that matches nothing. - pub fn index>(mut self, index: I) -> QueryIndex<'a, F, I, K> { - let label = self - .common - .call_context() - .and_then(|call| self.plan.query_label::(&index.spec(), call)); - QueryIndex { - common: self.common, - value: self.value, + let QueryVia { + common, + value, + op, index, - label, - } + } = self; + let inner = plan.resolve(QueryIndexBuilder { + common, + value, + index, + }); + QueryOp::new(inner, op) } } -impl<'a, F: 'static, X: Indexes, K: 'static> QueryUsing<'a, F, ValuePlan>, K> { - /// The query term for the operator `op`, through the index a plan that - /// declares its indexes answers it with ([`DefaultIndex`]), selected - /// from the plan's indexes by type: a plan that does not hold that - /// index does not compile here. - pub fn op(self, op: Op) -> QueryOp<'a, F, Op, Op::Via, K> - where - Op: DefaultIndex, - Op::Via: Clone, - X: Select, - { - self.op_via(op, Op::via()) +/// A chain deriving an index's own term, before its plan is named. Started +/// by [`StackCipher::query_index`]. +pub struct QueryIndexBuilder<'a, I, F, K> { + common: Common<'a, K>, + value: F, + index: I, +} + +impl<'a, I, F: 'static, K: 'static> QueryIndexBuilder<'a, I, F, K> { + /// Name the context the term is derived under, as + /// [`QueryBuilder::context`]. + pub fn context(mut self, context: impl IntoLabel) -> Self { + self.common.context = Some(context.into_label()); + self } - /// The query term for `op` through the plan's index of the type of - /// `index`, for a plan that holds an index other than the operator's - /// default. The plan's own index is used; `index` names its type. - pub fn op_via(self, _op: Op, _index: I) -> QueryOp<'a, F, Op, I, K> + /// Derive the index's term on `plan`, which must declare it + /// ([`Resolves`]); otherwise awaiting is + /// [`PlanError::IndexNotDeclared`] or [`PlanError::FieldType`], never a + /// term that matches nothing. + pub fn on(self, plan: &P) -> QueryIndex<'a, F, I, K> where - Op: Operator, - I: Supports + Clone, - X: Select, + I: Index, + P: Resolves, { - QueryOp::new(self.index::()) + plan.resolve(self) } +} - /// The term of the plan's index of type `I`, selected by type - /// ([`Indexes::select`]); `At` is inferred: `.index::()`. - pub fn index(mut self) -> QueryIndex<'a, F, I, K> - where - I: Index + Clone, - X: Select, - { - let index = self.plan.indexes().select::().clone(); - let label = self - .common +/// A plan a query for the operator `Op` over `F` runs on, and the index it +/// answers the operator with. Sealed: the three plan shapes implement it. +/// +/// A field of a fields plan and a one-value plan built `with` indexes +/// answer through the operator's [`DefaultIndex`]; a one-value plan for a +/// target type answers through the type's [`Answers`] impl. `At` is the +/// position of that index in a plan built `with` several, and is inferred. +#[diagnostic::on_unimplemented( + message = "`{Self}` does not answer `{Op}` over `{F}`", + label = "no index answers this operator on this plan", + note = "a field of a fields plan or a plan built `with` indexes answers an operator through its default index (`op::Eq`: `Equality`; `op::Matches`: `Match` with the default configuration, case-insensitive and over text only); a typed plan answers the operators its target type implements `Answers` for; `.via(index)` names an index the plan declares" +)] +pub trait Chooses: Resolves { + /// The index that answers `Op` on this plan. + type Via: Supports + Default; +} + +impl, F: 'static> Chooses for FieldPlan { + type Via = Op::Via; +} + +impl Chooses for ValuePlan> +where + Op: DefaultIndex, + Op::Via: Clone, + F: 'static, + X: Indexes + Select, +{ + type Via = Op::Via; +} + +impl Chooses for ValuePlan> +where + Op: Operator, + F: 'static, + T: Answers, + T::Via: Default, + Typed: ValueShape, +{ + type Via = T::Via; +} + +mod plan_sealed { + use super::{FieldPlan, Indexed, Typed, ValuePlan}; + pub trait Sealed {} + impl Sealed for FieldPlan {} + impl Sealed for ValuePlan> {} + impl Sealed for ValuePlan> {} +} + +/// A plan the index `I` over `F` can be queried on. Sealed: the three plan +/// shapes implement it, and each finds the label the write used. +/// +/// A field of a fields plan ([`Plan::field`](super::Plan::field)) and a +/// one-value plan for a target type take the index as given, and refuse +/// one the plan does not declare when the query is awaited +/// ([`PlanError::IndexNotDeclared`]). A one-value plan built `with` +/// indexes selects its own index of that type ([`Indexes::select`]), at +/// the position `At`, which is inferred; a plan without that index does +/// not compile. +#[diagnostic::on_unimplemented( + message = "`{Self}` cannot be queried through `{I}` over `{F}`", + label = "this plan does not declare the index, or is not a plan a query runs on", + note = "query on a field of a fields plan (`plan.field(..)`) or on a one-value plan; a plan built `with` indexes must hold one of this type" +)] +pub trait Resolves: plan_sealed::Sealed { + /// The chain for the index's term on this plan, under the label the + /// write used. + fn resolve<'a, K: 'static>( + &self, + query: QueryIndexBuilder<'a, I, F, K>, + ) -> QueryIndex<'a, F, I, K>; +} + +impl> Resolves for FieldPlan { + fn resolve<'a, K: 'static>( + &self, + query: QueryIndexBuilder<'a, I, F, K>, + ) -> QueryIndex<'a, F, I, K> { + let QueryIndexBuilder { + mut common, + value, + index, + } = query; + let label = common .call_context() - .and_then(|call| self.plan.resolve(call)); + .and_then(|call| self.query_label::(&index.spec(), call)); QueryIndex { - common: self.common, - value: self.value, + common, + value, index, label, } } } -impl<'a, F: 'static, T, K: 'static> QueryUsing<'a, F, ValuePlan>, K> +impl Resolves for ValuePlan> where - Typed: ValueShape, + X: Indexes + Select, + I: Index + Clone, { - /// The query term for the operator `op`, through the index the target - /// type answers it with ([`Answers`]). A target that does not answer - /// `op` does not compile here; one that answers it through an index it - /// does not declare is [`PlanError::IndexNotDeclared`] when awaited. - pub fn op(self, op: Op) -> QueryOp<'a, F, Op, T::Via, K> - where - Op: Operator, - T: Answers, - { - self.op_via(op, T::via()) - } - - /// The query term for `op` through `index`, which the target must - /// declare: for a target with no [`Answers`] impl for `op`, such as - /// an [`Encrypted`](crate::target::Encrypted) tuple of terms. - pub fn op_via(self, _op: Op, index: I) -> QueryOp<'a, F, Op, I, K> - where - Op: Operator, - I: Supports, - { - QueryOp::new(self.index(index)) + fn resolve<'a, K: 'static>( + &self, + query: QueryIndexBuilder<'a, I, F, K>, + ) -> QueryIndex<'a, F, I, K> { + let QueryIndexBuilder { + mut common, value, .. + } = query; + let index = self.indexes().select::().clone(); + let label = common.call_context().and_then(|call| self.resolve(call)); + QueryIndex { + common, + value, + index, + label, + } } +} - /// The term of `index`. The target must declare it - /// ([`EncryptFrom::indexes`]); otherwise awaiting is - /// [`PlanError::IndexNotDeclared`], never a term that matches nothing. - pub fn index>(mut self, index: I) -> QueryIndex<'a, F, I, K> { - let label = self - .common +impl> Resolves for ValuePlan> +where + Typed: ValueShape, +{ + fn resolve<'a, K: 'static>( + &self, + query: QueryIndexBuilder<'a, I, F, K>, + ) -> QueryIndex<'a, F, I, K> { + let QueryIndexBuilder { + mut common, + value, + index, + } = query; + let label = common .call_context() - .and_then(|call| self.plan.query_label(&index.spec(), call)); + .and_then(|call| self.query_label(&index.spec(), call)); QueryIndex { - common: self.common, - value: self.value, + common, + value, index, label, } @@ -808,7 +928,7 @@ pub struct QueryOp<'a, F, Op, I, K> { impl<'a, F, Op: Operator, I: Supports, K: 'static> QueryOp<'a, F, Op, I, K> { /// The chain for `inner`'s index, refused when awaited if the index /// cannot answer `Op` as it is configured. - fn new(mut inner: QueryIndex<'a, F, I, K>) -> Self { + fn new(mut inner: QueryIndex<'a, F, I, K>, _op: Op) -> Self { if let Err(error) = inner.index.check() { inner.label = Err(error); } @@ -945,8 +1065,9 @@ chain_options! { ['a, S, X, K: 'static] EncryptIndexed<'a, S, X, K>; ['a, S: 'static, K: 'static] EncryptFields<'a, S, K>; ['a, S: ?Sized, P, K: 'static] EncryptUsing<'a, S, P, K>; - ['a, F, K: 'static] QueryBuilder<'a, F, K>; - ['a, F, P, K: 'static] QueryUsing<'a, F, P, K>; + ['a, Op, F, K: 'static] QueryBuilder<'a, Op, F, K>; + ['a, Op, F, I, K: 'static] QueryVia<'a, Op, F, I, K>; + ['a, I, F, K: 'static] QueryIndexBuilder<'a, I, F, K>; ['a, F, I, K: 'static] QueryIndex<'a, F, I, K>; ['a, R, K: 'static] OpenBuilder<'a, R, K>; ['a, R, P, K: 'static] OpenUsing<'a, R, P, K>; @@ -979,8 +1100,9 @@ opaque_debug! { ['a, S, X, K] EncryptIndexed<'a, S, X, K>, "EncryptIndexed"; ['a, S: 'static, K: 'static] EncryptFields<'a, S, K>, "EncryptFields"; ['a, S: ?Sized, P, K] EncryptUsing<'a, S, P, K>, "EncryptUsing"; - ['a, F, K] QueryBuilder<'a, F, K>, "QueryBuilder"; - ['a, F, P, K] QueryUsing<'a, F, P, K>, "QueryUsing"; + ['a, Op, F, K] QueryBuilder<'a, Op, F, K>, "QueryBuilder"; + ['a, Op, F, I, K] QueryVia<'a, Op, F, I, K>, "QueryVia"; + ['a, I, F, K] QueryIndexBuilder<'a, I, F, K>, "QueryIndexBuilder"; ['a, F, I, K] QueryIndex<'a, F, I, K>, "QueryIndex"; ['a, F, Op, I, K] QueryOp<'a, F, Op, I, K>, "QueryOp"; ['a, R, K] OpenBuilder<'a, R, K>, "OpenBuilder"; @@ -1033,7 +1155,7 @@ impl std::fmt::Debug for All<'_, K, T> { /// let email = String::from("bob@example.com"); /// let (written, query_value) = stack_encrypt::all(( /// cipher.encrypt(&email).using(&email_plan), -/// cipher.query("bob@example.com").using(&email_plan).op(op::Eq), +/// cipher.query(op::Eq, "bob@example.com").on(&email_plan), /// )) /// .await?; /// assert_eq!(written.terms, query_value); diff --git a/packages/stack-encrypt/src/plan/error.rs b/packages/stack-encrypt/src/plan/error.rs index 85932f299..f029d4761 100644 --- a/packages/stack-encrypt/src/plan/error.rs +++ b/packages/stack-encrypt/src/plan/error.rs @@ -192,10 +192,10 @@ pub enum PlanError { )] #[diagnostic( code(stack_encrypt::plan_match_case), - help("Make `MatchCase::Case` agree with `MatchConfig::options().downcase`: `Insensitive` when it is true, `Sensitive` when it is false.") + help("Make `MatchCase::Case` agree with `MatchConfig::options().downcase`: `CaseInsensitive` when it is true, `CaseSensitive` when it is false.") )] MatchCase { - /// The case `MatchCase::Case` names: `Insensitive` or `Sensitive`. + /// The case `MatchCase::Case` names: `CaseInsensitive` or `CaseSensitive`. declared: &'static str, /// Whether the configuration's options fold case. folds: bool, diff --git a/packages/stack-encrypt/src/plan/mod.rs b/packages/stack-encrypt/src/plan/mod.rs index d7fc1512d..7e58c504f 100644 --- a/packages/stack-encrypt/src/plan/mod.rs +++ b/packages/stack-encrypt/src/plan/mod.rs @@ -87,7 +87,7 @@ //! .encrypt(&user) //! .context("users") //! .fields() -//! .encrypt_index::("email", (Equality, Match::default())) +//! .encrypt_index::("email", (Equality, Match)) //! .encrypt_index::("age", (Equality, Ore)) //! .encrypt::("notes") //! .passthrough::("id") @@ -158,7 +158,7 @@ //! //! let users_plan: Plan = Plan::context("users") //! .fields() -//! .encrypt_index::("email", (Equality, Match::default())) +//! .encrypt_index::("email", (Equality, Match)) //! .encrypt_index::("age", (Equality, Ore)) //! .encrypt::("notes") //! .passthrough::("id") @@ -170,7 +170,7 @@ //! //! // Query: derived under the label the write used, so it matches. //! let email_plan = users_plan.field("email")?; -//! let query_value = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; +//! let query_value = cipher.query(op::Eq, "bob@example.com").on(&email_plan).await?; //! assert_eq!(query_value, stored.terms.0); //! //! // Read: every field that can come back. @@ -252,7 +252,7 @@ //! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; //! let user = User { email: "bob@example.com".into(), nickname: "bob".into() }; //! let record: EncryptedUser = cipher.default_keyset().encrypt_as(&user, ().into()).await?; -//! let query_value = cipher.query("bob@example.com").using(&by_hand.field("email")?).op(op::Eq).await?; +//! let query_value = cipher.query(op::Eq, "bob@example.com").on(&by_hand.field("email")?).await?; //! assert_eq!(record.email.terms, query_value); //! # Ok(()) //! # } @@ -301,7 +301,7 @@ //! | `fields()` | each field picked out of the value (by name, or by a picker's accessor as `project` does), `zip`ped | //! | `extend(parts)` | the [`DeclaredContext`](crate::target::DeclaredContext) the description runs under | //! | `using(&plan)` | [`KeysetCipher::run`](crate::KeysetCipher::run) of [`Plan::encryption`] | -//! | `query(v).using(..).op(op)`, `.index(x)` | the answering index's operation alone, `.under(label)`; for `op`, mapped to its query term ([`Supports::query`](crate::op::Supports::query)) | +//! | `query(op, v).on(..)`, `query_index(x, v).on(..)` | the answering index's operation alone, `.under(label)`; for `op`, mapped to its query term ([`Supports::query`](crate::op::Supports::query)) | //! | `open(record).using(&plan)` | [`open`](crate::target::open) per field, run by `run_decryption` | //! | `.await`, [`all`] | `Pending::zip`, then one settle | //! @@ -319,9 +319,9 @@ mod values; pub use build::{FieldKind, FieldPlan, FieldsBuilder, IntoLabel, Opens, Plan, PlanContext, Runs}; pub use chain::{ - all, All, Batch, EncryptBuilder, EncryptFields, EncryptIndexed, EncryptUsing, + all, All, Batch, Chooses, EncryptBuilder, EncryptFields, EncryptIndexed, EncryptUsing, EncryptWithContext, KeysetChoice, OpenBuilder, OpenUsing, Operation, PlanKms, QueryBuilder, - QueryIndex, QueryOp, QueryUsing, + QueryIndex, QueryIndexBuilder, QueryOp, QueryVia, Resolves, }; pub use error::PlanError; pub use field_ref::{pick, FieldRef}; diff --git a/packages/stack-encrypt/src/plan/value.rs b/packages/stack-encrypt/src/plan/value.rs index 76ae45f23..9ff54b49f 100644 --- a/packages/stack-encrypt/src/plan/value.rs +++ b/packages/stack-encrypt/src/plan/value.rs @@ -52,10 +52,9 @@ impl Plan<(), ()> { /// let email = String::from("bob@example.com"); /// let (c, eq) = cipher.encrypt(&email).context("users/email").using(&email_plan).await?; /// let query_value = cipher - /// .query("bob@example.com") + /// .query(op::Eq, "bob@example.com") /// .context("users/email") - /// .using(&email_plan) - /// .op(op::Eq) + /// .on(&email_plan) /// .await?; /// assert_eq!(query_value, eq); /// let back = cipher.open((c, eq)).context("users/email").using(&email_plan).await?; @@ -152,7 +151,7 @@ impl ValueStart { /// Unlike a fields plan's context field, the context here is a `C`, not /// a [`Label`], so the plan does not run through the chain: /// `cipher.encrypt(..).using`, `cipher.open(..).using` and - /// `cipher.query(..).using` do not take it (it is neither + /// `cipher.query(..).on` do not take it (it is neither /// [`Runs`](super::Runs) nor [`Opens`](super::Opens)), and it has no /// query path. Run it as the derive does, with its two descriptions: /// `keyset.run(plan.encryption_with_context(), &value, context)` and diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 473d21775..2dba9df2c 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -642,27 +642,27 @@ impl PartialEq for MatchTerms { impl Eq for MatchTerms {} -/// The query term of a match index: a Bloom filter over the *needle's* +/// The query term of a match index: a Bloom filter over the *query's* /// tokens, where the stored [`MatchTerms`] is a filter over the whole -/// value's. A row matches when its stored filter holds every position of +/// stored value's. A row matches when its stored filter holds every position of /// the query's ([`MatchTerms::contains`]). /// /// The two are different types because they are different things, even /// where their bytes share an encoding: a query is never stored, and a /// stored filter is never sent as a query. ADR-0009 records why. A -/// `MatchQuery` comes from a query chain ([`op::Matches`](crate::op::Matches)), +/// `MatchQuery` comes from a query chain ([`op::Matches`](struct@crate::op::Matches)), /// never from a target. pub struct MatchQuery { terms: MatchTerms, } impl MatchQuery { - /// The query term of the filter `terms`, derived over a needle. + /// The query term of the filter `terms`, derived over the query's value. pub(crate) fn from_terms(terms: MatchTerms) -> Self { Self { terms } } - /// The set positions of the needle's filter, sorted ascending, no + /// The set positions of the query's filter, sorted ascending, no /// duplicates: what the server tests a stored filter for. pub fn positions(&self) -> &[u16] { self.terms.positions() @@ -673,7 +673,7 @@ impl MatchQuery { self.terms.to_bytes() } - /// The needle's filter. + /// The query's filter. pub(crate) fn filter(&self) -> &MatchTerms { &self.terms } diff --git a/packages/stack-encrypt/src/target/index.rs b/packages/stack-encrypt/src/target/index.rs index 145cf0be7..7006f16a8 100644 --- a/packages/stack-encrypt/src/target/index.rs +++ b/packages/stack-encrypt/src/target/index.rs @@ -115,7 +115,7 @@ impl fmt::Display for IndexSpec { /// /// Implemented for exactly the plaintexts the index's scheme is defined /// over, which is what makes an inapplicable index a compile error: there is -/// no `Index` for [`Match`], because match is defined over text. +/// no `Index` for [`Match`](struct@Match), because match is defined over text. /// /// An index does not seal anything. [`indexed`] puts a ciphertext beside the /// indexes of a field; [`operation`](Self::operation) on its own is the term @@ -163,18 +163,29 @@ pub struct Ope; /// The full-text match index, over text only, with the tokenizer and filter /// parameters its [`MatchConfig`] `O` fixes. /// -/// `Match::default()` is the default configuration ([`DefaultMatch`]); -/// `Match::::new()` is another. The configuration is on the type so -/// that a stored field and the query probing it agree on it by construction. -pub struct Match(PhantomData O>); +/// The value [`Match`](constant@Match) is the index under the default +/// configuration ([`DefaultMatch`]), as `PhantomData` is a value of its +/// type; `Match::::new()` is another. The configuration is on the +/// type so that a stored field and the query probing it agree on it by +/// construction. +pub struct Match { + config: PhantomData O>, +} + +/// The match index under the default configuration, as a value: +/// `.with((Equality, Match))`. +#[allow(non_upper_case_globals)] +pub const Match: Match = Match::new(); impl Match { /// The match index under the configuration `O`. pub const fn new() -> Self { - Self(PhantomData) + Self { + config: PhantomData, + } } } -impl Default for Match { +impl Default for Match { fn default() -> Self { Self::new() } diff --git a/packages/stack-encrypt/tests/derive_plan.rs b/packages/stack-encrypt/tests/derive_plan.rs index 018f35cda..26ae77162 100644 --- a/packages/stack-encrypt/tests/derive_plan.rs +++ b/packages/stack-encrypt/tests/derive_plan.rs @@ -130,9 +130,8 @@ async fn a_derived_record_as_a_typed_field_answers_its_queries() { let mut record = cipher.encrypt(&user).using(&users_plan).await.unwrap(); let email: Email = record.take("email").unwrap(); let query_value = cipher - .query("bob@example.com") - .using(&users_plan.field("email").unwrap()) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&users_plan.field("email").unwrap()) .await .unwrap(); assert_eq!(query_value, email.hm); diff --git a/packages/stack-encrypt/tests/index.rs b/packages/stack-encrypt/tests/index.rs index bd9e0e5e9..a61f28078 100644 --- a/packages/stack-encrypt/tests/index.rs +++ b/packages/stack-encrypt/tests/index.rs @@ -241,7 +241,7 @@ mod given_a_set_of_indexes { assert_eq!(>::specs(&Ore), [IndexSpec::Ore]); assert_eq!(>::specs(&Ope), [IndexSpec::Ope]); assert_eq!( - >::specs(&Match::default()), + >::specs(&Match), [IndexSpec::Match(MatchOptions::default())] ); assert_eq!( @@ -275,7 +275,7 @@ mod given_a_set_of_indexes { #[test] fn a_match_index_debugs_by_name_whatever_its_configuration() { - assert_eq!(format!("{:?}", Match::default()), "Match"); + assert_eq!(format!("{:?}", Match), "Match"); assert_eq!(format!("{:?}", Match::::new()), "Match"); let copied = Match::::new(); let again = copied; @@ -433,7 +433,7 @@ mod given_a_set_of_indexes { .unwrap(); assert_eq!(q, ope_t, "position 2"); - let four = (Equality, Ore, Ope, Match::default()); + let four = (Equality, Ore, Ope, Match); assert_eq!( Index::::spec(Indexes::::select::(&four)), IndexSpec::Match(MatchOptions::default()), @@ -662,7 +662,7 @@ mod given_encrypted_terms_as_a_target { #[tokio::test] async fn every_term_alone_lowers_to_its_index() { same_bytes::(7, Equality).await; - same_bytes::("hello world".into(), Match::default()).await; + same_bytes::("hello world".into(), Match).await; same_bytes::, _>("Hello World".into(), Match::::new()) .await; same_bytes::, _>(7, Ore).await; @@ -702,7 +702,7 @@ mod given_encrypted_terms_as_a_target { _, >( "Hello World".into(), - (Equality, Match::default(), Ore, Ope, Match::::new()), + (Equality, Match, Ore, Ope, Match::::new()), ) .await; } diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 03854af9a..487ffe056 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -24,7 +24,7 @@ impl MatchConfig for Exact { } } impl MatchCase for Exact { - type Case = op::Sensitive; + type Case = op::CaseSensitive; } #[derive(Clone)] @@ -64,7 +64,7 @@ fn person() -> Person { fn people_plan() -> Plan { Plan::context("people") .fields() - .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("email", (Equality, Match)) .encrypt_index::("name", Match::::new()) .build() .expect("a valid plan") @@ -86,16 +86,15 @@ async fn an_eq_query_is_the_stored_equality_term() { let email: Encrypted<(EqualityTerm, MatchTerms)> = row.take("email").unwrap(); let query: EqualityTerm = cipher - .query("bob@example.com") - .using(&plan.field("email").unwrap()) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&plan.field("email").unwrap()) .await .unwrap(); assert_eq!(query, email.terms.0, "HMAC equality sends what it stores"); } #[tokio::test] -async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_holds() { +async fn a_matches_query_is_a_filter_over_the_query_value_that_the_stored_filter_holds() { let cipher = stack_cipher().await; let plan = people_plan(); let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap(); @@ -104,22 +103,20 @@ async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_hold let email_plan = plan.field("email").unwrap(); let query: MatchQuery = cipher - .query("BOB") - .using(&email_plan) - .op(op::Matches::new()) + .query(op::Matches, "BOB") + .on(&email_plan) .await .unwrap(); assert!(stored.matches(&query), "a case-folding index matches BOB"); assert!( query.positions().len() < stored.positions().len(), - "the query is the needle's filter, not the value's" + "the query is the query value's filter, not the stored value's" ); // The operator derives the term the index does, under the same label. let by_index: MatchTerms = cipher - .query("BOB") - .using(&email_plan) - .index(Match::default()) + .query_index(Match, "BOB") + .on(&email_plan) .await .unwrap(); assert_eq!(query.to_bytes(), by_index.to_bytes()); @@ -127,9 +124,8 @@ async fn a_matches_query_is_a_filter_over_the_needle_that_the_stored_filter_hold assert_eq!(query.clone(), query); let elsewhere: MatchQuery = cipher - .query("zzz") - .using(&email_plan) - .op(op::Matches::new()) + .query(op::Matches, "zzz") + .on(&email_plan) .await .unwrap(); assert!(!stored.matches(&elsewhere)); @@ -144,30 +140,24 @@ async fn a_match_index_with_its_own_configuration_is_named_with_op_via() { let name: Encrypted> = row.take("name").unwrap(); let name_plan = plan.field("name").unwrap(); - let exact = |needle: &'static str| { + let exact = |value: &'static str| { cipher - .query(needle) - .using(&name_plan) - .op_via(op::Matches::::new(), Match::::new()) + .query(op::Matches.case_sensitive(), value) + .via(Match::::new()) + .on(&name_plan) }; let hit: MatchQuery = exact("Nguyen").await.unwrap(); assert!(name.terms.matches(&hit)); let miss: MatchQuery = exact("nguyen").await.unwrap(); assert!( !name.terms.matches(&miss), - "a case-sensitive index does not fold the needle" + "a case-sensitive index does not fold the query" ); // The operator's default index is the default configuration, which // this field does not declare. assert_eq!( - plan_error( - cipher - .query("Nguyen") - .using(&name_plan) - .op(op::Matches::new()) - .await - ), + plan_error(cipher.query(op::Matches, "Nguyen").on(&name_plan).await), PlanError::IndexOptions { field: "name".into(), declared: IndexSpec::Match(Exact::options()), @@ -183,9 +173,8 @@ async fn an_operator_the_field_has_no_index_for_is_refused_when_awaited() { assert_eq!( plan_error( cipher - .query("Alice") - .using(&plan.field("name").unwrap()) - .op(op::Eq) + .query(op::Eq, "Alice") + .on(&plan.field("name").unwrap()) .await ), PlanError::IndexNotDeclared { @@ -199,7 +188,7 @@ async fn an_operator_the_field_has_no_index_for_is_refused_when_awaited() { async fn a_one_value_plan_answers_from_its_indexes_by_type() { let cipher = stack_cipher().await; let plan = Plan::context("users/email") - .with::((Equality, Match::default())) + .with::((Equality, Match)) .build() .unwrap(); let stored: Encrypted<(EqualityTerm, MatchTerms)> = cipher @@ -209,16 +198,14 @@ async fn a_one_value_plan_answers_from_its_indexes_by_type() { .unwrap(); let eq = cipher - .query("bob@example.com") - .using(&plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&plan) .await .unwrap(); assert_eq!(eq, stored.terms.0); let matches = cipher - .query("example") - .using(&plan) - .op(op::Matches::new()) + .query(op::Matches, "example") + .on(&plan) .await .unwrap(); assert!(stored.terms.1.matches(&matches)); @@ -233,9 +220,9 @@ async fn a_one_value_plan_answers_from_its_indexes_by_type() { .await .unwrap(); let query = cipher - .query("Ali") - .using(&exact_plan) - .op_via(op::Matches::::new(), Match::::new()) + .query(op::Matches.case_sensitive(), "Ali") + .via(Match::::new()) + .on(&exact_plan) .await .unwrap(); assert!(name.terms.matches(&query)); @@ -255,9 +242,8 @@ async fn a_target_answers_an_operator_through_the_index_it_names() { .await .unwrap(); let query = cipher - .query("bob@example.com") - .using(&eq_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&eq_plan) .await .unwrap(); assert_eq!(query, stored); @@ -273,9 +259,8 @@ async fn a_target_answers_an_operator_through_the_index_it_names() { .await .unwrap(); let query = cipher - .query("bob") - .using(&match_plan) - .op(op::Matches::new()) + .query(op::Matches, "bob") + .on(&match_plan) .await .unwrap(); assert!(stored.matches(&query)); @@ -295,16 +280,16 @@ async fn a_target_with_several_terms_names_the_index_with_op_via() { .await .unwrap(); let eq = cipher - .query("bob@example.com") - .using(&plan) - .op_via(op::Eq, Equality) + .query(op::Eq, "bob@example.com") + .via(Equality) + .on(&plan) .await .unwrap(); assert_eq!(eq, stored.terms.0); let matches = cipher - .query("bob") - .using(&plan) - .op_via(op::Matches::new(), Match::default()) + .query(op::Matches, "bob") + .via(Match) + .on(&plan) .await .unwrap(); assert!(stored.terms.1.matches(&matches)); @@ -318,9 +303,9 @@ async fn a_target_with_several_terms_names_the_index_with_op_via() { assert_eq!( plan_error( cipher - .query("bob") - .using(&one_term) - .op_via(op::Matches::new(), Match::default()) + .query(op::Matches, "bob") + .via(Match) + .on(&one_term) .await ), PlanError::IndexNotDeclared { @@ -328,7 +313,7 @@ async fn a_target_with_several_terms_names_the_index_with_op_via() { index: "match" } ); - let eq = cipher.query("bob").using(&one_term).op(op::Eq).await; + let eq = cipher.query(op::Eq, "bob").on(&one_term).await; assert!(eq.is_ok(), "one term answers what the term answers"); } @@ -347,14 +332,12 @@ async fn operator_queries_settle_together_and_extend_like_the_write() { let (eq, matches) = stack_encrypt::all(( cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .extend(7u64), cipher - .query("bob") - .using(&email_plan) - .op(op::Matches::new()) + .query(op::Matches, "bob") + .on(&email_plan) .extend(7u64), )) .await @@ -363,9 +346,8 @@ async fn operator_queries_settle_together_and_extend_like_the_write() { assert!(email.terms.1.matches(&matches)); let unextended = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_ne!( @@ -380,7 +362,7 @@ async fn a_query_chain_debugs_without_its_value() { let plan = people_plan(); let email_plan = plan.field("email").unwrap(); let secret = "bob@example.com"; - let shown = format!("{:?}", cipher.query(secret).using(&email_plan).op(op::Eq)); + let shown = format!("{:?}", cipher.query(op::Eq, secret).on(&email_plan)); assert_eq!(shown, "QueryOp { .. }"); } @@ -396,7 +378,7 @@ impl MatchConfig for Mislabelled { } } impl MatchCase for Mislabelled { - type Case = op::Insensitive; + type Case = op::CaseInsensitive; } #[tokio::test] @@ -408,21 +390,21 @@ async fn a_match_case_that_disagrees_with_its_options_is_refused() { .unwrap(); let refused = plan_error( cipher - .query("Bob") - .using(&plan) - .op_via(op::Matches::new(), Match::::new()) + .query(op::Matches, "Bob") + .via(Match::::new()) + .on(&plan) .await, ); assert_eq!( refused, PlanError::MatchCase { - declared: "Insensitive", + declared: "CaseInsensitive", folds: false } ); assert_eq!( refused.to_string(), - "a match configuration declares the case Insensitive but its options keep case" + "a match configuration declares the case CaseInsensitive but its options keep case" ); } @@ -436,9 +418,8 @@ async fn a_refused_operator_query_loads_no_keyset() { .build() .unwrap(); let refused = cipher - .query("Alice") - .using(&plan.field("name").unwrap()) - .op(op::Eq) + .query(op::Eq, "Alice") + .on(&plan.field("name").unwrap()) .keyset("tenant") .await; assert_eq!( @@ -460,9 +441,8 @@ async fn a_match_query_debugs_as_its_positions() { let cipher = stack_cipher().await; let plan = people_plan(); let query = cipher - .query("bob") - .using(&plan.field("email").unwrap()) - .op(op::Matches::new()) + .query(op::Matches, "bob") + .on(&plan.field("email").unwrap()) .await .unwrap(); assert_eq!( @@ -485,17 +465,15 @@ async fn an_operator_query_runs_under_the_keyset_it_names() { .await .unwrap(); let under_tenant = cipher - .query("bob@example.com") - .using(&plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&plan) .keyset("tenant") .await .unwrap(); assert_eq!(under_tenant, stored.terms); let under_default = cipher - .query("bob@example.com") - .using(&plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&plan) .await .unwrap(); assert_ne!(under_default, stored.terms, "another keyset, another key"); diff --git a/packages/stack-encrypt/tests/plan.rs b/packages/stack-encrypt/tests/plan.rs index 76133e69d..d2f3ac879 100644 --- a/packages/stack-encrypt/tests/plan.rs +++ b/packages/stack-encrypt/tests/plan.rs @@ -102,7 +102,7 @@ type AgeTerms = (EqualityTerm, OreTerm); fn users_plan() -> Plan { Plan::context("users") .fields() - .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("email", (Equality, Match)) .encrypt_index::("age", (Equality, Ore)) .encrypt::("notes") .passthrough::("id") @@ -134,7 +134,7 @@ type HandOut = ( u64, ); fn hand_composed<'s, K: 'static>() -> Encryption<'s, User, HandOut, K, DeclaredContext> { - indexed::((Equality, Match::default())) + indexed::((Equality, Match)) .under(nonempty!("users").with("email")) .project(|u: &User| &u.email) .zip( @@ -159,7 +159,7 @@ async fn the_chain_is_the_hand_composed_combinators_field_for_field() { .encrypt(&user) .context("users") .fields() - .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("email", (Equality, Match)) .encrypt_index::("age", (Equality, Ore)) .encrypt::("notes") .passthrough::("id") @@ -255,7 +255,7 @@ async fn a_saved_plan_is_the_same_chain_and_decrypts_back() { .encrypt(&user) .context("users") .fields() - .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("email", (Equality, Match)) .encrypt_index::("age", (Equality, Ore)) .encrypt::("notes") .passthrough::("id") @@ -290,9 +290,8 @@ async fn a_query_derives_under_the_label_the_write_used() { let email_plan = users_plan.field("email").unwrap(); let age_plan = users_plan.field("age").unwrap(); let eq: EqualityTerm = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!( @@ -300,25 +299,18 @@ async fn a_query_derives_under_the_label_the_write_used() { "equality query bytes equal the stored term" ); let matching: MatchTerms = cipher - .query("bob@example.com") - .using(&email_plan) - .index(Match::default()) + .query_index(Match, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(matching.to_bytes(), email.terms.1.to_bytes()); - let ore: OreTerm = cipher - .query(&34u32) - .using(&age_plan) - .index(Ore) - .await - .unwrap(); + let ore: OreTerm = cipher.query_index(Ore, &34u32).on(&age_plan).await.unwrap(); assert_eq!(ore.to_bytes(), age.terms.1.to_bytes()); // A different value is a different term: the query is not a constant. let other = cipher - .query("eve@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "eve@example.com") + .on(&email_plan) .await .unwrap(); assert_ne!(other, email.terms.0); @@ -337,12 +329,12 @@ async fn a_query_extended_like_the_write_matches_it() { let email: Encrypted = row.take("email").unwrap(); let email_plan = users_plan.field("email").unwrap(); let query = |extend: Option| { - let chain = cipher.query("bob@example.com").using(&email_plan); + let chain = cipher.query(op::Eq, "bob@example.com"); let chain = match extend { Some(parts) => chain.extend(parts), None => chain, }; - chain.op(op::Eq) + chain.on(&email_plan) }; assert_eq!(query(Some(7)).await.unwrap(), email.terms.0); assert_ne!(query(None).await.unwrap(), email.terms.0); @@ -357,21 +349,21 @@ async fn a_query_against_an_undeclared_index_or_type_is_refused() { let notes_plan = users_plan.field("notes").unwrap(); assert_eq!( - plan_error(cipher.query(&34u32).using(&age_plan).index(Ope).await), + plan_error(cipher.query_index(Ope, &34u32).on(&age_plan).await), PlanError::IndexNotDeclared { field: "age".into(), index: "ope" } ); assert_eq!( - plan_error(cipher.query("x").using(¬es_plan).op(op::Eq).await), + plan_error(cipher.query(op::Eq, "x").on(¬es_plan).await), PlanError::IndexNotDeclared { field: "notes".into(), index: "eq" } ); assert_eq!( - plan_error(cipher.query(&34u64).using(&age_plan).op(op::Eq).await), + plan_error(cipher.query(op::Eq, &34u64).on(&age_plan).await), PlanError::FieldType { field: "age".into(), expected: "u32" @@ -403,19 +395,9 @@ async fn a_one_value_plan_writes_queries_by_type_and_reads_back() { assert_eq!(chained.terms.0, out.terms.0); assert_eq!(chained.terms.1.to_bytes(), out.terms.1.to_bytes()); - let eq = cipher - .query(&34u32) - .using(&age_plan) - .op(op::Eq) - .await - .unwrap(); + let eq = cipher.query(op::Eq, &34u32).on(&age_plan).await.unwrap(); assert_eq!(eq, out.terms.0); - let ore = cipher - .query(&34u32) - .using(&age_plan) - .index::() - .await - .unwrap(); + let ore = cipher.query_index(Ore, &34u32).on(&age_plan).await.unwrap(); assert_eq!(ore.to_bytes(), out.terms.1.to_bytes()); // Byte-identical to the combinator, under the same label. @@ -511,7 +493,7 @@ async fn a_one_value_plan_extended_on_write_needs_the_extension_to_query_and_ope assert_eq!(sent.lock().unwrap().generated(), ["(users/age)/7u64"]); let query = |parts: Option| { - let chain = cipher.query(&34u32).using(&age_plan).op(op::Eq); + let chain = cipher.query(op::Eq, &34u32).on(&age_plan); match parts { Some(parts) => chain.extend(parts), None => chain, @@ -546,7 +528,7 @@ async fn an_index_only_field_matches_its_query_and_mints_no_data_key() { let (cipher, sent) = recording_cipher().await; let tokens_plan: Plan = Plan::context("users") .fields() - .index::("email", (Equality, Match::default())) + .index::("email", (Equality, Match)) .encrypt::("age") .encrypt::("notes") .passthrough::("id") @@ -562,9 +544,8 @@ async fn an_index_only_field_matches_its_query_and_mints_no_data_key() { let email_plan = tokens_plan.field("email").unwrap(); let query = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(query, eq, "the query derives under the write's label"); @@ -862,9 +843,8 @@ async fn a_refused_chain_never_loads_the_keyset_it_names() { assert!(matches!( plan_error( cipher - .query(&34u32) - .using(&age_plan) - .index(Ope) + .query_index(Ope, &34u32) + .on(&age_plan) .keyset("tenant") .await ), @@ -964,13 +944,7 @@ async fn a_passthrough_name_need_not_be_a_label_segment() { let passthrough_plan = odd_plan.field("2fa_enabled").unwrap(); assert_eq!( - plan_error( - cipher - .query(&1u64) - .using(&passthrough_plan) - .op(op::Eq) - .await - ), + plan_error(cipher.query(op::Eq, &1u64).on(&passthrough_plan).await), PlanError::IndexNotDeclared { field: "2fa_enabled".into(), index: "eq" @@ -1230,9 +1204,8 @@ async fn match_options_must_agree_for_a_query_to_be_declared() { let email_plan = users_plan.field("email").unwrap(); let refused = plan_error( cipher - .query("bob") - .using(&email_plan) - .index(Match::::new()) + .query_index(Match::::new(), "bob") + .on(&email_plan) .await, ); assert_eq!( @@ -1260,10 +1233,7 @@ async fn all_settles_several_chains_in_one_request() { let (row, probe) = stack_encrypt::all(( cipher.encrypt(&user).using(&users_plan), - cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq), + cipher.query(op::Eq, "bob@example.com").on(&email_plan), )) .await .unwrap(); @@ -1748,9 +1718,14 @@ async fn a_chain_shows_no_value() { format!("{:?}", cipher.encrypt(&secret).context("x").with(Equality)), format!("{:?}", cipher.encrypt(&user()).context("x").fields()), format!("{:?}", cipher.encrypt(&user()).using(&users_plan)), - format!("{:?}", cipher.query(&secret)), - format!("{:?}", cipher.query(&secret).using(&email_plan)), - format!("{:?}", cipher.query(&secret).using(&email_plan).op(op::Eq)), + format!("{:?}", cipher.query(op::Eq, &secret)), + format!("{:?}", cipher.query(op::Eq, &secret).via(Equality)), + format!("{:?}", cipher.query(op::Eq, &secret).on(&email_plan)), + format!("{:?}", cipher.query_index(Equality, &secret)), + format!( + "{:?}", + cipher.query_index(Equality, &secret).on(&email_plan) + ), format!("{:?}", cipher.open(secret.clone())), format!("{:?}", cipher.open(FieldValues::new()).using(&users_plan)), format!( @@ -1873,16 +1848,14 @@ async fn every_chain_honours_its_named_keyset() { let email: Encrypted = row.take("email").unwrap(); let email_plan = users_plan.field("email").unwrap(); let under_tenant = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .keyset("tenant") .await .unwrap(); let under_default = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(under_tenant, email.terms.0); diff --git a/packages/stack-encrypt/tests/plan_grammar.rs b/packages/stack-encrypt/tests/plan_grammar.rs index 100c20f72..db9d94114 100644 --- a/packages/stack-encrypt/tests/plan_grammar.rs +++ b/packages/stack-encrypt/tests/plan_grammar.rs @@ -142,10 +142,9 @@ async fn a_plan_built_without_a_context_takes_the_calls() { let stored: Encrypted = record.take("email").unwrap(); let query_value = cipher - .query("bob@example.com") + .query(op::Eq, "bob@example.com") .context("tenants/acme") - .using(&email_plan) - .op(op::Eq) + .on(&email_plan) .await .unwrap(); assert_eq!( @@ -153,10 +152,9 @@ async fn a_plan_built_without_a_context_takes_the_calls() { "a query naming the same context matches" ); let elsewhere = cipher - .query("bob@example.com") + .query(op::Eq, "bob@example.com") .context("tenants/globex") - .using(&email_plan) - .op(op::Eq) + .on(&email_plan) .await .unwrap(); assert_ne!(elsewhere, stored.terms); @@ -186,9 +184,8 @@ async fn a_plan_run_with_no_context_is_refused_before_any_key_request() { assert_eq!( plan_error( cipher - .query("bob@example.com") - .using(&users_plan.field("email").unwrap()) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&users_plan.field("email").unwrap()) .await ), PlanError::NoContext @@ -211,7 +208,7 @@ async fn a_plan_run_with_no_context_is_refused_before_any_key_request() { PlanError::NoContext ); assert_eq!( - plan_error(cipher.query(&34u32).using(&age_plan).op(op::Eq).await), + plan_error(cipher.query(op::Eq, &34u32).on(&age_plan).await), PlanError::NoContext ); let age = cipher @@ -290,10 +287,9 @@ async fn a_context_given_twice_is_refused_at_the_call_before_any_key_request() { assert_eq!( plan_error( cipher - .query("bob@example.com") + .query(op::Eq, "bob@example.com") .context("users") - .using(&users_plan.field("email").unwrap()) - .op(op::Eq) + .on(&users_plan.field("email").unwrap()) .await ), two("the plan", "the call") @@ -340,10 +336,9 @@ async fn a_context_given_twice_is_refused_at_the_call_before_any_key_request() { assert_eq!( plan_error( cipher - .query(&34u32) + .query(op::Eq, &34u32) .context("users/age") - .using(&age_plan) - .op(op::Eq) + .on(&age_plan) .await ), two("the plan", "the call") @@ -366,10 +361,9 @@ async fn a_call_context_that_is_not_a_label_is_refused() { ))); assert!(bad(plan_error( cipher - .query("bob@example.com") + .query(op::Eq, "bob@example.com") .context("a//b") - .using(&users_plan.field("email").unwrap()) - .op(op::Eq) + .on(&users_plan.field("email").unwrap()) .await ))); assert!(bad(plan_error( @@ -837,10 +831,9 @@ async fn a_context_field_is_the_context_of_every_other_field() { // A query names the context field's value. let query_value = cipher - .query("bob@example.com") + .query(op::Eq, "bob@example.com") .context("tenants/acme") - .using(&records_plan.field("email").unwrap()) - .op(op::Eq) + .on(&records_plan.field("email").unwrap()) .await .unwrap(); assert_eq!(query_value, stored.terms); @@ -1091,7 +1084,7 @@ fn a_five_index_field_is_refused_by_both_spellings() { .fields::, FakeDataKeySource>() .encrypt_index( pick("v", |o: &One| &o.v), - (Equality, Match::default(), Ore, Ope, Match::::new()), + (Equality, Match, Ore, Ope, Match::::new()), ) .build(); assert_eq!(plan_error(data), refused()); @@ -1129,25 +1122,22 @@ async fn a_typed_field_answers_the_queries_its_target_declares_and_no_other() { let email_plan = users_plan.field("email").unwrap(); let query_value = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(query_value, stored.terms.0); let matched = cipher - .query("bob@example.com") - .using(&email_plan) - .index(Match::default()) + .query_index(Match, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(matched, stored.terms.1); assert_eq!( plan_error( cipher - .query("bob@example.com") - .using(&email_plan) - .index(Ore) + .query_index(Ore, "bob@example.com") + .on(&email_plan) .await ), PlanError::IndexNotDeclared { @@ -1158,9 +1148,8 @@ async fn a_typed_field_answers_the_queries_its_target_declares_and_no_other() { assert_eq!( plan_error( cipher - .query(&34u32) - .using(&users_plan.field("age").unwrap()) - .op(op::Eq) + .query(op::Eq, &34u32) + .on(&users_plan.field("age").unwrap()) .await ), PlanError::IndexNotDeclared { @@ -1184,9 +1173,8 @@ async fn a_typed_field_of_terms_alone_does_not_come_back() { let mut record = cipher.encrypt(&user()).using(&users_plan).await.unwrap(); let term: EqualityTerm = record.take("email").unwrap(); let query_value = cipher - .query("bob@example.com") - .using(&users_plan.field("email").unwrap()) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&users_plan.field("email").unwrap()) .await .unwrap(); assert_eq!(query_value, term); @@ -1333,18 +1321,16 @@ async fn a_one_value_typed_plan_answers_its_targets_queries_and_no_other() { let email = String::from("bob@example.com"); let (_, hm) = cipher.encrypt(&email).using(&email_plan).await.unwrap(); let query_value = cipher - .query("bob@example.com") - .using(&email_plan) - .op(op::Eq) + .query(op::Eq, "bob@example.com") + .on(&email_plan) .await .unwrap(); assert_eq!(query_value, hm); assert_eq!( plan_error( cipher - .query("bob@example.com") - .using(&email_plan) - .index(Ore) + .query_index(Ore, "bob@example.com") + .on(&email_plan) .await ), PlanError::IndexNotDeclared { @@ -1368,10 +1354,9 @@ async fn a_one_value_plan_quotes_no_call_context_in_its_refusals() { let error = plan_error( cipher - .query("bob@example.com") + .query_index(Ore, "bob@example.com") .context(marker) - .using(&plan) - .index(Ore) + .on(&plan) .await, ); assert_eq!( @@ -1631,9 +1616,8 @@ async fn a_pinned_identity_rescues_a_field_name_that_is_not_plain() { ); let value: Encrypted = record.take("0").unwrap(); let query_value = cipher - .query(&21u32) - .using(&readings_plan.field("0").unwrap()) - .op(op::Eq) + .query(op::Eq, &21u32) + .on(&readings_plan.field("0").unwrap()) .await .unwrap(); assert_eq!(query_value, value.terms); @@ -1684,10 +1668,9 @@ async fn a_pinned_identity_keys_a_call_context_plan_for_query_and_open() { assert_eq!(sent.lock().unwrap().generated(), ["reading/value"]); let stored: Encrypted = record.take("0").unwrap(); let query_value = cipher - .query(&21u32) + .query(op::Eq, &21u32) .context("reading") - .using(&readings_plan.field("0").unwrap()) - .op(op::Eq) + .on(&readings_plan.field("0").unwrap()) .await .unwrap(); assert_eq!(query_value, stored.terms); diff --git a/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.rs b/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.rs index 6b363cb74..e601f4dd2 100644 --- a/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.rs +++ b/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.rs @@ -4,5 +4,5 @@ use stack_encrypt::target::{indexed, Borrowed, Equality, Match}; use stack_kms::FakeDataKeySource; fn main() { - let _ = indexed::((Equality, Match::default())); + let _ = indexed::((Equality, Match)); } diff --git a/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.stderr b/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.stderr index cb38e177b..f2afb651f 100644 --- a/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.stderr +++ b/packages/stack-encrypt/tests/ui/match_in_a_tuple_on_integer.stderr @@ -1,8 +1,8 @@ error[E0277]: the trait bound `u32: AsRef` is not satisfied --> tests/ui/match_in_a_tuple_on_integer.rs:7:13 | -7 | let _ = indexed::((Equality, Match::default())); - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `AsRef` is not implemented for `u32` +7 | let _ = indexed::((Equality, Match)); + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `AsRef` is not implemented for `u32` | help: the trait `stack_encrypt::Index` is implemented for `stack_encrypt::Match` --> src/target/index.rs diff --git a/packages/stack-encrypt/tests/ui/match_on_integer.rs b/packages/stack-encrypt/tests/ui/match_on_integer.rs index 37298c7ee..39da28605 100644 --- a/packages/stack-encrypt/tests/ui/match_on_integer.rs +++ b/packages/stack-encrypt/tests/ui/match_on_integer.rs @@ -4,5 +4,5 @@ use stack_encrypt::target::{indexed, Borrowed, Match}; use stack_kms::FakeDataKeySource; fn main() { - let _ = indexed::(Match::default()); + let _ = indexed::(Match); } diff --git a/packages/stack-encrypt/tests/ui/match_on_integer.stderr b/packages/stack-encrypt/tests/ui/match_on_integer.stderr index 309411cc5..2843b7847 100644 --- a/packages/stack-encrypt/tests/ui/match_on_integer.stderr +++ b/packages/stack-encrypt/tests/ui/match_on_integer.stderr @@ -1,8 +1,8 @@ error[E0277]: the trait bound `u32: AsRef` is not satisfied --> tests/ui/match_on_integer.rs:7:13 | -7 | let _ = indexed::(Match::default()); - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `AsRef` is not implemented for `u32` +7 | let _ = indexed::(Match); + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `AsRef` is not implemented for `u32` | help: the trait `stack_encrypt::Index` is implemented for `stack_encrypt::Match` --> src/target/index.rs diff --git a/packages/stack-encrypt/tests/ui/op_case_mismatch.rs b/packages/stack-encrypt/tests/ui/op_case_mismatch.rs index ab1c0c2e4..6e1d2e9a5 100644 --- a/packages/stack-encrypt/tests/ui/op_case_mismatch.rs +++ b/packages/stack-encrypt/tests/ui/op_case_mismatch.rs @@ -1,18 +1,18 @@ //! The default match index folds case, so it answers only a //! case-insensitive `Matches`: asking it for a case-sensitive one does not -//! compile, rather than quietly folding the needle. +//! compile, rather than quietly folding the query. use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::{op, Match, Plan, StackCipher}; async fn query(cipher: &StackCipher) { let name_plan = Plan::context("users/name") - .with::(Match::default()) + .with::(Match) .build() .unwrap(); let _ = cipher - .query("Ali") - .using(&name_plan) - .op_via(op::Matches::::new(), Match::default()); + .query(op::Matches.case_sensitive(), "Ali") + .via(Match) + .on(&name_plan); } fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr b/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr index af453429e..fd7634477 100644 --- a/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr +++ b/packages/stack-encrypt/tests/ui/op_case_mismatch.stderr @@ -1,26 +1,37 @@ -error[E0308]: mismatched types - --> tests/ui/op_case_mismatch.rs:15:17 +error[E0271]: type mismatch resolving `::Case == CaseSensitive` + --> tests/ui/op_case_mismatch.rs:14:14 | -15 | .op_via(op::Matches::::new(), Match::default()); - | ------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Matches`, found `Matches` +14 | .via(Match) + | --- ^^^^^ expected `CaseSensitive`, found `CaseInsensitive` | | - | arguments to this method are incorrect + | required by a bound introduced by this call | - = note: expected struct `stack_encrypt::op::Matches` - found struct `stack_encrypt::op::Matches` -help: the return type of this call is `stack_encrypt::op::Matches` due to the type of the argument passed - --> tests/ui/op_case_mismatch.rs:12:13 +note: required by a bound in `QueryBuilder::<'a, Op, F, K>::via` + --> src/plan/chain.rs + | + | pub fn via>(self, index: I) -> QueryVia<'a, Op, F, I, K> { + | ^^^^^^^^^^^^^^^ required by this bound in `QueryBuilder::<'a, Op, F, K>::via` + +error[E0599]: the method `on` exists for struct `QueryVia<'_, stack_encrypt::op::Matches, std::string::String, stack_encrypt::Match, FakeDataKeySource>`, but its trait bounds were not satisfied + --> tests/ui/op_case_mismatch.rs:15:10 | 12 | let _ = cipher - | _____________^ -13 | | .query("Ali") -14 | | .using(&name_plan) -15 | | .op_via(op::Matches::::new(), Match::default()); - | |_________________-----------------------------------__________________^ - | | - | this argument influences the return type of `op_via` -note: method defined here - --> src/plan/chain.rs + | _____________- +13 | | .query(op::Matches.case_sensitive(), "Ali") +14 | | .via(Match) +15 | | .on(&name_plan); + | |_________-^^ + | + ::: src/target/index.rs + | + | pub struct Match { + | ---------------------------------- doesn't satisfy `_: Supports, String>` + | + ::: src/sem/mod.rs + | + | pub struct DefaultMatch; + | ----------------------- doesn't satisfy `<_ as MatchCase>::Case = CaseSensitive` | - | pub fn op_via(self, _op: Op, _index: I) -> QueryOp<'a, F, Op, I, K> - | ^^^^^^ + = note: the following trait bounds were not satisfied: + `::Case = CaseSensitive` + which is required by `stack_encrypt::Match: Supports, std::string::String>` diff --git a/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs b/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs index b92c7917c..8408d9b3f 100644 --- a/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs +++ b/packages/stack-encrypt/tests/ui/op_matches_on_integer.rs @@ -5,7 +5,7 @@ use stack_encrypt::{op, Equality, Plan, StackCipher}; async fn query(cipher: &StackCipher) { let age_plan = Plan::context("users/age").with::(Equality).build().unwrap(); - let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); + let _ = cipher.query(op::Matches, &34u32).on(&age_plan); } fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr b/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr index 966ebf8f9..5f15c0684 100644 --- a/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr +++ b/packages/stack-encrypt/tests/ui/op_matches_on_integer.stderr @@ -1,21 +1,73 @@ -error[E0271]: type mismatch resolving `>::Via == Equality` - --> tests/ui/op_matches_on_integer.rs:8:51 +error[E0277]: the trait bound `u32: AsRef` is not satisfied + --> tests/ui/op_matches_on_integer.rs:8:47 + | +8 | let _ = cipher.query(op::Matches, &34u32).on(&age_plan); + | ^^ the trait `AsRef` is not implemented for `u32` | -8 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); - | ^^ expected `Equality`, found `Match` +help: the following other types implement trait `Chooses` + --> src/plan/chain.rs | - = note: expected struct `stack_encrypt::Equality` - found struct `stack_encrypt::Match` + | / impl Chooses for ValuePlan> + | | where + | | Op: DefaultIndex, + | | Op::Via: Clone, + | | F: 'static, + | | X: Indexes + Select, + | |________________________________________^ `ValuePlan>` implements `Chooses` +... + | / impl Chooses for ValuePlan> + | | where + | | Op: Operator, + | | F: 'static, + | | T: Answers, + | | T::Via: Default, + | | Typed: ValueShape, + | |____________________________^ `ValuePlan>` implements `Chooses` + = note: required for `stack_encrypt::op::Matches` to implement `DefaultIndex` + = note: required for `ValuePlan>` to implement `Chooses` -error[E0277]: the trait bound `u32: AsRef` is not satisfied - --> tests/ui/op_matches_on_integer.rs:8:51 +error[E0277]: the trait bound `stack_encrypt::Equality: Select` is not satisfied + --> tests/ui/op_matches_on_integer.rs:8:47 | -8 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Matches::new()); - | ^^ the trait `AsRef` is not implemented for `u32` +8 | let _ = cipher.query(op::Matches, &34u32).on(&age_plan); + | ^^ the trait `Select` is not implemented for `stack_encrypt::Equality` | -help: the trait `DefaultIndex` is implemented for `stack_encrypt::op::Matches` - --> src/op.rs + = help: the following other types implement trait `Select`: + `(A, B)` implements `Select>` + `(A, B)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + and $N others + = note: required for `ValuePlan>` to implement `Chooses` + +error[E0277]: the trait bound `stack_encrypt::Equality: Select` is not satisfied + --> tests/ui/op_matches_on_integer.rs:8:50 | - | impl> DefaultIndex for Matches<::Case> { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - = note: required for `stack_encrypt::op::Matches` to implement `DefaultIndex` +8 | let _ = cipher.query(op::Matches, &34u32).on(&age_plan); + | -- ^^^^^^^^^ the trait `Select` is not implemented for `stack_encrypt::Equality` + | | + | required by a bound introduced by this call + | + = help: the following other types implement trait `Select`: + `(A, B)` implements `Select>` + `(A, B)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + and $N others + = note: required for `ValuePlan>` to implement `Chooses` +note: required by a bound in `QueryBuilder::<'a, Op, F, K>::on` + --> src/plan/chain.rs + | + | pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, P::Via, K> + | -- required by a bound in this associated function + | where + | P: Chooses, + | ^^^^^^^^^^^^^^^^^^ required by this bound in `QueryBuilder::<'a, Op, F, K>::on` diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs index 0b0bdc6e4..3dee1da94 100644 --- a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.rs @@ -10,7 +10,7 @@ async fn query(cipher: &StackCipher) { .encrypt_into::() .build() .unwrap(); - let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); + let _ = cipher.query(op::Matches, "bob").on(&email_plan); } fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr index 3a2dacda9..8fe0edf23 100644 --- a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr @@ -1,22 +1,15 @@ -error[E0308]: mismatched types - --> tests/ui/op_target_does_not_answer.rs:13:55 +error[E0277]: `EqualityTerm` does not answer `stack_encrypt::op::Matches` over `std::string::String` + --> tests/ui/op_target_does_not_answer.rs:13:46 | -13 | let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); - | -- ^^^^^^^^^^^^^^^^^^ expected `Eq`, found `Matches<_>` - | | - | arguments to this method are incorrect +13 | let _ = cipher.query(op::Matches, "bob").on(&email_plan); + | ^^ this target type does not answer the operator | - = note: expected struct `stack_encrypt::op::Eq` - found struct `stack_encrypt::op::Matches<_>` -help: the return type of this call is `stack_encrypt::op::Matches<_>` due to the type of the argument passed - --> tests/ui/op_target_does_not_answer.rs:13:13 + = note: a target answers the operators its type implements `Answers` for; `.via(index)` names an index the target declares +help: the trait `Answers` is not implemented for `EqualityTerm` + but trait `Answers` is implemented for it + --> src/op.rs | -13 | let _ = cipher.query("bob").using(&email_plan).op(op::Matches::new()); - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^------------------^ - | | - | this argument influences the return type of `op` -note: method defined here - --> src/plan/chain.rs - | - | pub fn op(self, op: Op) -> QueryOp<'a, F, Op, T::Via, K> - | ^^ + | impl Answers for EqualityTerm { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + = help: for that trait implementation, expected `stack_encrypt::op::Eq`, found `stack_encrypt::op::Matches` + = note: required for `ValuePlan>` to implement `Chooses` diff --git a/packages/stack-encrypt/tests/ui/pass/indexes.rs b/packages/stack-encrypt/tests/ui/pass/indexes.rs index 4079dd413..eca5a09c6 100644 --- a/packages/stack-encrypt/tests/ui/pass/indexes.rs +++ b/packages/stack-encrypt/tests/ui/pass/indexes.rs @@ -16,7 +16,7 @@ fn main() { let _ = indexed::(Equality); let _ = indexed::((Equality, Ore)); let _ = indexed::((Equality, Ore, Ope)); - let _ = indexed::(Match::default()); + let _ = indexed::(Match); let _ = indexed::(( Equality, Match::::new(), @@ -25,7 +25,7 @@ fn main() { )); let _ = indexed::(( Equality, - Match::default(), + Match, Ore, Ope, Match::::new(), diff --git a/packages/stack-encrypt/tests/ui/pass/plan_chain.rs b/packages/stack-encrypt/tests/ui/pass/plan_chain.rs index 4a51aa731..79347a38d 100644 --- a/packages/stack-encrypt/tests/ui/pass/plan_chain.rs +++ b/packages/stack-encrypt/tests/ui/pass/plan_chain.rs @@ -34,7 +34,7 @@ async fn chains(cipher: &StackCipher, person: &Person) -> Res .encrypt(person) .context("people") .fields() - .encrypt_index::("email", (Equality, Match::default())) + .encrypt_index::("email", (Equality, Match)) .encrypt::("age") .keyset("tenant") .extend(7u64) @@ -47,18 +47,18 @@ async fn chains(cipher: &StackCipher, person: &Person) -> Res let row = cipher.encrypt(person).using(&people_plan).await?; let _ = cipher.open(row).using(&people_plan).await?; let email_plan = people_plan.field("email")?; - let _ = cipher.query("bob@example.com").using(&email_plan).op(op::Eq).await?; + let _ = cipher.query(op::Eq, "bob@example.com").on(&email_plan).await?; let name_plan = Plan::context("people/name") - .with::(stack_encrypt::Match::default()) + .with::(stack_encrypt::Match) .build()?; - let _ = cipher.query("ali").using(&name_plan).op(op::Matches::new()).await?; + let _ = cipher.query(op::Matches, "ali").on(&name_plan).await?; let age_plan = Plan::context("people/age").with((Equality, Ore)).build()?; let ages = vec![1u32, 2]; let _ = cipher.encrypt(&ages).using(&age_plan).await?; - let _ = cipher.query(&3u32).using(&age_plan).index::().await?; + let _ = cipher.query_index(Ore, &3u32).on(&age_plan).await?; let _ = stack_encrypt::all(( cipher.encrypt(person).using(&people_plan), - cipher.query(&3u32).using(&age_plan).op(op::Eq), + cipher.query(op::Eq, &3u32).on(&age_plan), )) .await?; Ok(()) diff --git a/packages/stack-encrypt/tests/ui/plan_match_on_integer.rs b/packages/stack-encrypt/tests/ui/plan_match_on_integer.rs index ca5ba34f3..699d6163b 100644 --- a/packages/stack-encrypt/tests/ui/plan_match_on_integer.rs +++ b/packages/stack-encrypt/tests/ui/plan_match_on_integer.rs @@ -21,5 +21,5 @@ impl Field for User { fn main() { let _ = Plan::context("users") .fields::() - .encrypt_index::("age", Match::default()); + .encrypt_index::("age", Match); } diff --git a/packages/stack-encrypt/tests/ui/plan_match_on_integer.stderr b/packages/stack-encrypt/tests/ui/plan_match_on_integer.stderr index af4212052..35d448a70 100644 --- a/packages/stack-encrypt/tests/ui/plan_match_on_integer.stderr +++ b/packages/stack-encrypt/tests/ui/plan_match_on_integer.stderr @@ -1,7 +1,7 @@ error[E0277]: the trait bound `u32: AsRef` is not satisfied --> tests/ui/plan_match_on_integer.rs:24:10 | -24 | .encrypt_index::("age", Match::default()); +24 | .encrypt_index::("age", Match); | ^^^^^^^^^^^^^ the trait `AsRef` is not implemented for `u32` | help: the trait `stack_encrypt::Index` is implemented for `stack_encrypt::Match` diff --git a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs index ce1440102..26f9f5e92 100644 --- a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs +++ b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.rs @@ -6,7 +6,7 @@ use stack_encrypt::{Ore, Plan, StackCipher}; async fn query(cipher: &StackCipher) { let age_plan = Plan::context("users/age").with::(Ore).build().unwrap(); - let _ = cipher.query(&34u32).using(&age_plan).op(op::Eq); + let _ = cipher.query(op::Eq, &34u32).on(&age_plan); } fn main() {} diff --git a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr index 3a5c76063..a26c14f20 100644 --- a/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr +++ b/packages/stack-encrypt/tests/ui/plan_query_undeclared_index.stderr @@ -1,5 +1,45 @@ -error[E0271]: type mismatch resolving `>::Via == Ore` - --> tests/ui/plan_query_undeclared_index.rs:9:51 +error[E0277]: the trait bound `stack_encrypt::Ore: Select` is not satisfied + --> tests/ui/plan_query_undeclared_index.rs:9:42 | -9 | let _ = cipher.query(&34u32).using(&age_plan).op(op::Eq); - | ^^ expected `Ore`, found `Equality` +9 | let _ = cipher.query(op::Eq, &34u32).on(&age_plan); + | ^^ the trait `Select` is not implemented for `stack_encrypt::Ore` + | + = help: the following other types implement trait `Select`: + `(A, B)` implements `Select>` + `(A, B)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + and $N others + = note: required for `ValuePlan>` to implement `Chooses` + +error[E0277]: the trait bound `stack_encrypt::Ore: Select` is not satisfied + --> tests/ui/plan_query_undeclared_index.rs:9:45 + | +9 | let _ = cipher.query(op::Eq, &34u32).on(&age_plan); + | -- ^^^^^^^^^ the trait `Select` is not implemented for `stack_encrypt::Ore` + | | + | required by a bound introduced by this call + | + = help: the following other types implement trait `Select`: + `(A, B)` implements `Select>` + `(A, B)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + `(A, B, C, D)` implements `Select>` + and $N others + = note: required for `ValuePlan>` to implement `Chooses` +note: required by a bound in `QueryBuilder::<'a, Op, F, K>::on` + --> src/plan/chain.rs + | + | pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, P::Via, K> + | -- required by a bound in this associated function + | where + | P: Chooses, + | ^^^^^^^^^^^^^^^^^^ required by this bound in `QueryBuilder::<'a, Op, F, K>::on` From aa918613e58e83942dc6cbabb8e2f25fc9e8c881 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 22:16:54 +1100 Subject: [PATCH 11/38] feat(stack-encrypt): verify a match query's rows after decrypt A Bloom filter can hold every bit of a query without the value holding the needle, so a `Matches` query returns false positives. ADR-0009 decision 4 makes such a term `Probabilistic`, with `verify` as the operator's own meaning run on a decrypted value. - `op::Probabilistic { type Plaintext; fn verify(&self, &Plaintext) -> bool }`. Infallible: every outcome other than a match is `false`. - `MatchQuery` keeps the needle's tokens as the index cut them, through a new `Supports::carry` taken before the value is consumed. `verify` cuts the decrypted value the same way and checks every needle token is among its tokens. The tokens are zeroized on drop and never shown by `Debug`. - `Supports::Query` is bounded by `op::TermFor`: any query term answers `Eq`, only a probabilistic one answers `Matches`, so an index cannot claim a text match without saying how its answer is checked (tests/ui/op_inexact_operator_needs_probabilistic.rs). - `verify` on an exact term does not compile (tests/ui/op_verify_exact_term.rs). The tests plant a real Bloom false positive under a 32-bit filter and check `verify` drops it. Closes #1183 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 11 +- ...and-a-query-term-is-not-the-stored-term.md | 18 ++- packages/stack-encrypt/src/op.rs | 83 ++++++++++-- packages/stack-encrypt/src/plan/chain.rs | 3 +- packages/stack-encrypt/src/sem/mod.rs | 77 ++++++++++- packages/stack-encrypt/tests/op.rs | 126 +++++++++++++++--- ...op_inexact_operator_needs_probabilistic.rs | 35 +++++ ...nexact_operator_needs_probabilistic.stderr | 19 +++ .../tests/ui/op_verify_exact_term.rs | 16 +++ .../tests/ui/op_verify_exact_term.stderr | 5 + 10 files changed, 350 insertions(+), 43 deletions(-) create mode 100644 packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs create mode 100644 packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr create mode 100644 packages/stack-encrypt/tests/ui/op_verify_exact_term.rs create mode 100644 packages/stack-encrypt/tests/ui/op_verify_exact_term.stderr diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index d02d7bc31..e3e6f9161 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -36,6 +36,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`Match` is a value.** `.with((Equality, Match))` declares the match index under the default configuration, as `Equality` declares equality; `Match::::new()` is another configuration. +- **`verify` drops a match query's false positives after decrypt.** A + `MatchQuery` is `op::Probabilistic`: `query.verify(&value)` is `true` when + every token of the query's value, cut by the index's own tokenizer and + case folding, is a token of the decrypted value. Filter decrypted rows + with it. The query keeps its value's tokens to do this; they are zeroized on drop + and never shown by `Debug`. An exact term has no `verify`, and an index + can answer `op::Matches` only with a probabilistic term (`op::TermFor`). - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as @@ -85,9 +92,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `QueryBuilder`, `QueryVia`, `QueryIndexBuilder`, `QueryIndex` and `QueryOp`, and the plan shapes implement `plan::Chooses` and `plan::Resolves`. -- **`Match::default()` is `Match`.** The match index under the default +- **`Match` is `Match`.** The match index under the default configuration is the value `Match`, and `Default` is now implemented for - every `Match`, so `Match::default()` no longer infers the configuration. + every `Match`, so `Match` no longer infers the configuration. - **Update pattern matches for `dynamic::Error`.** `Context`, `Plan`, `Source`, and `Record` now carry diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 0cd8e8898..ad21cf9e1 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -144,9 +144,11 @@ query against a case-folding index is a type error, not an empty result. pub trait Operator: Copy + 'static {} pub trait Supports: Index { - type Query: QueryTerm; // what this index sends for this operator + type Query: TermFor; // what this index sends for `Op` + type Carry; // what the query keeps of the value fn check(&self) -> Result<(), PlanError>; - fn query(term: Self::Term, token: QueryToken) -> Self::Query; + fn carry(&self, value: &S) -> Self::Carry; + fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query; } ``` @@ -164,10 +166,14 @@ ciphertext; its `Term` is left and right. A Bloom index's `Supports::Query` is a filter over the needle (`MatchQuery`); its `Term` is a filter over the value (`MatchTerms`). -The bound tightens per operator. `Supports` requires -`Query: Probabilistic`; `Supports` requires `Query: Ranked`. An -index cannot claim an inexact operator without saying how its answer is -checked. +The bound tightens per operator, through `TermFor`: any `QueryTerm` +answers `Eq`, but only a `Probabilistic` one answers `Matches`, and only a +`Ranked` one will answer `Nearest`. An index cannot claim an inexact +operator without saying how its answer is checked. + +`carry` takes what the query term keeps of the value before the value is +consumed: nothing for an exact term, the query's tokens for a match query, +which `verify` needs and the filter cannot give back. ### 3. The domain type chooses the best index for an operator diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op.rs index 5f1d26d2e..b14b44ba2 100644 --- a/packages/stack-encrypt/src/op.rs +++ b/packages/stack-encrypt/src/op.rs @@ -55,7 +55,9 @@ use std::fmt; use std::marker::PhantomData; -use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms}; +use stack_kms::MaybeSend; + +use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens}; use crate::target::{Equality, Index, Match}; use crate::PlanError; @@ -199,13 +201,57 @@ pub trait QueryTerm {} impl QueryTerm for EqualityTerm {} impl QueryTerm for MatchQuery {} +/// A query term whose answer can hold **false positives and no false +/// negatives**: a row the server returns may not match, and a row that +/// matches is always returned. +/// +/// [`verify`](Self::verify) is the operator's own meaning, run on a +/// decrypted value: it decides whether a row the server returned is a real +/// match. The library never runs a query, so it cannot drop the false +/// positives itself; a caller filters the decrypted rows with `verify`. +/// +/// An exact term does not implement this, so `verify` on one is a compile +/// error rather than a method that is always `true`. +pub trait Probabilistic: QueryTerm { + /// The decrypted value `verify` reads: the plaintext the index was + /// declared over, which is not always the value the user wrote. + type Plaintext: ?Sized; + + /// Whether `value`, a row's decrypted value, really matches the query. + /// + /// Infallible: it runs on a value already decrypted and already the type + /// the index was declared over, so every outcome other than a match is + /// `false`, never an error. + fn verify(&self, value: &Self::Plaintext) -> bool; +} + +/// The query terms that may answer the operator `Op`: what +/// [`Supports::Query`] must be. +/// +/// Most operators take any [`QueryTerm`]. One whose answer is inherently +/// approximate requires more: a term that answers [`Matches`](struct@Matches) must be +/// [`Probabilistic`], so an index cannot claim a text match without saying +/// how its answer is checked. An operator defined in another crate states +/// its own requirement by implementing this. +#[diagnostic::on_unimplemented( + message = "`{Self}` cannot answer `{Op}`", + label = "this query term does not meet the operator's requirement", + note = "a term that answers `op::Matches` must implement `op::Probabilistic`" +)] +pub trait TermFor: QueryTerm {} + +impl TermFor for T {} +impl TermFor> for T {} + /// An index that answers the operator `Op` over plaintexts `S`, and the /// query term it sends for it. /// /// [`Index::Term`] is what the index derives; [`query`](Self::query) makes -/// the query term from it. For an equality index the two are the same -/// HMAC. For a match index they are not: the query is a filter over the -/// query's value, and its type says so. +/// the query term from it and from what [`carry`](Self::carry) kept of the +/// query's value. For an equality index the two are the same HMAC and +/// nothing is kept. For a match index they are not: the query is a filter +/// over the query's value, and it keeps that value's tokens so it can +/// [`verify`](Probabilistic::verify) a row. #[diagnostic::on_unimplemented( message = "`{Self}` does not answer `{Op}` over `{S}`", label = "this index cannot answer the operator", @@ -213,15 +259,20 @@ impl QueryTerm for MatchQuery {} )] pub trait Supports: Index { /// What this index sends for `Op`. - type Query: QueryTerm; + type Query: TermFor; + /// What the query term keeps of the query's value, taken before the + /// value is consumed: `()` for an exact term. + type Carry: MaybeSend + 'static; /// Why this index cannot answer `Op` as configured, if it cannot: a /// match index whose [`MatchCase`] disagrees with its options. fn check(&self) -> Result<(), PlanError>; - /// The query term, from the term the index derived for the query's - /// value. Only a query chain calls this: `QueryToken` cannot be made - /// outside this crate, so a stored term cannot be passed off as a - /// query. - fn query(term: Self::Term, token: QueryToken) -> Self::Query; + /// Take what the query term keeps of `value`. + fn carry(&self, value: &S) -> Self::Carry; + /// The query term, from what was kept and the term the index derived + /// for the query's value. Only a query chain calls this: `QueryToken` + /// cannot be made outside this crate, so no caller can pair one query's tokens + /// with another value's filter, or pass a stored term off as a query. + fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query; } /// Proof that [`Supports::query`] is called by a query chain. Only this @@ -238,10 +289,12 @@ impl QueryToken { impl Supports for Equality { type Query = EqualityTerm; + type Carry = (); fn check(&self) -> Result<(), PlanError> { Ok(()) } - fn query(term: EqualityTerm, _: QueryToken) -> EqualityTerm { + fn carry(&self, _: &S) {} + fn query((): (), term: EqualityTerm, _: QueryToken) -> EqualityTerm { term } } @@ -252,6 +305,7 @@ where O: MatchCase, { type Query = MatchQuery; + type Carry = QueryTokens; fn check(&self) -> Result<(), PlanError> { let folds = O::options().downcase; if folds == O::Case::FOLDS { @@ -263,8 +317,11 @@ where }) } } - fn query(term: MatchTerms, _: QueryToken) -> MatchQuery { - MatchQuery::from_terms(term) + fn carry(&self, value: &S) -> QueryTokens { + QueryTokens::of::(value.as_ref()) + } + fn query(tokens: QueryTokens, term: MatchTerms, _: QueryToken) -> MatchQuery { + MatchQuery::new(term, tokens) } } diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 1c7181829..6c9c880bc 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -980,9 +980,10 @@ where where 'a: 'p, { + let carry = self.inner.index.carry(&self.inner.value); self.inner .prepare(keyset, scoped) - .map(|term| I::query(term, QueryToken::new())) + .map(move |term| I::query(carry, term, QueryToken::new())) } } diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 2dba9df2c..004ac89db 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -128,6 +128,7 @@ mod tokenize; pub use tokenize::Tokenizer; +use std::collections::HashSet; use std::fmt; use std::marker::PhantomData; @@ -142,7 +143,7 @@ use vitaminc_prf::{ PrfVisitorError, SeqAccess, }; use vitaminc_protected::NonEmpty; -use zeroize::Zeroize; +use zeroize::{Zeroize, Zeroizing}; use stack_kms::MaybeSend; @@ -652,14 +653,24 @@ impl Eq for MatchTerms {} /// stored filter is never sent as a query. ADR-0009 records why. A /// `MatchQuery` comes from a query chain ([`op::Matches`](struct@crate::op::Matches)), /// never from a target. +/// +/// A Bloom filter can hold every bit of the query without the value holding +/// the query's text, so a match query is +/// [`Probabilistic`](crate::op::Probabilistic): it keeps the query's tokens, +/// as the index cut them, and [`verify`](crate::op::Probabilistic::verify) +/// checks a decrypted value against them. The tokens are plaintext: they +/// never leave the process, never appear in `Debug`, and are zeroized when +/// the query is dropped. pub struct MatchQuery { terms: MatchTerms, + tokens: QueryTokens, } impl MatchQuery { - /// The query term of the filter `terms`, derived over the query's value. - pub(crate) fn from_terms(terms: MatchTerms) -> Self { - Self { terms } + /// The query term of the filter `terms`, derived over the value whose + /// tokens are `tokens`. + pub(crate) fn new(terms: MatchTerms, tokens: QueryTokens) -> Self { + Self { terms, tokens } } /// The set positions of the query's filter, sorted ascending, no @@ -679,11 +690,32 @@ impl MatchQuery { } } +/// A row is a real match when every token of the query is a token of the +/// value, cut by the index's own tokenizer and case folding. That is the +/// meaning a Bloom match approximates, so `verify` keeps every row the +/// filter would have kept had it no false positives: order and repetition +/// do not count, and a match is not a substring test. +impl crate::op::Probabilistic for MatchQuery { + type Plaintext = str; + + fn verify(&self, value: &str) -> bool { + let options = O::options(); + let tokens = Zeroizing::new(tokenize::tokenize( + value, + options.tokenizer, + options.downcase, + )); + let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); + !self.tokens.0.is_empty() && self.tokens.0.iter().all(|t| have.contains(t.as_str())) + } +} + +/// Positions only: the tokens are plaintext. impl fmt::Debug for MatchQuery { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("MatchQuery") .field("positions", &self.terms.positions) - .finish() + .finish_non_exhaustive() } } @@ -691,10 +723,14 @@ impl Clone for MatchQuery { fn clone(&self) -> Self { Self { terms: self.terms.clone(), + tokens: self.tokens.clone(), } } } +/// Two queries are equal when they send the same filter. Under one key and +/// context the filter is a function of the query's tokens, so this never +/// compares the plaintext. impl PartialEq for MatchQuery { fn eq(&self, other: &Self) -> bool { self.terms == other.terms @@ -703,6 +739,24 @@ impl PartialEq for MatchQuery { impl Eq for MatchQuery {} +/// The query's tokens, as a match index cuts them from the query's value: +/// what a [`MatchQuery`] keeps to verify a row. Plaintext, so it is +/// zeroized on drop and has no `Debug`. Built only by a query chain. +#[derive(Clone)] +pub struct QueryTokens(Zeroizing>); + +impl QueryTokens { + /// The tokens of `text` under `O`'s tokenizer and case folding. + pub(crate) fn of(text: &str) -> Self { + let options = O::options(); + Self(Zeroizing::new(tokenize::tokenize( + text, + options.tokenizer, + options.downcase, + ))) + } +} + /// A [`PrfVisitor`] that folds a sequence of per-token PRF blocks into /// Bloom-filter bit positions: `k` little-endian 2-byte slices of each block, /// masked to the filter size. @@ -1277,6 +1331,19 @@ mod tests { use super::*; + #[test] + fn a_match_query_with_no_tokens_verifies_nothing() { + // A query chain refuses token-less text before a term exists, so + // this cannot be built through the API; the guard is for a query + // built any other way, which must never match every row. + let query = MatchQuery::::new( + MatchTerms::normalised(vec![1, 2, 3]), + QueryTokens(Zeroizing::new(Vec::new())), + ); + assert!(!crate::op::Probabilistic::verify(&query, "anything at all")); + assert!(!crate::op::Probabilistic::verify(&query, "")); + } + /// The Bloom fold reads a token *stream*: handed a map-shaped PRF input /// it refuses, rather than folding the entries' blocks as if they were /// tokens. diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 487ffe056..aa855c312 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -6,7 +6,7 @@ mod common; use common::{loads_counting_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; -use stack_encrypt::op::{self, MatchCase}; +use stack_encrypt::op::{self, MatchCase, Probabilistic}; use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms}; use stack_encrypt::target::{Encrypted, IndexSpec}; @@ -436,21 +436,6 @@ async fn a_refused_operator_query_loads_no_keyset() { ); } -#[tokio::test] -async fn a_match_query_debugs_as_its_positions() { - let cipher = stack_cipher().await; - let plan = people_plan(); - let query = cipher - .query(op::Matches, "bob") - .on(&plan.field("email").unwrap()) - .await - .unwrap(); - assert_eq!( - format!("{query:?}"), - format!("MatchQuery {{ positions: {:?} }}", query.positions()) - ); -} - #[tokio::test] async fn an_operator_query_runs_under_the_keyset_it_names() { let cipher = stack_cipher().await; @@ -478,3 +463,112 @@ async fn an_operator_query_runs_under_the_keyset_it_names() { .unwrap(); assert_ne!(under_default, stored.terms, "another keyset, another key"); } + +/// A match configuration with the smallest filter allowed, so a long value +/// sets nearly every bit and false positives are easy to find. +struct Tiny; +impl MatchConfig for Tiny { + fn options() -> MatchOptions { + MatchOptions { + m: 32, + ..Default::default() + } + } +} +impl MatchCase for Tiny { + type Case = op::CaseInsensitive; +} + +#[tokio::test] +async fn verify_keeps_every_real_match_and_folds_case_as_the_index_does() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let email_plan = plan.field("email").unwrap(); + let name_plan = plan.field("name").unwrap(); + + let query = cipher + .query(op::Matches, "BOB@Example") + .on(&email_plan) + .await + .unwrap(); + assert!(query.verify("bob@example.com")); + assert!(query.verify(&String::from("say hi to Bob@example.org"))); + assert!(!query.verify("alice@example.com")); + assert!(query.clone().verify("bob@example.com")); + + let exact = cipher + .query(op::Matches.case_sensitive(), "Nguyen") + .via(Match::::new()) + .on(&name_plan) + .await + .unwrap(); + assert!(exact.verify("Alice Nguyen")); + assert!( + !exact.verify("alice nguyen"), + "a case-sensitive index keeps case" + ); +} + +#[tokio::test] +async fn verify_drops_a_row_the_filter_matched_by_chance() { + let cipher = stack_cipher().await; + let plan = Plan::context("notes/body") + .with::(Match::::new()) + .build() + .unwrap(); + let value = "the quick brown fox jumps over the lazy dog and keeps running"; + let stored: Encrypted> = cipher + .encrypt(&value.to_owned()) + .using(&plan) + .await + .unwrap(); + + let real = cipher + .query(op::Matches, "lazy dog") + .via(Match::::new()) + .on(&plan) + .await + .unwrap(); + assert!(stored.terms.matches(&real) && real.verify(value)); + + // The value holds no digit, so no query with one is in it; a 32-bit + // filter this full holds some of their filters anyway. + let mut false_positive = None; + for i in 0..1000 { + let digits = format!("{i:04}"); + let query = cipher + .query(op::Matches, &digits) + .via(Match::::new()) + .on(&plan) + .await + .unwrap(); + if stored.terms.matches(&query) { + false_positive = Some(query); + break; + } + } + let query = false_positive.expect("a 32-bit filter this full matches some query by chance"); + assert!( + !query.verify(value), + "the server would return this row; verify does not keep it" + ); +} + +#[tokio::test] +async fn a_match_query_debugs_without_its_tokens() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let query = cipher + .query(op::Matches, "needle") + .on(&plan.field("email").unwrap()) + .await + .unwrap(); + let shown = format!("{query:?}"); + assert_eq!( + shown, + format!("MatchQuery {{ positions: {:?}, .. }}", query.positions()) + ); + for token in ["nee", "eed", "edl", "dle", "needle"] { + assert!(!shown.contains(token), "{shown}"); + } +} diff --git a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs new file mode 100644 index 000000000..c9509d133 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs @@ -0,0 +1,35 @@ +//! A text match is approximate, so an index that answers `Matches` must send +//! a term that says how its answer is checked: an exact term cannot answer +//! it. +use stack_encrypt::op; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::target::{CallerContext, ConsumeSource, Encryption, Index, IndexSpec}; +use stack_encrypt::Equality; + +struct Shortcut; + +impl Index for Shortcut { + type Term = EqualityTerm; + fn spec(&self) -> IndexSpec { + IndexSpec::Equality + } + fn operation<'s, K: 'static, M: ConsumeSource<'s, String>>( + &self, + ) -> Encryption<'s, String, EqualityTerm, K, CallerContext, M> { + Index::::operation(&Equality) + } +} + +impl op::Supports for Shortcut { + type Query = EqualityTerm; + type Carry = (); + fn check(&self) -> Result<(), stack_encrypt::PlanError> { + Ok(()) + } + fn carry(&self, _: &String) {} + fn query((): (), term: EqualityTerm, _: op::QueryToken) -> EqualityTerm { + term + } +} + +fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr new file mode 100644 index 000000000..ec190c313 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr @@ -0,0 +1,19 @@ +error[E0277]: `EqualityTerm` cannot answer `stack_encrypt::op::Matches` + --> tests/ui/op_inexact_operator_needs_probabilistic.rs:24:18 + | +24 | type Query = EqualityTerm; + | ^^^^^^^^^^^^ this query term does not meet the operator's requirement + | + = help: the trait `Probabilistic` is not implemented for `EqualityTerm` + = note: a term that answers `op::Matches` must implement `op::Probabilistic` +help: the trait `Probabilistic` is implemented for `MatchQuery` + --> src/sem/mod.rs + | + | impl crate::op::Probabilistic for MatchQuery { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + = note: required for `EqualityTerm` to implement `TermFor` +note: required by a bound in `stack_encrypt::op::Supports::Query` + --> src/op.rs + | + | type Query: TermFor; + | ^^^^^^^^^^^ required by this bound in `Supports::Query` diff --git a/packages/stack-encrypt/tests/ui/op_verify_exact_term.rs b/packages/stack-encrypt/tests/ui/op_verify_exact_term.rs new file mode 100644 index 000000000..d83af5055 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_verify_exact_term.rs @@ -0,0 +1,16 @@ +//! An exact query term has nothing to verify: the server's answer is the +//! answer, so `verify` on an equality term does not compile. +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::op::{self, Probabilistic}; +use stack_encrypt::{Equality, Plan, StackCipher}; + +async fn query(cipher: &StackCipher) { + let plan = Plan::context("users/email") + .with::(Equality) + .build() + .unwrap(); + let term = cipher.query(op::Eq, "bob").on(&plan).await.unwrap(); + let _ = term.verify("bob"); +} + +fn main() {} diff --git a/packages/stack-encrypt/tests/ui/op_verify_exact_term.stderr b/packages/stack-encrypt/tests/ui/op_verify_exact_term.stderr new file mode 100644 index 000000000..2ffdf93b3 --- /dev/null +++ b/packages/stack-encrypt/tests/ui/op_verify_exact_term.stderr @@ -0,0 +1,5 @@ +error[E0599]: no method named `verify` found for struct `EqualityTerm` in the current scope + --> tests/ui/op_verify_exact_term.rs:13:18 + | +13 | let _ = term.verify("bob"); + | ^^^^^^ method not found in `EqualityTerm` From 936f345c19839c3bae7293e8530b17c944a209b7 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 23:02:14 +1100 Subject: [PATCH 12/38] fix(stack-encrypt): address the review of match verify - `Supports::query` takes the sealed `QueryToken` (from the #1190 fix), so no caller can pair a needle with another value's filter and have `verify` drop the real rows. `Needle`'s doc says how one is joined. - `TermFor`'s doc and ADR-0009 say the "only a probabilistic term answers `Matches`" rule binds this crate's terms; a term in another crate can opt in by hand. - The tokenizer zeroizes its own copies of the text: the folded string, its characters, and each n-gram, built at its final size so it does not reallocate. `verify` runs it on every decrypted row. - `MatchQuery`'s `PartialEq` says two needles can share a filter and still verify differently. - ADR-0009 decision 4's code shows the trait as this PR ships it; `target_limit` joins it with #1184. - Tests: `verify` is a token test, not a substring test ("abcd" verifies "abc bcd"), and the standard tokenizer verifies whole words in any order. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 8 ++- ...and-a-query-term-is-not-the-stored-term.md | 9 ++- packages/stack-encrypt/src/op.rs | 11 +++- packages/stack-encrypt/src/sem/mod.rs | 19 +++++-- packages/stack-encrypt/src/sem/tokenize.rs | 23 ++++++-- packages/stack-encrypt/tests/op.rs | 56 ++++++++++++++++++- 6 files changed, 105 insertions(+), 21 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index e3e6f9161..bd3e71a7c 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -40,9 +40,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `MatchQuery` is `op::Probabilistic`: `query.verify(&value)` is `true` when every token of the query's value, cut by the index's own tokenizer and case folding, is a token of the decrypted value. Filter decrypted rows - with it. The query keeps its value's tokens to do this; they are zeroized on drop - and never shown by `Debug`. An exact term has no `verify`, and an index - can answer `op::Matches` only with a probabilistic term (`op::TermFor`). + with it. The query keeps its value's tokens to do this; they are zeroized + on drop and never shown by `Debug`, and the tokenizer now zeroizes its own + copies of the text. An exact term has no `verify`, and none of this + crate's indexes can answer `op::Matches` but with a probabilistic term + (`op::TermFor`). - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index ad21cf9e1..f3992ac95 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -168,8 +168,10 @@ ciphertext; its `Term` is left and right. A Bloom index's The bound tightens per operator, through `TermFor`: any `QueryTerm` answers `Eq`, but only a `Probabilistic` one answers `Matches`, and only a -`Ranked` one will answer `Nearest`. An index cannot claim an inexact -operator without saying how its answer is checked. +`Ranked` one will answer `Nearest`. None of this crate's indexes can claim +an inexact operator without saying how its answer is checked. The rule binds +this crate's terms through blanket impls; a term in another crate can opt +into `TermFor` by hand, so there it is a guide rather than a proof. `carry` takes what the query term keeps of the value before the value is consumed: nothing for an exact term, the query's tokens for a match query, @@ -215,7 +217,8 @@ pub trait QueryTerm {} pub trait Probabilistic: QueryTerm { type Plaintext; fn verify(&self, value: &Self::Plaintext) -> bool; - fn target_limit(&self, want: u64, stats: &Stats) -> Limit; + // `target_limit(&self, want, &Stats) -> Limit` joins it with decision 5 + // (#1184). } pub trait Ranked: QueryTerm { diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op.rs index b14b44ba2..08eb45082 100644 --- a/packages/stack-encrypt/src/op.rs +++ b/packages/stack-encrypt/src/op.rs @@ -230,9 +230,14 @@ pub trait Probabilistic: QueryTerm { /// /// Most operators take any [`QueryTerm`]. One whose answer is inherently /// approximate requires more: a term that answers [`Matches`](struct@Matches) must be -/// [`Probabilistic`], so an index cannot claim a text match without saying -/// how its answer is checked. An operator defined in another crate states -/// its own requirement by implementing this. +/// [`Probabilistic`], so none of this crate's indexes can claim a text +/// match without saying how its answer is checked. +/// +/// The rule binds this crate's terms, through the blanket impls below. A +/// term defined in another crate can implement this for an operator by +/// hand, and coherence lets it; the bound is a guide there, not a proof. An +/// operator defined in another crate states its requirement the same way, +/// one term type at a time. #[diagnostic::on_unimplemented( message = "`{Self}` cannot answer `{Op}`", label = "this query term does not meet the operator's requirement", diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 004ac89db..60a725f47 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -728,9 +728,11 @@ impl Clone for MatchQuery { } } -/// Two queries are equal when they send the same filter. Under one key and -/// context the filter is a function of the query's tokens, so this never -/// compares the plaintext. +/// Two queries are equal when they send the same filter. This compares no +/// plaintext. Two query values can share a filter by chance, as a short +/// value under a small filter does, and those queries are equal here yet +/// verify differently: equal queries select the same rows, not the same +/// matches. impl PartialEq for MatchQuery { fn eq(&self, other: &Self) -> bool { self.terms == other.terms @@ -741,7 +743,12 @@ impl Eq for MatchQuery {} /// The query's tokens, as a match index cuts them from the query's value: /// what a [`MatchQuery`] keeps to verify a row. Plaintext, so it is -/// zeroized on drop and has no `Debug`. Built only by a query chain. +/// zeroized on drop and has no `Debug`. +/// [`Supports::carry`](crate::op::Supports::carry) makes one; only a query +/// chain can join it to a filter +/// ([`Supports::query`](crate::op::Supports::query) takes a token no caller +/// can make), so one query's tokens never verify rows for another value's +/// filter. #[derive(Clone)] pub struct QueryTokens(Zeroizing>); @@ -1334,8 +1341,8 @@ mod tests { #[test] fn a_match_query_with_no_tokens_verifies_nothing() { // A query chain refuses token-less text before a term exists, so - // this cannot be built through the API; the guard is for a query - // built any other way, which must never match every row. + // this cannot reach a `MatchQuery` through the API; the guard is for + // one built any other way, which must never match every row. let query = MatchQuery::::new( MatchTerms::normalised(vec![1, 2, 3]), QueryTokens(Zeroizing::new(Vec::new())), diff --git a/packages/stack-encrypt/src/sem/tokenize.rs b/packages/stack-encrypt/src/sem/tokenize.rs index 1d5986fe5..8639de231 100644 --- a/packages/stack-encrypt/src/sem/tokenize.rs +++ b/packages/stack-encrypt/src/sem/tokenize.rs @@ -10,6 +10,8 @@ //! is that empty tokens are dropped (v1 keeps the empty strings its separator //! split produces, which only add noise bits to every filter). +use zeroize::Zeroizing; + /// How text is split into tokens before each token is run through the PRF. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum Tokenizer { @@ -45,12 +47,17 @@ impl Default for Tokenizer { /// term can never reach a query. /// /// [`match_terms`]: crate::StackCipher::match_terms +/// +/// The text is plaintext, so every copy made on the way is zeroized when it +/// is dropped: the folded text, its characters, and each n-gram, which is +/// built at its final size so it never reallocates. The tokens returned are +/// the caller's to zeroize. pub(crate) fn tokenize(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec { - let text = if downcase { + let text = Zeroizing::new(if downcase { text.to_lowercase() } else { text.to_string() - }; + }); match tokenizer { Tokenizer::Standard => text @@ -59,19 +66,27 @@ pub(crate) fn tokenize(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec< .map(str::to_string) .collect(), Tokenizer::Ngram { length } => { - let chars: Vec = text.chars().collect(); + let chars: Zeroizing> = Zeroizing::new(text.chars().collect()); // As in the v1 indexer's `process_ngram`: shorter than one gram // means no tokens (never a partial or whole-text token, which // could not match any stored gram). if chars.len() < length { Vec::new() } else { - chars.windows(length).map(|w| w.iter().collect()).collect() + chars.windows(length).map(gram).collect() } } } } +/// One n-gram as a `String` allocated at its final size, so building it +/// leaves no reallocated copy behind. +fn gram(chars: &[char]) -> String { + let mut gram = String::with_capacity(chars.iter().map(|c| c.len_utf8()).sum()); + gram.extend(chars); + gram +} + #[cfg(test)] mod tests { use super::*; diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index aa855c312..81f0aced5 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -133,7 +133,7 @@ async fn a_matches_query_is_a_filter_over_the_query_value_that_the_stored_filter } #[tokio::test] -async fn a_match_index_with_its_own_configuration_is_named_with_op_via() { +async fn a_match_index_with_its_own_configuration_is_named_with_via() { let cipher = stack_cipher().await; let plan = people_plan(); let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap(); @@ -267,7 +267,7 @@ async fn a_target_answers_an_operator_through_the_index_it_names() { } #[tokio::test] -async fn a_target_with_several_terms_names_the_index_with_op_via() { +async fn a_target_with_several_terms_names_the_index_with_via() { let cipher = stack_cipher().await; let plan = Plan::value::() .context("users/email") @@ -572,3 +572,55 @@ async fn a_match_query_debugs_without_its_tokens() { assert!(!shown.contains(token), "{shown}"); } } + +/// Whole words, as the v1 match indexer's standard tokenizer splits them. +struct Words; +impl MatchConfig for Words { + fn options() -> MatchOptions { + MatchOptions { + tokenizer: stack_encrypt::sem::Tokenizer::Standard, + ..Default::default() + } + } +} +impl MatchCase for Words { + type Case = op::CaseInsensitive; +} + +/// A Bloom match asks for the query's tokens, in any order and anywhere; +/// it is not a substring test, and `verify` must not become one, or it +/// would drop rows the index rightly returned. +#[tokio::test] +async fn verify_is_a_token_test_not_a_substring_test() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let query = cipher + .query(op::Matches, "abcd") + .on(&plan.field("email").unwrap()) + .await + .unwrap(); + assert!(query.verify("abc bcd"), "both 3-grams, never adjacent"); + assert!(!query.verify("abc"), "one 3-gram of two"); + + let words = Plan::context("notes/body") + .with::(Match::::new()) + .build() + .unwrap(); + let stored: Encrypted> = cipher + .encrypt(&"Hello, big World!".to_owned()) + .using(&words) + .await + .unwrap(); + let query = cipher + .query(op::Matches, "world hello") + .via(Match::::new()) + .on(&words) + .await + .unwrap(); + assert!(stored.terms.matches(&query)); + assert!(query.verify("Hello, big World!")); + assert!( + !query.verify("worldly hello"), + "a word is a token, not a prefix" + ); +} From d763ca16fc28f1af9d80b75eaf2f601d5a0b542c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 22:24:30 +1100 Subject: [PATCH 13/38] feat(stack-encrypt): size a match query's fetch with target_limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0009 decision 5: a caller who wants `want` real matches from a probabilistic query fetches `want + Poisson_quantile(λ = p·N, confidence)` rows, where `p` is the mean over stored filters of `f^b`, `f` a filter's fill and `b` the bits the query sets. - `Probabilistic::target_limit(&self, want, &Stats) -> Limit`. - `op::Stats::declared(rows)` uses a new `MatchConfig::expected_tokens()` prior, a default method returning `None`, so existing configurations compile unchanged; it is not part of the index's identity. - `op::Stats::measured(rows, histogram)` takes `(set bits, rows)` buckets from one sampled aggregate over the column, and takes the mean of the power, never the power of the mean fill. An empty histogram falls back to the prior. - `op::Limit` is `Rows(n)` or `Unbounded`, the latter when nothing is known of the stored filters or the confidence is not in (0, 1). `fetch_or(page)` reads either as a fetch size. - The quantile sums the Poisson mass exactly in log space from `λ − 10√λ`, so it costs about `12√λ` terms; past `λ = 1e7` it is the conservative `λ + 8√λ`. Unit tests reproduce the ADR's worked table and its mixed-length column, and pin the quantile to reference values computed with `lgamma`. Closes #1184 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 8 + ...and-a-query-term-is-not-the-stored-term.md | 35 +- .../stack-encrypt/src/{op.rs => op/mod.rs} | 16 + packages/stack-encrypt/src/op/stats.rs | 381 ++++++++++++++++++ packages/stack-encrypt/src/sem/mod.rs | 26 ++ packages/stack-encrypt/tests/op.rs | 102 ++++- ...nexact_operator_needs_probabilistic.stderr | 2 +- .../tests/ui/op_target_does_not_answer.stderr | 2 +- 8 files changed, 556 insertions(+), 16 deletions(-) rename packages/stack-encrypt/src/{op.rs => op/mod.rs} (95%) create mode 100644 packages/stack-encrypt/src/op/stats.rs diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index bd3e71a7c..501adc4a1 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -45,6 +45,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 copies of the text. An exact term has no `verify`, and none of this crate's indexes can answer `op::Matches` but with a probabilistic term (`op::TermFor`). +- **`target_limit` says how many rows a match query should fetch.** + `query.target_limit(want, &stats)` is `want` plus the false positives + expected at the confidence `stats` names (99% by default), as an + `op::Limit`. `op::Stats::declared(rows)` uses the new + `MatchConfig::expected_tokens()` prior; `op::Stats::measured(rows, + histogram)` uses a histogram of the stored filters' set-bit counts. With + neither, the limit is `Limit::Unbounded`, and the caller pages with a + cursor. - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index f3992ac95..16fb7c964 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -345,9 +345,10 @@ index declaration. The library takes either as numbers in `Stats` and does not care which it was given. A column has the prior from the day it is declared. -**Declared.** A match index declares `expected_tokens`, the typical token -count of the values it will hold, beside `m` and `k`: 20 for a name column, -200 for a notes column. From it `Stats::declared(rows)` takes +**Declared.** A match configuration declares `expected_tokens()`, the typical +token count of the values it will hold, beside the `m` and `k` of its +options: 20 for a name column, 200 for a notes column. It is not part of the +index's identity and changes no term. From it `Stats::declared(rows)` takes `f = 1 − e^(−k·expected_tokens/m)` and `p = f^b`. This is a prior, and `rows` is a planning estimate too. It is enough to warn before a query runs (a three-letter needle against a notes column) and to size a first fetch; it @@ -370,8 +371,16 @@ query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a page sample, which is fine for a distribution and unsuited to an exact count. It is also a sample of pages, so on a table of a few thousand rows a 1% sample can select none; below that size the clause is dropped and the whole -table is read, which costs nothing worth saving. `rows` comes from `SELECT reltuples FROM pg_class WHERE oid = -'users'::regclass`, a catalog read. +table is read, which costs nothing worth saving. `rows` comes from +`SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog +read. An empty histogram falls back to the declared prior. + +`target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the +statistics say nothing about the stored filters (no prior and no +histogram) or the confidence is not below 1. `fetch_or(page)` reads either +as a fetch size, for a caller that pages anyway. Past ten million expected +false positives the quantile is the conservative `λ + 8√λ`, since the query +is hopeless at that point. Where the numbers live and who keeps them in step was asked on the review, so it is written down. They are read from the server by the application, at @@ -440,7 +449,7 @@ shot, with a bound, and the completeness check from decision 5: ```rust let term = cipher.query(op::Matches, "ali").on(&users::name).await?; -let fetch = term.target_limit(10, &stats).fetch(); +let fetch = term.target_limit(10, &stats).fetch_or(50); let rows: Vec = sqlx::query_as( "SELECT id, name FROM users WHERE name @@ $1 ORDER BY id LIMIT $2", @@ -701,7 +710,7 @@ the operator is `@@`, column on the left. The query selects the ciphertext ```rust let term = cipher.query(op::Matches, "ali").on(&users::name).await?; -let fetch = term.target_limit(10, &stats).fetch(); +let fetch = term.target_limit(10, &stats).fetch_or(50); let rows: Vec = sqlx::query_as( "SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2", @@ -723,12 +732,12 @@ The `Stats` it used were read once, at startup, by the application: ```rust let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass") .fetch_one(&pool).await?; -let histogram: Vec<(i16, i64)> = sqlx::query_as( +let histogram: Vec<(i32, i64)> = sqlx::query_as( "SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1", ) .fetch_all(&pool).await?; -let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u16, n as u64))); +let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u32, n as u64))); ``` or, before anyone has measured, `Stats::declared(1_000_000)`, which uses the @@ -748,7 +757,7 @@ let rows: Vec = sqlx::query_as( ) .bind(sqlx::types::Json(&name)) .bind(sqlx::types::Json(&age)) -.bind(name.target_limit(10, &stats).fetch() as i64) +.bind(name.target_limit(10, &stats).fetch_or(50) as i64) .fetch_all(&pool).await?; let hits: Vec = cipher.open(rows).using(&users_plan).await? .into_iter().filter(|u| name.verify(&u.name)).take(10).collect(); @@ -890,7 +899,7 @@ it binds as a parameter in any library built on `database/sql` or pgx: ```go q, _ := users.Fields.Name.Matches(ctx, cipher, "ali") -limit := q.TargetLimit(10, stats).Fetch() +limit := q.TargetLimit(10, stats).FetchOr(50) rows, err := db.QueryContext(ctx, `SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2`, @@ -996,7 +1005,7 @@ TypeScript programs use: ```ts // pg -const limit = name.targetLimit(10, stats).fetch() +const limit = name.targetLimit(10, stats).fetchOr(50) const { rows } = await pool.query( 'SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2', [name, limit], @@ -1040,7 +1049,7 @@ const q = await ops.matches(users.name, 'ali') // { sql, verify, target const rows = await db.select().from(users) .where(q.sql) // name @@ '…'::eql_v4.query_text_match .orderBy(users.id) - .limit(q.targetLimit(10, stats).fetch()) + .limit(q.targetLimit(10, stats).fetchOr(50)) const hits = (await cipher.decrypt(users, rows)).filter((u) => q.verify(u.name)).slice(0, 10) ``` diff --git a/packages/stack-encrypt/src/op.rs b/packages/stack-encrypt/src/op/mod.rs similarity index 95% rename from packages/stack-encrypt/src/op.rs rename to packages/stack-encrypt/src/op/mod.rs index 08eb45082..82945a12d 100644 --- a/packages/stack-encrypt/src/op.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -57,6 +57,9 @@ use std::marker::PhantomData; use stack_kms::MaybeSend; +pub(crate) mod stats; +pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE}; + use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens}; use crate::target::{Equality, Index, Match}; use crate::PlanError; @@ -223,6 +226,19 @@ pub trait Probabilistic: QueryTerm { /// the index was declared over, so every outcome other than a match is /// `false`, never an error. fn verify(&self, value: &Self::Plaintext) -> bool; + + /// How many candidate rows to fetch so that, at the confidence `stats` + /// names, `want` of them [`verify`](Self::verify), when the table holds + /// that many matches. + /// + /// A probabilistic bound, not a guarantee: at 99% confidence one query + /// in a hundred draws more false positives than it allows for. The + /// candidate count says which case a fetch is in. If the server + /// returned fewer rows than the limit, the predicate is exhausted, and + /// fewer than `want` verified rows means fewer than `want` matches. If + /// it returned exactly the limit, the result is a page: carry on from + /// its last row with a cursor. + fn target_limit(&self, want: u64, stats: &Stats) -> Limit; } /// The query terms that may answer the operator `Op`: what diff --git a/packages/stack-encrypt/src/op/stats.rs b/packages/stack-encrypt/src/op/stats.rs new file mode 100644 index 000000000..edefda45d --- /dev/null +++ b/packages/stack-encrypt/src/op/stats.rs @@ -0,0 +1,381 @@ +//! How many rows to fetch for a probabilistic query: [`Stats`] in, +//! [`Limit`] out (ADR-0009, decision 5). +//! +//! A Bloom match returns every real match and some rows that matched by +//! chance. A non-matching row whose stored filter has fill `f` (the +//! fraction of its bits set) passes a query that sets `b` bits with +//! probability about `f^b`. Over a table the pass rate `p` is the *mean* of +//! each row's `f^b`, and the false positives a query scanning `N` rows +//! draws are `Poisson(λ = p·N)`. To expect `want` real matches at a given +//! confidence, fetch +//! +//! ```text +//! want + Poisson_quantile(λ, confidence) +//! ``` +//! +//! rows. The bound is probabilistic: at 99% confidence one query in a +//! hundred draws more false positives than the quantile. + +/// What a probabilistic query's [`target_limit`] knows of the table: how +/// many rows the predicate scans, how full their stored filters are, and +/// the confidence the fetch should meet. +/// +/// The numbers come from the server, read by the application at startup or +/// on a timer and kept as plain numbers. None of them is secret: the server +/// holds the filters and the catalog holds the row count. A stale `Stats` +/// moves the confidence, never the correctness, because +/// [`verify`](super::Probabilistic::verify) still decides every row. +/// +/// [`target_limit`]: super::Probabilistic::target_limit +#[derive(Clone, Debug, PartialEq)] +pub struct Stats { + rows: u64, + fill: Fill, + confidence: f64, +} + +/// Where a [`Stats`] takes the stored filters' fill from. +#[derive(Clone, Debug, PartialEq)] +enum Fill { + /// The index's declared `expected_tokens` prior. + Declared, + /// A histogram of set-bit counts: `(bits, rows)` pairs. + Measured(Vec<(u32, u64)>), +} + +/// The confidence a [`Stats`] targets unless told otherwise. +pub const DEFAULT_CONFIDENCE: f64 = 0.99; + +impl Stats { + /// Statistics from the index's declared prior + /// ([`MatchConfig::expected_tokens`](crate::sem::MatchConfig::expected_tokens)) + /// and an estimate of the rows the query scans, such as + /// `pg_class.reltuples`. + /// + /// The prior knows nothing about the spread of value lengths, so it + /// underestimates a column of mixed lengths; measure the column + /// ([`measured`](Self::measured)) once it holds data. An index that + /// declares no prior gives [`Limit::Unbounded`]. + pub fn declared(rows: u64) -> Self { + Self { + rows, + fill: Fill::Declared, + confidence: DEFAULT_CONFIDENCE, + } + } + + /// Statistics from a measured histogram of the stored filters' set-bit + /// counts, as `(bits, rows)` pairs, and an estimate of the rows the + /// query scans. + /// + /// One aggregate over the column gives the histogram, with at most `m` + /// buckets; a page sample bounds its cost on a large table: + /// + /// ```sql + /// SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) + /// FROM users TABLESAMPLE SYSTEM (1) + /// GROUP BY 1 + /// ``` + /// + /// A page sample can select no page of a small table, so drop the + /// `TABLESAMPLE` there. An empty histogram falls back to the declared + /// prior. + pub fn measured(rows: u64, histogram: impl IntoIterator) -> Self { + Self { + rows, + fill: Fill::Measured(histogram.into_iter().collect()), + confidence: DEFAULT_CONFIDENCE, + } + } + + /// Target `confidence` instead of 99%: the probability that the fetch + /// holds `want` real matches when the table has that many. Outside the + /// open interval `(0, 1)` there is no finite fetch to promise, and + /// [`target_limit`](super::Probabilistic::target_limit) answers + /// [`Limit::Unbounded`]. + pub fn with_confidence(mut self, confidence: f64) -> Self { + self.confidence = confidence; + self + } + + /// The rows the query scans. + pub fn rows(&self) -> u64 { + self.rows + } + + /// The confidence the fetch targets. + pub fn confidence(&self) -> f64 { + self.confidence + } + + /// The probability that one non-matching row passes a query that sets + /// `bits` of an `m`-bit filter, or `None` when nothing is known of the + /// stored filters' fill. `declared` is the fill of one stored filter + /// under the index's prior, if it declares one. + pub(crate) fn pass_rate(&self, bits: u32, m: u32, declared: Option) -> Option { + let measured = match &self.fill { + Fill::Measured(histogram) => mean_power(histogram, bits, m), + Fill::Declared => None, + }; + measured.or_else(|| declared.map(|fill| fill.powi(bits as i32))) + } + + /// The fetch for `want` real matches at pass rate `p`. + pub(crate) fn limit(&self, want: u64, pass_rate: Option) -> Limit { + let Some(p) = pass_rate else { + return Limit::Unbounded; + }; + let promisable = self.confidence > 0.0 && self.confidence < 1.0; + if !promisable || !p.is_finite() { + return Limit::Unbounded; + } + let lambda = p.clamp(0.0, 1.0) * self.rows as f64; + Limit::Rows(want.saturating_add(poisson_quantile(lambda, self.confidence))) + } +} + +/// The mean of `(bits_i / m)^b` over a histogram of set-bit counts, or +/// `None` for an empty one. A count above `m` (a histogram from another +/// configuration) is a full filter. +fn mean_power(histogram: &[(u32, u64)], b: u32, m: u32) -> Option { + let rows: u64 = histogram.iter().map(|&(_, rows)| rows).sum(); + if rows == 0 || m == 0 { + return None; + } + let total: f64 = histogram + .iter() + .map(|&(bits, rows)| { + let fill = (f64::from(bits) / f64::from(m)).min(1.0); + rows as f64 * fill.powi(b as i32) + }) + .sum(); + Some(total / rows as f64) +} + +/// The fill of one stored filter of `tokens` tokens, each setting `k` of +/// `m` bits: `1 − e^(−k·tokens/m)`. +pub(crate) fn expected_fill(tokens: u32, k: usize, m: u32) -> f64 { + 1.0 - (-(k as f64) * f64::from(tokens) / f64::from(m)).exp() +} + +/// How many rows a probabilistic query should fetch. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Limit { + /// Fetch this many rows: `want` and the false positives expected at the + /// stated confidence. + Rows(u64), + /// No finite fetch can be promised: the statistics say nothing about + /// the stored filters, or the confidence is not below 1. Page through + /// the candidates with a cursor until `want` rows verify. + Unbounded, +} + +impl Limit { + /// The rows to fetch, if a finite fetch can be promised. + pub fn rows(self) -> Option { + match self { + Self::Rows(rows) => Some(rows), + Self::Unbounded => None, + } + } + + /// The rows to fetch, or `page` when no finite fetch can be promised, + /// for a caller that pages either way. + pub fn fetch_or(self, page: u64) -> u64 { + self.rows().unwrap_or(page) + } +} + +/// The smallest `q` with `P(X ≤ q) ≥ confidence` for `X ~ Poisson(λ)`. +/// +/// Exact up to floating point: the probabilities are summed in log space +/// from where the mass begins, `λ − 10√λ` for a large `λ` (the mass below +/// it is under `1e-23`), so the work is about `12√λ` terms. Past `1e7` +/// expected false positives the query is hopeless anyway and the answer is +/// the conservative `λ + 8√λ`, which covers any confidence below +/// `1 − 1e-15`. +pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { + if lambda <= 0.0 || lambda.is_nan() { + return 0; + } + if lambda > 1e7 { + return (lambda + 8.0 * lambda.sqrt()).ceil() as u64; + } + // Saturates to 0 for a small `λ`, where the mass starts at 0. + let start = (lambda - 10.0 * lambda.sqrt()).floor().max(0.0) as u64; + let ln_lambda = lambda.ln(); + let mut log_term = -lambda + start as f64 * ln_lambda - ln_factorial(start); + let mut cdf = 0.0; + let mut k = start; + loop { + cdf += log_term.exp(); + // Past the mode, a term this small adds nothing a float can hold: + // the remaining mass is below any confidence that is not 1. + if cdf >= confidence || (k as f64 > lambda && log_term < -50.0) { + return k; + } + k += 1; + log_term += ln_lambda - (k as f64).ln(); + } +} + +/// `ln(n!)`: summed exactly below 20, Stirling's series above, where it is +/// accurate to about `1e-12`. +fn ln_factorial(n: u64) -> f64 { + if n < 20 { + return (2..=n).map(|i| (i as f64).ln()).sum(); + } + let x = n as f64 + 1.0; + (x - 0.5) * x.ln() - x + 0.5 * (2.0 * std::f64::consts::PI).ln() + 1.0 / (12.0 * x) + - 1.0 / (360.0 * x.powi(3)) + + 1.0 / (1260.0 * x.powi(5)) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Reference quantiles, computed exactly with `math.lgamma` in log space. + #[test] + fn the_poisson_quantile_matches_reference_values() { + for (lambda, confidence, expected) in [ + (0.034, 0.99, 1), + (0.439, 0.99, 3), + (1.0, 0.5, 1), + (1.0, 0.99, 4), + (10.0, 0.99, 18), + (57.8, 0.99, 76), + (100.0, 0.99, 124), + (245.0, 0.99, 282), + (1000.0, 0.999, 1099), + (3800.0, 0.99, 3944), + (7600.0, 0.99, 7804), + (1e6, 0.99, 1_002_327), + ] { + assert_eq!( + poisson_quantile(lambda, confidence), + expected, + "λ = {lambda}, confidence = {confidence}" + ); + } + assert_eq!(poisson_quantile(0.0, 0.99), 0); + assert_eq!(poisson_quantile(-1.0, 0.99), 0); + } + + #[test] + fn past_ten_million_the_quantile_is_conservative() { + let lambda = 4e7; + let q = poisson_quantile(lambda, 0.999_999); + assert!(q as f64 >= lambda + 4.9 * lambda.sqrt(), "{q}"); + assert!(q as f64 <= lambda + 8.0 * lambda.sqrt() + 1.0, "{q}"); + } + + #[test] + fn a_confidence_at_one_still_ends() { + // A float CDF may never reach 1; the tail cut-off ends the sum. + let q = poisson_quantile(5.0, 1.0); + assert!(q > 20, "{q}"); + } + + #[test] + fn ln_factorial_is_continuous_where_the_series_takes_over() { + let exact: f64 = (2..=20u64).map(|i| (i as f64).ln()).sum(); + assert!((ln_factorial(20) - exact).abs() < 1e-10); + assert_eq!(ln_factorial(0), 0.0); + assert_eq!(ln_factorial(1), 0.0); + assert!((ln_factorial(5) - 120f64.ln()).abs() < 1e-12); + } + + /// ADR-0009 decision 5's worked table: EQL v4 defaults (`k = 6`, + /// `m = 2048`), a million rows, 99%. + #[test] + fn the_declared_prior_reproduces_the_worked_table() { + let stats = Stats::declared(1_000_000); + for (tokens, bits, want_extra) in [(20, 6, 1), (200, 6, 7805), (200, 18, 3)] { + let fill = expected_fill(tokens, 6, 2048); + let p = stats.pass_rate(bits, 2048, Some(fill)); + assert_eq!( + stats.limit(10, p), + Limit::Rows(10 + want_extra), + "{tokens} stored tokens, {bits} query bits" + ); + } + } + + /// The mixed-length column of decision 5: half the rows hold 20 tokens, + /// half 200. The mean of `f^b` is what the bound needs; the mean fill + /// raised to `b` would promise a fetch about fifteen times too small. + #[test] + fn a_measured_histogram_uses_the_mean_of_the_power_not_the_power_of_the_mean() { + let short = (expected_fill(20, 6, 2048) * 2048.0).round() as u32; + let long = (expected_fill(200, 6, 2048) * 2048.0).round() as u32; + let stats = Stats::measured(1_000_000, [(short, 500_000), (long, 500_000)]); + let p = stats.pass_rate(6, 2048, None).unwrap(); + let lambda = p * 1e6; + assert!((3700.0..3900.0).contains(&lambda), "λ = {lambda}"); + + let mean_fill = (f64::from(short) + f64::from(long)) / 2.0 / 2048.0; + assert!(mean_fill.powi(6) * 1e6 < 300.0); + let Limit::Rows(rows) = stats.limit(10, Some(p)) else { + panic!("a measured column has a finite fetch"); + }; + assert!(rows > 3800, "{rows}"); + } + + #[test] + fn nothing_known_is_unbounded() { + let stats = Stats::declared(1_000_000); + assert_eq!( + stats.limit(10, stats.pass_rate(6, 2048, None)), + Limit::Unbounded + ); + + let empty = Stats::measured(1_000_000, []); + assert_eq!(empty.pass_rate(6, 2048, None), None); + assert_eq!( + empty.pass_rate(6, 2048, Some(0.5)), + Some(0.5f64.powi(6)), + "an empty histogram falls back to the prior" + ); + } + + #[test] + fn a_confidence_outside_the_open_interval_is_unbounded() { + for confidence in [0.0, 1.0, -0.5, 1.5, f64::NAN] { + let stats = Stats::declared(1000).with_confidence(confidence); + assert_eq!( + stats.limit(10, Some(0.01)), + Limit::Unbounded, + "{confidence}" + ); + } + let stats = Stats::declared(1000).with_confidence(0.5); + assert_eq!(stats.confidence(), 0.5); + assert_eq!(stats.rows(), 1000); + assert_eq!(stats.limit(10, Some(0.01)), Limit::Rows(10 + 10)); + } + + #[test] + fn a_count_above_the_filter_size_is_a_full_filter() { + let stats = Stats::measured(100, [(5000, 1)]); + assert_eq!(stats.pass_rate(6, 2048, None), Some(1.0)); + // Every row passes: λ = 100, whose 99% quantile is 124. + assert_eq!(stats.limit(1, Some(1.0)), Limit::Rows(1 + 124)); + } + + #[test] + fn a_limit_reads_back_as_rows_or_a_page() { + assert_eq!(Limit::Rows(12).rows(), Some(12)); + assert_eq!(Limit::Rows(12).fetch_or(50), 12); + assert_eq!(Limit::Unbounded.rows(), None); + assert_eq!(Limit::Unbounded.fetch_or(50), 50); + } + + #[test] + fn want_does_not_overflow() { + assert_eq!( + Stats::declared(10).limit(u64::MAX, Some(0.5)), + Limit::Rows(u64::MAX) + ); + } +} diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 60a725f47..3dc012738 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -471,6 +471,16 @@ impl MatchOptions { /// construction. Define your own by implementing this on a marker type. pub trait MatchConfig: Send + Sync + 'static { fn options() -> MatchOptions; + + /// The typical number of tokens in a value this index stores: 20 for a + /// name column, 200 for a notes column. A prior, used to size a query's + /// fetch ([`Stats::declared`](crate::op::Stats::declared)) before the + /// column's filters have been measured. It is not part of the index's + /// identity and changes no term. `None`, the default, declares no + /// prior. + fn expected_tokens() -> Option { + None + } } /// The default [`MatchConfig`]: [`MatchOptions::default`]. @@ -708,6 +718,22 @@ impl crate::op::Probabilistic for MatchQuery { let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); !self.tokens.0.is_empty() && self.tokens.0.iter().all(|t| have.contains(t.as_str())) } + + /// `want` plus the false positives expected among the rows `stats` + /// scans: the query sets `b` bits ([`positions`](Self::positions)), and + /// a stored filter of fill `f` holds them all with probability `f^b`. + /// The fill comes from the measured histogram in `stats`, or else from + /// the configuration's + /// [`expected_tokens`](MatchConfig::expected_tokens) prior; with + /// neither, the answer is + /// [`Limit::Unbounded`](crate::op::Limit::Unbounded). + fn target_limit(&self, want: u64, stats: &crate::op::Stats) -> crate::op::Limit { + let options = O::options(); + let declared = O::expected_tokens() + .map(|tokens| crate::op::stats::expected_fill(tokens, options.k, options.m)); + let bits = u32::try_from(self.positions().len()).unwrap_or(u32::MAX); + stats.limit(want, stats.pass_rate(bits, options.m, declared)) + } } /// Positions only: the tokens are plaintext. diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 81f0aced5..67ed9b22c 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -6,7 +6,7 @@ mod common; use common::{loads_counting_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; -use stack_encrypt::op::{self, MatchCase, Probabilistic}; +use stack_encrypt::op::{self, Limit, MatchCase, Probabilistic, Stats}; use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms}; use stack_encrypt::target::{Encrypted, IndexSpec}; @@ -624,3 +624,103 @@ async fn verify_is_a_token_test_not_a_substring_test() { "a word is a token, not a prefix" ); } + +/// EQL v4's match parameters, with a notes column's prior. +struct Notes; +impl MatchConfig for Notes { + fn options() -> MatchOptions { + MatchOptions { + k: 6, + m: 2048, + ..Default::default() + } + } + fn expected_tokens() -> Option { + Some(200) + } +} +impl MatchCase for Notes { + type Case = op::CaseInsensitive; +} + +#[tokio::test] +async fn target_limit_sizes_the_fetch_from_the_query_and_the_prior() { + let cipher = stack_cipher().await; + let plan = Plan::context("notes/body") + .with::(Match::::new()) + .build() + .unwrap(); + let query = |text: &'static str| { + cipher + .query(op::Matches, text) + .via(Match::::new()) + .on(&plan) + }; + let stats = Stats::declared(1_000_000); + + let short = query("ali").await.unwrap(); + let long = query("alice nguyen").await.unwrap(); + let (Limit::Rows(short_rows), Limit::Rows(long_rows)) = ( + short.target_limit(10, &stats), + long.target_limit(10, &stats), + ) else { + panic!("a declared prior gives a finite fetch"); + }; + assert!( + short_rows > 1_000, + "three letters against long text match thousands of rows by chance: {short_rows}" + ); + assert!(long_rows < 20, "a long query sets enough bits: {long_rows}"); + assert!(long_rows >= 10, "never fewer than `want`"); + + // A tighter confidence asks for more. + let strict = short.target_limit(10, &stats.clone().with_confidence(0.9999)); + assert!(strict.rows().unwrap() > short_rows); +} + +#[tokio::test] +async fn target_limit_without_a_prior_needs_a_measured_column() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let email_plan = plan.field("email").unwrap(); + let query = cipher + .query(op::Matches, "bob") + .on(&email_plan) + .await + .unwrap(); + + // The default configuration declares no prior. + assert_eq!( + query.target_limit(10, &Stats::declared(1_000)), + Limit::Unbounded + ); + + // Measure the column: the set-bit count of each stored filter. + let mut histogram = std::collections::BTreeMap::::new(); + for email in [ + "bob@example.com", + "alice@example.org", + "carol.smith@example.net", + ] { + let mut row = cipher + .encrypt(&Person { + email: email.into(), + name: "x".repeat(4), + }) + .using(&plan) + .await + .unwrap(); + let stored: Encrypted<(EqualityTerm, MatchTerms)> = row.take("email").unwrap(); + *histogram + .entry(stored.terms.1.positions().len() as u32) + .or_default() += 1; + } + let stats = Stats::measured(3, histogram); + let Limit::Rows(rows) = query.target_limit(1, &stats) else { + panic!("a measured column gives a finite fetch"); + }; + assert!( + (1..=4).contains(&rows), + "1 wanted, at most every row: {rows}" + ); +} diff --git a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr index ec190c313..ceaa770be 100644 --- a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr +++ b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr @@ -13,7 +13,7 @@ help: the trait `Probabilistic` is implemented for `MatchQuery` | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = note: required for `EqualityTerm` to implement `TermFor` note: required by a bound in `stack_encrypt::op::Supports::Query` - --> src/op.rs + --> src/op/mod.rs | | type Query: TermFor; | ^^^^^^^^^^^ required by this bound in `Supports::Query` diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr index 8fe0edf23..fa0080642 100644 --- a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr @@ -7,7 +7,7 @@ error[E0277]: `EqualityTerm` does not answer `stack_encrypt::op::Matches` over ` = note: a target answers the operators its type implements `Answers` for; `.via(index)` names an index the target declares help: the trait `Answers` is not implemented for `EqualityTerm` but trait `Answers` is implemented for it - --> src/op.rs + --> src/op/mod.rs | | impl Answers for EqualityTerm { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 33c52946e4ceb6a6d82762aff8ca69c334cfd46f Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 23:06:25 +1100 Subject: [PATCH 14/38] fix(stack-encrypt): address the review of target_limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The Poisson quantile is a bounded, normalised sum over `[λ − 10√λ, λ + 12√λ + 50]`. Each term is a log relative to the window's first, so a large `λ` no longer loses the starting term to cancellation, and the sums run from the top, where the terms are smallest, so a confidence near 1 is compared against a tail rounding did not swallow. It matches an exact reference to within one up to 1 − 1e-12 (was off by thousands at λ ≈ 4.75e6), and no mutation of it can loop forever. `ln_factorial` is gone. - A `Limit` never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest SQL `LIMIT`; `reltuples` of −1 read as a `u64` no longer overflows a bind. The ADR's statements use `greatest(reltuples, 0)`. - Histogram counts are summed as floats, so huge counts cannot overflow. - The independence assumption is stated in ADR-0009 decision 5 and on `target_limit`: a row sharing tokens with the needle passes more often, so the confidence is nominal on natural text; `verify` still decides. - Docs: the stats SQL filters NULL values; `target_limit` says a fetch that returned the limit is a page only when fewer than `want` verified; `expected_tokens` counts distinct tokens; decision 4's code shows `target_limit` on `Probabilistic`. - Tests: exact references at high confidence and at λ = 1e7 (which kills the threshold mutant), the `MAX_ROWS` cap, float-summed counts, and the measured-column test bounds the fetch against the histogram's own λ. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 5 +- ...and-a-query-term-is-not-the-stored-term.md | 27 +++- packages/stack-encrypt/src/op/mod.rs | 12 +- packages/stack-encrypt/src/op/stats.rs | 140 ++++++++++++------ packages/stack-encrypt/src/sem/mod.rs | 4 +- packages/stack-encrypt/tests/op.rs | 18 ++- 6 files changed, 138 insertions(+), 68 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 501adc4a1..a24a9934a 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -52,7 +52,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `MatchConfig::expected_tokens()` prior; `op::Stats::measured(rows, histogram)` uses a histogram of the stored filters' set-bit counts. With neither, the limit is `Limit::Unbounded`, and the caller pages with a - cursor. + cursor. A limit never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest + SQL `LIMIT`. The bound assumes a non-matching row shares no token with + the needle, so on natural text its confidence is nominal; `verify` still + decides every row. - **Errors now expose stable codes, help text, and structured details.** All error types implement `miette::Diagnostic` with codes such as diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 16fb7c964..5db67b16d 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -217,8 +217,7 @@ pub trait QueryTerm {} pub trait Probabilistic: QueryTerm { type Plaintext; fn verify(&self, value: &Self::Plaintext) -> bool; - // `target_limit(&self, want, &Stats) -> Limit` joins it with decision 5 - // (#1184). + fn target_limit(&self, want: u64, stats: &Stats) -> Limit; } pub trait Ranked: QueryTerm { @@ -319,6 +318,15 @@ positives); the mean fill, 0.25, raised to 6 predicts 245. The number the bound needs is therefore `p = mean(f_i^b)`, and the statistics the client holds must let it compute that for the `b` of each query. +The model assumes a non-matching row shares no token with the needle. The +same token sets the same bits in every filter, so a row holding part of the +needle has fewer bits left to chance: "police" already holds two of +"alice"'s three 3-grams, and passes at `f^6` rather than `f^18`. On natural +text the stated confidence is therefore nominal, and a measured histogram +cannot correct it, because it measures fill, not overlap. Correctness does +not depend on it: `verify` decides every row, and the candidate count says +whether a fetch was complete. + False positives are then `Poisson(λ = p·N)`, and the worst case that still satisfies the caller is exactly `want` true matches preceded by every false positive, so @@ -362,6 +370,7 @@ fills. One aggregate gives it, at most `m` buckets: ```sql SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) +WHERE name IS NOT NULL GROUP BY 1 ``` @@ -372,8 +381,10 @@ page sample, which is fine for a distribution and unsuited to an exact count. It is also a sample of pages, so on a table of a few thousand rows a 1% sample can select none; below that size the clause is dropped and the whole table is read, which costs nothing worth saving. `rows` comes from -`SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog -read. An empty histogram falls back to the declared prior. +`SELECT greatest(reltuples, 0) FROM pg_class WHERE oid = 'users'::regclass`, +a catalog read; `reltuples` is −1 for a table that has never been vacuumed +or analysed, hence the `greatest`. A `Limit` never exceeds `i64::MAX`, the +largest `LIMIT` PostgreSQL takes. An empty histogram falls back to the declared prior. `target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the statistics say nothing about the stored filters (no prior and no @@ -730,11 +741,11 @@ let hits: Vec = cipher.open(rows).using(&users_plan).await? The `Stats` it used were read once, at startup, by the application: ```rust -let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass") +let rows: i64 = sqlx::query_scalar("SELECT greatest(reltuples, 0)::bigint FROM pg_class WHERE oid = 'users'::regclass") .fetch_one(&pool).await?; let histogram: Vec<(i32, i64)> = sqlx::query_as( "SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) - FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1", + FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1", ) .fetch_all(&pool).await?; let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u32, n as u64))); @@ -1063,8 +1074,8 @@ application, or from the declared prior: ```ts const stats = Stats.measured( - (await pool.query("SELECT reltuples::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows, - (await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1')).rows, + (await pool.query("SELECT greatest(reltuples, 0)::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows, + (await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1')).rows, ) const prior = Stats.declared(1_000_000) // uses users.name's expectedTokens ``` diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index 82945a12d..dabaa3ec3 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -58,7 +58,7 @@ use std::marker::PhantomData; use stack_kms::MaybeSend; pub(crate) mod stats; -pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE}; +pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE, MAX_ROWS}; use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens}; use crate::target::{Equality, Index, Match}; @@ -232,12 +232,14 @@ pub trait Probabilistic: QueryTerm { /// that many matches. /// /// A probabilistic bound, not a guarantee: at 99% confidence one query - /// in a hundred draws more false positives than it allows for. The - /// candidate count says which case a fetch is in. If the server + /// in a hundred draws more false positives than it allows for, and the + /// model assumes a non-matching row shares no token with the needle, + /// which natural text does not honour, so the confidence is nominal. + /// The candidate count says which case a fetch is in. If the server /// returned fewer rows than the limit, the predicate is exhausted, and /// fewer than `want` verified rows means fewer than `want` matches. If - /// it returned exactly the limit, the result is a page: carry on from - /// its last row with a cursor. + /// it returned exactly the limit and fewer than `want` verified, the + /// result is a page: carry on from its last row with a cursor. fn target_limit(&self, want: u64, stats: &Stats) -> Limit; } diff --git a/packages/stack-encrypt/src/op/stats.rs b/packages/stack-encrypt/src/op/stats.rs index edefda45d..ad33323d7 100644 --- a/packages/stack-encrypt/src/op/stats.rs +++ b/packages/stack-encrypt/src/op/stats.rs @@ -46,11 +46,16 @@ enum Fill { /// The confidence a [`Stats`] targets unless told otherwise. pub const DEFAULT_CONFIDENCE: f64 = 0.99; +/// The largest fetch a [`Limit`] names: PostgreSQL's `LIMIT` is a signed +/// 64-bit integer. +pub const MAX_ROWS: u64 = i64::MAX as u64; + impl Stats { /// Statistics from the index's declared prior /// ([`MatchConfig::expected_tokens`](crate::sem::MatchConfig::expected_tokens)) /// and an estimate of the rows the query scans, such as - /// `pg_class.reltuples`. + /// `greatest(pg_class.reltuples, 0)` (`reltuples` is −1 for a table + /// never analysed). /// /// The prior knows nothing about the spread of value lengths, so it /// underestimates a column of mixed lengths; measure the column @@ -74,6 +79,7 @@ impl Stats { /// ```sql /// SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) /// FROM users TABLESAMPLE SYSTEM (1) + /// WHERE name IS NOT NULL /// GROUP BY 1 /// ``` /// @@ -130,7 +136,8 @@ impl Stats { return Limit::Unbounded; } let lambda = p.clamp(0.0, 1.0) * self.rows as f64; - Limit::Rows(want.saturating_add(poisson_quantile(lambda, self.confidence))) + let rows = want.saturating_add(poisson_quantile(lambda, self.confidence)); + Limit::Rows(rows.min(MAX_ROWS)) } } @@ -138,8 +145,9 @@ impl Stats { /// `None` for an empty one. A count above `m` (a histogram from another /// configuration) is a full filter. fn mean_power(histogram: &[(u32, u64)], b: u32, m: u32) -> Option { - let rows: u64 = histogram.iter().map(|&(_, rows)| rows).sum(); - if rows == 0 || m == 0 { + // Summed as floats: counts from a caller need not fit a `u64` total. + let rows: f64 = histogram.iter().map(|&(_, rows)| rows as f64).sum(); + if rows == 0.0 || m == 0 { return None; } let total: f64 = histogram @@ -149,7 +157,7 @@ fn mean_power(histogram: &[(u32, u64)], b: u32, m: u32) -> Option { rows as f64 * fill.powi(b as i32) }) .sum(); - Some(total / rows as f64) + Some(total / rows) } /// The fill of one stored filter of `tokens` tokens, each setting `k` of @@ -158,7 +166,8 @@ pub(crate) fn expected_fill(tokens: u32, k: usize, m: u32) -> f64 { 1.0 - (-(k as f64) * f64::from(tokens) / f64::from(m)).exp() } -/// How many rows a probabilistic query should fetch. +/// How many rows a probabilistic query should fetch. Never more than +/// [`MAX_ROWS`], the largest `LIMIT` PostgreSQL takes. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Limit { /// Fetch this many rows: `want` and the false positives expected at the @@ -188,47 +197,58 @@ impl Limit { /// The smallest `q` with `P(X ≤ q) ≥ confidence` for `X ~ Poisson(λ)`. /// -/// Exact up to floating point: the probabilities are summed in log space -/// from where the mass begins, `λ − 10√λ` for a large `λ` (the mass below -/// it is under `1e-23`), so the work is about `12√λ` terms. Past `1e7` -/// expected false positives the query is hopeless anyway and the answer is -/// the conservative `λ + 8√λ`, which covers any confidence below +/// The probabilities are summed over the window `[λ − 10√λ, λ + 12√λ + 50]`, +/// which holds all but about `1e-23` of the mass, and normalised by the +/// window's total. Each term is kept as a log relative to the window's +/// first, so no term carries the cancellation of `−λ + k·ln λ − ln k!` at a +/// large `λ`, and the sums run from the small end, so a confidence close to +/// 1 compares against a tail summed without losing it to rounding. The +/// answer is exact to within one at any confidence below `1 − 1e-12`. The +/// work is about `22√λ` terms. +/// Past `1e7` expected false positives the query is hopeless anyway and the +/// answer is the conservative `λ + 8√λ`, which covers any confidence below /// `1 − 1e-15`. +/// +/// `λ ≤ 0` and NaN have a window of one term at 0, so the answer is 0. pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { - if lambda <= 0.0 || lambda.is_nan() { - return 0; - } if lambda > 1e7 { return (lambda + 8.0 * lambda.sqrt()).ceil() as u64; } - // Saturates to 0 for a small `λ`, where the mass starts at 0. - let start = (lambda - 10.0 * lambda.sqrt()).floor().max(0.0) as u64; + let spread = lambda.sqrt(); + // A float-to-integer cast saturates, and maps NaN to 0. + let start = (lambda - 10.0 * spread).floor().max(0.0) as u64; + let end = (lambda + 12.0 * spread + 50.0).ceil() as u64; let ln_lambda = lambda.ln(); - let mut log_term = -lambda + start as f64 * ln_lambda - ln_factorial(start); - let mut cdf = 0.0; - let mut k = start; - loop { - cdf += log_term.exp(); - // Past the mode, a term this small adds nothing a float can hold: - // the remaining mass is below any confidence that is not 1. - if cdf >= confidence || (k as f64 > lambda && log_term < -50.0) { - return k; + let mut weights = Vec::with_capacity((end - start + 1) as usize); + let mut log_term = 0.0; + for k in start..=end { + if k > start { + log_term += ln_lambda - (k as f64).ln(); } - k += 1; - log_term += ln_lambda - (k as f64).ln(); + weights.push(log_term); } -} - -/// `ln(n!)`: summed exactly below 20, Stirling's series above, where it is -/// accurate to about `1e-12`. -fn ln_factorial(n: u64) -> f64 { - if n < 20 { - return (2..=n).map(|i| (i as f64).ln()).sum(); + let peak = weights.iter().copied().fold(f64::NEG_INFINITY, f64::max); + for weight in &mut weights { + *weight = (*weight - peak).exp(); + } + // Summed from the top down, where the terms are smallest, so the tail a + // high confidence compares against is not lost to rounding. + let total: f64 = weights.iter().rev().sum(); + let allowed = (1.0 - confidence) * total; + // Walking down from the top, `above` is the mass strictly above `k`. + // The answer is the lowest `k` whose mass above is within what the + // confidence allows. + let mut answer = end; + let mut above = 0.0; + for (offset, weight) in weights.iter().enumerate().rev() { + if above <= allowed { + answer = start + offset as u64; + above += weight; + } else { + break; + } } - let x = n as f64 + 1.0; - (x - 0.5) * x.ln() - x + 0.5 * (2.0 * std::f64::consts::PI).ln() + 1.0 / (12.0 * x) - - 1.0 / (360.0 * x.powi(3)) - + 1.0 / (1260.0 * x.powi(5)) + answer } #[cfg(test)] @@ -272,18 +292,29 @@ mod tests { #[test] fn a_confidence_at_one_still_ends() { - // A float CDF may never reach 1; the tail cut-off ends the sum. + // A float CDF may never reach 1; the window's end ends the sum. let q = poisson_quantile(5.0, 1.0); - assert!(q > 20, "{q}"); + assert!((20..=5 + 27 + 50).contains(&q), "{q}"); + assert_eq!(poisson_quantile(f64::NAN, 0.99), 0); } + /// High confidences over a large `λ`, where summing from `e^(−λ)` + /// loses precision: the window is normalised instead. References are + /// exact (mpmath's regularised incomplete gamma); the sum may land one + /// above, never below. #[test] - fn ln_factorial_is_continuous_where_the_series_takes_over() { - let exact: f64 = (2..=20u64).map(|i| (i as f64).ln()).sum(); - assert!((ln_factorial(20) - exact).abs() < 1e-10); - assert_eq!(ln_factorial(0), 0.0); - assert_eq!(ln_factorial(1), 0.0); - assert!((ln_factorial(5) - 120f64.ln()).abs() < 1e-12); + fn the_quantile_holds_at_high_confidence_and_large_lambda() { + for (lambda, confidence, exact) in [ + (3.4e6, 0.9999, 3_406_860), + (4_750_922.0, 1.0 - 1e-12, 4_766_263), + (1e7, 0.99, 10_007_357), + ] { + let q = poisson_quantile(lambda, confidence); + assert!( + q == exact || q == exact + 1, + "λ = {lambda}, confidence = {confidence}: {q}, exact {exact}" + ); + } } /// ADR-0009 decision 5's worked table: EQL v4 defaults (`k = 6`, @@ -372,10 +403,23 @@ mod tests { } #[test] - fn want_does_not_overflow() { + fn a_limit_fits_a_sql_limit() { assert_eq!( Stats::declared(10).limit(u64::MAX, Some(0.5)), - Limit::Rows(u64::MAX) + Limit::Rows(MAX_ROWS) + ); + // `reltuples` is −1 for a table never analysed; read as a `u64` it is + // the largest count there is. + assert_eq!( + Stats::declared(u64::MAX).limit(10, Some(0.5)), + Limit::Rows(MAX_ROWS) ); } + + #[test] + fn histogram_counts_do_not_overflow() { + let stats = Stats::measured(10, [(1024, u64::MAX), (2048, u64::MAX)]); + let p = stats.pass_rate(1, 2048, None).unwrap(); + assert!((p - 0.75).abs() < 1e-12, "{p}"); + } } diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 3dc012738..18ffe3f31 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -472,8 +472,8 @@ impl MatchOptions { pub trait MatchConfig: Send + Sync + 'static { fn options() -> MatchOptions; - /// The typical number of tokens in a value this index stores: 20 for a - /// name column, 200 for a notes column. A prior, used to size a query's + /// The typical number of distinct tokens in a value this index stores: + /// 20 for a name column, 200 for a notes column. A prior, used to size a query's /// fetch ([`Stats::declared`](crate::op::Stats::declared)) before the /// column's filters have been measured. It is not part of the index's /// identity and changes no term. `None`, the default, declares no diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 67ed9b22c..7976d9178 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -715,12 +715,22 @@ async fn target_limit_without_a_prior_needs_a_measured_column() { .entry(stored.terms.1.positions().len() as u32) .or_default() += 1; } - let stats = Stats::measured(3, histogram); - let Limit::Rows(rows) = query.target_limit(1, &stats) else { + // The pass rate the histogram gives this query, over a large table. + let bits = query.positions().len() as i32; + let rows_seen: u64 = histogram.values().sum(); + let p = histogram + .iter() + .map(|(&set, &n)| n as f64 * (f64::from(set) / 256.0).powi(bits)) + .sum::() + / rows_seen as f64; + let lambda = p * 1_000_000.0; + let stats = Stats::measured(1_000_000, histogram); + let Limit::Rows(rows) = query.target_limit(10, &stats) else { panic!("a measured column gives a finite fetch"); }; + let extra = (rows - 10) as f64; assert!( - (1..=4).contains(&rows), - "1 wanted, at most every row: {rows}" + extra >= lambda && extra <= lambda + 4.0 * lambda.sqrt() + 3.0, + "10 wanted, λ = {lambda}: {rows}" ); } From 98bf57023f6fc9d10ce19081156080fe407cd8d4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 13 Oct 2026 23:58:20 +1100 Subject: [PATCH 15/38] fix(stack-encrypt): bound the Poisson quantile's work, and pin its edges MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's mutants gate found one survivor and three timeouts in `poisson_quantile`: - Shifting each log term by the window's peak never mattered: relative to the window's first term no term exceeds about e^100, so `-` → `+` there was equivalent. The shift is gone. - A mutated threshold or window bound sent the `MAX_ROWS` test's λ of about 9e18 into a window of billions of terms. The window is now `22√λ + 50` wide from its start, capped at 2^20 terms whatever the input, so no mutation can hang a test. - The window's end is the answer when the confidence allows no mass (above 1), now pinned by a test. A focused `cargo mutants` run over `op/stats.rs` leaves no survivor and no timeout. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/src/op/stats.rs | 38 ++++++++++++++++---------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/packages/stack-encrypt/src/op/stats.rs b/packages/stack-encrypt/src/op/stats.rs index ad33323d7..dc2fa036a 100644 --- a/packages/stack-encrypt/src/op/stats.rs +++ b/packages/stack-encrypt/src/op/stats.rs @@ -197,10 +197,10 @@ impl Limit { /// The smallest `q` with `P(X ≤ q) ≥ confidence` for `X ~ Poisson(λ)`. /// -/// The probabilities are summed over the window `[λ − 10√λ, λ + 12√λ + 50]`, -/// which holds all but about `1e-23` of the mass, and normalised by the -/// window's total. Each term is kept as a log relative to the window's -/// first, so no term carries the cancellation of `−λ + k·ln λ − ln k!` at a +/// The probabilities are summed over a window from `λ − 10√λ` (or 0) that is +/// `22√λ + 50` wide, which holds all but about `1e-23` of the mass, and +/// normalised by the window's total. Each term is kept as a log relative to +/// the window's first, so no term carries the cancellation of `−λ + k·ln λ − ln k!` at a /// large `λ`, and the sums run from the small end, so a confidence close to /// 1 compares against a tail summed without losing it to rounding. The /// answer is exact to within one at any confidence below `1 − 1e-12`. The @@ -217,19 +217,21 @@ pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { let spread = lambda.sqrt(); // A float-to-integer cast saturates, and maps NaN to 0. let start = (lambda - 10.0 * spread).floor().max(0.0) as u64; - let end = (lambda + 12.0 * spread + 50.0).ceil() as u64; + // `22√λ + 50` past `start` reaches `λ + 12√λ + 50`. Capped, so no input + // makes the window large: below `1e7` it is at most about 70,000 terms. + let width = ((22.0 * spread + 50.0).ceil() as u64).min(MAX_WINDOW); let ln_lambda = lambda.ln(); - let mut weights = Vec::with_capacity((end - start + 1) as usize); + // Each term is `e^log` with `log` relative to the window's first term. + // From `λ − 10√λ` to the mode that ratio is at most about `e^50`, and + // from 0 to the mode of a `λ` below 100 about `e^λ`, so no term + // overflows and none of the mass is lost to underflow. + let mut weights = Vec::with_capacity(width as usize + 1); let mut log_term = 0.0; - for k in start..=end { + for k in start..=start + width { if k > start { log_term += ln_lambda - (k as f64).ln(); } - weights.push(log_term); - } - let peak = weights.iter().copied().fold(f64::NEG_INFINITY, f64::max); - for weight in &mut weights { - *weight = (*weight - peak).exp(); + weights.push(log_term.exp()); } // Summed from the top down, where the terms are smallest, so the tail a // high confidence compares against is not lost to rounding. @@ -238,7 +240,7 @@ pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { // Walking down from the top, `above` is the mass strictly above `k`. // The answer is the lowest `k` whose mass above is within what the // confidence allows. - let mut answer = end; + let mut answer = start + width; let mut above = 0.0; for (offset, weight) in weights.iter().enumerate().rev() { if above <= allowed { @@ -251,6 +253,10 @@ pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { answer } +/// The most terms [`poisson_quantile`] sums: far more than any `λ` below +/// `1e7` needs, and a bound on the work whatever it is handed. +const MAX_WINDOW: u64 = 1 << 20; + #[cfg(test)] mod tests { use super::*; @@ -294,8 +300,12 @@ mod tests { fn a_confidence_at_one_still_ends() { // A float CDF may never reach 1; the window's end ends the sum. let q = poisson_quantile(5.0, 1.0); - assert!((20..=5 + 27 + 50).contains(&q), "{q}"); + // The window starts at 0 and is `⌈22√5 + 50⌉` wide. + assert!((20..=100).contains(&q), "{q}"); assert_eq!(poisson_quantile(f64::NAN, 0.99), 0); + // A confidence above 1 allows no mass at all: the window's end. + assert_eq!(poisson_quantile(5.0, 1.5), 100); + assert_eq!(poisson_quantile(7600.0, 1.5), 6728 + 1968); } /// High confidences over a large `λ`, where summing from `e^(−λ)` From 0e032091e1959f9d388c68dbcd3b3b10c22b5109 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 00:49:30 +1100 Subject: [PATCH 16/38] fix(stack-encrypt): loop the Poisson window by offset, under the cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's mutants gate found `start..=start + width` → `start * width` surviving: the loop's range sat outside the cap on the window's width, so the mutation ran for billions of terms instead of failing. The loop now runs over `0..=width`, the capped width itself, and derives each `k` from it. A focused `cargo mutants` run over `op/stats.rs`: 91 mutants, 87 caught, 4 unviable, none missed, no timeouts. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/src/op/stats.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/packages/stack-encrypt/src/op/stats.rs b/packages/stack-encrypt/src/op/stats.rs index dc2fa036a..dd670a5b7 100644 --- a/packages/stack-encrypt/src/op/stats.rs +++ b/packages/stack-encrypt/src/op/stats.rs @@ -227,9 +227,9 @@ pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 { // overflows and none of the mass is lost to underflow. let mut weights = Vec::with_capacity(width as usize + 1); let mut log_term = 0.0; - for k in start..=start + width { - if k > start { - log_term += ln_lambda - (k as f64).ln(); + for offset in 0..=width { + if offset > 0 { + log_term += ln_lambda - ((start + offset) as f64).ln(); } weights.push(log_term.exp()); } From 8aed9a6839a63d77a9fb332d9d0a25ce494ff2f5 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 11:01:03 +1100 Subject: [PATCH 17/38] docs(stack-encrypt): the query, not the needle, in target_limit's prose Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 2 +- ...elect-the-index-and-a-query-term-is-not-the-stored-term.md | 4 ++-- packages/stack-encrypt/src/op/mod.rs | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index a24a9934a..25d492c76 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -54,7 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 neither, the limit is `Limit::Unbounded`, and the caller pages with a cursor. A limit never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest SQL `LIMIT`. The bound assumes a non-matching row shares no token with - the needle, so on natural text its confidence is nominal; `verify` still + the query, so on natural text its confidence is nominal; `verify` still decides every row. - **Errors now expose stable codes, help text, and structured details.** diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 5db67b16d..b061a7f6f 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -318,9 +318,9 @@ positives); the mean fill, 0.25, raised to 6 predicts 245. The number the bound needs is therefore `p = mean(f_i^b)`, and the statistics the client holds must let it compute that for the `b` of each query. -The model assumes a non-matching row shares no token with the needle. The +The model assumes a non-matching row shares no token with the query. The same token sets the same bits in every filter, so a row holding part of the -needle has fewer bits left to chance: "police" already holds two of +query has fewer bits left to chance: "police" already holds two of "alice"'s three 3-grams, and passes at `f^6` rather than `f^18`. On natural text the stated confidence is therefore nominal, and a measured histogram cannot correct it, because it measures fill, not overlap. Correctness does diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index dabaa3ec3..e8ffa6ebd 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -233,7 +233,7 @@ pub trait Probabilistic: QueryTerm { /// /// A probabilistic bound, not a guarantee: at 99% confidence one query /// in a hundred draws more false positives than it allows for, and the - /// model assumes a non-matching row shares no token with the needle, + /// model assumes a non-matching row shares no token with the query, /// which natural text does not honour, so the confidence is nominal. /// The candidate count says which case a fetch is in. If the server /// returned fewer rows than the limit, the predicate is exhausted, and From 0b8e9f4fb0c6efb5b735101571db02c3b0cd3736 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 17:49:12 +1100 Subject: [PATCH 18/38] feat(stack-encrypt): store a token-less match value as the empty filter, refuse only the query A match value that cuts no token under the index's tokenizer (empty or separator-only text, or text shorter than the n-gram length) used to fail term derivation on both the stored and the query side, so one two-letter name failed a whole row. ADR-0009 splits the two: the stored term is the empty filter, which no query matches, and only a query of such text is refused, because its filter would match every row. `match_term` now derives the empty filter for such text, with no PRF call, so `MatchTerms` as a target, `KeysetCipher::match_terms` and the `dynamic` record path all store it, and the value seals and opens as any other. The refusal moves to the query paths: `op::Supports::check` takes the query's value, the `Match` index refuses token-less text there with `TermError::EmptyTermText` when the query is awaited, before any term is derived or keyset loaded, and `dynamic::term` refuses it for a match kind. The error keeps its code and is reworded for the query. The Go module builds the stack-encrypt guest by path, so its round-trip test of the old refusal is rewritten to the new behaviour: a short `TextMatch` value seals, opens and matches no query, and a short match query is still `ErrTerm`. The package docs and the term-failure message follow. Two tests that used token-less text as a leaf that fails before any I/O now use a match configuration with invalid options, which still fails during the synchronous build. Closes #1185 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- languages/golang/encrypt/doc.go | 11 +- languages/golang/encrypt/records.go | 12 +- languages/golang/encrypt/roundtrip_test.go | 61 ++++++---- packages/stack-encrypt/CHANGELOG.md | 16 +++ ...and-a-query-term-is-not-the-stored-term.md | 9 +- packages/stack-encrypt/src/dynamic/term.rs | 53 ++++++++- packages/stack-encrypt/src/op/mod.rs | 44 ++++--- packages/stack-encrypt/src/plan/chain.rs | 17 ++- packages/stack-encrypt/src/sem/mod.rs | 65 +++++++---- packages/stack-encrypt/src/sem/tokenize.rs | 11 +- packages/stack-encrypt/tests/derive.rs | 44 +++++-- packages/stack-encrypt/tests/op.rs | 109 +++++++++++++++++- packages/stack-encrypt/tests/sem_terms.rs | 57 +++++---- packages/stack-encrypt/tests/target.rs | 19 ++- ...op_inexact_operator_needs_probabilistic.rs | 2 +- 15 files changed, 410 insertions(+), 120 deletions(-) diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 7b114f0e3..7036e1303 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -104,12 +104,11 @@ // [Match], [Ore] and [Ope]; [JSON] is declared and refused until the engine // derives it. Every stored type implements driver.Valuer and sql.Scanner. // -// A match index needs text with at least one token: the engine derives no -// match term for an empty string, separator-only text, or text shorter than -// the n-gram length (3 characters), because an empty term would match every -// row. Encrypt then fails for the whole batch with [ErrTerm], naming the row, -// the field and the index, so an optional or short value does not belong -// under match. +// A match index stores text with no token (an empty string, separator-only +// text, or text shorter than the n-gram length of 3 characters) as the empty +// filter: a valid value that no query matches, so an optional or short value +// can live under match. A match *query* of such text is refused with +// [ErrTerm], because an empty filter as a query would match every row. // // # Transport and auth // diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index 424184526..80f1f655f 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -470,11 +470,11 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, context string, p // locateTermFailure names the row, field and index behind an ErrTerm from a // record call. The guest's error says what was wrong with the term but not // which value or field it came from, so the host asks the engine again, one -// term at a time, which costs no key request: a term derives locally. The engine defines no match term for text that yields no -// token (empty, separator-only, or shorter than the n-gram), because an -// empty term would match every row; a program hands such a field a value or -// drops the match index. p is the plan without this cipher's extension, -// which Derive adds. +// term at a time, which costs no key request: a term derives locally. A +// stored value is never refused for its text (text with no token is stored +// as the empty filter), so a term failure on a record is the scheme refusing +// the value's kind under that index. p is the plan without this cipher's +// extension, which Derive adds. func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error { for i, row := range rows { for _, f := range p.Fields { @@ -483,7 +483,7 @@ func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows [ continue } if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) { - return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value (a match index needs text with at least one token; an empty or separator-only string has none)", err, i, f.Name, indexWord(o)) + return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value", err, i, f.Name, indexWord(o)) } } } diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go index b9a1522b3..c1658108b 100644 --- a/languages/golang/encrypt/roundtrip_test.go +++ b/languages/golang/encrypt/roundtrip_test.go @@ -260,37 +260,56 @@ func TestATamperedRecordDoesNotOpen(t *testing.T) { } } -// A match index needs text that yields a token: the engine defines no match -// term for an empty or separator-only string, because an empty term would -// match every row, and the typed Rust path refuses it the same way. The Go -// error names the row, the field and the index. Every other zero value -// seals: a zero integer under ore and equality, an empty string with no -// match index. -func TestAMatchIndexNeedsText(t *testing.T) { +// A match index stores text that yields no token as the empty filter: an +// empty or separator-only string, or one shorter than the n-gram, is a valid +// value that no query matches, and it seals and opens like any other. Only a +// match *query* of such text is refused, because an empty filter as a query +// would match every row. Every other zero value seals too: a zero integer +// under ore and equality, an empty string with no match index. +func TestAMatchIndexStoresShortText(t *testing.T) { c := deterministicClient(t) ctx := context.Background() cipher := c.DefaultKeyset() for name, u := range map[string]testusers.User{ - "the zero value": {}, - "an empty email": {ID: 1, Age: 34, Notes: "x"}, - "a separator email": {ID: 1, Age: 34, Email: " \t", Notes: "x"}, + "the zero value": {}, + "an empty email": {ID: 1, Age: 34, Notes: "x"}, + "a separator email": {ID: 1, Age: 34, Email: " \t", Notes: "x"}, + "a two-letter email": {ID: 1, Age: 34, Email: "ab", Notes: "x"}, } { - _, err := testusers.Encrypt(ctx, cipher, []testusers.User{{ID: 9, Age: 1, Email: "ok@example.com", Notes: "x"}, u}) - if !errors.Is(err, encrypt.ErrTerm) { - t.Fatalf("%s: %v, want ErrTerm", name, err) + rows := []testusers.User{{ID: 9, Age: 1, Email: "ok@example.com", Notes: "x"}, u} + encrypted, err := testusers.Encrypt(ctx, cipher, rows) + if err != nil { + t.Fatalf("%s: %v, want the row to seal", name, err) + } + if positions, err := encrypted[1].Email.Match.Positions(); err != nil || len(positions) != 0 { + t.Errorf("%s: the stored match term is %v %v, want the empty filter", name, positions, err) + } + back, err := testusers.Decrypt(ctx, cipher, encrypted) + if err != nil { + t.Fatalf("%s: decrypt: %v", name, err) } - for _, want := range []string{`value 1`, `field "email"`, "no match term"} { - if !strings.Contains(err.Error(), want) { - t.Errorf("%s: error %q does not name %s", name, err, want) - } + if back[1].Email != u.Email { + t.Errorf("%s: email opened as %q, want %q", name, back[1].Email, u.Email) + } + // No query matches the empty filter: a query has at least one + // position, and the empty filter holds none of them. + q, err := testusers.Fields.Email.Match(ctx, cipher, "example") + if err != nil { + t.Fatal(err) + } + if positions, _ := q.Positions(); len(positions) == 0 { + t.Fatalf("%s: a query with a token sets no bit", name) } } - // The field entry says the same for one value. - if _, err := testusers.Fields.Email.Encrypt(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) || !strings.Contains(err.Error(), `field "email"`) { + // The field entry stores one such value the same way. + if _, err := testusers.Fields.Email.Encrypt(ctx, cipher, ""); err != nil { t.Fatalf("Fields.Email.Encrypt(\"\"): %v", err) } - if _, err := testusers.Fields.Email.Match(ctx, cipher, ""); !errors.Is(err, encrypt.ErrTerm) { - t.Fatalf("Fields.Email.Match(\"\"): %v", err) + // A match query with no token is the one value the engine refuses. + for _, text := range []string{"", " \t", "ab"} { + if _, err := testusers.Fields.Email.Match(ctx, cipher, text); !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("Fields.Email.Match(%q): %v, want ErrTerm", text, err) + } } // Zero values the engine does seal. encrypted, err := testusers.Encrypt(ctx, cipher, []testusers.User{{Email: "zero@example.com"}}) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 25d492c76..ad23b7421 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -93,6 +93,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 under it. `FieldPlan::with_target`, `FieldPlan::target()`, `Plan::new_with`. +### Changed + +- **A match value with no tokens is stored as the empty filter; only a query + with none is refused.** Text that cuts no token under the match index's + tokenizer (empty or separator-only text, or text shorter than the n-gram + length) used to fail term derivation on both sides, so one two-letter name + failed a whole row. Now the stored term is the empty filter (`bf: []`), + which no query matches, and the value seals and opens as any other; the + `dynamic` record path stores it the same way. A *query* of such text is + still refused, with `TermError::EmptyTermText` as before, because its + filter would match every row: `cipher.query(op::Matches, ..)` refuses it + when the query is awaited, before a term exists and before any keyset is + loaded (`op::Supports::check` now takes the query's value), and + `dynamic::term` refuses it for a match kind. `KeysetCipher::match_terms` + derives the stored filter and no longer fails on such text (ADR-0009). + ### Breaking - **The query chain starts with the operator.** `cipher.query(v)` takes the diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index b061a7f6f..cf96fb9cf 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -255,11 +255,14 @@ times `want`, so it takes nothing else. `verify` is infallible. It runs on a value that is already decrypted and already the type the index was declared over, so the only question it can answer is "does this row match", and every other outcome is "no": a JSON -path absent from the document, text the index would have refused to store. +path absent from the document, text too short to hold the query's tokens. A `bool` is what `Iterator::filter` takes; an error would have no recovery except dropping the row, which is `false` by another name. The one input the -library refuses, a needle too short to tokenise, is refused at -`cipher.query`, before a term exists. +library refuses, a query value too short to tokenise, is refused when the query +is awaited, before a term exists. The stored side refuses nothing: a value +too short to tokenise is stored as the empty filter, which no query matches, +so a two-letter name is a valid value in a match column and `verify` is +never asked about it. `Plaintext` is the decrypted value the term is defined over, and that is not always the value the user wrote. For `Matches` it is the text. For diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index d02b272c0..3b75fe7b8 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -20,7 +20,9 @@ use vitaminc_protected::{Controlled, OpaqueDebug, Protected}; use zeroize::Zeroizing; use super::{utf8, Error, Reason, Value}; -use crate::sem::{CllwOpeEncrypt, CllwOreEncrypt, DefaultMatch, MatchOptions, Tokenizer}; +use crate::sem::{ + CllwOpeEncrypt, CllwOreEncrypt, DefaultMatch, MatchOptions, QueryTokens, TermError, Tokenizer, +}; use crate::target::{chosen, CallerContext, ConsumeSource, Encryption, Index, IndexSpec, Pending}; use crate::{IntoPrfContext, KeysetCipher, NonEmpty}; @@ -293,7 +295,11 @@ impl Scalar { /// [`Error::Term`] if the scheme defines no such term for the scalar /// ([`IndexSpec::supports`] is the table, and checking it first is how a /// binding turns this into a boundary rejection). [`Error::Cipher`] if the -/// derivation itself fails. +/// derivation itself fails, or for the one query value the engine refuses: +/// match text that cuts no token under the index's options +/// ([`TermError::EmptyTermText`]), whose filter would match every row. That +/// refusal is the query path's alone; the record path stores such a value +/// as the empty filter, which no query matches. pub async fn term<'c, K, D>( cipher: &KeysetCipher<'_, K>, scalar: Scalar, @@ -304,6 +310,11 @@ where K: DataKeySource + Sync + 'static, D: IntoPrfContext<'c>, { + if let (IndexSpec::Match(options), Scalar::Text(text)) = (kind, &scalar) { + if QueryTokens::under(text, options).is_empty() { + return Err(Error::Cipher(TermError::EmptyTermText.into())); + } + } scalar_term(cipher, scalar, kind, context)? .await .map(TermBytes::into_bytes) @@ -613,6 +624,44 @@ mod tests { } } + /// The query path refuses match text with no tokens; the record path, + /// which runs the same dispatch as an index of a `Value`, stores it as + /// the empty filter. Same scalar, same options, two answers, because a + /// query with no tokens would match every row and a stored value with + /// none is matched by no query. + #[tokio::test] + async fn match_text_with_no_tokens_is_refused_as_a_query_and_stored_empty() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/bio")).expect("context"); + for text in ["", " ", "hi"] { + let scalar = Scalar::of(&s(text), &default_match()).expect("a scalar"); + let refused = term(&keyset, scalar, &default_match(), ctx.clone()).await; + assert!( + matches!( + refused, + Err(Error::Cipher(crate::Error::Term(TermError::EmptyTermText))) + ), + "{text:?} as a query: {refused:?}" + ); + + let stored: TermBytes = keyset + .run( + Index::::operation::( + &default_match(), + ), + Value::new(s(text)), + CallerContext::from(nonempty!("users/bio")), + ) + .await + .unwrap_or_else(|error| panic!("{text:?} stores: {error}")); + assert!(stored.as_bytes().is_empty(), "{text:?} is the empty filter"); + } + // Text with a token is a query like any other. + let scalar = Scalar::of(&s("hello"), &default_match()).expect("a scalar"); + assert!(term(&keyset, scalar, &default_match(), ctx).await.is_ok()); + } + /// A `TermBytes` reads as the typed term's bytes, through every /// accessor: `as_bytes`, `as_ref` and `into_bytes` give the same /// 32 PRF bytes an `EqualityTerm` holds, for a term derived through diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index e8ffa6ebd..bb4dded0a 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -45,7 +45,13 @@ //! [`query_index`](crate::StackCipher::query_index), before any key is //! requested. //! So is a match configuration whose [`MatchCase`] disagrees with its -//! options. +//! options, and so is the one query value the library refuses: text that +//! cuts no token under the match index's tokenizer, whose filter would +//! match every row ([`TermError::EmptyTermText`]). The stored side has no +//! such refusal: a value with no tokens is stored as the empty filter, +//! which no query matches. +//! +//! [`TermError::EmptyTermText`]: crate::sem::TermError::EmptyTermText //! //! The module is meant to be used by path (`op::Eq`): an imported `Eq` //! would shadow the prelude's `Eq` trait. @@ -60,9 +66,11 @@ use stack_kms::MaybeSend; pub(crate) mod stats; pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE, MAX_ROWS}; -use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens}; +use crate::sem::{ + DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens, TermError, +}; use crate::target::{Equality, Index, Match}; -use crate::PlanError; +use crate::{Error, PlanError}; /// A query operator: a unit type naming what a query asks of the server. /// @@ -286,9 +294,13 @@ pub trait Supports: Index { /// What the query term keeps of the query's value, taken before the /// value is consumed: `()` for an exact term. type Carry: MaybeSend + 'static; - /// Why this index cannot answer `Op` as configured, if it cannot: a - /// match index whose [`MatchCase`] disagrees with its options. - fn check(&self) -> Result<(), PlanError>; + /// Why this index cannot answer `Op` as configured, or over `value`, if + /// it cannot: a match index whose [`MatchCase`] disagrees with its + /// options ([`PlanError::MatchCase`]), or a match query of text that + /// cuts no token ([`TermError::EmptyTermText`]), whose filter would + /// match every row. Run when the query is awaited, before any key is + /// requested; an index that refuses nothing returns `Ok(())`. + fn check(&self, value: &S) -> Result<(), Error>; /// Take what the query term keeps of `value`. fn carry(&self, value: &S) -> Self::Carry; /// The query term, from what was kept and the term the index derived @@ -313,7 +325,7 @@ impl QueryToken { impl Supports for Equality { type Query = EqualityTerm; type Carry = (); - fn check(&self) -> Result<(), PlanError> { + fn check(&self, _: &S) -> Result<(), Error> { Ok(()) } fn carry(&self, _: &S) {} @@ -329,16 +341,22 @@ where { type Query = MatchQuery; type Carry = QueryTokens; - fn check(&self) -> Result<(), PlanError> { + fn check(&self, value: &S) -> Result<(), Error> { let folds = O::options().downcase; - if folds == O::Case::FOLDS { - Ok(()) - } else { - Err(PlanError::MatchCase { + if folds != O::Case::FOLDS { + return Err(PlanError::MatchCase { declared: O::Case::NAME, folds, - }) + } + .into()); + } + // A query with no tokens would match every row. The stored side + // keeps such a value as the empty filter; the query side refuses it + // here, before any term is derived. + if QueryTokens::of::(value.as_ref()).is_empty() { + return Err(TermError::EmptyTermText.into()); } + Ok(()) } fn carry(&self, value: &S) -> QueryTokens { QueryTokens::of::(value.as_ref()) diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 6c9c880bc..87ada56e0 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -927,11 +927,9 @@ pub struct QueryOp<'a, F, Op, I, K> { impl<'a, F, Op: Operator, I: Supports, K: 'static> QueryOp<'a, F, Op, I, K> { /// The chain for `inner`'s index, refused when awaited if the index - /// cannot answer `Op` as it is configured. - fn new(mut inner: QueryIndex<'a, F, I, K>, _op: Op) -> Self { - if let Err(error) = inner.index.check() { - inner.label = Err(error); - } + /// cannot answer `Op` as it is configured or over the query's value + /// ([`Supports::check`]). + fn new(inner: QueryIndex<'a, F, I, K>, _op: Op) -> Self { Self { inner, op: PhantomData, @@ -970,7 +968,8 @@ where self.inner.take_keyset() } fn check(&self) -> Result<(), Error> { - self.inner.check() + self.inner.check()?; + self.inner.index.check(&self.inner.value) } fn prepare<'p>( self, @@ -980,6 +979,12 @@ where where 'a: 'p, { + // `check` ran before any keyset was resolved; a chain prepared on + // its own is held to the same refusal, so no term is derived from a + // value the index refuses. + if let Err(error) = self.inner.index.check(&self.inner.value) { + return Pending::failed(keyset, error); + } let carry = self.inner.index.carry(&self.inner.value); self.inner .prepare(keyset, scoped) diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 18ffe3f31..c768194cd 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -187,18 +187,20 @@ pub enum TermError { #[error("invalid match options: {0}")] #[diagnostic(code(stack_encrypt::invalid_match_options))] InvalidOptions(&'static str), - /// The text produced no tokens under the configured tokenizer — empty or - /// separator-only text, or (for n-grams, as in the v1 match indexer) text - /// shorter than the n-gram length. Rejected at generation time for both - /// the write and query paths: an empty term used as a query would - /// vacuously match every stored row, and a sub-gram-length probe could - /// never match anything (a silent false negative). + /// A match query's text produced no tokens under the index's tokenizer: + /// empty or separator-only text, or (for n-grams, as in the v1 match + /// indexer) text shorter than the n-gram length. A query with no tokens + /// would match every row, so the query chain refuses it when the query + /// is awaited, before a term exists. A *stored* value with no tokens is + /// not refused: it is stored as the empty filter, which no query + /// matches, so a short or empty value is a valid thing to hold in a + /// match column (ADR-0009). #[error( - "text produces no match tokens (empty, separator-only, or shorter than the n-gram length)" + "the query's text produces no match tokens (empty, separator-only, or shorter than the n-gram length)" )] #[diagnostic( code(stack_encrypt::empty_term_text), - help("A match term needs text with at least one token: a query with none would match every row, and one shorter than the n-gram length could match none. Match on longer text, or skip the match condition for this value.") + help("A match query needs text with at least one token: a query with none would match every row, and one shorter than the n-gram length could match none. Query on longer text, or leave the match condition out.") )] EmptyTermText, /// Term bytes do not decode under the term kind's frozen encoding — see @@ -556,9 +558,11 @@ impl MatchTerms { /// /// An empty `query` returns `false`: containment of zero positions is /// vacuously true, which would turn an empty query into a match-every-row - /// query. Term generation already refuses to build such a term - /// ([`TermError::EmptyTermText`]); this guards any other - /// (e.g. deserialized) source of an empty term. + /// query. A query chain already refuses to build such a query + /// ([`TermError::EmptyTermText`]); this guards any other (e.g. + /// deserialized) source of an empty one. The stored side is the other + /// way round: a value with no tokens is stored as the empty filter, and + /// holds no query's positions, so no query matches it. pub fn matches(&self, query: &MatchQuery) -> bool { self.contains(query.filter()) } @@ -781,13 +785,23 @@ pub struct QueryTokens(Zeroizing>); impl QueryTokens { /// The tokens of `text` under `O`'s tokenizer and case folding. pub(crate) fn of(text: &str) -> Self { - let options = O::options(); + Self::under(text, &O::options()) + } + + /// The tokens of `text` under `options`' tokenizer and case folding. + pub(crate) fn under(text: &str, options: &MatchOptions) -> Self { Self(Zeroizing::new(tokenize::tokenize( text, options.tokenizer, options.downcase, ))) } + + /// Whether the text cut no token at all: the one query the library + /// refuses, because its filter would match every row. + pub(crate) fn is_empty(&self) -> bool { + self.0.is_empty() + } } /// A [`PrfVisitor`] that folds a sequence of per-token PRF blocks into @@ -839,6 +853,13 @@ impl PrfVisitor<[u8; 32], P> for BloomVisitor { /// Derive a match term. Synchronous — see [`equality`]: validation and /// tokenization run before the PRF, the visitor folds blocks into positions. +/// +/// Text that cuts no token (empty, separator-only, or shorter than the +/// n-gram length) is the empty filter: no position set, no PRF call. Stored, +/// it is a value no query matches, since a query always has at least one +/// token and the empty filter holds none of its positions. This is the +/// *stored* side's answer; a query of such text is refused by the query +/// chain before it gets here ([`TermError::EmptyTermText`]). fn match_term( prf: &HmacSha256Prf, text: &str, @@ -848,7 +869,7 @@ fn match_term( let mask = options.validate()?; let tokens = tokenize::tokenize(text, options.tokenizer, options.downcase); if tokens.is_empty() { - return Err(TermError::EmptyTermText); + return Ok(MatchTerms::normalised(Vec::new())); } let context = Context::pae(&[MATCH_DOMAIN, context.as_bytes()]); @@ -1273,15 +1294,15 @@ impl KeysetCipher<'_, K> { /// to the target-directed path for the same `O`. For custom options, /// define a marker type implementing [`MatchConfig`]. /// - /// The same call serves both write time (index the stored text) and query - /// time (index the probe text, then test [`MatchTerms::contains`] - /// server-side). - /// - /// Fails with [`TermError::EmptyTermText`] — as - /// [`Error::Term`], once awaited — when the text - /// yields no tokens: empty or separator-only text, or an n-gram probe - /// shorter than the gram length (which could never match; see - /// [`Tokenizer::Ngram`]). + /// This derives the *stored* filter. Text that yields no tokens (empty + /// or separator-only text, or text shorter than the n-gram length; see + /// [`Tokenizer::Ngram`]) is the empty filter, which no query matches, + /// so a short value can be stored. A query is the other type, + /// [`MatchQuery`], and comes from the query chain + /// (`cipher.query(op::Matches, ..)`), which refuses a query of such + /// text ([`TermError::EmptyTermText`]) because its filter would match + /// every row; this method does not know which it is deriving and + /// refuses nothing. pub fn match_terms<'c, O>( &self, text: &str, diff --git a/packages/stack-encrypt/src/sem/tokenize.rs b/packages/stack-encrypt/src/sem/tokenize.rs index 8639de231..504e12536 100644 --- a/packages/stack-encrypt/src/sem/tokenize.rs +++ b/packages/stack-encrypt/src/sem/tokenize.rs @@ -17,11 +17,12 @@ use zeroize::Zeroizing; pub enum Tokenizer { /// Sliding character n-grams of the given length over the whole text /// (whitespace included). Text shorter than `length` yields **no tokens**, - /// exactly like the v1 match indexer — so a probe shorter than the gram - /// length is rejected by - /// [`match_terms`](crate::KeysetCipher::match_terms) rather than - /// silently never matching. This is the default, matching the existing - /// match indexer's 3-gram configuration. + /// exactly like the v1 match indexer — so a query shorter than the gram + /// length is refused by the query chain + /// ([`TermError::EmptyTermText`](crate::sem::TermError::EmptyTermText)) + /// rather than silently never matching, and a stored value shorter than + /// it is the empty filter, which no query matches. This is the default, + /// matching the existing match indexer's 3-gram configuration. Ngram { length: usize }, /// Split on the v1 match indexer's separator set — space, comma, /// semicolon, colon and exclamation mark — one token per run of diff --git a/packages/stack-encrypt/tests/derive.rs b/packages/stack-encrypt/tests/derive.rs index d5b87a823..5c4de8c96 100644 --- a/packages/stack-encrypt/tests/derive.rs +++ b/packages/stack-encrypt/tests/derive.rs @@ -10,7 +10,7 @@ use std::sync::atomic::Ordering as AtomicOrdering; use cllw_ore::CllwOreEncrypt; use common::{counting_cipher, recording_cipher, stack_cipher}; -use stack_encrypt::sem::{EqualityTerm, MatchTerms, OreTerm}; +use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, MatchTerms, OreTerm, TermError}; use stack_encrypt::target::{AeadContext, DecryptFrom, EncryptInto, Encrypted}; use stack_encrypt::{ nonempty, ContextPiece, DecryptField, DecryptInto, Decryptable, EncryptFrom, Error, @@ -436,20 +436,50 @@ async fn a_ciphertext_only_record_accepts_an_aead_only_context_like_the_leaf_doe assert!(matches!(result, Err(Error::Aead)), "{result:?}"); } +/// A match configuration whose options fail the scheme's bounds, so its +/// leaf fails during the synchronous build. +struct TooManyHashes; +impl MatchConfig for TooManyHashes { + fn options() -> MatchOptions { + MatchOptions { + k: 17, + ..Default::default() + } + } +} + +#[derive(EncryptFrom, DecryptInto)] +struct BadlyIndexedText { + c: StackCipherText, + m: MatchTerms, +} + #[tokio::test] async fn a_failed_field_fails_the_derived_record_before_any_io() { let (cipher, generates, _) = counting_cipher().await; let keyset = cipher.default_keyset(); - // Text that yields no match tokens fails that leaf during the - // synchronous build; the derived record is the zip of its fields, so it - // fails the same way and never mints the data key its ciphertext field - // would have wanted. - let result: Result = String::new() + // Invalid match options fail that leaf during the synchronous build; + // the derived record is the zip of its fields, so it fails the same way + // and never mints the data key its ciphertext field would have wanted. + let result: Result = "alice" + .to_string() .encrypt_into_with_context(&keyset, nonempty!("users/name")) .await; - assert!(matches!(result, Err(Error::Term(_)))); + assert!(matches!( + result, + Err(Error::Term(TermError::InvalidOptions(_))) + )); assert_eq!(generates.load(AtomicOrdering::SeqCst), 0); + + // Text with no match tokens is not a failure: the leaf is the empty + // filter, and the record seals as any other. + let stored: SearchableText = String::new() + .encrypt_into_with_context(&keyset, nonempty!("users/name")) + .await + .unwrap(); + assert!(stored.m.positions().is_empty()); + assert_eq!(generates.load(AtomicOrdering::SeqCst), 1); } // --- Structs: each field from one field of the plaintext, under its own context diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 7976d9178..f2cd7198b 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -8,7 +8,9 @@ use common::{loads_counting_cipher, stack_cipher}; use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::op::{self, Limit, MatchCase, Probabilistic, Stats}; use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; -use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms}; +use stack_encrypt::sem::{ + EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms, TermError, +}; use stack_encrypt::target::{Encrypted, IndexSpec}; use stack_encrypt::{Equality, Error, Match, Plan}; @@ -734,3 +736,108 @@ async fn target_limit_without_a_prior_needs_a_measured_column() { "10 wanted, λ = {lambda}: {rows}" ); } + +/// A value too short to cut a token (two letters under 3-grams, or nothing +/// at all) is a valid thing to store in a match column. ADR-0009: the +/// stored term is the empty filter, which no query matches, and the value +/// itself seals and opens as any other. +#[tokio::test] +async fn a_short_match_value_stores_as_the_empty_filter_and_no_query_finds_it() { + let cipher = stack_cipher().await; + let plan = Plan::context("users/name") + .with::((Equality, Match)) + .build() + .unwrap(); + let query = cipher.query(op::Matches, "alice").on(&plan).await.unwrap(); + + for short in ["Al", "", " "] { + let stored: Encrypted<(EqualityTerm, MatchTerms)> = cipher + .encrypt(&short.to_owned()) + .using(&plan) + .await + .unwrap_or_else(|error| panic!("{short:?} stores: {error}")); + assert!( + stored.terms.1.positions().is_empty(), + "{short:?} is stored as the empty filter" + ); + assert!( + !stored.terms.1.matches(&query), + "{short:?}: the empty filter holds none of a query's positions" + ); + assert!(!query.verify(short)); + let back: String = cipher.open(stored).using(&plan).await.unwrap(); + assert_eq!(back, short, "{short:?} opens as written"); + } + + // The equality term still finds it: only the match term is empty. + let stored: Encrypted<(EqualityTerm, MatchTerms)> = + cipher.encrypt(&"Al".to_owned()).using(&plan).await.unwrap(); + let eq = cipher.query(op::Eq, "Al").on(&plan).await.unwrap(); + assert_eq!(eq, stored.terms.0); +} + +/// A query with no tokens would match every row, so it is the one query +/// value the library refuses: when the query is awaited, before a term +/// exists and before any keyset is loaded. +#[tokio::test] +async fn a_match_query_with_no_tokens_is_refused_before_any_term_or_keyset() { + let (cipher, loads) = loads_counting_cipher().await; + let plan: Plan = Plan::context("people") + .fields() + .encrypt_index::("email", (Equality, Match)) + .encrypt_index::("name", Match::::new()) + .build() + .unwrap(); + let email_plan = plan.field("email").unwrap(); + + for text in ["", " ", "bo"] { + let refused = cipher + .query(op::Matches, text) + .on(&email_plan) + .keyset("tenant") + .await; + assert!( + matches!(refused, Err(Error::Term(TermError::EmptyTermText))), + "{text:?}: {refused:?}" + ); + } + // Through `via`, under another configuration, and in a batch. + let refused = cipher + .query(op::Matches.case_sensitive(), "Al") + .via(Match::::new()) + .on(&plan.field("name").unwrap()) + .await; + assert!(matches!( + refused, + Err(Error::Term(TermError::EmptyTermText)) + )); + let refused = stack_encrypt::all(( + cipher.query(op::Eq, "bob@example.com").on(&email_plan), + cipher.query(op::Matches, "bo").on(&email_plan), + )) + .await; + assert!(matches!( + refused, + Err(Error::Term(TermError::EmptyTermText)) + )); + assert_eq!( + loads.load(std::sync::atomic::Ordering::SeqCst), + 0, + "refused before the named keyset is loaded" + ); + assert_eq!( + TermError::EmptyTermText.to_string(), + "the query's text produces no match tokens (empty, separator-only, or shorter than the n-gram length)" + ); + + // The same text is a fine equality query, and a fine stored value. + assert!(cipher.query(op::Eq, "").on(&email_plan).await.is_ok()); + let row = cipher + .encrypt(&Person { + email: "bo".into(), + name: "Al".into(), + }) + .using(&plan) + .await; + assert!(row.is_ok(), "short values store: {row:?}"); +} diff --git a/packages/stack-encrypt/tests/sem_terms.rs b/packages/stack-encrypt/tests/sem_terms.rs index 1ce6b799f..e22d62d98 100644 --- a/packages/stack-encrypt/tests/sem_terms.rs +++ b/packages/stack-encrypt/tests/sem_terms.rs @@ -286,38 +286,47 @@ async fn match_rejects_invalid_options() { } #[tokio::test] -async fn match_rejects_text_that_yields_no_tokens() { - use stack_encrypt::sem::TermError; - +async fn match_stores_text_that_yields_no_tokens_as_the_empty_filter() { let cipher = generator().await; let gen = cipher.default_keyset(); + let query = gen + .match_terms::("hello", nonempty!("users/bio")) + .await + .unwrap(); - // An empty term used as a query would vacuously match every stored row. - for text in ["", " "] { + // Empty or separator-only text, and text shorter than the n-gram, cut + // no token. The stored side keeps such a value as the empty filter: a + // query always has a token, and the empty filter holds none of its + // positions, so no query matches the value. (A query of such text is + // refused by the query chain instead; `tests/op.rs` holds that.) + for text in ["", " ", "hi"] { + let stored = gen + .match_terms::(text, nonempty!("users/bio")) + .await + .unwrap_or_else(|error| panic!("{text:?} stores: {error}")); assert!( - matches!( - gen.match_terms::(text, nonempty!("users/bio")) - .await, - Err(Error::Term(TermError::EmptyTermText)) - ), - "{text:?} must be rejected" + stored.positions().is_empty(), + "{text:?} is the empty filter" + ); + assert!(stored.to_bytes().is_empty()); + assert!(!stored.contains(&query), "{text:?}: no query matches it"); + assert!( + !query.contains(&stored), + "{text:?}: and the empty filter, as a query, matches nothing" + ); + assert_eq!( + MatchTerms::::from_bytes(&stored.to_bytes()).unwrap(), + stored, + "{text:?}: the empty filter round-trips through its bytes" ); } - // A probe shorter than the n-gram length could never match a stored gram - // (v1 indexer semantics) — rejected instead of a silent false negative. - assert!(matches!( - gen.match_terms::("hi", nonempty!("users/bio")) - .await, - Err(Error::Term(TermError::EmptyTermText)) - )); - // Separator-only text under the Standard tokenizer. - assert!(matches!( - gen.match_terms::(" ,;:! ", nonempty!("users/bio")) - .await, - Err(Error::Term(TermError::EmptyTermText)) - )); + let stored = gen + .match_terms::(" ,;:! ", nonempty!("users/bio")) + .await + .unwrap(); + assert!(stored.positions().is_empty()); } #[tokio::test] diff --git a/packages/stack-encrypt/tests/target.rs b/packages/stack-encrypt/tests/target.rs index dba4be098..14e12f98b 100644 --- a/packages/stack-encrypt/tests/target.rs +++ b/packages/stack-encrypt/tests/target.rs @@ -178,6 +178,19 @@ async fn match_leaf_supports_containment_queries() { assert!(stored.contains(&query)); } +/// A match configuration whose options fail the scheme's bounds, so a term +/// under it fails during the synchronous build, before any I/O. +struct TooManyHashes; + +impl MatchConfig for TooManyHashes { + fn options() -> MatchOptions { + MatchOptions { + k: 17, + ..Default::default() + } + } +} + /// A custom type-level match configuration. struct SmallFilter; @@ -706,13 +719,13 @@ async fn a_column_stops_declaring_at_the_first_refused_row() { #[tokio::test] async fn a_failed_field_fails_the_record_before_any_kms_call() { // One failed field must not cause the record's other fields to mint - // data keys that are then thrown away. A match term over text that - // yields no tokens fails during the synchronous build, before any I/O. + // data keys that are then thrown away. A match term under invalid + // options fails during the synchronous build, before any I/O. let (cipher, generates, _) = counting_cipher().await; let cipher = cipher.default_keyset(); let b = "b".to_string(); - let zipped = MatchTerms::::encrypt_from(&"", &cipher, nonempty!("users/x")) + let zipped = MatchTerms::::encrypt_from(&"abc", &cipher, nonempty!("users/x")) .zip(StackCipherText::encrypt_from( &b, &cipher, diff --git a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs index c9509d133..9e74ad2cc 100644 --- a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs +++ b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.rs @@ -23,7 +23,7 @@ impl Index for Shortcut { impl op::Supports for Shortcut { type Query = EqualityTerm; type Carry = (); - fn check(&self) -> Result<(), stack_encrypt::PlanError> { + fn check(&self, _: &String) -> Result<(), stack_encrypt::Error> { Ok(()) } fn carry(&self, _: &String) {} From a9f42c13b5683a20f46c3beeda3e23e5d83c59b2 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 22:44:09 +1100 Subject: [PATCH 19/38] fix(stack-encrypt): address the review of the empty match filter match_terms is no longer a query. In 0.2.0 the same call served write time and query time and refused text that cuts no token; it now returns the empty filter for that text, and EQL treats an empty filter as contained by every stored filter. A caller who still sends its output as a query gets every row and no error, so the CHANGELOG entry moves to Breaking and says so plainly, naming cipher.query(op::Matches, ..) as the query. The query side validated the match options only when it derived the term, after the new no-token check had tokenised the text, so an n-gram length of zero panicked in the tokenizer instead of returning InvalidOptions. Both query prechecks, op::Supports::check and dynamic::term, now validate first, as the stored path always has. Go's term-failure locator asks the query path for each term, and the query path refuses short match text that a record stores as the empty filter. It could stop at an earlier row with a short email and name the wrong row. It now skips that one refusal by its code, and a test pins it with a short email in row 0. ADR-0009 and the Go package doc say that a stored empty filter tells a reader of the column the value is empty, separator-only or shorter than the n-gram length. The tokenizer's doc no longer says match_terms rejects that text. --- languages/golang/encrypt/doc.go | 6 ++- languages/golang/encrypt/locate_test.go | 41 ++++++++++++++++++ languages/golang/encrypt/records.go | 20 +++++++-- packages/stack-encrypt/CHANGELOG.md | 36 +++++++++------- ...and-a-query-term-is-not-the-stored-term.md | 5 ++- packages/stack-encrypt/src/dynamic/term.rs | 30 +++++++++++++ packages/stack-encrypt/src/op/mod.rs | 4 ++ packages/stack-encrypt/src/sem/tokenize.rs | 8 ++-- packages/stack-encrypt/tests/op.rs | 42 ++++++++++++++++++- 9 files changed, 166 insertions(+), 26 deletions(-) create mode 100644 languages/golang/encrypt/locate_test.go diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 7036e1303..01179f7db 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -107,8 +107,10 @@ // A match index stores text with no token (an empty string, separator-only // text, or text shorter than the n-gram length of 3 characters) as the empty // filter: a valid value that no query matches, so an optional or short value -// can live under match. A match *query* of such text is refused with -// [ErrTerm], because an empty filter as a query would match every row. +// can live under match. The empty filter tells anyone who can read the +// column that the value is one of those. A match *query* of such text is +// refused with [ErrTerm], because an empty filter as a query would match every +// row. // // # Transport and auth // diff --git a/languages/golang/encrypt/locate_test.go b/languages/golang/encrypt/locate_test.go new file mode 100644 index 000000000..04413a1f9 --- /dev/null +++ b/languages/golang/encrypt/locate_test.go @@ -0,0 +1,41 @@ +package encrypt + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// A record stores match text that cuts no token as the empty filter, so the +// term-failure locator must not blame a row for it. Derive asks the query +// path, which refuses that text; the locator skips that refusal and goes on +// to the rows after it. +func TestLocateTermFailureSkipsShortMatchText(t *testing.T) { + ctx := context.Background() + c, err := NewDeterministicClient(ctx, [32]byte{}) + if errors.Is(err, ErrDeterministicGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = c.Close() }) + cph := c.DefaultKeyset() + plan := usersPlan() + + // The query path refuses the short email, so without the skip the + // locator would name row 0 for it. + _, derr := cph.Derive(ctx, plan, "email", record.Match, "ab") + if !queryOnlyRefusal(record.Match, derr) { + t.Fatalf("Derive of a short email: %v, want the empty-term-text refusal", derr) + } + + sentinel := fmt.Errorf("%w: from the record call", ErrTerm) + rows := []record.Source{usersRow(1, "ab"), usersRow(2, "alice@example.com")} + if got := cph.locateTermFailure(ctx, plan, rows, sentinel); got != sentinel { + t.Fatalf("locateTermFailure = %v, want the record call's error unchanged", got) + } +} diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index 80f1f655f..39094109e 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -473,8 +473,10 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, context string, p // term at a time, which costs no key request: a term derives locally. A // stored value is never refused for its text (text with no token is stored // as the empty filter), so a term failure on a record is the scheme refusing -// the value's kind under that index. p is the plan without this cipher's -// extension, which Derive adds. +// the value's kind under that index. Derive asks the query path, which does +// refuse text with no token, so that one refusal is skipped here: it cannot +// be why the record failed. p is the plan without this cipher's extension, +// which Derive adds. func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error { for i, row := range rows { for _, f := range p.Fields { @@ -482,7 +484,7 @@ func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows [ if !o.IsTerm() { continue } - if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) { + if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) && !queryOnlyRefusal(o, derr) { return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value", err, i, f.Name, indexWord(o)) } } @@ -491,6 +493,18 @@ func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows [ return err } +// queryOnlyRefusal reports whether derr is the one term refusal the query +// path makes and the record path does not: match text that cuts no token, +// which a record stores as the empty filter. +func queryOnlyRefusal(o record.Output, derr error) bool { + var d *Diagnostic + return o == record.Match && errors.As(derr, &d) && d.Code == emptyTermTextCode +} + +// emptyTermTextCode is the stable code of the engine's refusal of match text +// that cuts no token. +const emptyTermTextCode = "stack_encrypt::empty_term_text" + func indexWord(o record.Output) string { switch o { case record.Equality: diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index ad23b7421..eb9a4464e 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -93,24 +93,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 under it. `FieldPlan::with_target`, `FieldPlan::target()`, `Plan::new_with`. -### Changed - -- **A match value with no tokens is stored as the empty filter; only a query - with none is refused.** Text that cuts no token under the match index's - tokenizer (empty or separator-only text, or text shorter than the n-gram - length) used to fail term derivation on both sides, so one two-letter name - failed a whole row. Now the stored term is the empty filter (`bf: []`), - which no query matches, and the value seals and opens as any other; the - `dynamic` record path stores it the same way. A *query* of such text is - still refused, with `TermError::EmptyTermText` as before, because its - filter would match every row: `cipher.query(op::Matches, ..)` refuses it - when the query is awaited, before a term exists and before any keyset is - loaded (`op::Supports::check` now takes the query's value), and - `dynamic::term` refuses it for a match kind. `KeysetCipher::match_terms` - derives the stored filter and no longer fails on such text (ADR-0009). - ### Breaking +- **A match value with no tokens is stored as the empty filter, so + `KeysetCipher::match_terms` is no longer a query.** In 0.2.0 the same call + served write time and query time, and refused text that cuts no token + under the match index's tokenizer (empty or separator-only text, or text + shorter than the n-gram length) with `TermError::EmptyTermText`. Now it + derives the stored filter only, and returns the empty filter (`bf: []`) + for such text with no error. **Do not send the output of `match_terms` as + a query:** EQL treats an empty filter as contained by every stored filter, + and its query domains do not refuse one, so that query returns every row + with no error anywhere. A query comes from `cipher.query(op::Matches, ..)`, + which still refuses such text with `TermError::EmptyTermText`, when the + query is awaited, before a term exists and before any keyset is loaded + (`op::Supports::check` now takes the query's value); `dynamic::term` + refuses it for a match kind the same way. The stored side is the change a + caller sees: one two-letter name no longer fails a whole row, the value + seals and opens as any other, and the `dynamic` record path stores it the + same way. Storing the empty filter tells a reader of the database that the + value is empty, separator-only or shorter than the n-gram length + (ADR-0009). + - **The query chain starts with the operator.** `cipher.query(v)` takes the operator first: `cipher.query(op::Eq, v).on(&plan)` replaces `cipher.query(v).using(&plan).equality()`, and `on` replaces `using` for diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index cf96fb9cf..64eb0903f 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -262,7 +262,10 @@ library refuses, a query value too short to tokenise, is refused when the query is awaited, before a term exists. The stored side refuses nothing: a value too short to tokenise is stored as the empty filter, which no query matches, so a two-letter name is a valid value in a match column and `verify` is -never asked about it. +never asked about it. The empty filter is visible to anyone who can read the +column: it tells them the value is empty, separator-only or shorter than the +n-gram length. A filter's fill already leaks its value's approximate length; +this is an exact class, and the price of storing such values at all. `Plaintext` is the decrypted value the term is defined over, and that is not always the value the user wrote. For `Matches` it is the text. For diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index 3b75fe7b8..db5da5081 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -311,6 +311,11 @@ where D: IntoPrfContext<'c>, { if let (IndexSpec::Match(options), Scalar::Text(text)) = (kind, &scalar) { + // Options the index cannot use are refused before the text is + // tokenised, as the record path refuses them. + let _mask = options + .validate() + .map_err(|error| Error::Cipher(error.into()))?; if QueryTokens::under(text, options).is_empty() { return Err(Error::Cipher(TermError::EmptyTermText.into())); } @@ -662,6 +667,31 @@ mod tests { assert!(term(&keyset, scalar, &default_match(), ctx).await.is_ok()); } + /// Options the index cannot use are refused before the query's text is + /// tokenised: an n-gram length of zero is `InvalidOptions`, not a panic + /// in the tokenizer. + #[tokio::test] + async fn a_match_query_under_invalid_options_is_refused_not_tokenised() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/bio")).expect("context"); + let zero = IndexSpec::Match(MatchOptions { + tokenizer: Tokenizer::Ngram { length: 0 }, + ..MatchOptions::default() + }); + let scalar = Scalar::of(&s("hello"), &zero).expect("a scalar"); + let refused = term(&keyset, scalar, &zero, ctx).await; + assert!( + matches!( + refused, + Err(Error::Cipher(crate::Error::Term( + TermError::InvalidOptions(_) + ))) + ), + "{refused:?}" + ); + } + /// A `TermBytes` reads as the typed term's bytes, through every /// accessor: `as_bytes`, `as_ref` and `into_bytes` give the same /// 32 PRF bytes an `EqualityTerm` holds, for a term derived through diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index bb4dded0a..686e2c2cb 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -350,6 +350,10 @@ where } .into()); } + // Options the index cannot use are refused before the text is + // tokenised, as term derivation refuses them: an n-gram length of + // zero would otherwise reach the tokenizer. + let _mask = O::options().validate()?; // A query with no tokens would match every row. The stored side // keeps such a value as the empty filter; the query side refuses it // here, before any term is derived. diff --git a/packages/stack-encrypt/src/sem/tokenize.rs b/packages/stack-encrypt/src/sem/tokenize.rs index 504e12536..735a218a5 100644 --- a/packages/stack-encrypt/src/sem/tokenize.rs +++ b/packages/stack-encrypt/src/sem/tokenize.rs @@ -44,10 +44,12 @@ impl Default for Tokenizer { /// are case-insensitive. /// /// May return no tokens (empty text, separator-only text, or text shorter -/// than the n-gram length); [`match_terms`] rejects that case so an empty -/// term can never reach a query. +/// than the n-gram length). A stored value with none is the empty filter; a +/// query with none is refused before a term exists, because its filter would +/// match every row (ADR-0009). /// -/// [`match_terms`]: crate::StackCipher::match_terms +/// The tokenizer must be valid: an n-gram length of zero panics. Callers +/// validate the [`MatchOptions`](super::MatchOptions) first. /// /// The text is plaintext, so every copy made on the way is zeroized when it /// is dropped: the folded text, its characters, and each n-gram, which is diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index f2cd7198b..aeaa4c3b9 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -9,7 +9,7 @@ use stack_encrypt::kms::FakeDataKeySource; use stack_encrypt::op::{self, Limit, MatchCase, Probabilistic, Stats}; use stack_encrypt::plan::{Field, FieldSchema, Fields, PlanError}; use stack_encrypt::sem::{ - EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms, TermError, + EqualityTerm, MatchConfig, MatchOptions, MatchQuery, MatchTerms, TermError, Tokenizer, }; use stack_encrypt::target::{Encrypted, IndexSpec}; use stack_encrypt::{Equality, Error, Match, Plan}; @@ -29,6 +29,20 @@ impl MatchCase for Exact { type Case = op::CaseSensitive; } +/// A match configuration the index cannot use: an n-gram length of zero. +struct ZeroGram; +impl MatchConfig for ZeroGram { + fn options() -> MatchOptions { + MatchOptions { + tokenizer: Tokenizer::Ngram { length: 0 }, + ..Default::default() + } + } +} +impl MatchCase for ZeroGram { + type Case = op::CaseInsensitive; +} + #[derive(Clone)] struct Person { email: String, @@ -841,3 +855,29 @@ async fn a_match_query_with_no_tokens_is_refused_before_any_term_or_keyset() { .await; assert!(row.is_ok(), "short values store: {row:?}"); } + +/// A match query under options the index cannot use is refused with +/// `InvalidOptions`, as storing under them is, and never reaches the +/// tokenizer: an n-gram length of zero would panic there. +#[tokio::test] +async fn a_match_query_under_invalid_options_is_refused_not_tokenised() { + let cipher = stack_cipher().await; + let plan = Plan::context("users/name") + .with::(Match::::new()) + .build() + .unwrap(); + let refused = cipher + .query(op::Matches, "alice") + .via(Match::::new()) + .on(&plan) + .await; + assert!( + matches!(refused, Err(Error::Term(TermError::InvalidOptions(_)))), + "{refused:?}" + ); + let stored = cipher.encrypt(&"alice".to_owned()).using(&plan).await; + assert!( + matches!(stored, Err(Error::Term(TermError::InvalidOptions(_)))), + "{stored:?}" + ); +} From 402f8b0f74ff28cd0583bc25ac0d950bfdbecdc9 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 14 Oct 2026 18:05:19 +1100 Subject: [PATCH 20/38] feat(stack-encrypt): the ordering operators, Between as one term, and no query_index ADR-0009 decisions 1 and 7. `op::Lt`, `Lte`, `Gt` and `Gte` take one value and `op::Between` a pair of bounds, inclusive at both ends. The ORE and OPE indexes answer them over the integer types (`op::Ordered`), ORE by default and OPE through `.via(Ope)`; an ordering on text does not compile, since text has no order term this crate produces. A comparison's query term is the index's own term for the value: this crate's ORE is CLLW, whose ciphertext has no left half to send alone, so what is sent is what is stored, as for an HMAC, and `Supports` is where the left-only term of block ORE (#1140) goes when it lands. A range's query term is one `sem::RangeQuery` carrying both bounds' terms, derived under the one label, read with `lo()` and `hi()`. A range whose low bound is above its high bound is refused when the query is awaited, before either term is derived and before any keyset is loaded (`TermError::InvertedRange`). An operator now says what a query takes through `op::Arity`: `One` plaintext or a `Pair` of them. `op::Input` relates the value a query is given to the plaintext the plan is checked against, and derives one term per plaintext. `Supports`, `Answers` and `DefaultIndex` are keyed by that input, so `Supports` reads as before and `Between` is `Supports`; `Supports::query` takes the input's terms. `StackCipher::query_index` goes, with `QueryIndexBuilder` and `QueryIndex`: every index a plan can declare is reachable through an operator now, and the chain holds its own input, index and label. `Resolves::resolve` takes the index and the call's context and returns the index and the label. ADR-0009's decision 2 shows the shape that ships, and its mentions of `query_index` are past tense. Closes #1186 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB --- packages/stack-encrypt/CHANGELOG.md | 28 ++ ...and-a-query-term-is-not-the-stored-term.md | 74 ++- packages/stack-encrypt/src/codes.rs | 2 + packages/stack-encrypt/src/op/mod.rs | 439 +++++++++++++++--- packages/stack-encrypt/src/plan/chain.rs | 349 +++++--------- packages/stack-encrypt/src/plan/mod.rs | 4 +- packages/stack-encrypt/src/sem/mod.rs | 64 ++- packages/stack-encrypt/tests/op.rs | 300 +++++++++++- packages/stack-encrypt/tests/plan.rs | 22 +- packages/stack-encrypt/tests/plan_grammar.rs | 23 +- .../tests/ui/op_between_takes_a_pair.rs | 16 + .../tests/ui/op_between_takes_a_pair.stderr | 37 ++ .../tests/ui/op_matches_on_integer.stderr | 23 +- .../tests/ui/op_order_on_text.rs | 16 + .../tests/ui/op_order_on_text.stderr | 18 + .../tests/ui/op_target_does_not_answer.stderr | 6 +- .../stack-encrypt/tests/ui/pass/plan_chain.rs | 3 +- 17 files changed, 1033 insertions(+), 391 deletions(-) create mode 100644 packages/stack-encrypt/tests/ui/op_between_takes_a_pair.rs create mode 100644 packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr create mode 100644 packages/stack-encrypt/tests/ui/op_order_on_text.rs create mode 100644 packages/stack-encrypt/tests/ui/op_order_on_text.stderr diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index eb9a4464e..842612346 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -45,6 +45,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 copies of the text. An exact term has no `verify`, and none of this crate's indexes can answer `op::Matches` but with a probabilistic term (`op::TermFor`). +- **The ordering operators, and `Between` as one term.** `op::Lt`, `Lte`, + `Gt` and `Gte` take one value and `op::Between` a pair of bounds, + `cipher.query(op::Between, &(18, 65)).on(&age_plan)`, inclusive at both + ends. They are answered over the integer types (`op::Ordered`) by the + `Ore` index by default and by `Ope` through `.via(Ope)`; an ordering on + text does not compile. A comparison's query term is the index's own term + for the value (`OreTerm`, `OpeTerm`); a range's is one `sem::RangeQuery` + carrying both bounds' terms, read with `lo()` and `hi()`, so no statement + can pair the low bound of one query with the high bound of another. A + range whose low bound is above its high bound is refused when the query + is awaited, before either term is derived (`TermError::InvertedRange`). + What a query takes is the operator's `op::Arity` (`op::One` or + `op::Pair`), and `op::Input` relates the value given to the plaintext the + plan is checked against (ADR-0009, decision 7). - **`target_limit` says how many rows a match query should fetch.** `query.target_limit(want, &stats)` is `want` plus the false positives expected at the confidence `stats` names (99% by default), as an @@ -115,6 +129,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 value is empty, separator-only or shorter than the n-gram length (ADR-0009). +- **`StackCipher::query_index` is gone**, with `QueryIndexBuilder` and + `QueryIndex`: every index a plan can declare is reachable through an + operator now that the orderings have arrived. `query_index(Ore, &v)` is + `query(op::Lt, &v)` (or any ordering; the term is the same), and + `query_index(Match, text)` is `query(op::Matches, text)`, whose + `MatchQuery` carries the same filter. +- **`op::Supports`, `op::Answers` and `op::DefaultIndex` are keyed by the + query's input**, `Supports` with `In: Input`, in + place of the plaintext; for every operator but `Between` the input is + the plaintext, so `Supports` reads as before. `Supports::check` + takes the input and returns `Error`, and `Supports::query` takes the + input's terms (`In::Terms`: one term, or a pair for a + range). `plan::Resolves::resolve` takes the index and the call's context + and returns the index and the label. - **The query chain starts with the operator.** `cipher.query(v)` takes the operator first: `cipher.query(op::Eq, v).on(&plan)` replaces `cipher.query(v).using(&plan).equality()`, and `on` replaces `using` for diff --git a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md index 64eb0903f..e15a07014 100644 --- a/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md +++ b/packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md @@ -141,30 +141,46 @@ query against a case-folding index is a type error, not an empty result. ### 2. An index declares the operators it supports, and the query term for each ```rust -pub trait Operator: Copy + 'static {} - -pub trait Supports: Index { - type Query: TermFor; // what this index sends for `Op` - type Carry; // what the query keeps of the value - fn check(&self) -> Result<(), PlanError>; - fn carry(&self, value: &S) -> Self::Carry; - fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query; +pub trait Operator: Copy + 'static { type Arity: Arity; } // `One` plaintext, or a `Pair` + +pub trait Supports>: Index { + type Query: TermFor; // what this index sends for `Op` + type Carry; // what the query keeps of the input + fn check(&self, input: &In) -> Result<(), Error>; // a refusal, before any term exists + fn carry(&self, input: &In) -> Self::Carry; + fn query(carry: Self::Carry, terms: In::Terms, token: QueryToken) -> Self::Query; } ``` -`S` is the plaintext type the index is over, and `Index::Term` is the -stored term it derives. `Query` is the query term, and `query` makes it from -the term the index derived for the query's value. `QueryToken` can be made -only inside the crate, so only a query chain calls `query`, and no caller can -turn a stored term into a query term by hand. `check` refuses a configuration -that cannot answer `Op` although its type says it can: a match index whose -declared case disagrees with its `downcase` option. +`In::Plaintext` is the plaintext type the index is over, and its +`Index::Term` is the stored term it derives. `Query` is the query term, and +`query` makes it from what was kept and the terms the index derived for the +query's input. `QueryToken` can be made only inside the crate, so only a +query chain calls `query`, and no caller can turn a stored term into a query +term by hand. `QueryTerm` and the two traits that refine it are defined in decision 4. `Query` is not `Term`. An ORE index's `Supports::Query` is the left -ciphertext; its `Term` is left and right. A Bloom index's -`Supports::Query` is a filter over the needle (`MatchQuery`); its -`Term` is a filter over the value (`MatchTerms`). +ciphertext; its `Term` is left and right. (This crate's ORE is CLLW today, +whose ciphertext has no left half to send alone, so until block ORE lands, +#1140, the query term is the ciphertext itself, as an HMAC's is; `Supports` +is where the left-only term goes when it arrives.) A Bloom index's +`Supports::Query` is a filter over the query's value +(`MatchQuery`); its `Term` is a filter over the value (`MatchTerms`). There is +no other path from a derived term to the server: +`cipher.query_index(index, v).on(&plan)`, which returned an index's own term, +went with the ordering operators (#1186). + +An operator says what a query takes through its arity. `Eq`, the orderings +and `Matches` take the plaintext the index is declared over +(`Supports`); `Between` takes a pair of them +(`Supports`); `Input` relates the input to that plaintext, +so the plan is checked against `S` either way, and the chain derives one +term per plaintext under the one label. `check` is where a query is refused +before any term exists: a match index whose declared case disagrees with its +`downcase` option, a `Between` whose bounds are the wrong way round, a match +query with no tokens. `AtPath` takes a path and `Op`'s input, and gains +its shape with the JSON index. The bound tightens per operator, through `TermFor`: any `QueryTerm` answers `Eq`, but only a `Probabilistic` one answers `Matches`, and only a @@ -520,10 +536,14 @@ the ranked loop is worth packaging; it is deferred, not refused. ### 7. `Between` is one term and one operand -`cipher.query(op::Between, (lo, hi))` produces one term carrying both ORE left -ciphertexts, inclusive at both ends, and the client refuses `lo > hi`. One -call derives both bounds, under one context, so a caller cannot pair a bound -from one column with a bound from another. +`cipher.query(op::Between, &(lo, hi)).on(&plan)` produces one term, +`sem::RangeQuery`, carrying the index's term for each bound (the ORE left +ciphertexts, once block ORE lands), inclusive at both ends, and the client +refuses `lo > hi` (`TermError::InvertedRange`) before either term is +derived; `(v, v)` is one value. One call derives both bounds, under one +context, so a caller cannot pair a bound from one column with a bound from +another. The ordering operators are answered over the integer types +(`op::Ordered`), by the ORE index by default and by OPE through `.via(Ope)`. In SQL the term stays one operand. EQL v4 gains a function that takes the column and the range, and a range query domain beside each ordered query @@ -651,9 +671,13 @@ EQL v4 gains `eql_v4.between(column, range)` and a range query domain per ordered query domain, ORE and OPE, under the inlining requirement of decision 7. That is EQL-side work this ADR asks for, not a deferral. -It is a breaking change to the query chain. `QueryUsing::index(..)` goes; -`QueryUsing::equality()` becomes `query(op::Eq, ..)`. The write side -(`encrypt`, targets, plans) does not change. +It is a breaking change to the query chain. `cipher.query(v).using(&plan)` +with `.equality()` becomes `cipher.query(op::Eq, v).on(&plan)`: the operator +first, and `on` for a query where a write says `using`. +`cipher.query_index(index, v).on(&plan)`, the index's own term, is gone: +every index a plan can declare is reachable through an operator, ORE and OPE +through the ordering operators (#1186). The write side (`encrypt`, targets, +plans) does not change. ## Deferred diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index 7d6929fc8..6ac2e381a 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -80,6 +80,7 @@ mod tests { TermError::Ore(_) => TermError::Ore(cllw_ore::Error), TermError::InvalidOptions(_) => TermError::InvalidOptions("k out of range"), TermError::EmptyTermText => TermError::EmptyTermText, + TermError::InvertedRange => TermError::InvertedRange, TermError::Bytes(_) => TermError::Bytes(TermBytesError::OddMatchTermsLength(3)), ])); errors.extend(boxed(variants![ @@ -385,6 +386,7 @@ mod tests { json!({ "len": 3 }), ), (Box::new(TermError::EmptyTermText), json!({})), + (Box::new(TermError::InvertedRange), json!({})), ( Box::new(TermBytesError::MalformedCllwCiphertext(5)), json!({ "len": 5 }), diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index 686e2c2cb..a50ad23f2 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -1,22 +1,27 @@ //! Operators: what a query asks of the server, as a type. //! -//! A query names an *operator* ([`Eq`](struct@Eq), [`Matches`](struct@Matches)), and the plan it -//! runs on finds the index that answers it. The operator is a unit type -//! passed by value, as an index is (`.via(Ore)`): +//! A query names an *operator* ([`Eq`](struct@Eq), [`Lt`], [`Between`], +//! [`Matches`](struct@Matches)), and the plan it runs on finds the index +//! that answers it. The operator is a unit type passed by value, as an +//! index is (`.via(Ore)`), and it decides what the query takes: one value +//! for most, a pair of bounds for `Between`: //! //! ``` //! # async fn example() -> Result<(), stack_encrypt::Error> { //! use stack_encrypt::kms::FakeDataKeySource; -//! use stack_encrypt::{op, Equality, Match, Plan, StackCipher}; +//! use stack_encrypt::{op, Equality, Match, Ore, Plan, StackCipher}; //! //! let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; //! let email_plan = Plan::context("users/email") //! .with::((Equality, Match)) //! .build()?; +//! let age_plan = Plan::context("users/age").with::(Ore).build()?; //! //! let exact = cipher.query(op::Eq, "bob@example.com").on(&email_plan).await?; //! let search = cipher.query(op::Matches, "bob").on(&email_plan).await?; -//! # let _ = (exact, search); +//! let under = cipher.query(op::Lt, &40u32).on(&age_plan).await?; +//! let range = cipher.query(op::Between, &(18u32, 65)).on(&age_plan).await?; +//! # let _ = (exact, search, under, range); //! # Ok(()) //! # } //! # tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(example()).unwrap(); @@ -24,34 +29,42 @@ //! //! Three traits connect an operator to an index, and ADR-0009 records why: //! -//! - [`Supports`] is on an index, for each operator it can answer -//! over plaintexts `S`, and names the [`QueryTerm`] it sends. That term is -//! not the stored term: an equality index sends the same HMAC it stores, -//! but a match index sends a filter over the query's value -//! ([`MatchQuery`]), where it stores a filter over the stored value -//! ([`MatchTerms`]). -//! - [`Answers`] is on a target type, a domain type such as an EQL +//! - [`Supports`] is on an index, for each operator it can answer +//! and the input `In` the query takes, and names the [`QueryTerm`] it +//! sends. That term is not the stored term: an equality index sends the +//! same HMAC it stores, but a match index sends a filter over the query's +//! value ([`MatchQuery`]), where it stores a filter over the stored value +//! ([`MatchTerms`]), and an ORE index sends one [`RangeQuery`] for +//! `Between`, carrying both bounds, where it stores one term per value. +//! - [`Answers`] is on a target type, a domain type such as an EQL //! type, and names the index that answers `Op` for it. It is written by //! hand per type, because the choice decides what the query term reveals. -//! - [`DefaultIndex`] is on an operator, and names the index a plan that +//! - [`DefaultIndex`] is on an operator, and names the index a plan that //! declares its indexes, rather than a target type, answers it with. //! +//! What a query takes is the operator's [`Arity`]: [`One`] plaintext, or a +//! [`Pair`] of them as a tuple. [`Input`] says, for the value a query is +//! given, which plaintext the index is declared over (`S` for `(S, S)`), +//! so the plan's type is checked against it. +//! //! What the types can see is checked at compile time: a match query on an -//! integer, an operator a target type does not answer, and, on a one-value -//! plan, an index the plan does not hold or a case-sensitive match against -//! a case-folding index. A field of a fields plan is data, so there an -//! index the field does not declare, or declares with other options, is -//! refused when the query is awaited, as with -//! [`query_index`](crate::StackCipher::query_index), before any key is -//! requested. -//! So is a match configuration whose [`MatchCase`] disagrees with its -//! options, and so is the one query value the library refuses: text that -//! cuts no token under the match index's tokenizer, whose filter would -//! match every row ([`TermError::EmptyTermText`]). The stored side has no -//! such refusal: a value with no tokens is stored as the empty filter, -//! which no query matches. +//! integer, an ordering on text, an operator a target type does not answer, +//! and, on a one-value plan, an index the plan does not hold or a +//! case-sensitive match against a case-folding index. A field of a fields +//! plan is data, so there an index the field does not declare, or declares +//! with other options, is refused when the query is awaited, before any +//! key is requested. So is a match configuration whose [`MatchCase`] +//! disagrees with its options, and so are the two query values the library +//! refuses: match text that cuts no token under the index's tokenizer, +//! whose filter would match every row ([`TermError::EmptyTermText`]), and +//! a `Between` whose low bound is above its high bound +//! ([`TermError::InvertedRange`]). The stored side has no such refusal: a +//! value with no tokens is stored as the empty filter, which no query +//! matches. //! //! [`TermError::EmptyTermText`]: crate::sem::TermError::EmptyTermText +//! [`TermError::InvertedRange`]: crate::sem::TermError::InvertedRange +//! [`RangeQuery`]: crate::sem::RangeQuery //! //! The module is meant to be used by path (`op::Eq`): an imported `Eq` //! would shadow the prelude's `Eq` trait. @@ -67,22 +80,194 @@ pub(crate) mod stats; pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE, MAX_ROWS}; use crate::sem::{ - DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens, TermError, + CllwOpeEncrypt, CllwOreEncrypt, DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, + MatchTerms, OpeTerm, OreTerm, QueryTokens, RangeQuery, TermError, }; -use crate::target::{Equality, Index, Match}; -use crate::{Error, PlanError}; +use crate::target::{DeclaredContext, Equality, Index, Match, Ope, Ore, Owned, Pending}; +use crate::{Error, KeysetCipher, Label, NonEmpty, PlanError}; -/// A query operator: a unit type naming what a query asks of the server. +/// A query operator: a unit type naming what a query asks of the server, +/// and how many plaintexts it asks about ([`Arity`]). /// /// The trait is open, as [`Index`] is: an operator defined in another crate /// is answered by the indexes that implement [`Supports`] for it. -pub trait Operator: Copy + 'static {} +pub trait Operator: Copy + 'static { + /// What a query for this operator takes: [`One`] plaintext, or a + /// [`Pair`] of them as a tuple. + type Arity: Arity; +} /// Equality: rows whose value equals the query's. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] pub struct Eq; -impl Operator for Eq {} +impl Operator for Eq { + type Arity = One; +} + +/// Less than: rows whose value is below the query's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Lt; + +/// Less than or equal: rows whose value is at or below the query's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Lte; + +/// Greater than: rows whose value is above the query's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Gt; + +/// Greater than or equal: rows whose value is at or above the query's. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Gte; + +/// A range: rows whose value is between the query's two bounds, inclusive +/// at both ends. It takes a [`Pair`], `cipher.query(op::Between, &(lo, hi))`, +/// derives both bounds under one context, and sends them as one term +/// ([`RangeQuery`]), so no statement can pair the +/// low bound of one query with the high bound of another. A pair whose low +/// bound is above its high bound is refused when the query is awaited +/// ([`TermError::InvertedRange`]); one +/// value is `(v, v)`. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Between; + +impl Operator for Lt { + type Arity = One; +} +impl Operator for Lte { + type Arity = One; +} +impl Operator for Gt { + type Arity = One; +} +impl Operator for Gte { + type Arity = One; +} +impl Operator for Between { + type Arity = Pair; +} + +/// How many plaintexts a query for an operator takes: [`One`] or a +/// [`Pair`]. Sealed: the chain derives a term per plaintext and joins them +/// as the arity says, so there are exactly these two. +pub trait Arity: sealed::Sealed + 'static {} + +/// One plaintext: the query is given an `S` (`Eq`, `Lt`, `Matches`). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct One; + +/// Two plaintexts: the query is given an `(S, S)` (`Between`). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Pair; + +impl Arity for One {} +impl Arity for Pair {} + +/// What a query of arity `A` is given, and the plaintext its index is +/// declared over: an `S` is an input of [`One`] over `S`, and an `(S, S)` +/// an input of [`Pair`] over `S`. The chain's value type is the input, and +/// the plan is checked against the plaintext. +/// +/// Those are the two impls. The arity decides how the chain derives terms +/// from the input, which is what [`derive`](Self::derive) does. +pub trait Input: Sized + 'static { + /// The plaintext the index is declared over. + type Plaintext: 'static; + /// The index's terms for this input: one, or a pair of them. + type Terms: 'static; + + /// Derive the index's term for each plaintext of the input under + /// `label`, as one pending. Only a query chain calls this. + #[doc(hidden)] + fn derive<'a, 'p, I, K>( + self, + index: &I, + keyset: &'p KeysetCipher<'a, K>, + label: Label, + extend: DeclaredContext, + ) -> Pending<'p, Self::Terms, K> + where + I: Index, + K: 'static; +} + +impl Input for S { + type Plaintext = S; + type Terms = T; + + fn derive<'a, 'p, I, K>( + self, + index: &I, + keyset: &'p KeysetCipher<'a, K>, + label: Label, + extend: DeclaredContext, + ) -> Pending<'p, I::Term, K> + where + I: Index, + K: 'static, + { + keyset.run( + index.operation::().under(NonEmpty::from(label)), + self, + extend, + ) + } +} + +/// Both bounds under the one label, settled together. +impl Input for (S, S) { + type Plaintext = S; + type Terms = (T, T); + + fn derive<'a, 'p, I, K>( + self, + index: &I, + keyset: &'p KeysetCipher<'a, K>, + label: Label, + extend: DeclaredContext, + ) -> Pending<'p, (I::Term, I::Term), K> + where + I: Index, + K: 'static, + { + let (lo, hi) = self; + let lo = keyset.run( + index + .operation::() + .under(NonEmpty::from(label.clone())), + lo, + extend.clone(), + ); + let hi = keyset.run( + index.operation::().under(NonEmpty::from(label)), + hi, + extend, + ); + lo.zip(hi) + } +} + +/// A plaintext the ordering operators ([`Lt`], [`Lte`], [`Gt`], [`Gte`], +/// [`Between`]) are answered over: the integer types, whose ORE and OPE +/// terms order as the values do and whose order key is the value itself. +/// Text is not one: its order term is a catalog question (ADR-0009, ADR-0002), +/// and an ordering on a text plan does not compile. +pub trait Ordered: + PartialOrd + + CllwOreEncrypt + + CllwOpeEncrypt + + Send + + 'static +{ +} + +macro_rules! ordered { + ($($int:ty),+ $(,)?) => {$( + impl Ordered for $int {} + )+}; +} +ordered!(u8, u16, u32, u64, u128, i8, i16, i32, i64, i128); /// Text search: rows whose text contains the query's tokens, under the /// match index's tokenizer. Not containment: a Bloom match is order- and @@ -148,12 +333,16 @@ impl fmt::Debug for Matches { } } -impl Operator for Matches {} +impl Operator for Matches { + type Arity = One; +} mod sealed { pub trait Sealed {} impl Sealed for super::CaseInsensitive {} impl Sealed for super::CaseSensitive {} + impl Sealed for super::One {} + impl Sealed for super::Pair {} } /// The case a text operator is asked in: [`CaseInsensitive`] or @@ -211,6 +400,9 @@ pub trait QueryTerm {} impl QueryTerm for EqualityTerm {} impl QueryTerm for MatchQuery {} +impl QueryTerm for OreTerm {} +impl QueryTerm for OpeTerm {} +impl QueryTerm for RangeQuery {} /// A query term whose answer can hold **false positives and no false /// negatives**: a row the server returns may not match, and a row that @@ -272,42 +464,57 @@ pub trait Probabilistic: QueryTerm { pub trait TermFor: QueryTerm {} impl TermFor for T {} +impl TermFor for T {} +impl TermFor for T {} +impl TermFor for T {} +impl TermFor for T {} +impl TermFor for T {} impl TermFor> for T {} -/// An index that answers the operator `Op` over plaintexts `S`, and the +/// An index that answers the operator `Op` for the input `In`, and the /// query term it sends for it. /// -/// [`Index::Term`] is what the index derives; [`query`](Self::query) makes -/// the query term from it and from what [`carry`](Self::carry) kept of the -/// query's value. For an equality index the two are the same HMAC and -/// nothing is kept. For a match index they are not: the query is a filter -/// over the query's value, and it keeps that value's tokens so it can -/// [`verify`](Probabilistic::verify) a row. +/// `In` is what the query is given: the plaintext the index is declared +/// over for an operator of arity [`One`] (`Supports`), a pair of +/// them for [`Between`] (`Supports`); [`Input`] relates +/// the two. +/// +/// [`Index::Term`] is what the index derives, once per plaintext of the +/// input; [`query`](Self::query) makes the query term from those and from +/// what [`carry`](Self::carry) kept of the input. For an equality index +/// the two are the same HMAC and nothing is kept. For a match index they +/// are not: the query is a filter over the query's value, and it keeps +/// that value's tokens so it can [`verify`](Probabilistic::verify) a row. +/// For `Between` the two terms of the bounds become one +/// [`RangeQuery`]. #[diagnostic::on_unimplemented( - message = "`{Self}` does not answer `{Op}` over `{S}`", - label = "this index cannot answer the operator", - note = "an equality index answers `op::Eq`; a match index answers `op::Matches` over text, in the case its configuration fixes" + message = "`{Self}` does not answer `{Op}` given `{In}`", + label = "this index cannot answer the operator for this input", + note = "an equality index answers `op::Eq`; a match index answers `op::Matches` over text, in the case its configuration fixes; an ORE or OPE index answers `op::Lt`, `Lte`, `Gt` and `Gte` over an integer, and `op::Between` over a pair of them" )] -pub trait Supports: Index { +pub trait Supports>: Index { /// What this index sends for `Op`. type Query: TermFor; - /// What the query term keeps of the query's value, taken before the - /// value is consumed: `()` for an exact term. + /// What the query term keeps of the input, taken before the input is + /// consumed: `()` for an exact term. type Carry: MaybeSend + 'static; - /// Why this index cannot answer `Op` as configured, or over `value`, if + /// Why this index cannot answer `Op` as configured, or for `input`, if /// it cannot: a match index whose [`MatchCase`] disagrees with its - /// options ([`PlanError::MatchCase`]), or a match query of text that - /// cuts no token ([`TermError::EmptyTermText`]), whose filter would - /// match every row. Run when the query is awaited, before any key is - /// requested; an index that refuses nothing returns `Ok(())`. - fn check(&self, value: &S) -> Result<(), Error>; - /// Take what the query term keeps of `value`. - fn carry(&self, value: &S) -> Self::Carry; - /// The query term, from what was kept and the term the index derived - /// for the query's value. Only a query chain calls this: `QueryToken` - /// cannot be made outside this crate, so no caller can pair one query's tokens - /// with another value's filter, or pass a stored term off as a query. - fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query; + /// options ([`PlanError::MatchCase`]), a match query of text that cuts + /// no token ([`TermError::EmptyTermText`]), whose filter would match + /// every row, or a range whose low bound is above its high bound + /// ([`TermError::InvertedRange`]). + /// Run when the query is awaited, before any key is requested; an + /// index that refuses nothing returns `Ok(())`. + fn check(&self, input: &In) -> Result<(), Error>; + /// Take what the query term keeps of `input`. + fn carry(&self, input: &In) -> Self::Carry; + /// The query term, from what was kept and the terms the index derived + /// for the input's plaintexts. Only a query chain calls this: + /// `QueryToken` cannot be made outside this crate, so no caller can + /// pair one query's tokens with another value's filter, or pass a + /// stored term off as a query. + fn query(carry: Self::Carry, terms: In::Terms, token: QueryToken) -> Self::Query; } /// Proof that [`Supports::query`] is called by a query chain. Only this @@ -322,7 +529,7 @@ impl QueryToken { } } -impl Supports for Equality { +impl Supports for Equality { type Query = EqualityTerm; type Carry = (); fn check(&self, _: &S) -> Result<(), Error> { @@ -336,7 +543,7 @@ impl Supports for Equality { impl Supports, S> for Match where - S: AsRef, + S: AsRef + 'static, O: MatchCase, { type Query = MatchQuery; @@ -370,7 +577,68 @@ where } } -/// A target type that answers the operator `Op` over plaintexts `S`, and +/// The ordering operators on the ORE and OPE indexes, over the integer +/// types ([`Ordered`]). The query term for a comparison is the index's own +/// term for the value: this crate's ORE is CLLW, whose ciphertext has no +/// left half to send alone, so what is sent is what is stored, as for an +/// HMAC. `Between` sends both bounds' terms as one `RangeQuery`. +macro_rules! orders { + ($($index:ident => $term:ident),+ $(,)?) => {$( + orders!(@one $index => $term; Lt, Lte, Gt, Gte); + impl Supports for $index { + type Query = RangeQuery<$term>; + type Carry = (); + fn check(&self, (lo, hi): &(S, S)) -> Result<(), Error> { + if lo > hi { + Err(TermError::InvertedRange.into()) + } else { + Ok(()) + } + } + fn carry(&self, _: &(S, S)) {} + fn query((): (), (lo, hi): ($term, $term), _: QueryToken) -> Self::Query { + RangeQuery::new(lo, hi) + } + } + impl Answers for $term { + type Via = $index; + } + )+}; + (@one $index:ident => $term:ident; $($op:ident),+) => {$( + impl Supports<$op, S> for $index { + type Query = $term; + type Carry = (); + fn check(&self, _: &S) -> Result<(), Error> { + Ok(()) + } + fn carry(&self, _: &S) {} + fn query((): (), term: $term, _: QueryToken) -> $term { + term + } + } + impl Answers<$op, S> for $term { + type Via = $index; + } + )+}; +} +orders!(Ore => OreTerm, Ope => OpeTerm); + +/// The ordering operators default to the ORE index; an OPE index is named +/// with `.via(Ope)`. +macro_rules! default_ore { + ($($op:ident),+ $(,)?) => {$( + impl DefaultIndex for $op { + type Via = Ore; + } + )+}; +} +default_ore!(Lt, Lte, Gt, Gte); + +impl DefaultIndex<(S, S)> for Between { + type Via = Ore; +} + +/// A target type that answers the operator `Op` for the input `In`, and /// the index it answers it with. /// /// Written by hand per type: a type with two indexes that both answer an @@ -379,23 +647,23 @@ where /// that does not answer an operator has no impl, and the query does not /// compile. #[diagnostic::on_unimplemented( - message = "`{Self}` does not answer `{Op}` over `{S}`", + message = "`{Self}` does not answer `{Op}` given `{In}`", label = "this target type does not answer the operator", note = "a target answers the operators its type implements `Answers` for; `.via(index)` names an index the target declares" )] -pub trait Answers { +pub trait Answers> { /// The index that answers `Op` for this type. An index is a unit type, /// so its `Default` is the index. - type Via: Supports + Default; + type Via: Supports + Default; } -impl Answers for EqualityTerm { +impl Answers for EqualityTerm { type Via = Equality; } impl Answers, S> for MatchTerms where - S: AsRef, + S: AsRef + 'static, O: MatchCase, { type Via = Match; @@ -403,14 +671,18 @@ where /// A ciphertext with one term beside it, the shape of a `plaintext = ..` /// record without the struct, answers what its term answers. -impl> Answers for (crate::StackCipherText, T) { +impl, T: Answers> Answers + for (crate::StackCipherText, T) +{ type Via = T::Via; } /// A ciphertext with one term, as [`indexed`](crate::target::indexed) /// produces it, answers what its term answers. One with several terms /// names the index with `.via(index)`. -impl> Answers for crate::target::Encrypted { +impl, T: Answers> Answers + for crate::target::Encrypted +{ type Via = T::Via; } @@ -424,20 +696,20 @@ impl> Answers for crate::target::Encry /// [`Answers`] impl is not consulted, and a field whose type answers /// through another index needs `via`. #[diagnostic::on_unimplemented( - message = "`{Self}` has no default index over `{S}`", + message = "`{Self}` has no default index given `{In}`", label = "no default index answers this operator here", - note = "`op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; name another index with `.via(index)`" + note = "`op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; `op::Lt`, `Lte`, `Gt`, `Gte` and `Between` default to `Ore` over an integer, and `Between` takes a pair; name another index, such as `Ope`, with `.via(index)`" )] -pub trait DefaultIndex: Operator { +pub trait DefaultIndex>: Operator { /// The index. An index is a unit type, so its `Default` is the index. - type Via: Supports + Default; + type Via: Supports + Default; } -impl DefaultIndex for Eq { +impl DefaultIndex for Eq { type Via = Equality; } -impl> DefaultIndex for Matches<::Case> { +impl + 'static> DefaultIndex for Matches<::Case> { type Via = Match; } @@ -466,5 +738,22 @@ mod tests { "Matches" ); assert_eq!(format!("{:?}", Eq), "Eq"); + assert_eq!(format!("{:?}", Between), "Between"); + } + + /// The ordering operators default to ORE, and a range is refused only + /// when its bounds are the wrong way round: `(v, v)` is one value. + #[test] + fn a_range_is_checked_before_any_term_is_derived() { + let ok = |pair: (u32, u32)| Supports::::check(&Ore, &pair).is_ok(); + assert!(ok((1, 2))); + assert!(ok((2, 2))); + assert!(!ok((3, 2))); + assert!(matches!( + Supports::::check(&Ope, &(1, -1)), + Err(Error::Term(TermError::InvertedRange)) + )); + assert!(Supports::::check(&Ore, &7).is_ok()); + assert!(Supports::::check(&Ope, &-7).is_ok()); } } diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 87ada56e0..36c5ac037 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -11,10 +11,10 @@ use super::build::{FieldPlan, FieldsBuilder, IntoLabel, Opens, PlanContext, Runs use super::field_ref::FieldRef; use super::value::{Indexed, Typed, ValuePlan, ValueShape}; use super::PlanError; -use crate::op::{Answers, DefaultIndex, Operator, QueryToken, Supports}; +use crate::op::{Answers, DefaultIndex, Input, Operator, QueryToken, Supports}; use crate::target::{ ciphertext, Borrowed, CallerContext, DeclaredContext, DecryptField, Decryptable, EncryptFrom, - Index, Indexes, Owned, Pending, PendingFuture, Select, Whole, + Index, Indexes, Pending, PendingFuture, Select, Whole, }; use crate::{Error, KeysetCipher, Label, LabelError, NonEmpty, StackCipher, StackCipherText}; @@ -286,36 +286,20 @@ impl StackCipher { /// index the plan declares beside that one is named first, with /// [`via`](QueryBuilder::via). /// - /// The value is copied once (`to_owned`), so `query(op::Eq, "bob@example.com")` + /// The value is what the operator takes ([`Input`]): one plaintext, or + /// for `op::Between` a pair of them, `cipher.query(op::Between, &(18, 65))`. + /// It is copied once (`to_owned`), so `query(op::Eq, "bob@example.com")` /// queries a `String` field. pub fn query<'a, Op, Q>(&'a self, op: Op, value: &Q) -> QueryBuilder<'a, Op, F, K> where Op: Operator, Q: ToOwned + ?Sized, { + let _: Op = op; QueryBuilder { common: Common::new(self), value: value.to_owned(), - op, - } - } - - /// Start a chain that derives the term of `index` over `value`: the - /// index's own term, not a query term, so nothing says what it may be - /// compared with. For an index no operator answers yet ([`Ore`] and - /// [`Ope`]); a query for an operator is [`query`](Self::query), and this - /// goes when the ordering operators arrive. - /// - /// [`Ore`]: crate::target::Ore - /// [`Ope`]: crate::target::Ope - pub fn query_index<'a, I, Q>(&'a self, index: I, value: &Q) -> QueryIndexBuilder<'a, I, F, K> - where - Q: ToOwned + ?Sized, - { - QueryIndexBuilder { - common: Common::new(self), - value: value.to_owned(), - index, + op: PhantomData, } } @@ -621,10 +605,10 @@ type F = ::Owned; pub struct QueryBuilder<'a, Op, F, K> { common: Common<'a, K>, value: F, - op: Op, + op: PhantomData Op>, } -impl<'a, Op: Operator, F: 'static, K: 'static> QueryBuilder<'a, Op, F, K> { +impl<'a, Op: Operator, F: Input, K: 'static> QueryBuilder<'a, Op, F, K> { /// Name the context the term is derived under, for a plan built /// without one: the context the write's call named, or for a plan with /// a context field, the value that field held. A plan built with its @@ -636,9 +620,9 @@ impl<'a, Op: Operator, F: 'static, K: 'static> QueryBuilder<'a, Op, F, K> { /// Answer the operator through `index` rather than the index the plan /// chooses for it ([`Chooses`]): a match index with its own - /// configuration, or an index of a target type that has no - /// [`Answers`] impl for the operator. The plan named next must declare - /// it, as [`on`](Self::on) says. + /// configuration, an OPE index where ORE is the default, or an index + /// of a target type that has no [`Answers`] impl for the operator. The + /// plan named next must declare it, as [`on`](Self::on) says. pub fn via>(self, index: I) -> QueryVia<'a, Op, F, I, K> { QueryVia { common: self.common, @@ -655,10 +639,11 @@ impl<'a, Op: Operator, F: 'static, K: 'static> QueryBuilder<'a, Op, F, K> { /// label the write used. /// /// What the plan's type can see is checked here: an operator it does - /// not answer, or a one-value plan without the index, does not compile. - /// A field of a fields plan is data, so there an index the field does - /// not declare, or declares with other options, is refused when the - /// query is awaited ([`PlanError::IndexNotDeclared`], + /// not answer, a value that is not what the operator takes over the + /// plan's plaintext, or a one-value plan without the index, does not + /// compile. A field of a fields plan is data, so there an index the + /// field does not declare, or declares with other options, is refused + /// when the query is awaited ([`PlanError::IndexNotDeclared`], /// [`PlanError::IndexOptions`]), before any key is requested. pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, P::Via, K> where @@ -673,11 +658,16 @@ impl<'a, Op: Operator, F: 'static, K: 'static> QueryBuilder<'a, Op, F, K> { pub struct QueryVia<'a, Op, F, I, K> { common: Common<'a, K>, value: F, - op: Op, + op: PhantomData Op>, index: I, } -impl<'a, Op: Operator, F: 'static, I: Supports, K: 'static> QueryVia<'a, Op, F, I, K> { +impl<'a, Op, F, I, K: 'static> QueryVia<'a, Op, F, I, K> +where + Op: Operator, + F: Input, + I: Supports, +{ /// Query on `plan` through the named index, which the plan must /// declare ([`Resolves`]): a one-value plan without it does not /// compile; a field of a fields plan without it is refused when the @@ -685,90 +675,73 @@ impl<'a, Op: Operator, F: 'static, I: Supports, K: 'static> QueryVia<'a, /// own index of that type, so the named index says only which. pub fn on(self, plan: &P) -> QueryOp<'a, F, Op, I, K> where - P: Resolves, + P: Resolves, { let QueryVia { - common, + mut common, value, op, index, } = self; - let inner = plan.resolve(QueryIndexBuilder { + let (index, label) = match common.call_context() { + Ok(call) => plan.resolve(index, call), + Err(error) => (index, Err(error)), + }; + QueryOp { common, value, index, - }); - QueryOp::new(inner, op) - } -} - -/// A chain deriving an index's own term, before its plan is named. Started -/// by [`StackCipher::query_index`]. -pub struct QueryIndexBuilder<'a, I, F, K> { - common: Common<'a, K>, - value: F, - index: I, -} - -impl<'a, I, F: 'static, K: 'static> QueryIndexBuilder<'a, I, F, K> { - /// Name the context the term is derived under, as - /// [`QueryBuilder::context`]. - pub fn context(mut self, context: impl IntoLabel) -> Self { - self.common.context = Some(context.into_label()); - self - } - - /// Derive the index's term on `plan`, which must declare it - /// ([`Resolves`]); otherwise awaiting is - /// [`PlanError::IndexNotDeclared`] or [`PlanError::FieldType`], never a - /// term that matches nothing. - pub fn on(self, plan: &P) -> QueryIndex<'a, F, I, K> - where - I: Index, - P: Resolves, - { - plan.resolve(self) + label, + op, + } } } -/// A plan a query for the operator `Op` over `F` runs on, and the index it -/// answers the operator with. Sealed: the three plan shapes implement it. +/// A plan a query for the operator `Op`, given `In`, runs on, and the +/// index it answers the operator with. Sealed: the three plan shapes +/// implement it. /// /// A field of a fields plan and a one-value plan built `with` indexes /// answer through the operator's [`DefaultIndex`]; a one-value plan for a -/// target type answers through the type's [`Answers`] impl. `At` is the -/// position of that index in a plan built `with` several, and is inferred. +/// target type answers through the type's [`Answers`] impl. The plan's +/// plaintext is the input's ([`Input::Plaintext`]): `S` for a query given +/// an `S`, and for a `Between` given an `(S, S)`. `At` is the position of +/// that index in a plan built `with` several, and is inferred. #[diagnostic::on_unimplemented( - message = "`{Self}` does not answer `{Op}` over `{F}`", + message = "`{Self}` does not answer `{Op}` given `{In}`", label = "no index answers this operator on this plan", - note = "a field of a fields plan or a plan built `with` indexes answers an operator through its default index (`op::Eq`: `Equality`; `op::Matches`: `Match` with the default configuration, case-insensitive and over text only); a typed plan answers the operators its target type implements `Answers` for; `.via(index)` names an index the plan declares" + note = "a field of a fields plan or a plan built `with` indexes answers an operator through its default index (`op::Eq`: `Equality`; `op::Matches`: `Match` with the default configuration, case-insensitive and over text only; `op::Lt`, `Lte`, `Gt`, `Gte` and `Between`: `Ore` over an integer, `Between` given a pair); a typed plan answers the operators its target type implements `Answers` for; `.via(index)` names an index the plan declares" )] -pub trait Chooses: Resolves { +pub trait Chooses, At>: + Resolves +{ /// The index that answers `Op` on this plan. - type Via: Supports + Default; + type Via: Supports + Default; } -impl, F: 'static> Chooses for FieldPlan { +impl, In: Input> Chooses for FieldPlan { type Via = Op::Via; } -impl Chooses for ValuePlan> +impl Chooses for ValuePlan> where - Op: DefaultIndex, + Op: DefaultIndex, Op::Via: Clone, - F: 'static, - X: Indexes + Select, + In: Input, + S: 'static, + X: Indexes + Select, { type Via = Op::Via; } -impl Chooses for ValuePlan> +impl Chooses for ValuePlan> where Op: Operator, - F: 'static, - T: Answers, + In: Input, + S: 'static, + T: Answers, T::Via: Default, - Typed: ValueShape, + Typed: ValueShape, { type Via = T::Via; } @@ -781,8 +754,9 @@ mod plan_sealed { impl Sealed for ValuePlan> {} } -/// A plan the index `I` over `F` can be queried on. Sealed: the three plan -/// shapes implement it, and each finds the label the write used. +/// A plan the index `I` over the plaintext `S` can be queried on. Sealed: +/// the three plan shapes implement it, and each finds the label the write +/// used. /// /// A field of a fields plan ([`Plan::field`](super::Plan::field)) and a /// one-value plan for a target type take the index as given, and refuse @@ -792,102 +766,66 @@ mod plan_sealed { /// the position `At`, which is inferred; a plan without that index does /// not compile. #[diagnostic::on_unimplemented( - message = "`{Self}` cannot be queried through `{I}` over `{F}`", + message = "`{Self}` cannot be queried through `{I}` over `{S}`", label = "this plan does not declare the index, or is not a plan a query runs on", note = "query on a field of a fields plan (`plan.field(..)`) or on a one-value plan; a plan built `with` indexes must hold one of this type" )] -pub trait Resolves: plan_sealed::Sealed { - /// The chain for the index's term on this plan, under the label the - /// write used. - fn resolve<'a, K: 'static>( - &self, - query: QueryIndexBuilder<'a, I, F, K>, - ) -> QueryIndex<'a, F, I, K>; +pub trait Resolves: plan_sealed::Sealed { + /// The index the query runs through on this plan, and the label the + /// write used, or why the plan refuses the query. `call` is the context + /// the call named, if any. + fn resolve(&self, index: I, call: Option

for Encrypted { +impl DecryptInto

for Encrypted { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { open(self.ciphertext, context) @@ -507,7 +507,7 @@ impl + 'static, Terms> DecryptInto

for Encrypted EncryptFrom for Encrypted where - S: crate::Encrypt + Clone, + S: crate::Plaintext + Clone, Terms: TermSet, { type Context = CallerContext; @@ -621,7 +621,7 @@ pub fn indexed<'s, S, K, M, X>( indexes: X, ) -> Encryption<'s, S, Encrypted, K, CallerContext, M> where - S: crate::Encrypt + 's, + S: crate::Plaintext + 's, K: 'static, M: ConsumeSource<'s, S> + ShareSource<'s, S>, X: Indexes, diff --git a/packages/stack-encrypt/src/target/mod.rs b/packages/stack-encrypt/src/target/mod.rs index 565e370d1..349cc34ce 100644 --- a/packages/stack-encrypt/src/target/mod.rs +++ b/packages/stack-encrypt/src/target/mod.rs @@ -95,7 +95,7 @@ mod source; pub mod transcode; mod tuples; -pub(crate) use self::core::{decipher_pending, seal_pending}; +pub(crate) use self::core::{bind_pending, seal_pending}; pub use context::{AeadContext, CallerContext, DeclaredContext, ExpectedContext, Extends}; pub use index::{ indexed, At, Encrypted, Equality, Index, IndexSpec, Indexes, Match, Ope, Ore, Select, TermSet, diff --git a/packages/stack-encrypt/src/target/operations.rs b/packages/stack-encrypt/src/target/operations.rs index ce772f9a9..eae1d175c 100644 --- a/packages/stack-encrypt/src/target/operations.rs +++ b/packages/stack-encrypt/src/target/operations.rs @@ -449,7 +449,7 @@ where /// `Encrypt` consumes the plaintext. In [`Owned`](super::Owned) mode it is /// handed over, so `S` need not be `Clone`; in the default [`Borrowed`] mode /// it is cloned once, which is what `M: ConsumeSource<'s, S>` asks. -pub fn ciphertext<'s, S: crate::Encrypt + 's, K: 'static, M: ConsumeSource<'s, S>>( +pub fn ciphertext<'s, S: crate::Plaintext + 's, K: 'static, M: ConsumeSource<'s, S>>( ) -> Encryption<'s, S, StackCipherText, K, AeadContext, M> { Encryption { build: Box::new( @@ -647,7 +647,7 @@ impl Decryption { } /// The canonical opening operation: retrieve the tree's keys and decode `P` /// through its own Vitamin C `Decrypt` implementation, under `context`. -pub fn open + 'static, K: 'static>( +pub fn open( tree: StackCipherText, context: impl Into, ) -> Decryption { @@ -878,7 +878,7 @@ impl DecryptFrom for T {} // `Borrowed: ConsumeSource<'s, S>`, spelled out. The constructors themselves // (`ciphertext`, `equality`, `ore`, `ope`) ask only for the scheme's own // capability, and run a non-`Clone` plaintext in `Owned` mode. -impl EncryptFrom for StackCipherText { +impl EncryptFrom for StackCipherText { type Context = AeadContext; fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where @@ -887,7 +887,7 @@ impl EncryptFrom for StackCipherText { ciphertext() } } -impl + 'static> DecryptInto

for StackCipherText { +impl DecryptInto

for StackCipherText { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { open(self, context) diff --git a/packages/stack-encrypt/src/target/transcode.rs b/packages/stack-encrypt/src/target/transcode.rs index c99071d09..62a9008c2 100644 --- a/packages/stack-encrypt/src/target/transcode.rs +++ b/packages/stack-encrypt/src/target/transcode.rs @@ -10,7 +10,7 @@ //! leaf or marker is authenticated ciphertext, but passthrough metadata and //! terms are not, and a visitor must not present them as such. use crate::sem::{EqualityTerm, MatchConfig, MatchTerms, OpeTerm, OreTerm}; -use crate::{BoxedPassthrough, CipherText, Error, SealedValue, StackCipherText}; +use crate::{CipherText, Error, PassthroughFrame, SealedValue, StackCipherText}; /// An encrypted output that can drive a destination visitor. /// @@ -36,7 +36,9 @@ pub trait Reader: Sized { pub trait Visitor: Sized { /// The destination this visitor builds. type Value; - /// One sealed leaf: a scalar's ciphertext. + /// One sealed leaf: a scalar's ciphertext. An absent optional is one + /// too, a sealed null, so opening it under the wrong context fails + /// like any leaf. /// /// # Errors /// @@ -60,15 +62,6 @@ pub trait Visitor: Sized { fn map(self, _: R) -> Result { Err(Error::UnsupportedShape) } - /// The sealed marker of an absent optional. It is ciphertext, not a - /// null: opening it under the wrong context fails like any leaf. - /// - /// # Errors - /// - /// Refuses with [`Error::UnsupportedShape`] unless overridden. - fn absent(self, _: SealedValue) -> Result { - Err(Error::UnsupportedShape) - } /// The sealed marker of an empty sequence. /// /// # Errors @@ -86,12 +79,13 @@ pub trait Visitor: Sized { Err(Error::UnsupportedShape) } /// Passthrough metadata the plaintext carried alongside its encrypted - /// fields. Not authenticated: the reader hands it over as it was given. + /// fields, as the frame of its subtree. Not authenticated: the reader + /// hands it over as it was given. /// /// # Errors /// /// Refuses with [`Error::UnsupportedShape`] unless overridden. - fn passthrough(self, _: BoxedPassthrough) -> Result { + fn passthrough(self, _: PassthroughFrame) -> Result { Err(Error::UnsupportedShape) } /// An equality term. One-way, and not authenticated. @@ -183,7 +177,6 @@ impl Reader for StackCipherText { CipherText::Single(leaf) => visitor.sealed(leaf), CipherText::Sequence(items) => visitor.sequence(items.into_iter()), CipherText::Map(entries) => visitor.map(entries.into_iter()), - CipherText::None(marker) => visitor.absent(marker), CipherText::EmptySequence(marker) => visitor.empty_sequence(marker), CipherText::EmptyMap(marker) => visitor.empty_map(marker), CipherText::Passthrough(value) => visitor.passthrough(value), diff --git a/packages/stack-encrypt/tests/common/mod.rs b/packages/stack-encrypt/tests/common/mod.rs index 8f0af0ea9..b41b75ba9 100644 --- a/packages/stack-encrypt/tests/common/mod.rs +++ b/packages/stack-encrypt/tests/common/mod.rs @@ -233,3 +233,20 @@ pub async fn deterministic_cipher(seed: [u8; 32]) -> StackCipher(value: P) -> stack_encrypt::PassthroughFrame { + use vitaminc_protected::Controlled; + stack_encrypt::PassthroughFrame( + vitaminc_aead::to_frame(value) + .expect("a passthrough value frames") + .risky_unwrap(), + ) +} + +/// The `u32` a passthrough payload carries. +pub fn passthrough_u32(payload: &stack_encrypt::PassthroughFrame) -> u32 { + use vitaminc_aead::Decrypt; + let frame = payload.parse().expect("one frame"); + vitaminc_aead::Plaintext::visit(frame, u32::sink()).expect("a u32") +} diff --git a/packages/stack-encrypt/tests/fixtures/README.md b/packages/stack-encrypt/tests/fixtures/README.md index 13c8fa4c4..c7a90d045 100644 --- a/packages/stack-encrypt/tests/fixtures/README.md +++ b/packages/stack-encrypt/tests/fixtures/README.md @@ -11,10 +11,9 @@ same seed. `tests/record_lowering.rs` opens each record with the other author and checks that both derive the terms it holds. That cross-opening is the proof that `dynamic::record` is a lowering into the plan builder and not a second executor (ADR-0007, amended 2026-10-06). A Go test reads the -same file later, once the Go SDK's generated code is the third author. A -chain over bare `u32` or `String` fields derives the same terms but a -different leaf encoding, which the other reader cannot open or tell apart; -`dynamic::record`'s docs say why the lowering does not bridge that. +same file later, once the Go SDK's generated code is the third author. A +chain over bare `u32` or `String` fields seals the same leaves too: every +leaf is tagged with its type (vitaminc ADR 0001), whoever wrote it. Regenerate with `STACK_ENCRYPT_UPDATE_FIXTURES=1 cargo test -p stack-encrypt --all-features --test record_lowering`. Only the sealed bytes diff --git a/packages/stack-encrypt/tests/fixtures/record_lowering.json b/packages/stack-encrypt/tests/fixtures/record_lowering.json index 9aa62a478..b0744958c 100644 --- a/packages/stack-encrypt/tests/fixtures/record_lowering.json +++ b/packages/stack-encrypt/tests/fixtures/record_lowering.json @@ -47,7 +47,7 @@ "passthrough": "tenants/acme" }, "text": { - "c": "0100000000000000000000000000000000cf1a45d1517a08c45d6ae1c989ae99de2000da620d1dbdbba9372d1439b6d44f7cc4169e9a255387aa180d6a95fb17926d7001a7e8ada1c849706c1f1b5c56ad878ea5deb68ebefecb98a8816cbe658c20b33989cc", + "c": "0100000000000000000000000000000000cf1a45d1517a08c45d6ae1c989ae99de2000da620d1dbdbba9372d1439b6d44f7cc4169e9a255387aa180d6a95fb17926d70027eff9bae3dcbbf74666b87b43afcdeeb5d90fe4c4e316142e647677cb9597d6cf9c0", "eq": "0d84fbd44d82b5150351556d6d25a038c70cf0c696e2499a71b659a7732a9d08" } }, @@ -59,7 +59,7 @@ "passthrough": "tenants/acme" }, "text": { - "c": "010000000000000000000000000000000048a99626f697c02001f07da2edb19f4220009868b833b6220fa0d13b612af03af1f38fee082c7555b6e7fbf8e13ccb70f80701dc7cf368a02a3bd466d24b4037597aeae058b641ff5a2c73736f98d8d49ae10a3d9c", + "c": "010000000000000000000000000000000048a99626f697c02001f07da2edb19f4220009868b833b6220fa0d13b612af03af1f38fee082c7555b6e7fbf8e13ccb70f80702c1591a387d3f6fd6c83d4b547b857ec46a3ef86919ace434e5ca69581ce699cf0765", "eq": "0d84fbd44d82b5150351556d6d25a038c70cf0c696e2499a71b659a7732a9d08" } } @@ -125,12 +125,12 @@ "records": { "lowering": { "age": { - "c": "01000000000000000000000000000000004d5626148f77c4c6d9da0ddbdf7663a2200021162641133485216f389886cbd7c6e9b25afa0f5f166f985b23384e81ffcfac010686f7787f57b4c7a2192c3499ea1b890f9f4623d84b72b5f40266b10b7f607a41", + "c": "01000000000000000000000000000000004d5626148f77c4c6d9da0ddbdf7663a2200021162641133485216f389886cbd7c6e9b25afa0f5f166f985b23384e81ffcfac02b8821c190a544161b6ac13825768cd6b494515bdd96c26715ffecd7321277d13d6", "eq": "5433335e609685a33b29c98369e68cd5d92e838aa566a3ff1df5c2d251b63885", "ore": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a605cbf3f1" }, "email": { - "c": "0100000000000000000000000000000000b2afa77357a61488146c0edaa0246f65200043956c4f4bba857028c28165dfce9c63d904b738bbb039b6e11c805112d4a1ee01ec122b5ee8771a6a37fc17312ff3f7a56e93885e19d4bb189950aa37475eb74f63f5758c4f87a3494c5e46ff", + "c": "0100000000000000000000000000000000b2afa77357a61488146c0edaa0246f65200043956c4f4bba857028c28165dfce9c63d904b738bbb039b6e11c805112d4a1ee02d3ce6ecc6cc31f82d07a829142cf39b2fcd9a742f89627f316622416f49df065d381337c5da03a8f3ab5b84d", "eq": "ec534cfbf2336785378c3761ff88623c46361b2ba7c5fa83930ebacae49cf646", "match": "06000d0013001d002200260029002a003000320043005a006000660068006c0077008f00960098009e00c200c500d600d900de00e300e700e900eb00ec00f200fa00" }, @@ -138,17 +138,17 @@ "passthrough": 42 }, "notes": { - "c": "0100000000000000000000000000000000723f7a7f14b07197168aa13bc52010b3200072a73194b45af604cc6201b5a57554b05cc7ed6ad22b9102cbcb71bd9067849601a2ae59c2cfca81b0ecde824090e2331f2c389202676855068652e9e566dfccb9a55e9973a9301c" + "c": "0100000000000000000000000000000000723f7a7f14b07197168aa13bc52010b3200072a73194b45af604cc6201b5a57554b05cc7ed6ad22b9102cbcb71bd90678496020f72b5054a1a4467aa441215b4af28502c55a2590aae2a844ea40392f5fd64e679614a072b8736" } }, "typed_chain": { "age": { - "c": "010000000000000000000000000000000092e427c5e176b60261add6ce80fd02df20008d231453c97fdec97a0b6559ac8f509129d20163f262d40ce5d77cf1ebe0cad501155707def46648a0c3685910bea6d3ec9c63aeb8c1e864092d47a3bd0269d135c9", + "c": "010000000000000000000000000000000092e427c5e176b60261add6ce80fd02df20008d231453c97fdec97a0b6559ac8f509129d20163f262d40ce5d77cf1ebe0cad502c51f3fa24db5e34fadc3553b401d674928b00126a41cb7f30da889259f472223e8", "eq": "5433335e609685a33b29c98369e68cd5d92e838aa566a3ff1df5c2d251b63885", "ore": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a605cbf3f1" }, "email": { - "c": "0100000000000000000000000000000000cf92191950409f9fe82a2b11888ffc1d2000ca4d5bbafbff919af4d3a95c7cc046d632fd4c28785b238f4b6f7ea3aa08482901a3e671679e0e1ac106a28ad8a27e3ae143a6ad5c4b8280dbf6818d1bab32fa45cb0790df8551cec1c9bb3187", + "c": "0100000000000000000000000000000000cf92191950409f9fe82a2b11888ffc1d2000ca4d5bbafbff919af4d3a95c7cc046d632fd4c28785b238f4b6f7ea3aa08482902a44da283fdefa835c68377a191ccf172943720b07e7779cc903619869844eb9cfed80a8a7f9034e69385ddb1", "eq": "ec534cfbf2336785378c3761ff88623c46361b2ba7c5fa83930ebacae49cf646", "match": "06000d0013001d002200260029002a003000320043005a006000660068006c0077008f00960098009e00c200c500d600d900de00e300e700e900eb00ec00f200fa00" }, @@ -156,7 +156,7 @@ "passthrough": 42 }, "notes": { - "c": "0100000000000000000000000000000000562a41e032316ff2d1264a9e9f5c3b842000de33c8d1f2caaab8b90f691161a57eeb1328f7f1c9b5be76dec83e8a53765f11015924202425d1e3be9754306c2e290e350232d5b5623190ebcc69feb55fb566fe4b4db024f1ebcc" + "c": "0100000000000000000000000000000000562a41e032316ff2d1264a9e9f5c3b842000de33c8d1f2caaab8b90f691161a57eeb1328f7f1c9b5be76dec83e8a53765f1102c76bf0a61684b320c316ad94b01eb76bb52b3c8493f21a73012de6289580bf355b7192c61500e0" } } } diff --git a/packages/stack-encrypt/tests/index.rs b/packages/stack-encrypt/tests/index.rs index a61f28078..d5a89ec5b 100644 --- a/packages/stack-encrypt/tests/index.rs +++ b/packages/stack-encrypt/tests/index.rs @@ -622,8 +622,9 @@ mod given_encrypted_terms_as_a_target { /// opens through the other's reader. async fn same_bytes(value: S, indexes: X) where - S: stack_encrypt::Encrypt - + for<'a> stack_encrypt::Decrypt<'a> + S: stack_encrypt::Plaintext + + stack_encrypt::Decrypt + + Send + Clone + PartialEq + std::fmt::Debug diff --git a/packages/stack-encrypt/tests/plan_grammar.rs b/packages/stack-encrypt/tests/plan_grammar.rs index b56486a57..67d76cdd9 100644 --- a/packages/stack-encrypt/tests/plan_grammar.rs +++ b/packages/stack-encrypt/tests/plan_grammar.rs @@ -22,7 +22,7 @@ use stack_encrypt::target::{ EncryptFrom, Encrypted, Encryption, IndexSpec, Indexes, TermSet, }; use stack_encrypt::{ - nonempty, Decrypt, Encrypt, Equality, Error, ErrorPayload, Label, Match, NonEmpty, Ope, Ore, + nonempty, Decrypt, Equality, Error, ErrorPayload, Label, Match, NonEmpty, Ope, Ore, Plaintext, Plan, StackCipherText, }; @@ -960,7 +960,7 @@ struct One { /// other plan. async fn same_field_bytes(value: F, indexes: X) where - F: Encrypt + Decrypt<'static> + Clone + Send + Sync + PartialEq + Debug + 'static, + F: Plaintext + Decrypt + Clone + Send + Sync + PartialEq + Debug + 'static, Terms: TermSet + PartialEq + Debug + Send, X: Indexes + Clone + Send + Sync + 'static, { diff --git a/packages/stack-encrypt/tests/record_lowering.rs b/packages/stack-encrypt/tests/record_lowering.rs index dce2f8a5c..7f84e5999 100644 --- a/packages/stack-encrypt/tests/record_lowering.rs +++ b/packages/stack-encrypt/tests/record_lowering.rs @@ -3,8 +3,8 @@ //! records the other sealed, and both derive the same bytes for each term. //! The chain's fields are `dynamic::Value`s, the plaintext type a binding's //! values have, so the two authors write one declaration over one type; a -//! chain over bare `u32`/`String` fields shares the terms and not the leaves -//! (see `dynamic::record`). `tests/fixtures/record_lowering.json` holds one +//! chain over bare `u32`/`String` fields seals the same tagged leaves (see +//! `dynamic::record`). `tests/fixtures/record_lowering.json` holds one //! record from each author, sealed under a deterministic key source so the //! bytes open in any process, and its README gives the schema a Go test //! reads later. @@ -178,7 +178,7 @@ fn json_of_record(tree: StackCipherText) -> Json { let value = match (key.as_str(), node) { ("c", CipherText::Single(leaf)) => json!(hex(&leaf.to_bytes())), ("passthrough", CipherText::Passthrough(payload)) => { - match *payload.downcast::().expect("a value") { + match FfiValue::from_frame(&payload.0).expect("a value") { FfiValue::UInt32(v) => json!(v), FfiValue::String(s) => { json!(String::from_utf8(s.risky_ref().to_vec()).expect("utf8")) @@ -187,7 +187,7 @@ fn json_of_record(tree: StackCipherText) -> Json { } } (_, CipherText::Passthrough(payload)) => { - match *payload.downcast::().expect("a value") { + match FfiValue::from_frame(&payload.0).expect("a value") { FfiValue::Bytes(bytes) => json!(hex(bytes.risky_ref())), _ => panic!("a term is bytes"), } @@ -201,6 +201,14 @@ fn json_of_record(tree: StackCipherText) -> Json { Json::Object(out) } +/// A value as a passthrough node: its frame, in the clear. +fn passthrough(value: FfiValue) -> StackCipherText { + use vitaminc_protected::Controlled; + CipherText::Passthrough(stack_encrypt::PassthroughFrame( + value.to_frame().expect("a value frames").risky_unwrap(), + )) +} + fn record_of_json(record: &Json) -> StackCipherText { let fields = record.as_object().expect("an object"); CipherText::Map( @@ -216,20 +224,16 @@ fn record_of_json(record: &Json) -> StackCipherText { SealedValue::from_bytes(&unhex(value.as_str().expect("hex"))) .expect("a frozen leaf"), ), - "passthrough" => CipherText::Passthrough(Box::new(match value { + "passthrough" => passthrough(match value { Json::String(text) => FfiValue::String(text.as_str().into()), _ => FfiValue::UInt32( u32::try_from(value.as_u64().expect("an integer")) .expect("u32"), ), - }) - as stack_encrypt::BoxedPassthrough), - _ => CipherText::Passthrough(Box::new(FfiValue::Bytes( - vitaminc_protected::Protected::new(unhex( - value.as_str().expect("hex"), - )), - )) - as stack_encrypt::BoxedPassthrough), + }), + _ => passthrough(FfiValue::Bytes(vitaminc_protected::Protected::new( + unhex(value.as_str().expect("hex")), + ))), }; (key.clone(), node) }) @@ -248,10 +252,7 @@ fn record_of_typed(mut values: FieldValues) -> StackCipherText { let notes: StackCipherText = values.take("notes").expect("notes"); let id: u32 = values.take("id").expect("id"); let term = |bytes: Vec| -> StackCipherText { - CipherText::Passthrough( - Box::new(FfiValue::Bytes(vitaminc_protected::Protected::new(bytes))) - as stack_encrypt::BoxedPassthrough, - ) + passthrough(FfiValue::Bytes(vitaminc_protected::Protected::new(bytes))) }; CipherText::Map(vec![ ( @@ -275,9 +276,7 @@ fn record_of_typed(mut values: FieldValues) -> StackCipherText { "id".into(), CipherText::Map(vec![( "passthrough".into(), - CipherText::Passthrough( - Box::new(FfiValue::UInt32(id)) as stack_encrypt::BoxedPassthrough - ), + passthrough(FfiValue::UInt32(id)), )]), ), ]) @@ -303,7 +302,7 @@ fn typed_of_record(tree: StackCipherText) -> FieldValues { let CipherText::Passthrough(payload) = node else { panic!("a passthrough node"); }; - let FfiValue::UInt32(id) = *payload.downcast::().expect("a value") + let FfiValue::UInt32(id) = FfiValue::from_frame(&payload.0).expect("a value") else { panic!("a u32"); }; @@ -576,9 +575,6 @@ fn record_of_note(mut values: FieldValues) -> StackCipherText { let tenant: Value = values.take("tenant").expect("tenant"); let text: Encrypted = values.take("text").expect("text"); let id: u32 = values.take("id").expect("id"); - let passthrough = |value: FfiValue| -> StackCipherText { - CipherText::Passthrough(Box::new(value) as stack_encrypt::BoxedPassthrough) - }; CipherText::Map(vec![ ( "tenant".into(), @@ -628,7 +624,7 @@ fn note_of_record(tree: StackCipherText) -> FieldValues { let CipherText::Passthrough(payload) = node else { panic!("a passthrough node"); }; - match *payload.downcast::().expect("a value") { + match FfiValue::from_frame(&payload.0).expect("a value") { FfiValue::UInt32(id) => { let _ = values.insert(&name, id); } diff --git a/packages/stack-encrypt/tests/roundtrip.rs b/packages/stack-encrypt/tests/roundtrip.rs index 985f65792..96b115d01 100644 --- a/packages/stack-encrypt/tests/roundtrip.rs +++ b/packages/stack-encrypt/tests/roundtrip.rs @@ -7,11 +7,11 @@ use std::collections::HashMap; use stack_encrypt::{ - BoxedPassthrough, Cipher, CipherText, ContextTag, Element, Encrypt, IntoAad, SealedValue, - StackCipher, + CipherText, ContextTag, PassthroughFrame, Plaintext, SealedValue, StackCipher, }; use stack_kms::FakeDataKeySource; -use vitaminc_aead::{MapCipher, Passthrough}; +use vitaminc_aead::plaintext::{MapSink, Sink}; +use vitaminc_aead::Decrypt; use vitaminc_protected::{Controlled, Protected}; async fn cipher() -> StackCipher { @@ -126,9 +126,8 @@ async fn option_none_roundtrips() { #[tokio::test] async fn protected_roundtrip() { - // Exercises the `Protected` Decrypt impl, the sole user of `Decipher::map_ok`. - // (`Vec` would encrypt element-wise as a sequence of `u8`, not as bytes, - // so a string leaf is used here.) + // Exercises the `Protected` Decrypt impl, which builds the value inside + // the wrapper. let cipher = cipher().await; let keyset = cipher.default_keyset(); let secret = Protected::new("classified".to_string()); @@ -150,12 +149,15 @@ async fn nested_vec_roundtrips() { assert_eq!(pt, nested); } +/// The context a `ContextTag` binds is an AAD like any other: a value +/// sealed under `ContextTag::aad(tag)` opens under the same, and only the +/// same, tag. #[tokio::test] async fn context_tag_binds_and_roundtrips() { let cipher = cipher().await; let keyset = cipher.default_keyset(); let ct = keyset - .encrypt(ContextTag::new("token".to_string(), "user:42"), ()) + .encrypt("token".to_string(), ContextTag::aad("user:42")) .await .expect("encrypt"); @@ -172,7 +174,7 @@ async fn context_tag_wrong_context_fails() { let cipher = cipher().await; let keyset = cipher.default_keyset(); let ct = keyset - .encrypt(ContextTag::new("token".to_string(), "user:42"), ()) + .encrypt("token".to_string(), ContextTag::aad("user:42")) .await .expect("encrypt"); @@ -218,8 +220,7 @@ async fn empty_marker_does_not_decode_under_wrong_aad() { } /// A hand-written record carrying one field in the clear, through the -/// type-erased `passthrough_entry_boxed` a cipher-generic `Encrypt` has to -/// use. +/// sink's passthrough. struct Row { id: u32, email: String, @@ -243,22 +244,27 @@ impl Row { } } -impl Encrypt for Row { - fn encrypt_with_aad<'a, C, A>(self, cipher: C, aad: A) -> Result - where - C: Cipher, - A: IntoAad<'a>, - { - let mut map = cipher - .encrypt_map(aad) - .passthrough_entry_boxed("id", Box::new(self.id))?; +impl Plaintext for Row { + fn visit(self, sink: S) -> Result { + let mut map = sink.map(None)?; + map.entry("id")?.passthrough(self.id)?; if self.repeat_id { - map = map.passthrough_entry_boxed("id", Box::new(self.id))?; + map.entry("id")?.passthrough(self.id)?; } - map.encrypt_entry("email", self.email)?.end() + self.email.visit(map.entry("email")?)?; + map.end() } } +/// A passthrough payload: the frame of `value`, in the clear. +fn clear(value: u32) -> PassthroughFrame { + PassthroughFrame( + vitaminc_aead::to_frame(value) + .expect("a u32 frames") + .risky_unwrap(), + ) +} + /// A passthrough entry lands in the map as given, beside the sealed ones; /// a key given twice is refused like any repeated map key, since a map the /// cipher would refuse to open must never be produced. @@ -271,10 +277,15 @@ async fn a_passthrough_map_entry_is_stored_in_the_clear_once() { panic!("a record encrypts as a map: {ct:?}"); }; let entry = |name: &str| entries.iter().find(|(key, _)| key == name).map(|(_, v)| v); - assert!( - matches!(entry("id"), Some(CipherText::Passthrough(v)) if v.downcast_ref::() == Some(&7)), - "{entries:?}" - ); + let Some(CipherText::Passthrough(id)) = entry("id") else { + panic!("the id rides in the clear: {entries:?}"); + }; + let id = id + .parse() + .expect("one frame") + .visit(u32::sink()) + .expect("a u32"); + assert_eq!(id, 7, "the passthrough carries the value as given"); assert!( matches!(entry("email"), Some(CipherText::Single(_))), "{entries:?}" @@ -288,12 +299,11 @@ async fn a_passthrough_map_entry_is_stored_in_the_clear_once() { /// none — no entries at all, or only passthroughs, which carry no tag — /// would verify under *any* AAD. The encrypt side never produces one /// (emptiness is the authenticated `Empty*` marker); the decrypt side must -/// refuse to open one, even for a type that reads passthroughs back. +/// refuse to open one, on the tree's shape, before any sink sees it. #[tokio::test] async fn a_container_with_nothing_sealed_in_it_does_not_open() { let cipher = cipher().await; let aad = b"users".as_slice(); - let clear = |value: u32| CipherText::Passthrough(Box::new(value) as BoxedPassthrough); let result: Result, _> = cipher.decrypt(CipherText::Sequence(vec![]), aad).await; assert!(result.is_err(), "an entry-less sequence proves nothing"); @@ -301,15 +311,24 @@ async fn a_container_with_nothing_sealed_in_it_does_not_open() { cipher.decrypt(CipherText::Map(vec![]), aad).await; assert!(result.is_err(), "an entry-less map proves nothing"); - let result: Result>, _> = cipher - .decrypt(CipherText::Sequence(vec![clear(1), clear(2)]), aad) + let result: Result, _> = cipher + .decrypt( + CipherText::Sequence(vec![ + CipherText::Passthrough(clear(1)), + CipherText::Passthrough(clear(2)), + ]), + aad, + ) .await; assert!( result.is_err(), "an all-passthrough sequence proves nothing" ); - let result: Result>, _> = cipher - .decrypt(CipherText::Map(vec![("id".to_string(), clear(1))]), aad) + let result: Result, _> = cipher + .decrypt( + CipherText::Map(vec![("id".to_string(), CipherText::Passthrough(clear(1)))]), + aad, + ) .await; assert!(result.is_err(), "an all-passthrough map proves nothing"); } @@ -360,94 +379,6 @@ async fn sequence_element_cannot_be_rehomed_as_scalar() { ); } -#[tokio::test] -async fn element_roundtrips_under_bare_caller_aad() { - // `Element` derives `for_sequence_element` inside its own Encrypt/Decrypt - // impls. Both sides must honour that derivation: the decipher opens the leaf - // under the AAD the Decrypt drive supplies, not a pre-derived one. - let cipher = cipher().await; - let keyset = cipher.default_keyset(); - let ct = keyset - .encrypt(Element("row".to_string()), b"users".as_slice()) - .await - .expect("encrypt"); - let pt: Element = cipher - .decrypt(ct, b"users".as_slice()) - .await - .expect("Element must round-trip under the bare caller AAD"); - assert_eq!(pt.into_inner(), "row"); -} - -#[tokio::test] -async fn element_interchanges_with_vec_element() { - // A row sealed as one element of a `Vec` decrypts alone as `Element` - // under the same caller AAD (Element's documented use-case), and a lone - // `Element` ciphertext decrypts as a one-element `Vec`. - let cipher = cipher().await; - let keyset = cipher.default_keyset(); - let aad = b"users".as_slice(); - - let ct = keyset - .encrypt(vec!["a".to_string(), "b".to_string()], aad) - .await - .expect("encrypt"); - let second = match ct { - CipherText::Sequence(mut items) => items.remove(1), - other => other, - }; - let pt: Element = cipher - .decrypt(second, aad) - .await - .expect("spliced element must decrypt as Element"); - assert_eq!(pt.into_inner(), "b"); - - let lone = keyset - .encrypt(Element("c".to_string()), aad) - .await - .expect("encrypt"); - let wrapped = CipherText::Sequence(vec![lone]); - let pt: Vec = cipher - .decrypt(wrapped, aad) - .await - .expect("lone Element must decrypt as a one-element Vec"); - assert_eq!(pt, vec!["c".to_string()]); -} - -#[tokio::test] -async fn element_fails_under_wrong_caller_aad() { - let cipher = cipher().await; - let keyset = cipher.default_keyset(); - let ct = keyset - .encrypt(Element("row".to_string()), b"users".as_slice()) - .await - .expect("encrypt"); - let result: Result, _> = cipher.decrypt(ct, b"orders".as_slice()).await; - assert!( - result.is_err(), - "Element under the wrong caller AAD must not decrypt" - ); -} - -/// The derivation that binds a sequence element to its position is the -/// library's, applied by `Element` itself — there is no entry point that -/// lets a caller retrieve keys under one context and authenticate under -/// another, so a batched row is read back by naming the type, not by -/// reconstructing the AAD. -#[tokio::test] -async fn a_batched_element_opens_by_naming_the_type() { - let cipher = cipher().await; - let keyset = cipher.default_keyset(); - let ct = keyset - .encrypt(Element("row".to_string()), b"users".as_slice()) - .await - .expect("encrypt"); - let opened: Element = cipher - .decrypt(ct, b"users".as_slice()) - .await - .expect("Element applies its own derivation on open"); - assert_eq!(opened.into_inner(), "row"); -} - #[tokio::test] async fn wrong_shape_fails() { // A scalar ciphertext must not decode as a sequence. diff --git a/packages/stack-encrypt/tests/source_mode.rs b/packages/stack-encrypt/tests/source_mode.rs index 0489248b0..6780d5c67 100644 --- a/packages/stack-encrypt/tests/source_mode.rs +++ b/packages/stack-encrypt/tests/source_mode.rs @@ -1,6 +1,6 @@ //! How a description receives its plaintext (`target::{Borrowed, Owned}`). //! -//! Vitamin C's `Encrypt` and `PrfValue` consume their input. A description +//! Vitamin C's by-value `Plaintext` walk and `PrfValue` consume their input. A description //! handed a borrow (the default, and what every `EncryptFrom` declaration //! runs in) clones before an operation consumes; one handed the value //! (`Owned`) gives it to the operation. These tests hold the two claims that @@ -21,9 +21,9 @@ use stack_encrypt::sem::{EqualityTerm, MatchTerms}; use stack_encrypt::target::{ ciphertext, equality, matching, AeadContext, Borrowed, CallerContext, Encryption, Owned, }; -use stack_encrypt::{nonempty, Encrypt, NonEmpty, StackCipherText}; +use stack_encrypt::{nonempty, NonEmpty, Plaintext, StackCipherText}; use stack_kms::FakeDataKeySource; -use vitaminc_aead::{Cipher, IntoAad}; +use vitaminc_aead::plaintext::Sink; use vitaminc_prf::{IntoPrfContext, Prf, PrfValue, PrfVisitor}; use vitaminc_protected::{Controlled, Protected}; @@ -52,13 +52,9 @@ impl Secret { Self(Protected::new(text.to_string())) } } -impl Encrypt for Secret { - fn encrypt_with_aad<'a, C, A>(self, cipher: C, aad: A) -> Result - where - C: Cipher, - A: IntoAad<'a>, - { - self.0.encrypt_with_aad(cipher, aad) +impl Plaintext for Secret { + fn visit(self, sink: S) -> Result { + self.0.visit(sink) } } impl PrfValue for Secret { @@ -103,13 +99,9 @@ impl Clone for Counted { } } } -impl Encrypt for Counted { - fn encrypt_with_aad<'a, C, A>(self, cipher: C, aad: A) -> Result - where - C: Cipher, - A: IntoAad<'a>, - { - self.text.encrypt_with_aad(cipher, aad) +impl Plaintext for Counted { + fn visit(self, sink: S) -> Result { + self.text.visit(sink) } } impl PrfValue for Counted { @@ -186,7 +178,7 @@ async fn a_plaintext_that_is_not_clone_derives_a_match_term_alone() { #[cfg(feature = "dynamic")] #[tokio::test] async fn a_zeroizing_ffi_value_seals_through_ciphertext_alone() { - use vitaminc_aead_value::FfiValue; + use stack_encrypt::dynamic::FfiValue; let cipher = stack_cipher().await; let keyset = cipher.default_keyset(); @@ -237,7 +229,7 @@ async fn one_owned_operation_makes_no_copy() { fn sealed_and_indexed<'s, S, M>( ) -> Encryption<'s, S, (StackCipherText, EqualityTerm), FakeDataKeySource, CallerContext, M> where - S: Encrypt + PrfValue + 's, + S: Plaintext + PrfValue + 's, M: stack_encrypt::target::ConsumeSource<'s, S> + stack_encrypt::target::ShareSource<'s, S>, { ciphertext().accepting().zip(equality()) @@ -350,13 +342,9 @@ impl Drop for Dropped { let _ = self.drops.fetch_add(1, Ordering::SeqCst); } } -impl Encrypt for Dropped { - fn encrypt_with_aad<'a, C, A>(mut self, cipher: C, aad: A) -> Result - where - C: Cipher, - A: IntoAad<'a>, - { - std::mem::take(&mut self.text).encrypt_with_aad(cipher, aad) +impl Plaintext for Dropped { + fn visit(mut self, sink: S) -> Result { + std::mem::take(&mut self.text).visit(sink) } } impl PrfValue for Dropped { diff --git a/packages/stack-encrypt/tests/transcode.rs b/packages/stack-encrypt/tests/transcode.rs index 23c011cef..8f9033f90 100644 --- a/packages/stack-encrypt/tests/transcode.rs +++ b/packages/stack-encrypt/tests/transcode.rs @@ -5,11 +5,12 @@ use stack_encrypt::sem::EqualityTerm; use stack_encrypt::target::transcode::{MapReader, Reader, SequenceReader, Transcode, Visitor}; use stack_encrypt::target::{self, CallerContext, ExpectedContext}; use stack_encrypt::{ - nonempty, Cipher, CipherText, ContextPiece, DecryptField, DecryptInto, Decryptable, Decryption, - Encrypt, EncryptFrom, Encryption, Error, IntoAad, IntoContext, MaybeEmpty, NonEmpty, + nonempty, CipherText, ContextPiece, DecryptField, DecryptInto, Decryptable, Decryption, + EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty, NonEmpty, PassthroughFrame, Plaintext, SealedValue, StackCipherText, }; use std::sync::atomic::Ordering; +use vitaminc_aead::plaintext::Sink; #[derive(Clone, Debug, PartialEq)] struct Identifier { @@ -50,7 +51,7 @@ impl Transcode for StoredLeaf { LeafVisitor } } -impl EncryptFrom for StoredLeaf { +impl EncryptFrom for StoredLeaf { type Context = CallerContext; fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where @@ -59,7 +60,7 @@ impl EncryptFrom for StoredLeaf { target::ciphertext().accepting().transcode() } } -impl + 'static> DecryptInto

for StoredLeaf { +impl DecryptInto

for StoredLeaf { type Context = CallerContext; fn decryption(self, context: Self::Context) -> Decryption { match SealedValue::from_bytes(&self.0) { @@ -222,13 +223,14 @@ async fn empty_stored_identifier_is_refused_before_any_key_is_retrieved() { #[derive(Clone)] struct WithoutSerde(String); -impl Encrypt for WithoutSerde { - fn encrypt_with_aad<'a, C: Cipher, A: IntoAad<'a>>( - self, - cipher: C, - aad: A, - ) -> Result { - self.0.encrypt_with_aad(cipher, aad) +impl Plaintext for WithoutSerde { + fn visit(self, sink: S) -> Result { + self.0.visit(sink) + } +} +impl Plaintext for &WithoutSerde { + fn visit(self, sink: S) -> Result { + (&self.0).visit(sink) } } #[tokio::test] @@ -248,7 +250,7 @@ async fn ciphertext_uses_plaintexts_native_contract_without_serde() { .unwrap(); assert_eq!( opened, "native", - "a plaintext without Serde seals through its own Encrypt impl and opens canonically" + "a plaintext without Serde seals through its own Plaintext impl and opens canonically" ); let text = String::from("borrowed"); let result: StoredLeaf = keyset @@ -283,7 +285,7 @@ async fn scalar_destination_refuses_a_sequence() { // A target whose declaration carries every context it needs asks its caller // for none: `Context = ()`. struct FixedLeaf(StoredLeaf); -impl EncryptFrom for FixedLeaf { +impl EncryptFrom for FixedLeaf { type Context = (); fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where @@ -298,7 +300,7 @@ impl EncryptFrom for FixedLeaf { .accepting() } } -impl + 'static> DecryptInto

for FixedLeaf { +impl DecryptInto

for FixedLeaf { type Context = (); fn decryption(self, (): ()) -> Decryption { self.0.decryption(nonempty!("fixed/leaf").into()) @@ -337,10 +339,9 @@ enum Stored { Value(SealedValue), List(Vec), Object(Vec<(String, Stored)>), - Absent(SealedValue), EmptyList(SealedValue), EmptyObject(SealedValue), - Metadata(stack_encrypt::BoxedPassthrough), + Metadata(PassthroughFrame), } struct TreeVisitor; impl Visitor for TreeVisitor { @@ -362,16 +363,13 @@ impl Visitor for TreeVisitor { } Ok(Stored::Object(output)) } - fn absent(self, marker: SealedValue) -> Result { - Ok(Stored::Absent(marker)) - } fn empty_sequence(self, marker: SealedValue) -> Result { Ok(Stored::EmptyList(marker)) } fn empty_map(self, marker: SealedValue) -> Result { Ok(Stored::EmptyObject(marker)) } - fn passthrough(self, value: stack_encrypt::BoxedPassthrough) -> Result { + fn passthrough(self, value: PassthroughFrame) -> Result { Ok(Stored::Metadata(value)) } } @@ -383,7 +381,6 @@ impl Stored { Self::Object(v) => { CipherText::Map(v.into_iter().map(|(k, v)| (k, v.native())).collect()) } - Self::Absent(v) => CipherText::None(v), Self::EmptyList(v) => CipherText::EmptySequence(v), Self::EmptyObject(v) => CipherText::EmptyMap(v), Self::Metadata(v) => CipherText::Passthrough(v), @@ -441,7 +438,7 @@ fn native_readers_report_the_number_of_remaining_entries() { } } - let value = || CipherText::Passthrough(Box::new(7_u32) as stack_encrypt::BoxedPassthrough); + let value = || CipherText::Passthrough(common::passthrough(7_u32)); let sequence: StackCipherText = CipherText::Sequence(vec![value(), value(), value()]); assert_eq!( sequence.read(LengthVisitor).unwrap(), @@ -521,14 +518,14 @@ async fn native_readers_preserve_map_keys_and_authenticated_markers() { opened.is_err(), "an empty-map marker is sealed under its context, not an unauthenticated empty container" ); - let stored = CipherText::Passthrough(Box::new(17u32) as stack_encrypt::BoxedPassthrough) + let stored = CipherText::Passthrough(common::passthrough(17u32)) .read(TreeVisitor) .unwrap(); let Stored::Metadata(value) = stored else { panic!("metadata") }; assert_eq!( - *value.downcast::().unwrap(), + common::passthrough_u32(&value), 17, "passthrough metadata reaches the destination as it was given" ); diff --git a/packages/stack-encrypt/tests/ui/decrypt_field_not_decryptable.stderr b/packages/stack-encrypt/tests/ui/decrypt_field_not_decryptable.stderr index 4a434533e..68a157b03 100644 --- a/packages/stack-encrypt/tests/ui/decrypt_field_not_decryptable.stderr +++ b/packages/stack-encrypt/tests/ui/decrypt_field_not_decryptable.stderr @@ -14,7 +14,7 @@ help: the trait `DecryptField` is not implemented for `Opaqu `(A, B, C)` implements `DecryptField` `(A, B, C, D)` implements `DecryptField` `(A, B, C, D, E)` implements `DecryptField` - `CipherText>` implements `DecryptField` + `CipherText` implements `DecryptField` `Encrypted` implements `DecryptField` `EqualityTerm` implements `DecryptField` `MatchTerms` implements `DecryptField` @@ -38,7 +38,7 @@ help: the trait `Decryptable` is not implemented for `Opaque` (A, B, C) (A, B, C, D) (A, B, C, D, E) - CipherText> + CipherText Encrypted EqualityTerm MatchTerms diff --git a/packages/stack-encrypt/tests/ui/drop_record.stderr b/packages/stack-encrypt/tests/ui/drop_record.stderr index acf842fcd..c3667678c 100644 --- a/packages/stack-encrypt/tests/ui/drop_record.stderr +++ b/packages/stack-encrypt/tests/ui/drop_record.stderr @@ -5,4 +5,4 @@ error[E0509]: cannot move out of type `Rec`, which implements the `Drop` trait | ^^^^^^^^^^^^^^^^^^ | | | cannot move out of here - | move occurs because value has type `CipherText>`, which does not implement the `Copy` trait + | move occurs because value has type `CipherText`, which does not implement the `Copy` trait diff --git a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs index 8c808dca1..2573c44ca 100644 --- a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs +++ b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs @@ -3,10 +3,10 @@ //! one place a copy is unavoidable — the first side gets a clone, the last //! the value — and the refusal names `zip`, not the operations. use stack_encrypt::target::{ciphertext, equality, CallerContext, Owned}; -use stack_encrypt::{nonempty, Encrypt, KeysetCipher}; +use stack_encrypt::{nonempty, KeysetCipher, Plaintext}; use stack_kms::FakeDataKeySource; -fn seal_and_index( +fn seal_and_index( keyset: &KeysetCipher<'_, FakeDataKeySource>, value: S, ) { diff --git a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr index aca006b38..ca42a4fa4 100644 --- a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr +++ b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr @@ -15,5 +15,5 @@ note: required by a bound in `Encryption::<'s, S, T, K, Ctx, M>::zip` | ^^^^^^^^^^^^^^^^^^ required by this bound in `Encryption::<'s, S, T, K, Ctx, M>::zip` help: consider further restricting type parameter `S` with trait `Clone` | - 9 | fn seal_and_index( - | +++++++++++++++++++ + 9 | fn seal_and_index( + | +++++++++++++++++++ diff --git a/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs b/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs index 4ae01d79f..f15dcfedd 100644 --- a/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs +++ b/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs @@ -5,10 +5,10 @@ use stack_encrypt::sem::{CllwOpeEncrypt, CllwOreEncrypt, EqualityTerm, MatchTerm use stack_encrypt::target::{ ciphertext, equality, matching, ope, ore, CallerContext, Owned, Pending, }; -use stack_encrypt::{nonempty, Encrypt, KeysetCipher, StackCipherText}; +use stack_encrypt::{nonempty, KeysetCipher, Plaintext, StackCipherText}; use stack_kms::FakeDataKeySource; -fn seal<'a, S: Encrypt>( +fn seal<'a, S: Plaintext>( keyset: &'a KeysetCipher<'_, FakeDataKeySource>, value: S, ) -> Pending<'a, StackCipherText, FakeDataKeySource> { diff --git a/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs b/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs index 3ff7a83e1..78f5c82f6 100644 --- a/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs +++ b/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs @@ -5,7 +5,7 @@ use stack_kms::FakeDataKeySource; #[derive(EncryptFrom)] struct Probe { hm: EqualityTerm } -// No Encrypt bound: producing a term requires only the PRF capability. +// No Plaintext bound: producing a term requires only the PRF capability. fn query(cipher:&StackCipher, value:&S) { let keyset=cipher.default_keyset(); let _ = keyset.encrypt_as::<_,Probe>(value,nonempty!("column").into()); diff --git a/packages/stack-encrypt/tests/ui/plaintext_needs_encrypt.stderr b/packages/stack-encrypt/tests/ui/plaintext_needs_encrypt.stderr index dd4c11198..749218206 100644 --- a/packages/stack-encrypt/tests/ui/plaintext_needs_encrypt.stderr +++ b/packages/stack-encrypt/tests/ui/plaintext_needs_encrypt.stderr @@ -1,4 +1,4 @@ -error[E0277]: the trait bound `SerdeOnly: stack_encrypt::Encrypt` is not satisfied +error[E0277]: the trait bound `SerdeOnly: stack_encrypt::Plaintext` is not satisfied --> tests/ui/plaintext_needs_encrypt.rs:9:33 | 9 | let _=keyset.encrypt_as::<_,Target>(&SerdeOnly {value:"x".into()},nonempty!("column").into()); @@ -6,22 +6,22 @@ error[E0277]: the trait bound `SerdeOnly: stack_encrypt::Encrypt` is not satisfi | | | required by a bound introduced by this call | -help: the trait `stack_encrypt::Encrypt` is not implemented for `SerdeOnly` +help: the trait `stack_encrypt::Plaintext` is not implemented for `SerdeOnly` --> tests/ui/plaintext_needs_encrypt.rs:4:1 | 4 | struct SerdeOnly { value: String } | ^^^^^^^^^^^^^^^^ - = help: the following other types implement trait `stack_encrypt::Encrypt`: - &str - ContextTag - FfiValue - HashMap - Vec - Vec - [u8; N] - stack_encrypt::Element + = help: the following other types implement trait `stack_encrypt::Plaintext`: + &&'b T + &'a BTreeMap + &'a HashMap + &'a Vec + &'a [T] + &'a std::option::Option + &'a vitaminc_protected::protected::Protected + &FfiValue and $N others - = note: required for `CipherText>` to implement `EncryptFrom` + = note: required for `CipherText` to implement `EncryptFrom` = note: 1 redundant requirement hidden = note: required for `Target` to implement `EncryptFrom` note: required by a bound in `target::operations::>::encrypt_as` diff --git a/packages/stack-encrypt/tests/ui/struct_field_must_open.rs b/packages/stack-encrypt/tests/ui/struct_field_must_open.rs index 5394b539f..cf829679f 100644 --- a/packages/stack-encrypt/tests/ui/struct_field_must_open.rs +++ b/packages/stack-encrypt/tests/ui/struct_field_must_open.rs @@ -1,13 +1,29 @@ //! Every field of a `struct = ..` record opens back into the plaintext //! field it is derived from (`DecryptField`), even when //! the record derives `EncryptFrom` alone: the record's plan seals and -//! opens each field. A borrowed plaintext field cannot be opened into, and -//! a target with no `DecryptField` cannot open. +//! opens each field. A plaintext field with no `Decrypt`, such as one that +//! borrows its text, cannot be opened into, and a target with no +//! `DecryptField` cannot open. use stack_encrypt::target::{Decryptable, Encryption, AeadContext}; -use stack_encrypt::{EncryptFrom, StackCipherText}; +use stack_encrypt::{EncryptFrom, Plaintext, StackCipherText}; +use vitaminc_aead::plaintext::Sink; + +/// Text it borrows: it seals, and there is nothing to open it into. +#[derive(Clone)] +struct Name(&'static str); +impl Plaintext for Name { + fn visit(self, sink: S) -> Result { + self.0.visit(sink) + } +} +impl Plaintext for &Name { + fn visit(self, sink: S) -> Result { + self.0.visit(sink) + } +} struct Profile { - name: &'static str, + name: Name, bio: String, } diff --git a/packages/stack-encrypt/tests/ui/struct_field_must_open.stderr b/packages/stack-encrypt/tests/ui/struct_field_must_open.stderr index 40c9d95bf..59383c476 100644 --- a/packages/stack-encrypt/tests/ui/struct_field_must_open.stderr +++ b/packages/stack-encrypt/tests/ui/struct_field_must_open.stderr @@ -1,18 +1,27 @@ -error[E0277]: the trait bound `&str: Decrypt<'static>` is not satisfied - --> tests/ui/struct_field_must_open.rs:29:11 - | -29 | name: StackCipherText, - | ^^^^^^^^^^^^^^^ the trait `Decrypt<'static>` is not implemented for `&str` - | -help: the trait `Decrypt<'static>` is not implemented for `&str` - but trait `Decrypt<'_>` is implemented for `std::string::String` - --> $CARGO/vitaminc-aead-$VERSION/src/decrypt/impls.rs - | - | impl<'c> Decrypt<'c> for String { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - = help: for that trait implementation, expected `std::string::String`, found `&str` - = note: required for `CipherText>` to implement `DecryptInto<&str>` - = note: required for `CipherText>` to implement `DecryptField<&str, CallerContext>` +error[E0277]: the trait bound `Name: FromScalar` is not satisfied + --> tests/ui/struct_field_must_open.rs:45:11 + | +45 | name: StackCipherText, + | ^^^^^^^^^^^^^^^ unsatisfied trait bound + | +help: the trait `FromScalar` is not implemented for `Name` + --> tests/ui/struct_field_must_open.rs:13:1 + | +13 | struct Name(&'static str); + | ^^^^^^^^^^^ + = help: the following other types implement trait `FromScalar`: + Vec + [u8; N] + bool + f32 + f64 + i128 + i16 + i32 + and $N others + = note: required for `Name` to implement `Decrypt` + = note: required for `CipherText` to implement `DecryptInto` + = note: required for `CipherText` to implement `DecryptField` note: required by a bound in `FieldsBuilder::::encrypt_into` --> src/plan/build.rs | @@ -23,22 +32,22 @@ note: required by a bound in `FieldsBuilder::::encrypt_into` | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `FieldsBuilder::::encrypt_into` error[E0277]: the trait bound `Opaque: DecryptField` is not satisfied - --> tests/ui/struct_field_must_open.rs:30:10 + --> tests/ui/struct_field_must_open.rs:46:10 | -30 | bio: Opaque, +46 | bio: Opaque, | ^^^^^^ unsatisfied trait bound | help: the trait `DecryptField` is not implemented for `Opaque` - --> tests/ui/struct_field_must_open.rs:15:1 + --> tests/ui/struct_field_must_open.rs:31:1 | -15 | struct Opaque(#[allow(dead_code)] StackCipherText); +31 | struct Opaque(#[allow(dead_code)] StackCipherText); | ^^^^^^^^^^^^^ = help: the following other types implement trait `DecryptField`: `(A, B)` implements `DecryptField` `(A, B, C)` implements `DecryptField` `(A, B, C, D)` implements `DecryptField` `(A, B, C, D, E)` implements `DecryptField` - `CipherText>` implements `DecryptField` + `CipherText` implements `DecryptField` `Encrypted` implements `DecryptField` `EqualityTerm` implements `DecryptField` `MatchTerms` implements `DecryptField` diff --git a/packages/stack-guest-abi/Cargo.toml b/packages/stack-guest-abi/Cargo.toml index 3d3179922..a7ed2db77 100644 --- a/packages/stack-guest-abi/Cargo.toml +++ b/packages/stack-guest-abi/Cargo.toml @@ -16,18 +16,18 @@ publish = false [dependencies] zeroize = { workspace = true } # `last_error`: an error is any miette diagnostic plus its structured fields -# (a serde_json map, as `ErrorPayload::payload` gives them), encoded in the -# one transport codec every input and output already uses. +# (a serde_json map, as `ErrorPayload::payload` gives them), encoded as a +# `HostValue` in the one transport frame every input and output uses. miette = { workspace = true } serde_json = { workspace = true } thiserror = { workspace = true } -vitaminc-aead-value = { workspace = true } - -# Only `transport` (wasm32-only) needs these: the import's error type and -# `OpaqueDebug` for its response, whose body can carry wrapped key material -# or a credential and so never prints. Declared for that target alone, so a -# native build carries nothing it does not use and `cargo udeps` sees none. -[target.'cfg(target_arch = "wasm32")'.dependencies] +# `value`: a `HostValue` describes itself to vitaminc's plaintext model and +# crosses to the host as a transport frame. +vitaminc-aead = { workspace = true } +vitaminc-aead-frame = { workspace = true } +# `HostValue::Bytes` holds its bytes in `Protected`, and `transport` +# (wasm32-only) uses `OpaqueDebug` for its response, whose body can carry +# wrapped key material or a credential and so never prints. vitaminc-protected = { workspace = true } [dev-dependencies] diff --git a/packages/stack-guest-abi/src/abi.rs b/packages/stack-guest-abi/src/abi.rs index 90d363e35..bc04e172e 100644 --- a/packages/stack-guest-abi/src/abi.rs +++ b/packages/stack-guest-abi/src/abi.rs @@ -71,7 +71,7 @@ pub fn export(f: impl FnOnce() -> Result, u32>) -> u64 { } /// The full error behind the most recent failed export, as a packed buffer -/// result: a transport-codec object of `code`, `message`, `help`, `url` +/// result: a transport-frame map of `code`, `message`, `help`, `url` /// (when set), `severity`, `fields` and `causes` (see /// [`last_error::encode`]). Zero when there is none — no export has failed /// since the last success, or the error was already handed over. diff --git a/packages/stack-guest-abi/src/call.rs b/packages/stack-guest-abi/src/call.rs index a215404c0..c1b5bdc49 100644 --- a/packages/stack-guest-abi/src/call.rs +++ b/packages/stack-guest-abi/src/call.rs @@ -93,7 +93,7 @@ pub fn take_last_error() -> Option> { mod tests { use super::*; use crate::status::{STATUS_ENCODING, STATUS_INTERNAL}; - use vitaminc_aead_value::{transport as codec, FfiValue}; + use crate::value::HostValue; /// A packed result, read the way the Go host reads it. #[derive(Debug, PartialEq)] @@ -117,16 +117,8 @@ mod tests { /// The recorded error's code, handed over as the host would get it. fn last_code() -> Option { let bytes = take_last_error()?; - let value = codec::decode_value(&mut codec::Reader::new(&bytes)).expect("decodes"); - let FfiValue::Object(entries) = value else { - return None; - }; - entries.into_iter().find_map(|(key, value)| match value { - FfiValue::String(code) if key == "code" => { - String::from_utf8(code.risky_ref().to_vec()).ok() - } - _ => None, - }) + let value = HostValue::decode(&bytes).expect("decodes"); + value.get("code")?.as_text().map(str::to_owned) } #[test] diff --git a/packages/stack-guest-abi/src/last_error.rs b/packages/stack-guest-abi/src/last_error.rs index ff8ed8c1d..89db704f4 100644 --- a/packages/stack-guest-abi/src/last_error.rs +++ b/packages/stack-guest-abi/src/last_error.rs @@ -5,7 +5,7 @@ //! the fast path a host acts on, byte-compatible with the vitaminc guest. //! Beside it, the export records the error that produced the number — its //! miette code, message, help, URL, severity, structured fields and causes — -//! as one value in the transport codec every input and output already uses +//! as one value in the transport frame every input and output already uses //! ([`encode`]). A host that wants the detail calls `se_last_error` after a //! non-zero status; a call that succeeds pays nothing. //! @@ -43,10 +43,10 @@ use std::cell::Cell; use std::error::Error; use miette::{Diagnostic, Severity}; -use vitaminc_aead_value::{transport as codec, FfiValue}; use crate::buffers; use crate::status::{STATUS_ENCODING, STATUS_INTERNAL, STATUS_STATE}; +use crate::value::HostValue; thread_local! { /// The registered buffer holding the encoded last error, if any, and @@ -82,7 +82,7 @@ pub const UNDESCRIBED_CAUSE: &str = "an error from another library"; #[non_exhaustive] pub enum GuestError { /// The input failed the export's own validation before any library - /// saw it: bytes that are not the transport codec, a pointer/length + /// saw it: bytes that are not a transport frame, a pointer/length /// pair outside linear memory, a config or option of the wrong shape. #[error("malformed input: {0}")] #[diagnostic(code(stack_guest_abi::malformed_input))] @@ -191,7 +191,7 @@ pub fn encode( error: &dyn Diagnostic, fields: serde_json::Map, describe: &dyn Fn(&(dyn Error + 'static)) -> Option, -) -> FfiValue { +) -> HostValue { let mut entries = Vec::with_capacity(7); if let Some(code) = error.code() { entries.push(("code".to_owned(), text(code.to_string()))); @@ -211,18 +211,13 @@ pub fn encode( entries.push(("severity".to_owned(), text(severity))); entries.push(( "fields".to_owned(), - FfiValue::Object( - fields - .into_iter() - .map(|(key, value)| (key, json_value(value))) - .collect(), - ), + HostValue::from_json(serde_json::Value::Object(fields)), )); entries.push(( "causes".to_owned(), - FfiValue::Array(causes(error, describe)), + HostValue::List(causes(error, describe)), )); - FfiValue::Object(entries) + HostValue::Map(entries) } /// One link of the cause chain. @@ -239,7 +234,7 @@ enum Cause<'a> { fn causes( error: &dyn Diagnostic, describe: &dyn Fn(&(dyn Error + 'static)) -> Option, -) -> Vec { +) -> Vec { let mut out = Vec::new(); let mut next = next_of(error); while let Some(cause) = next { @@ -263,7 +258,7 @@ fn causes( entry.push(("code".to_owned(), text(code))); } entry.push(("message".to_owned(), text(message))); - out.push(FfiValue::Object(entry)); + out.push(HostValue::Map(entry)); next = after; } out @@ -279,35 +274,8 @@ fn next_of(diagnostic: &dyn Diagnostic) -> Option> { } /// A string leaf. -fn text(value: impl Into) -> FfiValue { - FfiValue::String(value.into().into()) -} - -/// A structured field as a transport value. -fn json_value(value: serde_json::Value) -> FfiValue { - match value { - serde_json::Value::Null => FfiValue::Null, - serde_json::Value::Bool(value) => FfiValue::Bool(value), - serde_json::Value::Number(number) => { - if let Some(value) = number.as_u64() { - FfiValue::UInt64(value) - } else if let Some(value) = number.as_i64() { - FfiValue::Int64(value) - } else { - FfiValue::Float64(number.as_f64().unwrap_or(f64::NAN)) - } - } - serde_json::Value::String(value) => text(value), - serde_json::Value::Array(items) => { - FfiValue::Array(items.into_iter().map(json_value).collect()) - } - serde_json::Value::Object(entries) => FfiValue::Object( - entries - .into_iter() - .map(|(key, value)| (key, json_value(value))) - .collect(), - ), - } +fn text(value: impl Into) -> HostValue { + HostValue::text(value) } /// Record `error` as the last error for `status`, with its structured @@ -331,14 +299,14 @@ pub fn record_with( describe: &dyn Fn(&(dyn Error + 'static)) -> Option, status: u32, ) -> u32 { - let mut bytes = Vec::new(); // An error that does not encode (it cannot: every leaf is a string, a - // number or a container) leaves the last error empty rather than half - // written; the status still says what happened. - if codec::encode_value(encode(error, fields, describe), &mut bytes).is_err() { + // number or a container, and every key is distinct) leaves the last + // error empty rather than half written; the status still says what + // happened. + let Ok(bytes) = encode(error, fields, describe).encode() else { clear(); return status; - } + }; store(bytes, status); status } @@ -417,22 +385,16 @@ pub fn take() -> Option> { mod tests { use super::*; - fn decode(bytes: &[u8]) -> FfiValue { - codec::decode_value(&mut codec::Reader::new(bytes)).expect("an encoded error decodes") + fn decode(bytes: &[u8]) -> HostValue { + HostValue::decode(bytes).expect("an encoded error decodes") } - fn get<'a>(value: &'a FfiValue, key: &str) -> Option<&'a FfiValue> { - match value { - FfiValue::Object(entries) => entries.iter().find(|(k, _)| k == key).map(|(_, v)| v), - _ => None, - } + fn get<'a>(value: &'a HostValue, key: &str) -> Option<&'a HostValue> { + value.get(key) } - fn string(value: Option<&FfiValue>) -> Option { - match value? { - FfiValue::String(s) => String::from_utf8(s.risky_ref().to_vec()).ok(), - _ => None, - } + fn string(value: Option<&HostValue>) -> Option { + value?.as_text().map(str::to_owned) } #[derive(Debug, thiserror::Error, Diagnostic)] @@ -473,9 +435,9 @@ mod tests { assert_eq!(string(get(&value, "severity")).as_deref(), Some("error")); assert!(matches!( get(get(&value, "fields").expect("fields"), "count"), - Some(FfiValue::UInt64(2)) + Some(HostValue::UInt(2)) )); - let Some(FfiValue::Array(causes)) = get(&value, "causes") else { + let Some(HostValue::List(causes)) = get(&value, "causes") else { panic!("causes is a list"); }; assert_eq!(causes.len(), 2); @@ -504,7 +466,7 @@ mod tests { struct Wrapper(#[source] Foreign); let _ = record(&Wrapper(Foreign), serde_json::Map::new(), STATUS_INTERNAL); let value = decode(&take().expect("recorded")); - let Some(FfiValue::Array(causes)) = get(&value, "causes") else { + let Some(HostValue::List(causes)) = get(&value, "causes") else { panic!("causes is a list"); }; assert_eq!( @@ -550,7 +512,7 @@ mod tests { ); assert!(matches!( get(get(&value, "fields").expect("fields"), "status"), - Some(FfiValue::UInt64(42)) + Some(HostValue::UInt(42)) )); } diff --git a/packages/stack-guest-abi/src/lib.rs b/packages/stack-guest-abi/src/lib.rs index f635c2855..6b4db1a3f 100644 --- a/packages/stack-guest-abi/src/lib.rs +++ b/packages/stack-guest-abi/src/lib.rs @@ -54,8 +54,8 @@ //! both are tested natively and the guests' leak tests run the real //! lifecycle. //! - [`last_error`] — the full error behind the most recent failed export: -//! its code, message, help, fields and causes, encoded once in the -//! transport codec for `se_last_error`. The status number stays the fast +//! its code, message, help, fields and causes, encoded once as a +//! transport frame for `se_last_error`. The status number stays the fast //! path; this is the detail a host asks for after it. //! - [`status`] — the status table. **One numbering for every guest**: the //! codes below keep their values for good, and a guest that needs more @@ -65,6 +65,8 @@ //! call and gets both response buffers back through the registry. //! - [`headers`] — the `name: value` line format request and response //! headers cross the import in. +//! - [`value`] — [`HostValue`](value::HostValue), a value a guest hands its +//! host, and the transport frame it crosses as. //! //! What is deliberately *not* here: anything that names a crate a guest is //! built over. The ZeroKMS connection over the transport, the token import @@ -102,6 +104,7 @@ pub mod call; pub mod headers; pub mod last_error; pub mod status; +pub mod value; #[cfg(target_arch = "wasm32")] pub mod abi; diff --git a/packages/stack-guest-abi/src/value.rs b/packages/stack-guest-abi/src/value.rs new file mode 100644 index 000000000..013013566 --- /dev/null +++ b/packages/stack-guest-abi/src/value.rs @@ -0,0 +1,350 @@ +//! [`HostValue`]: a value a guest hands its host, and how it crosses the +//! boundary. +//! +//! Every structured output a guest returns (a list of workspace ids, the +//! fields of a token, the last error) crosses as one transport frame, the +//! encoding vitaminc's `Frame` and `FrameWriter` define and the Go module +//! reads with `vcffi`. A `HostValue` describes itself to vitaminc's +//! [`Plaintext`] model, so [`encode`](HostValue::encode) is vitaminc's own +//! frame writer, and [`decode`](HostValue::decode) reads a frame back +//! through a [`Sink`], for the tests that check what a host is given. + +use vitaminc_aead::plaintext::{ + visit_map, visit_seq, Float, Logical, MapSink, Physical, Plaintext, SeqSink, Sink, SourceError, +}; +use vitaminc_aead_frame::{to_frame, Frame, FrameError, FrameWriteError}; +use vitaminc_protected::{Controlled, Protected}; + +/// A value a guest hands its host: a scalar, a list or a map with string +/// keys. +/// +/// Integers are 64-bit, unsigned where the value cannot be negative, so the +/// host reads one integer type per field. Bytes are held in [`Protected`], +/// as a client key is, and wiped when the value drops. +pub enum HostValue { + /// No value. + Null, + /// A boolean. + Bool(bool), + /// An unsigned integer. + UInt(u64), + /// A signed integer. + Int(i64), + /// A float. + Float(f64), + /// Text. + Text(String), + /// Bytes. + Bytes(Protected>), + /// A list. + List(Vec), + /// A map, in the order its entries were given. + Map(Vec<(String, HostValue)>), +} + +impl HostValue { + /// Text, from anything that converts to a `String`. + pub fn text(value: impl Into) -> Self { + Self::Text(value.into()) + } + + /// A structured field as a host value: a JSON number becomes an + /// unsigned integer if it is one, then a signed integer, then a float. + pub fn from_json(value: serde_json::Value) -> Self { + match value { + serde_json::Value::Null => Self::Null, + serde_json::Value::Bool(value) => Self::Bool(value), + serde_json::Value::Number(number) => { + if let Some(value) = number.as_u64() { + Self::UInt(value) + } else if let Some(value) = number.as_i64() { + Self::Int(value) + } else { + Self::Float(number.as_f64().unwrap_or(f64::NAN)) + } + } + serde_json::Value::String(value) => Self::Text(value), + serde_json::Value::Array(items) => { + Self::List(items.into_iter().map(Self::from_json).collect()) + } + serde_json::Value::Object(entries) => Self::Map( + entries + .into_iter() + .map(|(key, value)| (key, Self::from_json(value))) + .collect(), + ), + } + } + + /// This value as a transport frame. + /// + /// The frame is returned bare, as every guest output is: it is copied + /// into a registered buffer, which the registry wipes when the host + /// releases it. + /// + /// # Errors + /// + /// [`FrameWriteError`] for a map with a repeated key or a value nested + /// deeper than a frame carries. A guest builds its outputs itself, so + /// either is a bug in the guest. + pub fn encode(&self) -> Result, FrameWriteError> { + to_frame(self).map(Protected::risky_unwrap) + } + + /// Read one value from a transport frame. + /// + /// # Errors + /// + /// [`DecodeError`] when the bytes are not exactly one frame, or the + /// frame holds a scalar a `HostValue` does not (a 32-bit float, a date). + pub fn decode(bytes: &[u8]) -> Result { + let frame = Frame::parse(bytes).map_err(DecodeError::Frame)?; + frame.visit(ValueSink(Root)) + } + + /// The entry under `key`, if this is a map that has one. + pub fn get(&self, key: &str) -> Option<&HostValue> { + match self { + Self::Map(entries) => entries.iter().find(|(k, _)| k == key).map(|(_, v)| v), + _ => None, + } + } + + /// The text, if this is text. + pub fn as_text(&self) -> Option<&str> { + match self { + Self::Text(text) => Some(text), + _ => None, + } + } +} + +/// Why [`HostValue::decode`] refused its bytes. +#[derive(Debug, thiserror::Error)] +pub enum DecodeError { + /// The bytes are not exactly one well-formed frame. + #[error(transparent)] + Frame(FrameError), + /// The frame holds a scalar no `HostValue` holds. + #[error("the frame holds a scalar a host value does not")] + Scalar, + /// The frame's source failed (it cannot: a parsed frame is complete). + #[error("the frame could not be read")] + Source, +} + +impl vitaminc_aead::plaintext::Error for DecodeError { + fn source(_: SourceError) -> Self { + Self::Source + } +} + +/// Walks the value, leaving it where it is. +impl Plaintext for &HostValue { + fn visit(self, sink: S) -> Result { + match self { + HostValue::Null => sink.scalar(None, Physical::Null), + HostValue::Bool(b) => sink.scalar(None, Physical::Bool(*b)), + HostValue::UInt(u) => sink.scalar(None, Physical::Int((*u).into())), + HostValue::Int(i) => sink.scalar(None, Physical::Int((*i).into())), + HostValue::Float(f) => sink.scalar(None, Physical::Float(Float::F64(*f))), + HostValue::Text(text) => sink.scalar(None, Physical::Text(text)), + HostValue::Bytes(bytes) => sink.scalar(None, Physical::Bytes(bytes.risky_ref())), + HostValue::List(items) => visit_seq(sink, Some(items.len()), items), + HostValue::Map(entries) => visit_map( + sink, + Some(entries.len()), + entries.iter().map(|(key, value)| (key.as_str(), value)), + ), + } + } +} + +/// Where a finished value goes: back to the caller, or into its parent. +trait Put { + type Ok; + fn put(self, value: HostValue) -> Self::Ok; +} + +struct Root; + +impl Put for Root { + type Ok = HostValue; + fn put(self, value: HostValue) -> HostValue { + value + } +} + +struct Element<'s>(&'s mut Vec); + +impl Put for Element<'_> { + type Ok = (); + fn put(self, value: HostValue) { + self.0.push(value); + } +} + +struct Entry<'s>(&'s mut Vec<(String, HostValue)>, String); + +impl Put for Entry<'_> { + type Ok = (); + fn put(self, value: HostValue) { + self.0.push((self.1, value)); + } +} + +/// Builds the value a frame describes. +struct ValueSink(O); + +impl Sink for ValueSink { + type Ok = O::Ok; + type Error = DecodeError; + type Seq = ValueSeq; + type Map = ValueMap; + + fn scalar(self, ty: Option, value: Physical<'_>) -> Result { + let value = match (ty, value) { + (None, Physical::Null) => HostValue::Null, + (None, Physical::Bool(b)) => HostValue::Bool(b), + (None, Physical::Int(int)) => { + if let Some(u) = int.to_u128().and_then(|u| u64::try_from(u).ok()) { + HostValue::UInt(u) + } else if let Some(i) = int.to_i128().and_then(|i| i64::try_from(i).ok()) { + HostValue::Int(i) + } else { + return Err(DecodeError::Scalar); + } + } + (None, Physical::Float(Float::F64(f))) => HostValue::Float(f), + (None, Physical::Text(text)) => HostValue::Text(text.to_owned()), + (None, Physical::Bytes(bytes)) => HostValue::Bytes(Protected::new(bytes.to_vec())), + _ => return Err(DecodeError::Scalar), + }; + Ok(self.0.put(value)) + } + + fn seq(self, _: Option) -> Result, DecodeError> { + Ok(ValueSeq { + out: self.0, + items: Vec::new(), + }) + } + + fn map(self, _: Option) -> Result, DecodeError> { + Ok(ValueMap { + out: self.0, + entries: Vec::new(), + }) + } + + /// A guest never hands its host a passthrough. + fn passthrough(self, _: P) -> Result { + Err(DecodeError::Scalar) + } +} + +struct ValueSeq { + out: O, + items: Vec, +} + +impl SeqSink for ValueSeq { + type Ok = O::Ok; + type Error = DecodeError; + type Element<'s> + = ValueSink> + where + Self: 's; + + fn element(&mut self) -> Result>, DecodeError> { + Ok(ValueSink(Element(&mut self.items))) + } + + fn end(self) -> Result { + Ok(self.out.put(HostValue::List(self.items))) + } +} + +struct ValueMap { + out: O, + entries: Vec<(String, HostValue)>, +} + +impl MapSink for ValueMap { + type Ok = O::Ok; + type Error = DecodeError; + type Entry<'s> + = ValueSink> + where + Self: 's; + + fn entry(&mut self, key: &str) -> Result>, DecodeError> { + Ok(ValueSink(Entry(&mut self.entries, key.to_owned()))) + } + + fn end(self) -> Result { + Ok(self.out.put(HostValue::Map(self.entries))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_value_reads_back_as_it_was_written() { + let value = HostValue::Map(vec![ + ("null".into(), HostValue::Null), + ("yes".into(), HostValue::Bool(true)), + ("big".into(), HostValue::UInt(u64::MAX)), + ("negative".into(), HostValue::Int(-2)), + ("half".into(), HostValue::Float(0.5)), + ("name".into(), HostValue::text("alice")), + ("key".into(), HostValue::Bytes(Protected::new(vec![1, 2]))), + ( + "ids".into(), + HostValue::List(vec![HostValue::text("a"), HostValue::List(vec![])]), + ), + ]); + let bytes = value.encode().expect("encodes"); + let back = HostValue::decode(&bytes).expect("decodes"); + assert!(matches!(back.get("null"), Some(HostValue::Null))); + assert!(matches!(back.get("yes"), Some(HostValue::Bool(true)))); + assert!(matches!(back.get("big"), Some(HostValue::UInt(u64::MAX)))); + assert!(matches!(back.get("negative"), Some(HostValue::Int(-2)))); + assert!(matches!(back.get("half"), Some(HostValue::Float(f)) if *f == 0.5)); + assert_eq!(back.get("name").and_then(HostValue::as_text), Some("alice")); + assert!(matches!(back.get("key"), Some(HostValue::Bytes(b)) if b.risky_ref() == &[1, 2])); + assert!(matches!(back.get("ids"), Some(HostValue::List(items)) if items.len() == 2)); + } + + #[test] + fn a_map_with_a_repeated_key_does_not_encode() { + let value = HostValue::Map(vec![ + ("a".into(), HostValue::Null), + ("a".into(), HostValue::Null), + ]); + assert!(value.encode().is_err()); + } + + #[test] + fn bytes_that_are_not_one_frame_do_not_decode() { + assert!(matches!(HostValue::decode(&[]), Err(DecodeError::Frame(_)))); + } + + #[test] + fn json_numbers_keep_their_integer_type() { + assert!(matches!( + HostValue::from_json(serde_json::json!(2)), + HostValue::UInt(2) + )); + assert!(matches!( + HostValue::from_json(serde_json::json!(-2)), + HostValue::Int(-2) + )); + assert!(matches!( + HostValue::from_json(serde_json::json!(0.5)), + HostValue::Float(f) if f == 0.5 + )); + } +} From 4008caa11e493c2e463669d1e0b46c8f0fa38875 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 01:01:34 +1100 Subject: [PATCH 24/38] test(stack-encrypt): build the fuzz crate against the same vitaminc, with no absent marker to fuzz The fuzz crate is its own workspace and compiles stack-encrypt by path, so it carries the same temporary vitaminc pin. check_record's model loses the None node, which ADR 0001 retires, and builds its passthroughs as frames. --- packages/stack-encrypt/fuzz/Cargo.lock | 96 +++++++++++++------ packages/stack-encrypt/fuzz/Cargo.toml | 12 +++ .../fuzz/fuzz_targets/check_record.rs | 22 +++-- 3 files changed, 91 insertions(+), 39 deletions(-) diff --git a/packages/stack-encrypt/fuzz/Cargo.lock b/packages/stack-encrypt/fuzz/Cargo.lock index 086027d53..635728183 100644 --- a/packages/stack-encrypt/fuzz/Cargo.lock +++ b/packages/stack-encrypt/fuzz/Cargo.lock @@ -1520,6 +1520,15 @@ dependencies = [ "syn 2.0.114", ] +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + [[package]] name = "proc-macro-error-attr2" version = "2.0.0" @@ -2033,7 +2042,7 @@ dependencies = [ "thiserror 1.0.69", "uuid", "vitaminc-aead", - "vitaminc-aead-value", + "vitaminc-aead-frame", "vitaminc-encrypt", "vitaminc-hmac", "vitaminc-prf", @@ -2283,6 +2292,36 @@ dependencies = [ "syn 2.0.114", ] +[[package]] +name = "toml_datetime" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.17+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" +dependencies = [ + "winnow", +] + [[package]] name = "tracing" version = "0.1.44" @@ -2474,8 +2513,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "vitaminc" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35042fb7a81d72dc43c53fec835e61027f18b4d01cc0f7e12d6c6a6ffbf3bfb9" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead", "vitaminc-context", @@ -2488,11 +2526,12 @@ dependencies = [ [[package]] name = "vitaminc-aead" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62cacd4dae485e98cfe5407e919187a8468dd933ea4234be5115b7d13a30f3c6" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bytes", + "futures-util", "serde", + "thiserror 2.0.18", "vitaminc-aead-derive", "vitaminc-context", "vitaminc-protected", @@ -2503,8 +2542,7 @@ dependencies = [ [[package]] name = "vitaminc-aead-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff34056b70a8417ad3bf9db73b2bd3be97f448794566c6de2080bd8a7cd9b6b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -2512,21 +2550,17 @@ dependencies = [ ] [[package]] -name = "vitaminc-aead-value" +name = "vitaminc-aead-frame" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d78cdadbc6374c3bd7c19fb4d99abca693b96953b18fef8c608efbe092cf3ea" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead", - "vitaminc-protected", - "zeroize", ] [[package]] name = "vitaminc-context" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7df52501fe33cdb3ee9e155b5242fac184c037d147e58c96e5143ed011252a8" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "vitaminc-protected", @@ -2535,8 +2569,7 @@ dependencies = [ [[package]] name = "vitaminc-encrypt" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fbd0e1748055e381e02033e08af385de879c4dcf32e0f6739ac0c0cf0c6f13d" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "aes-gcm", "aws-lc-rs", @@ -2549,8 +2582,7 @@ dependencies = [ [[package]] name = "vitaminc-hmac" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15fe558a5e92d4f2b3f811c5b67d863d9255da5f66d08a71d4f0dd3d0011c563" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "hmac", "sha2 0.11.0", @@ -2562,8 +2594,7 @@ dependencies = [ [[package]] name = "vitaminc-prf" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712d31d46d76b38c6b62deb35f516fc5636a35f90f39c0213d0ca9a052b4400b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "thiserror 2.0.18", @@ -2574,8 +2605,7 @@ dependencies = [ [[package]] name = "vitaminc-protected" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7b514f605a63f88874ba46ffea81775fad64994475ab239cb19da159e884b7b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bitvec", "digest 0.11.3", @@ -2591,9 +2621,9 @@ dependencies = [ [[package]] name = "vitaminc-protected-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cb0a3d11afbe2c909f0cfc9996fe37fd89b1196eb0b8d89a66e28716360ee09" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ + "proc-macro-crate", "proc-macro2", "quote", "syn 3.0.3", @@ -2602,8 +2632,7 @@ dependencies = [ [[package]] name = "vitaminc-random" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7b1bfab35ee322de37269c6fd94d8cd94047a66f97cf6722e53e3321482167" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "chacha20", "getrandom 0.4.2", @@ -2617,8 +2646,7 @@ dependencies = [ [[package]] name = "vitaminc-random-derives" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0915e2bfb417ce11e23123fb7ec122af5d27d8cfd22d6ad6a2cf7a1c2dbb4712" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -2628,8 +2656,7 @@ dependencies = [ [[package]] name = "vitaminc-traits" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b31ec300e351a7a59e64899d0a2d7e7227cfa74ac0514f4e745acb2cc9335cee" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "anyhow", "bytes", @@ -2991,6 +3018,15 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + [[package]] name = "wit-bindgen" version = "0.51.0" diff --git a/packages/stack-encrypt/fuzz/Cargo.toml b/packages/stack-encrypt/fuzz/Cargo.toml index 5a3bca512..8db1c8589 100644 --- a/packages/stack-encrypt/fuzz/Cargo.toml +++ b/packages/stack-encrypt/fuzz/Cargo.toml @@ -63,3 +63,15 @@ bench = false [workspace] resolver = "2" + +# Temporary, as in the root Cargo.toml: the vitaminc crates stack-encrypt +# builds on come from the branch that implements vitaminc ADR 0001 +# (cipherstash/vitaminc PRs #439 to #445) until vitaminc releases it. +[patch.crates-io] +vitaminc = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/packages/stack-encrypt/fuzz/fuzz_targets/check_record.rs b/packages/stack-encrypt/fuzz/fuzz_targets/check_record.rs index 39e2a375d..d7277890c 100644 --- a/packages/stack-encrypt/fuzz/fuzz_targets/check_record.rs +++ b/packages/stack-encrypt/fuzz/fuzz_targets/check_record.rs @@ -35,9 +35,11 @@ use stack_encrypt::dynamic::record::{check_record, plan_with, Plan}; use stack_encrypt::dynamic::{ FfiValue, Output, TargetDescriptor, TargetError, TargetResolver, ValueKind, }; -use stack_encrypt::{KeysetCipher, Label, Pending, SealedValue, StackCipher, StackCipherText}; +use stack_encrypt::{ + KeysetCipher, Label, PassthroughFrame, Pending, SealedValue, StackCipher, StackCipherText, +}; use uuid::Uuid; -use vitaminc_protected::Protected; +use vitaminc_protected::{Controlled, Protected}; /// The resolver plans parse under: one producible type, `TextEq` over /// strings, and nothing that runs — `check_record` never reaches a cipher. @@ -264,7 +266,6 @@ impl PlanSpec { #[derive(Arbitrary, Debug)] enum Tree { Single, - None, EmptySequence, EmptyMap, Passthrough, @@ -284,17 +285,20 @@ fn leaf() -> SealedValue { .clone() } +/// A value as a passthrough node: its frame, in the clear. +fn passthrough(value: FfiValue) -> StackCipherText { + let frame = value.to_frame().expect("a scalar frames"); + StackCipherText::Passthrough(PassthroughFrame(frame.risky_unwrap())) +} + impl Tree { fn into_ciphertext(self) -> StackCipherText { match self { Tree::Single => StackCipherText::Single(leaf()), - Tree::None => StackCipherText::None(leaf()), Tree::EmptySequence => StackCipherText::EmptySequence(leaf()), Tree::EmptyMap => StackCipherText::EmptyMap(leaf()), - Tree::Passthrough => StackCipherText::Passthrough(Box::new(FfiValue::Null)), - Tree::PassthroughBytes => StackCipherText::Passthrough(Box::new(FfiValue::Bytes( - Protected::new(b"{}".to_vec()), - ))), + Tree::Passthrough => passthrough(FfiValue::Null), + Tree::PassthroughBytes => passthrough(FfiValue::Bytes(Protected::new(b"{}".to_vec()))), Tree::Sequence(items) => { StackCipherText::Sequence(items.into_iter().map(Tree::into_ciphertext).collect()) } @@ -320,7 +324,7 @@ impl Tree { .all(|(at, (name, _))| !entries[..at].iter().any(|(prior, _)| prior == name)); unique && entries.iter().all(|(_, node)| node.is_clean()) } - Tree::Single | Tree::None | Tree::EmptySequence | Tree::EmptyMap => true, + Tree::Single | Tree::EmptySequence | Tree::EmptyMap => true, } } } From 596344131b107cb74aef1d997dd02a398d5c565b Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 01:01:34 +1100 Subject: [PATCH 25/38] feat(golang)!: carry transport frames between the Go module and both guests Both WASI guests build against the same temporary vitaminc pin as the root workspace, and every value crosses as a transport frame. - The encrypt guest decodes inputs with `FfiValue::from_frame`, writes outputs through vitaminc's `FrameWriter` into a buffer still sized before the first byte, and reads and writes ciphertext trees with `vitaminc_aead_frame`'s codec, a passthrough as its frame. - The credential guest's outputs are `HostValue`s. - The Go module moves vcffi and vcvalue to the same vitaminc commit, whose container tags and leaf kinds match the frame. The absent leaf kind is gone from the binding's leaf set. vcffi sends an 8- or 16-bit integer at its own width, which the engine reads as the 32-bit integer of its sign, so a generated field seals as it did. --- languages/golang/auth/guest/Cargo.lock | 92 +++++++++++----- languages/golang/auth/guest/Cargo.toml | 15 ++- languages/golang/auth/guest/src/lib.rs | 4 +- languages/golang/auth/guest/src/ops.rs | 56 +++++----- languages/golang/auth/guest/tests/leak.rs | 32 +++--- languages/golang/encrypt/ciphertext.go | 5 +- languages/golang/encrypt/guest/Cargo.lock | 102 +++++++++++++----- languages/golang/encrypt/guest/Cargo.toml | 24 +++-- languages/golang/encrypt/guest/src/abi.rs | 9 +- languages/golang/encrypt/guest/src/config.rs | 2 +- languages/golang/encrypt/guest/src/lib.rs | 5 +- languages/golang/encrypt/guest/src/ops.rs | 98 ++++++++--------- languages/golang/encrypt/guest/src/options.rs | 2 +- languages/golang/encrypt/guest/src/status.rs | 2 +- languages/golang/encrypt/guest/src/targets.rs | 14 +-- languages/golang/encrypt/guest/tests/leak.rs | 37 ++++--- .../golang/encrypt/guest/tests/native_ops.rs | 96 +++++++++-------- languages/golang/encrypt/unit_test.go | 1 - languages/golang/go.mod | 4 +- languages/golang/go.sum | 4 + languages/golang/stashgen/emit.go | 7 +- 21 files changed, 360 insertions(+), 251 deletions(-) diff --git a/languages/golang/auth/guest/Cargo.lock b/languages/golang/auth/guest/Cargo.lock index 1616b6d77..141489710 100644 --- a/languages/golang/auth/guest/Cargo.lock +++ b/languages/golang/auth/guest/Cargo.lock @@ -1365,6 +1365,15 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + [[package]] name = "proc-macro-error-attr2" version = "2.0.0" @@ -1858,7 +1867,6 @@ dependencies = [ "stack-profile", "tempfile", "url", - "vitaminc-aead-value", "vitaminc-protected", "zeroize", ] @@ -1870,7 +1878,8 @@ dependencies = [ "miette", "serde_json", "thiserror 1.0.69", - "vitaminc-aead-value", + "vitaminc-aead", + "vitaminc-aead-frame", "vitaminc-protected", "zeroize", ] @@ -2066,6 +2075,36 @@ dependencies = [ "syn 3.0.6", ] +[[package]] +name = "toml_datetime" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.17+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" +dependencies = [ + "winnow", +] + [[package]] name = "tracing" version = "0.1.44" @@ -2247,8 +2286,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "vitaminc" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35042fb7a81d72dc43c53fec835e61027f18b4d01cc0f7e12d6c6a6ffbf3bfb9" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead", "vitaminc-context", @@ -2261,11 +2299,12 @@ dependencies = [ [[package]] name = "vitaminc-aead" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62cacd4dae485e98cfe5407e919187a8468dd933ea4234be5115b7d13a30f3c6" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bytes", + "futures-util", "serde", + "thiserror 2.0.20", "vitaminc-aead-derive", "vitaminc-context", "vitaminc-protected", @@ -2276,8 +2315,7 @@ dependencies = [ [[package]] name = "vitaminc-aead-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff34056b70a8417ad3bf9db73b2bd3be97f448794566c6de2080bd8a7cd9b6b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -2285,21 +2323,17 @@ dependencies = [ ] [[package]] -name = "vitaminc-aead-value" +name = "vitaminc-aead-frame" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d78cdadbc6374c3bd7c19fb4d99abca693b96953b18fef8c608efbe092cf3ea" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead", - "vitaminc-protected", - "zeroize", ] [[package]] name = "vitaminc-context" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7df52501fe33cdb3ee9e155b5242fac184c037d147e58c96e5143ed011252a8" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "vitaminc-protected", @@ -2308,8 +2342,7 @@ dependencies = [ [[package]] name = "vitaminc-encrypt" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fbd0e1748055e381e02033e08af385de879c4dcf32e0f6739ac0c0cf0c6f13d" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "aes-gcm", "aws-lc-rs", @@ -2322,8 +2355,7 @@ dependencies = [ [[package]] name = "vitaminc-protected" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7b514f605a63f88874ba46ffea81775fad64994475ab239cb19da159e884b7b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bitvec", "digest 0.11.3", @@ -2339,9 +2371,9 @@ dependencies = [ [[package]] name = "vitaminc-protected-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cb0a3d11afbe2c909f0cfc9996fe37fd89b1196eb0b8d89a66e28716360ee09" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ + "proc-macro-crate", "proc-macro2", "quote", "syn 3.0.6", @@ -2350,8 +2382,7 @@ dependencies = [ [[package]] name = "vitaminc-random" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7b1bfab35ee322de37269c6fd94d8cd94047a66f97cf6722e53e3321482167" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "chacha20", "getrandom 0.4.3", @@ -2365,8 +2396,7 @@ dependencies = [ [[package]] name = "vitaminc-random-derives" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0915e2bfb417ce11e23123fb7ec122af5d27d8cfd22d6ad6a2cf7a1c2dbb4712" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -2376,8 +2406,7 @@ dependencies = [ [[package]] name = "vitaminc-traits" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b31ec300e351a7a59e64899d0a2d7e7227cfa74ac0514f4e745acb2cc9335cee" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "anyhow", "bytes", @@ -2622,6 +2651,15 @@ version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + [[package]] name = "wit-bindgen" version = "0.57.1" diff --git a/languages/golang/auth/guest/Cargo.toml b/languages/golang/auth/guest/Cargo.toml index d3928e995..39973dab4 100644 --- a/languages/golang/auth/guest/Cargo.toml +++ b/languages/golang/auth/guest/Cargo.toml @@ -18,6 +18,17 @@ publish = false [workspace] +# Temporary, as in the root Cargo.toml: the vitaminc crates stack-guest-abi +# and the stack crates build on come from the branch that implements +# vitaminc ADR 0001 (cipherstash/vitaminc PRs #439 to #445) until vitaminc +# releases it. +[patch.crates-io] +vitaminc = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } + [lib] # cdylib: the .wasm guest module. rlib: lets the ops/status modules # unit-test natively (`cargo test` here, no wasm toolchain needed). @@ -41,10 +52,6 @@ stack-guest-abi = { path = "../../../../packages/stack-guest-abi" } # hostname and the process id are native-only there, and creating a device # identity is CLI territory. stack-profile = { path = "../../../../packages/stack-profile" } -# The FFI codec for the structured results (a list of workspace ids, the -# fields of a token): the one codec every binding carries. Same vitaminc -# version the crypto guest builds against. -vitaminc-aead-value = "0.5.0" # The deserialized form of `secretkey.json` is wiped when it drops. zeroize = { version = "1", features = ["derive"] } diff --git a/languages/golang/auth/guest/src/lib.rs b/languages/golang/auth/guest/src/lib.rs index fb9986d4c..a95e5c511 100644 --- a/languages/golang/auth/guest/src/lib.rs +++ b/languages/golang/auth/guest/src/lib.rs @@ -58,8 +58,8 @@ //! //! Profile inputs are UTF-8 strings: the store directory (a guest path under //! the mount), a workspace id, a filename. Outputs are either a UTF-8 string -//! (an id, a path) or a value in vitaminc's FFI codec -//! (`vitaminc_aead_value::transport`): a list of ids, or the fields of +//! (an id, a path) or a vitaminc transport frame +//! ([`HostValue`](stack_guest_abi::value::HostValue)): a list of ids, or the fields of //! `secretkey.json`, `auth.json` or `device.json` as an object. The client //! key in `secretkey.json` crosses as the text the file holds; the Go side //! wraps it in its opaque `ClientKey` and wipes its transport copy. A diff --git a/languages/golang/auth/guest/src/ops.rs b/languages/golang/auth/guest/src/ops.rs index 6a32bfa5d..ab8bb36ed 100644 --- a/languages/golang/auth/guest/src/ops.rs +++ b/languages/golang/auth/guest/src/ops.rs @@ -30,8 +30,8 @@ use serde::Deserialize; use stack_auth::Token; +use stack_guest_abi::value::HostValue; use stack_profile::{DeviceIdentity, ProfileStore}; -use vitaminc_aead_value::{transport as codec, FfiValue}; use zeroize::{Zeroize, ZeroizeOnDrop}; use stack_guest_abi::last_error::{internal, malformed}; @@ -76,18 +76,18 @@ fn text(bytes: &[u8]) -> Result<&str, u32> { std::str::from_utf8(bytes).map_err(|_| malformed("an input is not UTF-8")) } -fn string(value: impl Into) -> FfiValue { - FfiValue::String(value.into().into()) +fn string(value: impl Into) -> HostValue { + HostValue::text(value) } -fn optional(value: Option<&str>) -> FfiValue { - value.map_or(FfiValue::Null, string) +fn optional(value: Option<&str>) -> HostValue { + value.map_or(HostValue::Null, string) } -fn encode(value: FfiValue) -> Result, u32> { - let mut out = Vec::new(); - codec::encode_value(value, &mut out).map_err(|_| internal("an output did not encode"))?; - Ok(out) +fn encode(value: HostValue) -> Result, u32> { + value + .encode() + .map_err(|_| internal("an output did not encode")) } /// The current workspace id, as text. @@ -122,7 +122,7 @@ pub fn list_workspaces(dir: &[u8]) -> Result, u32> { let ids = store(dir)? .list_workspaces() .map_err(|e| fail_profile(&e))?; - encode(FfiValue::Array(ids.into_iter().map(string).collect())) + encode(HostValue::List(ids.into_iter().map(string).collect())) } /// The directory of the store scoped to workspace `id`, as text: what @@ -165,11 +165,11 @@ pub fn secret_key(dir: &[u8]) -> Result, u32> { // fields cannot be moved out of it directly. let client_id = std::mem::take(&mut file.client_id); let client_key = std::mem::take(&mut file.client_key); - encode(FfiValue::Object(vec![ + encode(HostValue::Map(vec![ ("client_id".to_string(), string(client_id)), ( "client_key".to_string(), - FfiValue::Bytes(client_key.into_bytes().into()), + HostValue::Bytes(client_key.into_bytes().into()), ), ])) } @@ -184,7 +184,7 @@ pub fn token(dir: &[u8]) -> Result, u32> { let token: Token = store(dir)? .load(AUTH_FILENAME) .map_err(|e| fail_profile(&e))?; - encode(FfiValue::Object(vec![ + encode(HostValue::Map(vec![ ( "access_token".to_string(), string(token.access_token().as_str()), @@ -192,7 +192,7 @@ pub fn token(dir: &[u8]) -> Result, u32> { ("token_type".to_string(), string(token.token_type())), ( "expires_at".to_string(), - FfiValue::UInt64(token.expires_at()), + HostValue::UInt(token.expires_at()), ), ("region".to_string(), optional(token.region())), ("client_id".to_string(), optional(token.client_id())), @@ -214,7 +214,7 @@ pub fn has_token(dir: &[u8]) -> Result, u32> { /// `{device_instance_id, device_name}`. Creating one is the CLI's. pub fn device_identity(dir: &[u8]) -> Result, u32> { let identity = DeviceIdentity::load(&store(dir)?).map_err(|e| fail_profile(&e))?; - encode(FfiValue::Object(vec![ + encode(HostValue::Map(vec![ ( "device_instance_id".to_string(), string(identity.device_instance_id.to_string()), @@ -235,12 +235,12 @@ mod tests { t.path().to_str().unwrap().as_bytes().to_vec() } - fn decode(bytes: &[u8]) -> FfiValue { - codec::decode_value(&mut codec::Reader::new(bytes)).unwrap() + fn decode(bytes: &[u8]) -> HostValue { + HostValue::decode(bytes).unwrap() } - fn field<'a>(value: &'a FfiValue, key: &str) -> &'a FfiValue { - let FfiValue::Object(entries) = value else { + fn field<'a>(value: &'a HostValue, key: &str) -> &'a HostValue { + let HostValue::Map(entries) = value else { panic!("not an object"); }; entries @@ -250,16 +250,16 @@ mod tests { .unwrap_or_else(|| panic!("no field {key}")) } - fn as_text(value: &FfiValue) -> String { - let FfiValue::String(s) = value else { + fn as_text(value: &HostValue) -> String { + let HostValue::Text(s) = value else { panic!("not a string"); }; - String::from_utf8(s.risky_ref().to_vec()).unwrap() + s.clone() } - fn as_bytes(value: &FfiValue) -> Vec { + fn as_bytes(value: &HostValue) -> Vec { use vitaminc_protected::Controlled; - let FfiValue::Bytes(b) = value else { + let HostValue::Bytes(b) = value else { panic!("not bytes"); }; b.risky_ref().to_vec() @@ -314,7 +314,7 @@ mod tests { assert_eq!(current_workspace(&d).unwrap(), WS_A.as_bytes()); let listed = decode(&list_workspaces(&d).unwrap()); - let FfiValue::Array(items) = listed else { + let HostValue::List(items) = listed else { panic!("not an array"); }; let ids: Vec = items.iter().map(as_text).collect(); @@ -411,11 +411,11 @@ mod tests { assert_eq!(as_text(field(&tok, "token_type")), "Bearer"); assert!(matches!( field(&tok, "expires_at"), - FfiValue::UInt64(1_800_000_000) + HostValue::UInt(1_800_000_000) )); assert_eq!(as_text(field(&tok, "region")), "ap-southeast-2"); - assert!(matches!(field(&tok, "client_id"), FfiValue::Null)); - let FfiValue::Object(entries) = tok else { + assert!(matches!(field(&tok, "client_id"), HostValue::Null)); + let HostValue::Map(entries) = tok else { panic!("not an object"); }; assert!( diff --git a/languages/golang/auth/guest/tests/leak.rs b/languages/golang/auth/guest/tests/leak.rs index 6d08d4cf5..2b9f23082 100644 --- a/languages/golang/auth/guest/tests/leak.rs +++ b/languages/golang/auth/guest/tests/leak.rs @@ -25,7 +25,7 @@ use stack_auth::AuthError; use stack_auth_guest::ops; use stack_auth_guest::status::fail_auth; use stack_guest_abi::call; -use vitaminc_aead_value::{transport as codec, FfiValue}; +use stack_guest_abi::value::HostValue; const ACCESS_TOKEN: &str = "leak-marker-access-token"; const ACCESS_KEY: &str = "leak-marker-access-key"; @@ -43,18 +43,18 @@ fn failure( what: &str, expect: &[&str], step: impl FnOnce() -> Result, -) -> FfiValue { +) -> HostValue { let _ = call::run(step).expect_err(what); let bytes = call::take_last_error().unwrap_or_else(|| panic!("{what}: no error recorded")); - let error = codec::decode_value(&mut codec::Reader::new(&bytes)).expect("an error decodes"); + let error = HostValue::decode(&bytes).expect("an error decodes"); assert_eq!(codes(&error), expect, "{what}: the codes it reaches"); error } /// An error's code, then its causes' codes, in order. -fn codes(value: &FfiValue) -> Vec { +fn codes(value: &HostValue) -> Vec { let mut out: Vec = text(get(value, "code")).into_iter().collect(); - if let Some(FfiValue::Array(causes)) = get(value, "causes") { + if let Some(HostValue::List(causes)) = get(value, "causes") { out.extend(causes.iter().filter_map(|cause| text(get(cause, "code")))); } out @@ -71,19 +71,19 @@ fn check(text: &[u8], at: &str, found: &mut Vec) { } } -fn leaks(value: &FfiValue, path: &str, found: &mut Vec) { +fn leaks(value: &HostValue, path: &str, found: &mut Vec) { match value { - FfiValue::String(text) => check(text.risky_ref(), path, found), - FfiValue::Bytes(bytes) => { + HostValue::Text(text) => check(text.as_bytes(), path, found), + HostValue::Bytes(bytes) => { use vitaminc_protected::Controlled; check(bytes.risky_ref(), path, found); } - FfiValue::Array(items) => { + HostValue::List(items) => { for (at, item) in items.iter().enumerate() { leaks(item, &format!("{path}[{at}]"), found); } } - FfiValue::Object(entries) => { + HostValue::Map(entries) => { for (key, item) in entries { check(key.as_bytes(), &format!("{path} key"), found); leaks(item, &format!("{path}.{key}"), found); @@ -93,16 +93,16 @@ fn leaks(value: &FfiValue, path: &str, found: &mut Vec) { } } -fn get<'a>(value: &'a FfiValue, key: &str) -> Option<&'a FfiValue> { +fn get<'a>(value: &'a HostValue, key: &str) -> Option<&'a HostValue> { match value { - FfiValue::Object(entries) => entries.iter().find(|(k, _)| k == key).map(|(_, v)| v), + HostValue::Map(entries) => entries.iter().find(|(k, _)| k == key).map(|(_, v)| v), _ => None, } } -fn text(value: Option<&FfiValue>) -> Option { +fn text(value: Option<&HostValue>) -> Option { match value? { - FfiValue::String(s) => String::from_utf8(s.risky_ref().to_vec()).ok(), + HostValue::Text(s) => Some(s.clone()), _ => None, } } @@ -111,7 +111,7 @@ fn dir(path: &Path) -> Vec { path.to_str().expect("utf8 temp dir").as_bytes().to_vec() } -fn scenarios() -> Vec<(&'static str, FfiValue)> { +fn scenarios() -> Vec<(&'static str, HostValue)> { let root = tempfile::tempdir().expect("temp dir"); let store = dir(root.path()); let workspace = root.path().join("workspaces").join("AAAAAAAAAAAAAAAA"); @@ -268,7 +268,7 @@ fn a_profile_json_error_points_at_the_line() { Some("stack_profile::json") ); let fields = get(&error, "fields").expect("fields"); - assert!(matches!(get(fields, "line"), Some(FfiValue::UInt64(2)))); + assert!(matches!(get(fields, "line"), Some(HostValue::UInt(2)))); assert!( text(get(&error, "help")).is_some(), "the help says how to fix it" diff --git a/languages/golang/encrypt/ciphertext.go b/languages/golang/encrypt/ciphertext.go index 242d85d67..7d70475d8 100644 --- a/languages/golang/encrypt/ciphertext.go +++ b/languages/golang/encrypt/ciphertext.go @@ -50,8 +50,9 @@ func scanBytes(kind string, src any) ([]byte, error) { } // otherLeaf is a leaf of a kind the SDK does not store: the authenticated -// markers for an absent value, an empty sequence and an empty map, which a -// record field never produces. Decoding one is an error at the use site. +// markers for an empty sequence and an empty map, which a record field never +// produces. Decoding one is an error at the use site. An absent value is a +// sealed null, a Ciphertext like any other value. type otherLeaf struct { kind vcffi.LeafKind bytes []byte diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 09815fb6a..20dc422e4 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -1486,6 +1486,15 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + [[package]] name = "proc-macro-error-attr2" version = "2.0.0" @@ -2076,7 +2085,7 @@ dependencies = [ "thiserror 1.0.69", "uuid", "vitaminc-aead", - "vitaminc-aead-value", + "vitaminc-aead-frame", "vitaminc-encrypt", "vitaminc-hmac", "vitaminc-prf", @@ -2109,7 +2118,7 @@ dependencies = [ "stack-guest-abi", "stack-kms", "uuid", - "vitaminc-aead-value", + "vitaminc-aead-frame", "vitaminc-protected", "zeroize", "zerokms-protocol", @@ -2122,7 +2131,8 @@ dependencies = [ "miette", "serde_json", "thiserror 1.0.69", - "vitaminc-aead-value", + "vitaminc-aead", + "vitaminc-aead-frame", "vitaminc-protected", "zeroize", ] @@ -2357,6 +2367,36 @@ dependencies = [ "syn 3.0.4", ] +[[package]] +name = "toml_datetime" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.17+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" +dependencies = [ + "winnow", +] + [[package]] name = "tracing" version = "0.1.44" @@ -2570,8 +2610,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "vitaminc" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35042fb7a81d72dc43c53fec835e61027f18b4d01cc0f7e12d6c6a6ffbf3bfb9" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead", "vitaminc-context", @@ -2584,11 +2623,12 @@ dependencies = [ [[package]] name = "vitaminc-aead" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62cacd4dae485e98cfe5407e919187a8468dd933ea4234be5115b7d13a30f3c6" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bytes", + "futures-util", "serde", + "thiserror 2.0.20", "vitaminc-aead-derive", "vitaminc-context", "vitaminc-protected", @@ -2599,14 +2639,21 @@ dependencies = [ [[package]] name = "vitaminc-aead-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff34056b70a8417ad3bf9db73b2bd3be97f448794566c6de2080bd8a7cd9b6b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", "syn 3.0.4", ] +[[package]] +name = "vitaminc-aead-frame" +version = "0.5.1" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" +dependencies = [ + "vitaminc-aead", +] + [[package]] name = "vitaminc-aead-value" version = "0.5.1" @@ -2621,8 +2668,7 @@ dependencies = [ [[package]] name = "vitaminc-context" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7df52501fe33cdb3ee9e155b5242fac184c037d147e58c96e5143ed011252a8" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "vitaminc-protected", @@ -2631,8 +2677,7 @@ dependencies = [ [[package]] name = "vitaminc-encrypt" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fbd0e1748055e381e02033e08af385de879c4dcf32e0f6739ac0c0cf0c6f13d" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "aes-gcm", "aws-lc-rs", @@ -2645,8 +2690,7 @@ dependencies = [ [[package]] name = "vitaminc-hmac" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15fe558a5e92d4f2b3f811c5b67d863d9255da5f66d08a71d4f0dd3d0011c563" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "hmac", "sha2 0.11.0", @@ -2658,8 +2702,7 @@ dependencies = [ [[package]] name = "vitaminc-prf" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712d31d46d76b38c6b62deb35f516fc5636a35f90f39c0213d0ca9a052b4400b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "thiserror 2.0.20", @@ -2670,8 +2713,7 @@ dependencies = [ [[package]] name = "vitaminc-protected" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7b514f605a63f88874ba46ffea81775fad64994475ab239cb19da159e884b7b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bitvec", "digest 0.11.3", @@ -2687,9 +2729,9 @@ dependencies = [ [[package]] name = "vitaminc-protected-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cb0a3d11afbe2c909f0cfc9996fe37fd89b1196eb0b8d89a66e28716360ee09" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ + "proc-macro-crate", "proc-macro2", "quote", "syn 3.0.4", @@ -2698,8 +2740,7 @@ dependencies = [ [[package]] name = "vitaminc-random" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7b1bfab35ee322de37269c6fd94d8cd94047a66f97cf6722e53e3321482167" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "chacha20", "getrandom 0.4.3", @@ -2713,8 +2754,7 @@ dependencies = [ [[package]] name = "vitaminc-random-derives" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0915e2bfb417ce11e23123fb7ec122af5d27d8cfd22d6ad6a2cf7a1c2dbb4712" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -2724,8 +2764,7 @@ dependencies = [ [[package]] name = "vitaminc-traits" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b31ec300e351a7a59e64899d0a2d7e7227cfa74ac0514f4e745acb2cc9335cee" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "anyhow", "bytes", @@ -2979,6 +3018,15 @@ version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + [[package]] name = "wit-bindgen" version = "0.57.1" diff --git a/languages/golang/encrypt/guest/Cargo.toml b/languages/golang/encrypt/guest/Cargo.toml index ef9814ecf..4cef89332 100644 --- a/languages/golang/encrypt/guest/Cargo.toml +++ b/languages/golang/encrypt/guest/Cargo.toml @@ -17,6 +17,18 @@ publish = false [workspace] +# Temporary, as in the root Cargo.toml: the vitaminc crates this guest and +# stack-encrypt build on come from the branch that implements vitaminc ADR +# 0001 (cipherstash/vitaminc PRs #439 to #445) until vitaminc releases it. +[patch.crates-io] +vitaminc = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } + [lib] # cdylib: the .wasm guest module. rlib: lets the ops/config/status modules # unit-test natively (`cargo test` here, no wasm toolchain needed). @@ -61,12 +73,12 @@ stack-kms = { path = "../../../../packages/stack-kms", default-features = false eql-bindings = { path = "../../../../packages/eql/crates/eql-bindings", features = ["stack-encrypt"], optional = true } zerokms-protocol = "=0.12.31" -# The FFI codec + `FfiValue` tree, shared with vitaminc's own guest — one -# codec, not a fork. Same vitaminc version stack-encrypt builds against -# (see the comment in packages/stack-encrypt/Cargo.toml). A different -# version here fails at the `FfiValue: Decrypt` bound, since the aead crate -# would be duplicated. -vitaminc-aead-value = "0.5.1" +# The transport frame and ciphertext codec, shared with vitaminc's own +# guest — one codec, not a fork. Same vitaminc version stack-encrypt builds +# against (see the comment in packages/stack-encrypt/Cargo.toml). A +# different version here fails at the `FfiValue: Decrypt` bound, since the +# aead crate would be duplicated. +vitaminc-aead-frame = "0.5.1" vitaminc-protected = "0.5.1" futures = { version = "0.3", default-features = false, features = ["executor"] } diff --git a/languages/golang/encrypt/guest/src/abi.rs b/languages/golang/encrypt/guest/src/abi.rs index 5e5d1c675..7abbdd660 100644 --- a/languages/golang/encrypt/guest/src/abi.rs +++ b/languages/golang/encrypt/guest/src/abi.rs @@ -72,12 +72,11 @@ use std::cell::{Cell, RefCell}; use std::panic::{catch_unwind, AssertUnwindSafe}; use futures::executor::block_on; +use stack_encrypt::dynamic::FfiValue; use stack_encrypt::{KeysetCipher, StackCipher}; use stack_guest_abi::abi::{export, input, take_plaintext, wipe_input}; use stack_guest_abi::buffers; use stack_guest_abi::last_error::{self, malformed, out_of_order}; -use vitaminc_aead_value::transport as codec; -use vitaminc_aead_value::FfiValue; use crate::ops; use crate::options::{parse_options, parse_selector, scope_for, KeysetSelector, Side}; @@ -111,8 +110,8 @@ thread_local! { /// Decode one codec-encoded input. fn decode(bytes: &[u8]) -> Result { - codec::decode_value(&mut codec::Reader::new(bytes)) - .map_err(|_| malformed("an input is not a value in the transport codec")) + FfiValue::from_frame(bytes) + .map_err(|_| malformed("an input is not a value in the transport frame")) } /// Run `f` with the instance's cipher, or report `STATUS_STATE` when there @@ -336,7 +335,7 @@ pub unsafe extern "C" fn se_keyset(sel_ptr: *const u8, sel_len: u32) -> u64 { /// /// The context is one part — a string, bytes, or an `i32`/`i64`/`u32`/`u64` /// — or an array of parts, which may nest as deep as the transport codec -/// allows (`vitaminc_aead_value::transport::MAX_DEPTH` levels, counted from +/// allows (`vitaminc_aead_frame::MAX_DEPTH` levels, counted from /// the root of the encoded value; deeper is refused as `STATUS_ENCODING` /// before the context is parsed). [`stack_encrypt::dynamic::context`] is the one home of /// that grammar and of which Rust context each shape spells. diff --git a/languages/golang/encrypt/guest/src/config.rs b/languages/golang/encrypt/guest/src/config.rs index 800461444..e31f1e7d7 100644 --- a/languages/golang/encrypt/guest/src/config.rs +++ b/languages/golang/encrypt/guest/src/config.rs @@ -25,9 +25,9 @@ use std::num::NonZeroUsize; +use stack_encrypt::dynamic::FfiValue; use stack_kms::{ClientKey, ZeroKmsEndpoint}; use uuid::Uuid; -use vitaminc_aead_value::FfiValue; use zeroize::Zeroizing; /// A parse failure, carrying which key was at fault. Maps to diff --git a/languages/golang/encrypt/guest/src/lib.rs b/languages/golang/encrypt/guest/src/lib.rs index dfe7fbbdb..e402e14d0 100644 --- a/languages/golang/encrypt/guest/src/lib.rs +++ b/languages/golang/encrypt/guest/src/lib.rs @@ -78,8 +78,9 @@ //! //! On wasm32 `vitaminc-encrypt` uses its pure-Rust (RustCrypto `aes-gcm`) //! backend; the trade-offs are documented there. Values cross the boundary -//! in the vitaminc FFI codec (`vitaminc_aead_value::transport` — an FFI -//! encoding, not a storage format); the leaves inside a ciphertext tree are +//! as vitaminc transport frames and ciphertext trees in vitaminc's +//! ciphertext codec (`vitaminc_aead_frame` — FFI encodings, not a storage +//! format); the leaves inside a ciphertext tree are //! the *frozen* `SealedValue` byte encoding from Phase 2, so a leaf lifted //! out of a tree is exactly what a database column holds. diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index 9b0a8c9ac..ed08a9236 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -4,9 +4,10 @@ //! module wires them to the session table and the packed ABI; nothing in //! here knows about linear memory. //! -//! Values and ciphertext trees cross the boundary in the vitaminc FFI codec -//! (`vitaminc_aead_value::transport`) — the same codec the vitaminc guest -//! uses, so the Go side carries exactly one codec. A ciphertext tree's +//! Values cross the boundary as vitaminc transport frames, and ciphertext +//! trees in vitaminc's ciphertext codec (`vitaminc_aead_frame`) — the same +//! encodings the vitaminc guest uses, so the Go side carries exactly one +//! codec. A ciphertext tree's //! leaves are re-encoded through [`SealedValue::to_bytes`] / //! [`SealedValue::from_bytes`]: the codec sees an opaque byte-string leaf, //! and the bytes inside it are the frozen storage encoding a database column @@ -35,15 +36,15 @@ //! ABI's numeric term kinds, and the mapping from a library error to a //! status code. -use stack_encrypt::dynamic::{self, Scalar, Scope}; +use stack_encrypt::dynamic::{self, FfiValue, Scalar, Scope}; use stack_encrypt::sem::MatchOptions; use stack_encrypt::target::IndexSpec; use stack_encrypt::ErrorPayload; use stack_encrypt::{ - BoxedPassthrough, CipherText, KeysetCipher, Label, SealedValue, StackCipherText, + CipherText, KeysetCipher, Label, Node, Plaintext, SealedValue, StackCipherText, }; use stack_kms::DataKeySource; -use vitaminc_aead_value::{transport as codec, FfiValue}; +use vitaminc_aead_frame::{decode_ciphertext, encode_ciphertext, FrameWriter}; use vitaminc_protected::Controlled; use stack_guest_abi::last_error; @@ -63,7 +64,7 @@ pub const TERM_OPE: u32 = 4; /// A ciphertext tree whose leaves are the frozen [`SealedValue`] byte /// encoding — the shape that crosses the FFI codec. -type BytesTree = CipherText, BoxedPassthrough>; +type BytesTree = Node>; // ============================================================================= // Terms @@ -78,7 +79,7 @@ type BytesTree = CipherText, BoxedPassthrough>; /// /// The context is one part — a string, bytes, or an `i32`/`i64`/`u32`/`u64` /// — or an array of parts, nested as deep as the transport codec allows -/// ([`codec::MAX_DEPTH`] levels from the root of the encoded value; deeper +/// ([`MAX_DEPTH`](vitaminc_aead_frame::MAX_DEPTH) levels from the root of the encoded value; deeper /// is [`STATUS_ENCODING`] before the context is parsed), exactly as a plan /// field's; [`dynamic::context`] is the one home of that grammar. Shape is identity: /// `[x]` is a PAE-framed list and `x` is not, so a probe takes the context @@ -374,31 +375,32 @@ pub mod validate { // ============================================================================= fn decode_value(bytes: &[u8]) -> Result { - codec::decode_value(&mut codec::Reader::new(bytes)) - .map_err(|_| last_error::malformed("an input is not a value in the transport codec")) + FfiValue::from_frame(bytes) + .map_err(|_| last_error::malformed("an input is not a value in the transport frame")) } /// Encode a value tree into a buffer sized **before** the first byte is /// written. /// /// This buffer is plaintext on the decrypt path, and a `Vec` grown by the -/// codec's pushes would leave partial plaintext in every abandoned +/// frame writer's pushes would leave partial plaintext in every abandoned /// allocation a reallocation could not extend in place — memory nothing /// wipes, undercutting the guarantee the registry makes about the buffer it /// eventually hands the host. Reserving the exact encoded length up front -/// means the encoder never reallocates, and `register`'s `into_boxed_slice` +/// means the writer never reallocates, and `register`'s `into_boxed_slice` /// (capacity == length) does not copy either. fn encode_value(value: FfiValue) -> Result, u32> { let mut out = exact_buffer(value_encoded_len(&value))?; - codec::encode_value(value, &mut out).map_err(|_| STATUS_ENCODING)?; + (&value) + .visit(FrameWriter::new(&mut out)) + .map_err(|_| STATUS_ENCODING)?; Ok(out) } fn decode_tree(bytes: &[u8]) -> Result { - let tree: BytesTree = - codec::decode_ciphertext_boxed(&mut codec::Reader::new(bytes)).map_err(|_| { - last_error::malformed("the record is not a ciphertext tree in the transport codec") - })?; + let tree: BytesTree = decode_ciphertext(bytes).map_err(|_| { + last_error::malformed("the record is not a ciphertext tree in the transport codec") + })?; // Structural only — a decoded leaf proves nothing until its AEAD opens // (see the `SealedValue` docs). map_leaves(tree, &mut |l: Vec| { @@ -413,20 +415,19 @@ fn decode_tree(bytes: &[u8]) -> Result { fn encode_tree(tree: StackCipherText) -> Result, u32> { let tree = map_leaves(tree, &mut |l: SealedValue| Ok::<_, u32>(l.to_bytes()))?; let mut out = exact_buffer(tree_encoded_len(&tree))?; - codec::encode_ciphertext_boxed(tree, &mut out).map_err(|_| STATUS_ENCODING)?; + encode_ciphertext(&tree, &mut out).map_err(|_| STATUS_ENCODING)?; Ok(out) } /// Rebuild a ciphertext tree with every leaf run through `leaf`, keeping the -/// structure (and the passthrough payloads) untouched. One definition for +/// structure (and the passthrough frames) untouched. One definition for /// both directions of the frozen [`SealedValue`] leaf encoding. fn map_leaves( - tree: CipherText, + tree: Node, leaf: &mut impl FnMut(A) -> Result, -) -> Result, E> { +) -> Result, E> { Ok(match tree { CipherText::Single(l) => CipherText::Single(leaf(l)?), - CipherText::None(l) => CipherText::None(leaf(l)?), CipherText::EmptySequence(l) => CipherText::EmptySequence(leaf(l)?), CipherText::EmptyMap(l) => CipherText::EmptyMap(leaf(l)?), CipherText::Sequence(items) => CipherText::Sequence( @@ -460,9 +461,9 @@ fn exact_buffer(len: Option) -> Result, u32> { Ok(out) } -/// Exact byte length of the codec's encoding of `value`. `None` on overflow -/// or on a length the codec's `u32` frames cannot express — the encode would -/// fail on those anyway, so the caller reports an encoding error. +/// Exact byte length of the frame of `value`. `None` on overflow or on a +/// length the frame's `u32` lengths cannot express — the encode would fail +/// on those anyway, so the caller reports an encoding error. fn value_encoded_len(value: &FfiValue) -> Option { // tag byte + fixed payload, or tag + u32 length prefix + payload. let framed = |len: usize| u32::try_from(len).ok().and_then(|_| len.checked_add(5)); @@ -483,18 +484,14 @@ fn value_encoded_len(value: &FfiValue) -> Option { } } -/// Exact byte length of the codec's encoding of a ciphertext tree. The -/// passthrough payloads are read (not consumed) through `Any::downcast_ref`, -/// matching what `encode_ciphertext_boxed` will re-home them to; a payload -/// that is not an [`FfiValue`] is `None`, which is the same rejection the -/// encoder would make. +/// Exact byte length of the codec's encoding of a ciphertext tree. A +/// passthrough is its node byte and its frame, written as it is. fn tree_encoded_len(tree: &BytesTree) -> Option { let framed = |len: usize| u32::try_from(len).ok().and_then(|_| len.checked_add(5)); match tree { - CipherText::Single(l) - | CipherText::None(l) - | CipherText::EmptySequence(l) - | CipherText::EmptyMap(l) => framed(l.len()), + CipherText::Single(l) | CipherText::EmptySequence(l) | CipherText::EmptyMap(l) => { + framed(l.len()) + } CipherText::Sequence(items) => items .iter() .try_fold(5usize, |acc, item| acc.checked_add(tree_encoded_len(item)?)), @@ -502,9 +499,7 @@ fn tree_encoded_len(tree: &BytesTree) -> Option { acc.checked_add(framed(key.len())?.checked_sub(1)?)? .checked_add(tree_encoded_len(value)?) }), - CipherText::Passthrough(p) => { - value_encoded_len((**p).downcast_ref::()?)?.checked_add(1) - } + CipherText::Passthrough(p) => p.0.len().checked_add(1), } } @@ -513,8 +508,8 @@ mod tests { use super::*; use vitaminc_protected::Protected; - // `value_encoded_len` / `tree_encoded_len` re-derive the codec's framing - // arithmetic; the codec exports no `encoded_len` of its own, so these + // `value_encoded_len` / `tree_encoded_len` re-derive the frame's and the + // codec's arithmetic; neither exports an `encoded_len` of its own, so these // pins are the only thing that fails if the two drift. Drift is not a // cosmetic bug: an undersized reservation makes `encode_value` // reallocate mid-encode, leaving unwiped partial plaintext in the @@ -556,9 +551,8 @@ mod tests { fn value_encoded_len_matches_the_codec_exactly() { for (i, value) in every_value_shape().into_iter().enumerate() { let expected = value_encoded_len(&value).expect("encodable shape"); - let mut out = Vec::new(); - codec::encode_value(value, &mut out).expect("codec encode"); - assert_eq!(out.len(), expected, "shape {i}"); + let out = value.to_frame().expect("frame encode"); + assert_eq!(out.risky_ref().len(), expected, "shape {i}"); } } @@ -568,10 +562,9 @@ mod tests { let trees: Vec = vec![ leaf(b""), leaf(&[7u8; 40]), - CipherText::None(vec![1, 2]), CipherText::EmptySequence(vec![3]), CipherText::EmptyMap(Vec::new()), - CipherText::Sequence(vec![leaf(b"a"), CipherText::None(vec![9])]), + CipherText::Sequence(vec![leaf(b"a"), CipherText::EmptyMap(vec![9])]), CipherText::Map(vec![ ("name".to_string(), leaf(b"ct")), ( @@ -579,14 +572,17 @@ mod tests { CipherText::Map(vec![("x".to_string(), leaf(b"y"))]), ), ]), - CipherText::Passthrough( - Box::new(FfiValue::Bytes(Protected::new(vec![1, 2, 3]))) as BoxedPassthrough - ), + CipherText::Passthrough(stack_encrypt::PassthroughFrame( + FfiValue::Bytes(Protected::new(vec![1, 2, 3])) + .to_frame() + .expect("a frame") + .risky_unwrap(), + )), ]; for (i, tree) in trees.into_iter().enumerate() { let expected = tree_encoded_len(&tree).expect("encodable shape"); let mut out = Vec::new(); - codec::encode_ciphertext_boxed(tree, &mut out).expect("codec encode"); + encode_ciphertext(&tree, &mut out).expect("codec encode"); assert_eq!(out.len(), expected, "tree {i}"); } } @@ -594,9 +590,7 @@ mod tests { // ---- the EQL target path, native, over FakeDataKeySource ----------------- fn encoded(value: FfiValue) -> Vec { - let mut out = Vec::new(); - codec::encode_value(value, &mut out).expect("encode"); - out + value.to_frame().expect("encode").risky_unwrap() } fn text(value: &str) -> FfiValue { @@ -749,7 +743,7 @@ mod tests { let CipherText::Passthrough(payload) = &outputs[0].1 else { panic!("a passthrough") }; - let FfiValue::Bytes(bytes) = payload.downcast_ref::().expect("a value") else { + let FfiValue::Bytes(bytes) = FfiValue::from_frame(&payload.0).expect("a value") else { panic!("bytes") }; let eql: serde_json::Value = serde_json::from_slice(bytes.risky_ref()).expect("JSON"); diff --git a/languages/golang/encrypt/guest/src/options.rs b/languages/golang/encrypt/guest/src/options.rs index 02db276e1..569b4d05c 100644 --- a/languages/golang/encrypt/guest/src/options.rs +++ b/languages/golang/encrypt/guest/src/options.rs @@ -41,11 +41,11 @@ //! it is objects, strings, bytes and nothing else, and this module is its //! one home. The Go bindings plan points here. +use stack_encrypt::dynamic::FfiValue; use stack_encrypt::dynamic::Scope; use stack_encrypt::{KeysetCipher, Label, StackCipher}; use stack_kms::{IdentifiedBy, IndexKeySource}; use uuid::Uuid; -use vitaminc_aead_value::FfiValue; use vitaminc_protected::Controlled; use zerokms_protocol::Name; diff --git a/languages/golang/encrypt/guest/src/status.rs b/languages/golang/encrypt/guest/src/status.rs index 8425272e4..649ecddc8 100644 --- a/languages/golang/encrypt/guest/src/status.rs +++ b/languages/golang/encrypt/guest/src/status.rs @@ -414,8 +414,8 @@ mod tests { /// such. Swapping the arms, or dropping the `Other` one, fails here. #[test] fn target_refusals_are_encoding_and_a_resolver_failure_is_internal() { + use stack_encrypt::dynamic::ValueKind; use stack_encrypt::dynamic::{Error, TargetError}; - use vitaminc_aead_value::ValueKind; let refusals = [ TargetError::NoTargets { name: "TextEq".into(), diff --git a/languages/golang/encrypt/guest/src/targets.rs b/languages/golang/encrypt/guest/src/targets.rs index 4afc43340..5c570fde9 100644 --- a/languages/golang/encrypt/guest/src/targets.rs +++ b/languages/golang/encrypt/guest/src/targets.rs @@ -60,7 +60,7 @@ impl TargetResolver for EqlTargets { name: &str, keyset: &'a stack_encrypt::KeysetCipher<'_, K>, label: &stack_encrypt::Label, - plaintext: vitaminc_aead_value::FfiValue, + plaintext: stack_encrypt::dynamic::FfiValue, ) -> Result, K>, TargetError> { eql_bindings::encryption::targets::encrypt(name, keyset, label, plaintext).map_err(convert) } @@ -71,7 +71,7 @@ impl TargetResolver for EqlTargets { cipher: &'a stack_encrypt::StackCipher, label: &stack_encrypt::Label, stored: &[u8], - ) -> Result, TargetError> { + ) -> Result, TargetError> { eql_bindings::encryption::targets::decrypt(name, cipher, label, stored).map_err(convert) } @@ -80,7 +80,7 @@ impl TargetResolver for EqlTargets { name: &str, keyset: &'a stack_encrypt::KeysetCipher<'_, K>, label: &stack_encrypt::Label, - plaintext: vitaminc_aead_value::FfiValue, + plaintext: stack_encrypt::dynamic::FfiValue, ) -> Result, K>, TargetError> { eql_bindings::encryption::targets::query(name, keyset, label, plaintext).map_err(convert) } @@ -169,7 +169,7 @@ mod tests { assert!(text_eq.producible); assert_eq!( text_eq.plaintext, - Some(vitaminc_aead_value::ValueKind::String) + Some(stack_encrypt::dynamic::ValueKind::String) ); assert_eq!(text_eq.indexes, ["eq"]); assert_eq!(text_eq.query.as_deref(), Some("TextEqQuery")); @@ -203,10 +203,10 @@ mod tests { assert!(matches!( convert(Eql::Plaintext { target: "TextEq", - expected: vitaminc_aead_value::ValueKind::String, - found: Some(vitaminc_aead_value::ValueKind::UInt64), + expected: stack_encrypt::dynamic::ValueKind::String, + found: Some(stack_encrypt::dynamic::ValueKind::UInt64), }), - TargetError::Plaintext { target, expected: Some(vitaminc_aead_value::ValueKind::String), .. } if target == "TextEq" + TargetError::Plaintext { target, expected: Some(stack_encrypt::dynamic::ValueKind::String), .. } if target == "TextEq" )); } diff --git a/languages/golang/encrypt/guest/tests/leak.rs b/languages/golang/encrypt/guest/tests/leak.rs index 286520f51..46cb0816d 100644 --- a/languages/golang/encrypt/guest/tests/leak.rs +++ b/languages/golang/encrypt/guest/tests/leak.rs @@ -21,8 +21,9 @@ use std::borrow::Cow; use std::future::IntoFuture; use stack_auth::{SecretToken, ServiceToken}; +use stack_encrypt::dynamic::FfiValue; use stack_encrypt::dynamic::Scope; -use stack_encrypt::{CipherText, Descriptor, StackCipher}; +use stack_encrypt::{CipherText, Descriptor, Node, PassthroughFrame, StackCipher}; use stack_encrypt_guest::options::{parse_options, Side}; use stack_encrypt_guest::status::fail_error; use stack_encrypt_guest::{config, ops}; @@ -32,7 +33,7 @@ use stack_kms::{ GenerateKeyPayload, IndexKeySource, LoadKeysetError, RetrieveKeyError, RetrieveKeyPayload, }; use uuid::Uuid; -use vitaminc_aead_value::{transport as codec, FfiValue}; +use vitaminc_aead_frame::encode_ciphertext; use zerokms_protocol::{IdentifiedBy, UnverifiedContext, ViturRequestError}; const PLAINTEXT: &str = "leak-marker-plaintext"; @@ -127,9 +128,13 @@ fn failing() -> StackCipher { // ============================================================================= fn encode(value: FfiValue) -> Vec { - let mut out = Vec::new(); - codec::encode_value(value, &mut out).expect("encode"); - out + use vitaminc_protected::Controlled; + value.to_frame().expect("encode").risky_unwrap() +} + +/// A value as a passthrough node: its frame, in the clear. +fn clear(value: FfiValue) -> Node> { + CipherText::Passthrough(PassthroughFrame(encode(value))) } fn s(value: &str) -> FfiValue { @@ -191,13 +196,13 @@ fn sealed(cipher: &StackCipher, plan: &[u8]) -> Vec { } /// A one-leaf ciphertext tree whose leaf is `leaf`, under `email`/`c`. -fn tree_with_leaf(leaf: CipherText, Box>) -> Vec { - let tree: CipherText, Box> = CipherText::Map(vec![( +fn tree_with_leaf(leaf: Node>) -> Vec { + let tree: Node> = CipherText::Map(vec![( "email".to_owned(), CipherText::Map(vec![("c".to_owned(), leaf)]), )]); let mut out = Vec::new(); - codec::encode_ciphertext_boxed(tree, &mut out).expect("encode tree"); + encode_ciphertext(&tree, &mut out).expect("encode tree"); out } @@ -218,7 +223,7 @@ fn failure( ) -> FfiValue { let _ = call::run(step).expect_err(what); let bytes = call::take_last_error().unwrap_or_else(|| panic!("{what}: no error recorded")); - let error = codec::decode_value(&mut codec::Reader::new(&bytes)).expect("an error decodes"); + let error = FfiValue::from_frame(&bytes).expect("an error decodes"); assert_eq!(codes(&error), expect, "{what}: the codes it reaches"); error } @@ -488,8 +493,8 @@ fn scenarios() -> Vec<(&'static str, FfiValue)> { "record passthrough", &["stack_encrypt::dynamic_record"], || { - let record = tree_with_leaf(CipherText::Passthrough(Box::new( - FfiValue::Bytes(vitaminc_protected::Protected::new(CIPHERTEXT.to_vec())), + let record = tree_with_leaf(clear(FfiValue::Bytes( + vitaminc_protected::Protected::new(CIPHERTEXT.to_vec()), ))); block_on(ops::decrypt_record( Scope::Client(&cipher), @@ -651,19 +656,17 @@ fn scenarios() -> Vec<(&'static str, FfiValue)> { "email", obj(vec![("context", label("email")), ("target", s("TextEq"))]), )])); - let tree: CipherText, Box> = CipherText::Map(vec![( + let tree: Node> = CipherText::Map(vec![( "email".to_owned(), CipherText::Map(vec![( "eql".to_owned(), - CipherText::Passthrough(Box::new(FfiValue::Bytes( - vitaminc_protected::Protected::new( - [&b"\""[..], CIPHERTEXT, &b"\""[..]].concat(), - ), + clear(FfiValue::Bytes(vitaminc_protected::Protected::new( + [&b"\""[..], CIPHERTEXT, &b"\""[..]].concat(), ))), )]), )]); let mut record = Vec::new(); - codec::encode_ciphertext_boxed(tree, &mut record).expect("encode tree"); + encode_ciphertext(&tree, &mut record).expect("encode tree"); block_on(ops::decrypt_record( Scope::Client(&cipher), &record, diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 5a2f86639..8cafb754d 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -17,9 +17,13 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use std::future::IntoFuture; +use stack_encrypt::dynamic::FfiValue; use stack_encrypt::dynamic::Scope; use stack_encrypt::sem::DefaultMatch; -use stack_encrypt::{nonempty, CipherText, Encrypt, Label, SealedValue, StackCipher}; +use stack_encrypt::{ + nonempty, CipherText, Label, Node, PassthroughFrame, PendingStackCipherText, SealedValue, + StackCipher, +}; use stack_encrypt_guest::ops::{self, TERM_EQUALITY, TERM_MATCH, TERM_OPE, TERM_ORE}; use stack_encrypt_guest::status::{ STATUS_AUTH, STATUS_CONTEXT_MISMATCH, STATUS_ENCODING, STATUS_FOREIGN_KEYSET, @@ -29,7 +33,7 @@ use stack_kms::{ RetrieveKeyPayload, }; use uuid::Uuid; -use vitaminc_aead_value::{transport as codec, FfiValue}; +use vitaminc_aead_frame::{decode_ciphertext, encode_ciphertext}; use vitaminc_protected::{Controlled, Protected}; use zerokms_protocol::{IdentifiedBy, UnverifiedContext}; @@ -93,19 +97,29 @@ fn cipher() -> StackCipher { } fn encode(value: FfiValue) -> Vec { - let mut out = Vec::new(); - codec::encode_value(value, &mut out).expect("encode value"); - out + value.to_frame().expect("encode value").risky_unwrap() } fn decode(bytes: &[u8]) -> FfiValue { - codec::decode_value(&mut codec::Reader::new(bytes)).expect("decode value") + FfiValue::from_frame(bytes).expect("decode value") } -/// Decode a guest ciphertext buffer with the passthrough payload kept as a -/// plain [`FfiValue`], so tests can inspect term nodes directly. -fn decode_tree(bytes: &[u8]) -> CipherText, FfiValue> { - codec::decode_ciphertext(&mut codec::Reader::new(bytes)).expect("decode ciphertext tree") +/// Decode a guest ciphertext buffer with its leaves as bytes, so tests can +/// inspect the nodes directly. +fn decode_tree(bytes: &[u8]) -> Node> { + decode_ciphertext(bytes).expect("decode ciphertext tree") +} + +/// Encode a ciphertext tree as the guest reads it. +fn encode_tree(tree: &Node>) -> Vec { + let mut out = Vec::new(); + encode_ciphertext(tree, &mut out).expect("encode ciphertext tree"); + out +} + +/// A value as a passthrough node: its frame, in the clear. +fn clear(value: FfiValue) -> Node> { + CipherText::Passthrough(PassthroughFrame(encode(value))) } fn text(value: &FfiValue) -> &str { @@ -194,8 +208,11 @@ fn single_field_plan(field: &str, context: FfiValue) -> Vec { } /// The bytes of a term node in a decoded record tree. -fn term_bytes(node: &CipherText, FfiValue>) -> Vec { - let CipherText::Passthrough(FfiValue::Bytes(b)) = node else { +fn term_bytes(node: &Node>) -> Vec { + let CipherText::Passthrough(frame) = node else { + panic!("expected a passthrough term node"); + }; + let FfiValue::Bytes(b) = decode(&frame.0) else { panic!("expected a passthrough bytes term node"); }; b.risky_ref().to_vec() @@ -456,12 +473,11 @@ fn a_forged_passthrough_ciphertext_slot_is_rejected_not_decrypted() { }; for (key, slot) in outputs.iter_mut() { if key == "c" { - *slot = CipherText::Passthrough(FfiValue::UInt32(99)); + *slot = clear(FfiValue::UInt32(99)); } } } - let mut forged = Vec::new(); - codec::encode_ciphertext(&CipherText::Map(fields), &mut forged).expect("re-encode"); + let forged = encode_tree(&CipherText::Map(fields)); assert_eq!( block_on(ops::decrypt_record( @@ -666,9 +682,8 @@ fn a_natively_sealed_field_under_an_extended_context_opens_through_a_plan() { let keyset = cipher.default_keyset(); let native = nonempty!("users").with("age").with(7u64); let sealed = block_on( - FfiValue::UInt32(34) - .encrypt_with_aad(&keyset, native) - .expect("encrypt") + PendingStackCipherText::plan(FfiValue::UInt32(34), native) + .expect("plan") .seal(&keyset, native), ) .expect("seal"); @@ -677,12 +692,11 @@ fn a_natively_sealed_field_under_an_extended_context_opens_through_a_plan() { }; // Shape the tree the way `encrypt_record` writes it: field → { c: leaf }. - let tree: CipherText, FfiValue> = CipherText::Map(vec![( + let tree: Node> = CipherText::Map(vec![( "age".to_string(), CipherText::Map(vec![("c".to_string(), CipherText::Single(leaf.to_bytes()))]), )]); - let mut record = Vec::new(); - codec::encode_ciphertext(&tree, &mut record).expect("encode tree"); + let record = encode_tree(&tree); let plan_with = |context: FfiValue| single_field_plan("age", context); @@ -954,21 +968,15 @@ fn a_mixed_keyset_record_batch_opens_through_any_one_call_per_keyset() { )) .expect("encrypt globex row"); let batch = { - let (CipherText::Map(a), CipherText::Map(g)) = ( - codec::decode_ciphertext_boxed::>(&mut codec::Reader::new(&acme_rows)) - .expect("decode"), - codec::decode_ciphertext_boxed::>(&mut codec::Reader::new(&globex_rows)) - .expect("decode"), - ) else { + let (CipherText::Map(a), CipherText::Map(g)) = + (decode_tree(&acme_rows), decode_tree(&globex_rows)) + else { panic!("a single record is a map"); }; - let mut out = Vec::new(); - codec::encode_ciphertext_boxed( - CipherText::Sequence(vec![CipherText::Map(a), CipherText::Map(g)]), - &mut out, - ) - .expect("encode batch"); - out + encode_tree(&CipherText::Sequence(vec![ + CipherText::Map(a), + CipherText::Map(g), + ])) }; let before = cipher.kms().retrieve_calls.load(Ordering::SeqCst); @@ -1198,7 +1206,7 @@ fn record_tree_validation_refuses_what_decrypt_record_refuses() { .expect("encrypt record"); assert_eq!(ops::validate::record_tree(&record, &plan, None), Ok(())); - type Node = CipherText, FfiValue>; + type Node = stack_encrypt::Node>; // The tree is not `Clone`; every variant decodes the record afresh. let fields = || { let CipherText::Map(fields) = decode_tree(&record) else { @@ -1206,11 +1214,7 @@ fn record_tree_validation_refuses_what_decrypt_record_refuses() { }; fields }; - let re_encode = |tree: Node| { - let mut out = Vec::new(); - codec::encode_ciphertext(&tree, &mut out).expect("re-encode"); - out - }; + let re_encode = |tree: Node| encode_tree(&tree); let with_age = |edit: &dyn Fn(&mut Node)| { let mut fields = fields(); for (field, node) in &mut fields { @@ -1227,7 +1231,7 @@ fn record_tree_validation_refuses_what_decrypt_record_refuses() { }; for (key, slot) in outputs.iter_mut() { if key == "c" { - *slot = CipherText::Passthrough(FfiValue::UInt32(99)); + *slot = clear(FfiValue::UInt32(99)); } } }); @@ -1237,16 +1241,14 @@ fn record_tree_validation_refuses_what_decrypt_record_refuses() { }; outputs.retain(|(key, _)| key != "c"); }); - let not_a_map = with_age(&|node| *node = CipherText::Passthrough(FfiValue::Null)); + let not_a_map = with_age(&|node| *node = clear(FfiValue::Null)); let missing_field = { let mut fields = fields(); fields.retain(|(field, _)| field != "age"); CipherText::Map(fields) }; - let batch_of_non_records = CipherText::Sequence(vec![ - CipherText::Map(fields()), - CipherText::Passthrough(FfiValue::Null), - ]); + let batch_of_non_records = + CipherText::Sequence(vec![CipherText::Map(fields()), clear(FfiValue::Null)]); for (label, tree) in [ ("a forged passthrough under c", forged_c), @@ -1423,7 +1425,7 @@ fn a_context_field_record_opens_under_the_context_it_stores() { let CipherText::Sequence(rows) = decode_tree(&record) else { panic!("a batch is a sequence"); }; - let eq_of = |row: &CipherText, FfiValue>| { + let eq_of = |row: &Node>| { let CipherText::Map(fields) = row else { panic!("a row is a map"); }; diff --git a/languages/golang/encrypt/unit_test.go b/languages/golang/encrypt/unit_test.go index d016d8e6b..23083a74f 100644 --- a/languages/golang/encrypt/unit_test.go +++ b/languages/golang/encrypt/unit_test.go @@ -150,7 +150,6 @@ func TestTermsAndLeavesScanAndValue(t *testing.T) { func TestLeafSetKeepsStackEncryptLeavesDistinct(t *testing.T) { ct := map[string]any{ "a": Ciphertext{1}, - "b": otherLeaf{kind: vcffi.LeafNone, bytes: []byte{2}}, "c": otherLeaf{kind: vcffi.LeafEmptySeq, bytes: []byte{3}}, "d": otherLeaf{kind: vcffi.LeafEmptyMap, bytes: []byte{4}}, "p": vcvalue.Plain{V: "clear"}, diff --git a/languages/golang/go.mod b/languages/golang/go.mod index 751d6f869..52569fa25 100644 --- a/languages/golang/go.mod +++ b/languages/golang/go.mod @@ -3,8 +3,8 @@ module github.com/cipherstash/stack/languages/golang go 1.26.0 require ( - github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc - github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc + github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20261015123909-2e895cb9caed + github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20261015123909-2e895cb9caed github.com/tetratelabs/wazero v1.12.0 golang.org/x/oauth2 v0.37.0 ) diff --git a/languages/golang/go.sum b/languages/golang/go.sum index d9d2a09ce..e8211d9d1 100644 --- a/languages/golang/go.sum +++ b/languages/golang/go.sum @@ -1,7 +1,11 @@ github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc h1:iGbTgv9Kx5SaTI2tPxaxwDvS5BXZUO565N00cxRVwC8= github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= +github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20261015123909-2e895cb9caed h1:supKop+UADso55MvBLebaBjP2vyilWmbAxOtFZJOnQ8= +github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20261015123909-2e895cb9caed/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc h1:vlrjoILAURGfpBWucVZEywK22k4lfky1xGDIKV6kqNg= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= +github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20261015123909-2e895cb9caed h1:do5JX9ovxiHzLc9G4wcybbr4f3xaVAYeIKFnOJk9NKw= +github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20261015123909-2e895cb9caed/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index 1cd90257b..f72ea02b3 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -208,9 +208,10 @@ func (w *writer) shapeStruct(name string, fields []shapeField) { } // kindExpr is the gensupport kind of a Go type: the wire type the field -// seals as. The mapping is vcffi's: int8, int16 and int32 travel as int32, -// int and int64 as int64, the unsigned ones likewise, []byte as bytes. A -// composite is untyped and seals as one self-describing value. +// seals as. int8, int16 and int32 seal as int32, int and int64 as int64, the +// unsigned ones likewise, []byte as bytes: vcffi sends a narrower integer at +// its own width, and the engine reads it as the 32-bit integer of its sign. +// A composite is untyped and seals as one self-describing value. func kindExpr(t GoType) string { switch t.Kind { case KindString: From 0f661bc6e9202af41ceffcf02208548b36a4a8a3 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 01:41:22 +1100 Subject: [PATCH 26/38] test(stack-encrypt): pin each cipher half's refusal and a host's u8, and drop the unpinnable capacity hint cargo-mutants found the unused half of MintedKeys and BoundKeys, and the UINT8 arm of FfiValue's sink, untested: no Rust source walks a u8 as an int, so the test reads a host's frame of one. The sink's capacity hint changed nothing a test can see, so it is gone. --- packages/stack-encrypt/src/cipher.rs | 23 +++++++++++++++++++ .../stack-encrypt/src/dynamic/ffi_value.rs | 20 ++++++++-------- 2 files changed, 33 insertions(+), 10 deletions(-) diff --git a/packages/stack-encrypt/src/cipher.rs b/packages/stack-encrypt/src/cipher.rs index 716f7d019..7f94d9664 100644 --- a/packages/stack-encrypt/src/cipher.rs +++ b/packages/stack-encrypt/src/cipher.rs @@ -1549,6 +1549,29 @@ mod tests { } } + /// Each half of the cipher does one thing: a minted key never opens a + /// leaf, and a key bound to a leaf for opening never seals one. + #[test] + fn each_half_of_the_cipher_refuses_the_other_halfs_work() { + let leaf = + SealedValue::from_parts(Uuid::nil(), [0; 16], vec![1; 4], vec![2; 32]).expect("a leaf"); + let mut keys = std::iter::empty::(); + let minted = MintedKeys { + keyset_id: Uuid::nil(), + keys: RefCell::new(&mut keys), + }; + let aad = Context::from_encoded(b"aad"); + assert!(minted.open(leaf, &aad).is_err(), "a minted key never opens"); + assert!( + minted.seal(Protected::new(vec![1]), &aad).is_err(), + "and seals nothing once the batch is spent" + ); + assert!( + BoundKeys.seal(Protected::new(vec![1]), &aad).is_err(), + "a bound key never seals" + ); + } + /// Byte-level pin for the [`leaf_aad`] derivation. This is part of the /// frozen leaf format: a change to the domain label, the version byte, /// the keyset id's place, the piece order, or the PAE framing makes diff --git a/packages/stack-encrypt/src/dynamic/ffi_value.rs b/packages/stack-encrypt/src/dynamic/ffi_value.rs index d3d1795e5..231d5db48 100644 --- a/packages/stack-encrypt/src/dynamic/ffi_value.rs +++ b/packages/stack-encrypt/src/dynamic/ffi_value.rs @@ -290,17 +290,17 @@ impl Sink for ValueSink { Ok(self.0.put(value)) } - fn seq(self, len: Option) -> Result, DecodeError> { + fn seq(self, _: Option) -> Result, DecodeError> { Ok(ValueSeq { out: self.0, - items: Vec::with_capacity(capacity(len)), + items: Vec::new(), }) } - fn map(self, len: Option) -> Result, DecodeError> { + fn map(self, _: Option) -> Result, DecodeError> { Ok(ValueMap { out: self.0, - entries: Vec::with_capacity(capacity(len)), + entries: Vec::new(), }) } @@ -310,12 +310,6 @@ impl Sink for ValueSink { } } -/// A length hint is the source's word, so it only seeds a capacity, and -/// a bounded one. -fn capacity(len: Option) -> usize { - len.unwrap_or(0).min(1024) -} - /// The sequence sink of [`ValueSink`]. pub struct ValueSeq { out: O, @@ -619,6 +613,12 @@ mod tests { read(vitaminc_aead::to_frame(u16::MAX).expect("frames")), FfiValue::UInt32(u16::MAX.into()) ); + // No Rust source walks a `u8` as an int (`Vec` is bytes), but a + // host's frame can carry one: the `UINT8` tag and its byte. + assert_eq!( + FfiValue::from_frame(&[vitaminc_aead::tags::UINT8, 200]).expect("reads back"), + FfiValue::UInt32(200) + ); } /// A frame holding a scalar no variant holds is refused, not read as From f21a23879ca4fe89ad80c2fb5a0649a5503d55a2 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 01:49:16 +1100 Subject: [PATCH 27/38] feat(eql)!: build eql-bindings' stack-encrypt feature on vitaminc ADR 0001 so the EQL jobs and the EQL guest stay green eql-bindings' `stack-encrypt` feature and tests/encryption compile stack-encrypt by path, so they move with it. - The EQL workspace carries the same temporary vitaminc pin as the root, the fuzz crate and both guests: every workspace that compiles stack-encrypt now resolves one vitaminc. - `encryption::targets` and the generated `v3::targets` take `FfiValue` and `ValueKind` from `stack_encrypt::dynamic` (the feature turns on stack-encrypt's `dynamic`); the generator emits the new import. `vitaminc-aead-value` is gone from both manifests. - The `Ciphertext` target takes `Plaintext` in place of `Encrypt` and `Decrypt + Send + 'static` in place of `Decrypt<'static>`. - The encrypt guest's lockfile drops `vitaminc-aead-value`, which only its `eql` build still pulled in. No other EQL code names a vitaminc API. --- languages/golang/encrypt/guest/Cargo.lock | 12 ----- packages/eql/Cargo.lock | 54 ++++++++----------- packages/eql/Cargo.toml | 14 +++++ packages/eql/crates/eql-bindings/CHANGELOG.md | 7 +++ packages/eql/crates/eql-bindings/Cargo.toml | 12 ++--- packages/eql/crates/eql-bindings/README.md | 2 +- .../eql/crates/eql-bindings/src/encryption.rs | 10 ++-- .../eql-bindings/src/encryption/targets.rs | 2 +- .../eql/crates/eql-bindings/src/v3/targets.rs | 2 +- .../eql/crates/eql-codegen/src/targets.rs | 2 +- packages/eql/tests/encryption/Cargo.toml | 9 ++-- .../eql/tests/encryption/tests/targets.rs | 2 +- .../eql/tests/encryption/tests/text_eq.rs | 18 ++++--- 13 files changed, 71 insertions(+), 75 deletions(-) diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 20dc422e4..9d0d082b1 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -709,7 +709,6 @@ dependencies = [ "stack-encrypt", "thiserror 2.0.20", "ts-rs", - "vitaminc-aead-value", "vitaminc-prf", "zeroize", ] @@ -2654,17 +2653,6 @@ dependencies = [ "vitaminc-aead", ] -[[package]] -name = "vitaminc-aead-value" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d78cdadbc6374c3bd7c19fb4d99abca693b96953b18fef8c608efbe092cf3ea" -dependencies = [ - "vitaminc-aead", - "vitaminc-protected", - "zeroize", -] - [[package]] name = "vitaminc-context" version = "0.5.1" diff --git a/packages/eql/Cargo.lock b/packages/eql/Cargo.lock index 1c63b7750..c4451d716 100644 --- a/packages/eql/Cargo.lock +++ b/packages/eql/Cargo.lock @@ -1204,7 +1204,6 @@ dependencies = [ "stack-encrypt", "thiserror 2.0.20", "ts-rs", - "vitaminc-aead-value", "vitaminc-prf", "zeroize", ] @@ -1246,7 +1245,7 @@ dependencies = [ "stack-kms", "tokio", "uuid", - "vitaminc-aead-value", + "vitaminc-aead 0.5.1", ] [[package]] @@ -4240,6 +4239,7 @@ dependencies = [ "thiserror 1.0.69", "uuid", "vitaminc-aead 0.5.1", + "vitaminc-aead-frame", "vitaminc-encrypt 0.5.1", "vitaminc-hmac", "vitaminc-prf", @@ -4696,9 +4696,9 @@ dependencies = [ [[package]] name = "toml_parser" -version = "1.1.2+spec-1.1.0" +version = "1.1.5+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" dependencies = [ "winnow 1.0.3", ] @@ -5031,8 +5031,7 @@ dependencies = [ [[package]] name = "vitaminc" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35042fb7a81d72dc43c53fec835e61027f18b4d01cc0f7e12d6c6a6ffbf3bfb9" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead 0.5.1", "vitaminc-context", @@ -5058,11 +5057,12 @@ dependencies = [ [[package]] name = "vitaminc-aead" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62cacd4dae485e98cfe5407e919187a8468dd933ea4234be5115b7d13a30f3c6" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bytes", + "futures-util", "serde", + "thiserror 2.0.20", "vitaminc-aead-derive", "vitaminc-context", "vitaminc-protected 0.5.1", @@ -5073,8 +5073,7 @@ dependencies = [ [[package]] name = "vitaminc-aead-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ff34056b70a8417ad3bf9db73b2bd3be97f448794566c6de2080bd8a7cd9b6b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -5082,21 +5081,17 @@ dependencies = [ ] [[package]] -name = "vitaminc-aead-value" +name = "vitaminc-aead-frame" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d78cdadbc6374c3bd7c19fb4d99abca693b96953b18fef8c608efbe092cf3ea" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "vitaminc-aead 0.5.1", - "vitaminc-protected 0.5.1", - "zeroize", ] [[package]] name = "vitaminc-context" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7df52501fe33cdb3ee9e155b5242fac184c037d147e58c96e5143ed011252a8" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "vitaminc-protected 0.5.1", @@ -5119,8 +5114,7 @@ dependencies = [ [[package]] name = "vitaminc-encrypt" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fbd0e1748055e381e02033e08af385de879c4dcf32e0f6739ac0c0cf0c6f13d" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "aes-gcm", "aws-lc-rs", @@ -5133,8 +5127,7 @@ dependencies = [ [[package]] name = "vitaminc-hmac" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15fe558a5e92d4f2b3f811c5b67d863d9255da5f66d08a71d4f0dd3d0011c563" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "hmac 0.13.0", "sha2 0.11.0", @@ -5146,8 +5139,7 @@ dependencies = [ [[package]] name = "vitaminc-prf" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712d31d46d76b38c6b62deb35f516fc5636a35f90f39c0213d0ca9a052b4400b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "mutants", "thiserror 2.0.20", @@ -5173,8 +5165,7 @@ dependencies = [ [[package]] name = "vitaminc-protected" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7b514f605a63f88874ba46ffea81775fad64994475ab239cb19da159e884b7b" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "bitvec", "digest 0.11.3", @@ -5201,9 +5192,9 @@ dependencies = [ [[package]] name = "vitaminc-protected-derive" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cb0a3d11afbe2c909f0cfc9996fe37fd89b1196eb0b8d89a66e28716360ee09" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ + "proc-macro-crate", "proc-macro2", "quote", "syn 3.0.3", @@ -5226,8 +5217,7 @@ dependencies = [ [[package]] name = "vitaminc-random" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7b1bfab35ee322de37269c6fd94d8cd94047a66f97cf6722e53e3321482167" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "chacha20", "getrandom 0.4.2", @@ -5252,8 +5242,7 @@ dependencies = [ [[package]] name = "vitaminc-random-derives" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0915e2bfb417ce11e23123fb7ec122af5d27d8cfd22d6ad6a2cf7a1c2dbb4712" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "proc-macro2", "quote", @@ -5279,8 +5268,7 @@ dependencies = [ [[package]] name = "vitaminc-traits" version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b31ec300e351a7a59e64899d0a2d7e7227cfa74ac0514f4e745acb2cc9335cee" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" dependencies = [ "anyhow", "bytes", diff --git a/packages/eql/Cargo.toml b/packages/eql/Cargo.toml index ab1492558..e12e2ec0c 100644 --- a/packages/eql/Cargo.toml +++ b/packages/eql/Cargo.toml @@ -35,3 +35,17 @@ default-members = ["tests/sqlx"] [workspace.lints.rust] dead_code = "deny" unused_imports = "deny" + +# Temporary, as in the repository root's Cargo.toml: eql-bindings' +# `stack-encrypt` feature and tests/encryption compile stack-encrypt by path, +# so the vitaminc crates it builds on come from the branch that implements +# vitaminc ADR 0001 (cipherstash/vitaminc PRs #439 to #445) until vitaminc +# releases it. +[patch.crates-io] +vitaminc = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index 242167c93..c4ea4cf1e 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -35,6 +35,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **The `stack-encrypt` feature builds on vitaminc ADR 0001.** + `encryption::targets` takes `FfiValue` and `ValueKind` from + `stack_encrypt::dynamic` (the feature turns on stack-encrypt's `dynamic` + feature) in place of `vitaminc-aead-value`, which is gone, and the + `Ciphertext` target takes any `Plaintext` plaintext and opens into any + `Decrypt + Send` type, as stack-encrypt's own targets do. + - **Query-operand domain names switched to the `query_` prefix, homed in the `eql_v3` schema.** The SQL domain names carried by the query twins — in `DomainType::sql_domain`, the names `QueryPayload::parse` diff --git a/packages/eql/crates/eql-bindings/Cargo.toml b/packages/eql/crates/eql-bindings/Cargo.toml index 602ff7851..e4b2023b7 100644 --- a/packages/eql/crates/eql-bindings/Cargo.toml +++ b/packages/eql/crates/eql-bindings/Cargo.toml @@ -43,18 +43,15 @@ schemars = "1" # from the PACKAGED crate against the registry stack-encrypt instead: # `cargo publish --dry-run --all-features` in test-eql.yml at PR time, and # `publish_all_features` in ../../release-plz.toml at release time. -stack-encrypt = { path = "../../../stack-encrypt", version = "0.2.0", optional = true, default-features = false } +# `dynamic`: `FfiValue` / `ValueKind`, the runtime plaintext a plan field +# target takes and the kind names a plan's `"type"` key spells +# (`encryption::targets`). +stack-encrypt = { path = "../../../stack-encrypt", version = "0.2.0", optional = true, default-features = false, features = ["dynamic"] } # The vitaminc line stack-encrypt is built on. `PrfValue` is named from this # crate directly (stack-encrypt re-exports only `IntoPrfContext` and # `PrfContext`), and two vitaminc lines in one graph do not interoperate, so # the requirement is stack-encrypt's. vitaminc-prf = { version = "0.5.0", optional = true } -# `FfiValue` / `ValueKind`: the runtime plaintext a plan field target takes -# and the kind names a plan's `"type"` key spells (`encryption::targets`). -# Named from vitaminc directly, like `PrfValue` above, so the feature needs -# nothing from stack-encrypt's own `dynamic` feature; the same line -# stack-encrypt and the Go guest build against. -vitaminc-aead-value = { version = "0.5.1", optional = true } # The plaintext a target reads out of its `Protected` runtime value is held # in `Zeroizing` until it is sealed, so the copy is wiped like the original. zeroize = { version = "1", optional = true } @@ -67,7 +64,6 @@ default = [] stack-encrypt = [ "dep:stack-encrypt", "dep:vitaminc-prf", - "dep:vitaminc-aead-value", "dep:zeroize", "dep:thiserror", "dep:base64", diff --git a/packages/eql/crates/eql-bindings/README.md b/packages/eql/crates/eql-bindings/README.md index 823db57d4..830329531 100644 --- a/packages/eql/crates/eql-bindings/README.md +++ b/packages/eql/crates/eql-bindings/README.md @@ -192,7 +192,7 @@ unknown one with "no such EQL type". ```rust,ignore use eql_bindings::encryption::targets; use stack_encrypt::Label; -use vitaminc_aead_value::FfiValue; +use stack_encrypt::dynamic::FfiValue; let column = Label::new(["users", "email"])?; // the field's context: table/column let stored: Vec = targets::encrypt("TextEq", &keyset, &column, FfiValue::String("alice@example.com".into()))?.await?; diff --git a/packages/eql/crates/eql-bindings/src/encryption.rs b/packages/eql/crates/eql-bindings/src/encryption.rs index fccab8307..356be754e 100644 --- a/packages/eql/crates/eql-bindings/src/encryption.rs +++ b/packages/eql/crates/eql-bindings/src/encryption.rs @@ -62,7 +62,7 @@ //! An encrypted EQL record is not a plaintext value: //! ```compile_fail //! use eql_bindings::v3::text::TextEq; -//! fn require() {} +//! fn require() {} //! require::(); //! ``` //! The text domain accepts text, and requires a concrete column identifier: @@ -85,8 +85,8 @@ use stack_encrypt::target::transcode::{Reader, Transcode, Visitor}; use stack_encrypt::target::{self, AeadContext, CallerContext}; use stack_encrypt::{ CipherText, ContextPiece, Decrypt, DecryptField, DecryptInto, Decryptable, Decryption, - Describe, Description, Encrypt, EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty, - NonEmpty, SealedValue, StackCipherText, + Describe, Description, EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty, NonEmpty, + Plaintext, SealedValue, StackCipherText, }; use vitaminc_prf::PrfValue; @@ -174,7 +174,7 @@ impl Reader for Ciphertext { } } -impl EncryptFrom for Ciphertext { +impl EncryptFrom for Ciphertext { type Context = AeadContext; fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where @@ -195,7 +195,7 @@ impl Visitor for NativeCiphertextVisitor { Ok(CipherText::Single(leaf)) } } -impl + 'static> DecryptInto

for Ciphertext { +impl DecryptInto

for Ciphertext { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { match self.read(NativeCiphertextVisitor) { diff --git a/packages/eql/crates/eql-bindings/src/encryption/targets.rs b/packages/eql/crates/eql-bindings/src/encryption/targets.rs index 06dc034f1..83fded6d9 100644 --- a/packages/eql/crates/eql-bindings/src/encryption/targets.rs +++ b/packages/eql/crates/eql-bindings/src/encryption/targets.rs @@ -79,12 +79,12 @@ use std::fmt; use serde::{de::DeserializeOwned, Serialize}; +use stack_encrypt::dynamic::{FfiValue, ValueKind}; use stack_encrypt::kms::MaybeSend; use stack_encrypt::target::ExpectedContext; use stack_encrypt::{ DecryptInto, EncryptFrom, Error, KeysetCipher, Label, NonEmpty, Pending, StackCipher, }; -use vitaminc_aead_value::{FfiValue, ValueKind}; use crate::v3::targets::{decrypt_named, encrypt_named, query_named, TARGETS}; use crate::Identifier; diff --git a/packages/eql/crates/eql-bindings/src/v3/targets.rs b/packages/eql/crates/eql-bindings/src/v3/targets.rs index e767f3715..18ca472ae 100644 --- a/packages/eql/crates/eql-bindings/src/v3/targets.rs +++ b/packages/eql/crates/eql-bindings/src/v3/targets.rs @@ -4,8 +4,8 @@ use crate::encryption::targets::{ open_target, refuse, refuse_query, run_target, Opener, Target, TargetError, }; use crate::Identifier; +use stack_encrypt::dynamic::FfiValue; use stack_encrypt::{KeysetCipher, NonEmpty, Pending}; -use vitaminc_aead_value::FfiValue; /// Every EQL type a plan may name as a target, in /// `eql-domains::CATALOG` order: one per stored domain (every flat /// scalar domain plus the SteVec document). Query twins are not diff --git a/packages/eql/crates/eql-codegen/src/targets.rs b/packages/eql/crates/eql-codegen/src/targets.rs index 959a71a86..077d4af09 100644 --- a/packages/eql/crates/eql-codegen/src/targets.rs +++ b/packages/eql/crates/eql-codegen/src/targets.rs @@ -270,7 +270,7 @@ fn render_targets_from(rows: &[Row]) -> String { let file = quote! { #![doc = #mod_doc] - use vitaminc_aead_value::FfiValue; + use stack_encrypt::dynamic::FfiValue; use crate::encryption::targets::{#helpers refuse, refuse_query, Opener, Target, TargetError}; use crate::Identifier; diff --git a/packages/eql/tests/encryption/Cargo.toml b/packages/eql/tests/encryption/Cargo.toml index 993b27f7e..41f04494b 100644 --- a/packages/eql/tests/encryption/Cargo.toml +++ b/packages/eql/tests/encryption/Cargo.toml @@ -17,10 +17,11 @@ eql-bindings = { path = "../../crates/eql-bindings", features = ["stack-encrypt" # The parity oracle for the generated target table (tests/targets.rs): the # table must name every stored catalog domain, in catalog order. eql-domains = { path = "../../crates/eql-domains" } -stack-encrypt = { path = "../../../stack-encrypt", default-features = false } -# `FfiValue`: the runtime plaintext the target dispatch takes. The same line -# eql-bindings names. -vitaminc-aead-value = "0.5.1" +# `dynamic`: `FfiValue`, the runtime plaintext the target dispatch takes, +# as eql-bindings names it. +stack-encrypt = { path = "../../../stack-encrypt", default-features = false, features = ["dynamic"] } +# `plaintext::Sink`, for a plaintext type with its own `Plaintext` impl. +vitaminc-aead = "0.5.1" stack-kms = { path = "../../../stack-kms", default-features = false, features = ["test-support"] } serde = "1" serde_json = "1" diff --git a/packages/eql/tests/encryption/tests/targets.rs b/packages/eql/tests/encryption/tests/targets.rs index e5528b43a..8cb9a1e63 100644 --- a/packages/eql/tests/encryption/tests/targets.rs +++ b/packages/eql/tests/encryption/tests/targets.rs @@ -9,8 +9,8 @@ use eql_bindings::encryption::targets::{self as targets, Opener, Target, TargetE use eql_bindings::v3::text::{TextEq, TextEqQuery}; use eql_bindings::{v3, Identifier}; use eql_domains::{Shape, Term, CATALOG}; +use stack_encrypt::dynamic::{FfiValue, ValueKind}; use stack_encrypt::Label; -use vitaminc_aead_value::{FfiValue, ValueKind}; fn email() -> Label { Label::new(["users", "email"]).unwrap() diff --git a/packages/eql/tests/encryption/tests/text_eq.rs b/packages/eql/tests/encryption/tests/text_eq.rs index 86a2a2395..0691d5a57 100644 --- a/packages/eql/tests/encryption/tests/text_eq.rs +++ b/packages/eql/tests/encryption/tests/text_eq.rs @@ -6,7 +6,8 @@ use eql_bindings::v3::terms::{Ciphertext, Hmac256}; use eql_bindings::v3::text::{TextEq, TextEqQuery}; use eql_bindings::{Identifier, SchemaVersion}; use stack_encrypt::target::transcode::{Reader, Transcode}; -use stack_encrypt::{Cipher, Encrypt, IntoAad, NonEmpty}; +use stack_encrypt::{NonEmpty, Plaintext}; +use vitaminc_aead::plaintext::Sink; fn column() -> NonEmpty { Identifier::for_column("users", "email").unwrap() @@ -288,13 +289,14 @@ async fn a_column_batches_generation_and_retrieval_under_the_same_descriptors() #[derive(Clone)] struct WithoutSerde(String); -impl Encrypt for WithoutSerde { - fn encrypt_with_aad<'a, C: Cipher, A: IntoAad<'a>>( - self, - cipher: C, - aad: A, - ) -> Result { - self.0.encrypt_with_aad(cipher, aad) +impl Plaintext for WithoutSerde { + fn visit(self, sink: S) -> Result { + self.0.visit(sink) + } +} +impl Plaintext for &WithoutSerde { + fn visit(self, sink: S) -> Result { + (&self.0).visit(sink) } } #[tokio::test] From 871db7b9bf58a3334cbd856c2c35ae39f65397c3 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 02:28:20 +1100 Subject: [PATCH 28/38] feat(stack-encrypt)!: derive equality and match terms through vitaminc-normalize so text compares in Unicode 16 NFC An equality term is now the PRF over the value's vitaminc-normalize encoding under vitaminc's domain for its kind, read through that crate's Terms sink: one term per number whatever its width, text in Unicode 16 NFC, and no term for a boolean, a null or a container. Match text is tokenized in the same normal form. The equality and match domains move to /v2 and the pins and the record fixture are regenerated; ORE and OPE do not move. vitaminc-normalize is not on crates.io, so it joins the vitaminc patch blocks of every workspace that builds stack-encrypt. Part of #1140 --- Cargo.lock | 72 ++++++ Cargo.toml | 7 +- packages/stack-encrypt/Cargo.toml | 5 + packages/stack-encrypt/fuzz/Cargo.lock | 78 ++++++ packages/stack-encrypt/fuzz/Cargo.toml | 1 + packages/stack-encrypt/src/codes.rs | 11 + packages/stack-encrypt/src/dynamic/value.rs | 231 ------------------ packages/stack-encrypt/src/op/mod.rs | 14 +- packages/stack-encrypt/src/sem/mod.rs | 191 +++++++++++++-- packages/stack-encrypt/src/sem/tokenize.rs | 43 ++++ packages/stack-encrypt/src/target/index.rs | 4 +- .../stack-encrypt/src/target/operations.rs | 7 +- .../tests/fixtures/record_lowering.json | 32 +-- packages/stack-encrypt/tests/frozen_bytes.rs | 2 +- packages/stack-encrypt/tests/op.rs | 43 ++++ packages/stack-encrypt/tests/term_bytes.rs | 46 +++- .../tests/ui/op_target_does_not_answer.stderr | 4 +- .../tests/ui/owned_fan_out_needs_clone.rs | 2 +- .../tests/ui/owned_fan_out_needs_clone.stderr | 4 +- .../tests/ui/pass/owned_without_clone.rs | 2 +- .../tests/ui/pass/query_only_and_borrowed.rs | 7 +- 21 files changed, 507 insertions(+), 299 deletions(-) delete mode 100644 packages/stack-encrypt/src/dynamic/value.rs diff --git a/Cargo.lock b/Cargo.lock index 123d5c85a..4d6050eaa 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -351,6 +351,30 @@ dependencies = [ "zeroize", ] +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.3", +] + [[package]] name = "brotli" version = "8.0.2" @@ -2210,6 +2234,14 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" +[[package]] +name = "orderable-bytes" +version = "0.1.1" +source = "git+https://github.com/cipherstash/ore.rs?rev=b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5#b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5" +dependencies = [ + "rust_decimal", +] + [[package]] name = "parking_lot" version = "0.12.5" @@ -2755,6 +2787,23 @@ dependencies = [ "serde", ] +[[package]] +name = "rust_decimal" +version = "1.43.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a" +dependencies = [ + "arrayvec", + "borsh", + "bytes", + "num-traits", + "rand 0.8.6", + "rand 0.9.3", + "serde", + "serde_json", + "wasm-bindgen", +] + [[package]] name = "rustc-hash" version = "2.1.1" @@ -3302,6 +3351,7 @@ dependencies = [ "vitaminc-aead-frame", "vitaminc-encrypt", "vitaminc-hmac", + "vitaminc-normalize", "vitaminc-prf", "vitaminc-protected", "zeroize", @@ -3949,6 +3999,12 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.23" @@ -4187,6 +4243,20 @@ dependencies = [ "zeroize", ] +[[package]] +name = "vitaminc-normalize" +version = "0.5.1" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" +dependencies = [ + "orderable-bytes", + "rust_decimal", + "thiserror 2.0.18", + "unicode-general-category", + "unicode-normalization", + "vitaminc-aead", + "vitaminc-protected", +] + [[package]] name = "vitaminc-prf" version = "0.5.1" @@ -4194,7 +4264,9 @@ source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove- dependencies = [ "mutants", "thiserror 2.0.18", + "vitaminc-aead", "vitaminc-context", + "vitaminc-normalize", "vitaminc-protected", ] diff --git a/Cargo.toml b/Cargo.toml index 30bb41e57..2b4b5017c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -127,7 +127,11 @@ vitaminc-aead = "0.5.1" vitaminc-aead-frame = "0.5.1" vitaminc-encrypt = "0.5.1" vitaminc-hmac = "0.5.1" -vitaminc-prf = "0.5.1" +# Not on crates.io yet: the `[patch.crates-io]` block below supplies it. +# Normalized equality terms (Unicode 16 NFC text, one integer encoding for +# every width), which `vitaminc-prf`'s `normalize` feature derives over. +vitaminc-normalize = "0.5.1" +vitaminc-prf = { version = "0.5.1", features = ["normalize"] } vitaminc-protected = "0.5.1" # Temporary: every vitaminc crate this workspace uses comes from the branch @@ -142,5 +146,6 @@ vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "fea vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-normalize = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/packages/stack-encrypt/Cargo.toml b/packages/stack-encrypt/Cargo.toml index 0be8a4526..8e483388f 100644 --- a/packages/stack-encrypt/Cargo.toml +++ b/packages/stack-encrypt/Cargo.toml @@ -34,6 +34,11 @@ vitaminc-aead = { workspace = true } vitaminc-aead-frame = { workspace = true, optional = true } vitaminc-encrypt = { workspace = true } vitaminc-hmac = { workspace = true } +# Equality terms derive over `vitaminc-normalize`'s encoding of a value +# (the workspace turns on vitaminc-prf's `normalize` feature): one term per +# number whatever its width, text in Unicode 16 NFC. Match text is +# normalized by the same crate before it is tokenized. +vitaminc-normalize = { workspace = true } vitaminc-prf = { workspace = true } vitaminc-protected = { workspace = true } serde = { workspace = true } diff --git a/packages/stack-encrypt/fuzz/Cargo.lock b/packages/stack-encrypt/fuzz/Cargo.lock index 635728183..d09ae9648 100644 --- a/packages/stack-encrypt/fuzz/Cargo.lock +++ b/packages/stack-encrypt/fuzz/Cargo.lock @@ -234,6 +234,30 @@ dependencies = [ "zeroize", ] +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.3", +] + [[package]] name = "bumpalo" version = "3.19.1" @@ -306,6 +330,12 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + [[package]] name = "chacha20" version = "0.10.0" @@ -1411,6 +1441,14 @@ dependencies = [ "pathdiff", ] +[[package]] +name = "orderable-bytes" +version = "0.1.1" +source = "git+https://github.com/cipherstash/ore.rs?rev=b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5#b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5" +dependencies = [ + "rust_decimal", +] + [[package]] name = "parking_lot" version = "0.12.5" @@ -1776,6 +1814,23 @@ dependencies = [ "serde", ] +[[package]] +name = "rust_decimal" +version = "1.43.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a" +dependencies = [ + "arrayvec", + "borsh", + "bytes", + "num-traits", + "rand 0.8.6", + "rand 0.9.3", + "serde", + "serde_json", + "wasm-bindgen", +] + [[package]] name = "rustc_version" version = "0.4.1" @@ -2045,6 +2100,7 @@ dependencies = [ "vitaminc-aead-frame", "vitaminc-encrypt", "vitaminc-hmac", + "vitaminc-normalize", "vitaminc-prf", "vitaminc-protected", "zeroize", @@ -2366,6 +2422,12 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.23" @@ -2591,6 +2653,20 @@ dependencies = [ "zeroize", ] +[[package]] +name = "vitaminc-normalize" +version = "0.5.1" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" +dependencies = [ + "orderable-bytes", + "rust_decimal", + "thiserror 2.0.18", + "unicode-general-category", + "unicode-normalization", + "vitaminc-aead", + "vitaminc-protected", +] + [[package]] name = "vitaminc-prf" version = "0.5.1" @@ -2598,7 +2674,9 @@ source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove- dependencies = [ "mutants", "thiserror 2.0.18", + "vitaminc-aead", "vitaminc-context", + "vitaminc-normalize", "vitaminc-protected", ] diff --git a/packages/stack-encrypt/fuzz/Cargo.toml b/packages/stack-encrypt/fuzz/Cargo.toml index 8db1c8589..e5badc6b0 100644 --- a/packages/stack-encrypt/fuzz/Cargo.toml +++ b/packages/stack-encrypt/fuzz/Cargo.toml @@ -73,5 +73,6 @@ vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "fea vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-normalize = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index 6ac2e381a..baa3bed3d 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -81,6 +81,10 @@ mod tests { TermError::InvalidOptions(_) => TermError::InvalidOptions("k out of range"), TermError::EmptyTermText => TermError::EmptyTermText, TermError::InvertedRange => TermError::InvertedRange, + TermError::NoEqualityTerm => TermError::NoEqualityTerm, + TermError::Normalize(_) => { + TermError::Normalize(vitaminc_normalize::NormalizeError::UnassignedCodePoint) + }, TermError::Bytes(_) => TermError::Bytes(TermBytesError::OddMatchTermsLength(3)), ])); errors.extend(boxed(variants![ @@ -387,6 +391,13 @@ mod tests { ), (Box::new(TermError::EmptyTermText), json!({})), (Box::new(TermError::InvertedRange), json!({})), + (Box::new(TermError::NoEqualityTerm), json!({})), + ( + Box::new(TermError::Normalize( + vitaminc_normalize::NormalizeError::UnassignedCodePoint, + )), + json!({}), + ), ( Box::new(TermBytesError::MalformedCllwCiphertext(5)), json!({ "len": 5 }), diff --git a/packages/stack-encrypt/src/dynamic/value.rs b/packages/stack-encrypt/src/dynamic/value.rs deleted file mode 100644 index e632d892e..000000000 --- a/packages/stack-encrypt/src/dynamic/value.rs +++ /dev/null @@ -1,231 +0,0 @@ -//! An [`FfiValue`] as a plan field. See [`Value`]. - -use std::fmt; - -use vitaminc_aead::plaintext::Sink; -use vitaminc_aead::{Decrypt, Plaintext}; -use vitaminc_protected::{Controlled, Protected}; - -use super::{FfiValue, Put, ValueSink}; - -use crate::plan::IntoLabel; -use crate::{Label, LabelError}; - -/// A runtime value as the plaintext of a plan field: what every field -/// lowered from data holds, whatever its declared type, and what a Rust -/// chain's field holds when its rows must open from a binding. -/// -/// The engine hands a borrowed field to every operation that consumes it -/// and clones it on the way in, as it does a `String` field of a derived -/// record, so a field's plaintext type is `Clone`. [`FfiValue`] is not: -/// vitaminc keeps its leaves in [`Protected`] and gives no copy out -/// implicitly. This wrapper is the copy made explicit, once, here: its -/// `Clone` rebuilds the tree leaf by leaf into fresh `Protected` payloads, -/// which wipe on drop as the originals do, so a clone is under the same -/// custody as the value it was made from. A record of a few fields clones -/// each once per operation that consumes it, as the derive's does. -/// -/// It seals and opens as the [`FfiValue`] it wraps, in vitaminc's -/// self-describing tagged leaf encoding — the same leaf a bare `u32` or -/// `String` of the same value seals as. Its `Debug` names the type and -/// nothing else: the value is plaintext. -pub struct Value(FfiValue); - -impl Value { - /// Wrap a value. - pub fn new(value: FfiValue) -> Self { - Self(value) - } - - /// The value. - pub fn get(&self) -> &FfiValue { - &self.0 - } - - /// Unwrap the value. - pub fn into_inner(self) -> FfiValue { - self.0 - } -} - -impl From for Value { - fn from(value: FfiValue) -> Self { - Self(value) - } -} - -impl From for FfiValue { - fn from(value: Value) -> Self { - value.0 - } -} - -impl fmt::Debug for Value { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str("Value") - } -} - -impl Clone for Value { - fn clone(&self) -> Self { - Self(duplicate(&self.0)) - } -} - -/// A deep copy of a value, each byte-bearing leaf into a fresh -/// [`Protected`]. Exhaustive, so a variant added to [`FfiValue`] has to say -/// how it is copied. -fn duplicate(value: &FfiValue) -> FfiValue { - match value { - FfiValue::Null => FfiValue::Null, - FfiValue::Undefined => FfiValue::Undefined, - FfiValue::Bool(v) => FfiValue::Bool(*v), - FfiValue::Int32(v) => FfiValue::Int32(*v), - FfiValue::Int64(v) => FfiValue::Int64(*v), - FfiValue::UInt32(v) => FfiValue::UInt32(*v), - FfiValue::UInt64(v) => FfiValue::UInt64(*v), - FfiValue::Float32(v) => FfiValue::Float32(*v), - FfiValue::Float64(v) => FfiValue::Float64(*v), - // Valid UTF-8 by `Utf8String`'s construction invariant. Were it not, - // the copy is a bytes leaf: a different type the plan then refuses, - // never a string that reads differently from its original. - FfiValue::String(s) => match std::str::from_utf8(s.risky_ref()) { - Ok(text) => FfiValue::String(text.into()), - Err(_) => FfiValue::Bytes(Protected::new(s.risky_ref().to_vec())), - }, - FfiValue::Bytes(b) => FfiValue::Bytes(Protected::new(b.risky_ref().to_vec())), - FfiValue::Array(items) => FfiValue::Array(items.iter().map(duplicate).collect()), - FfiValue::Object(entries) => FfiValue::Object( - entries - .iter() - .map(|(key, value)| (key.clone(), duplicate(value))) - .collect(), - ), - FfiValue::Passthrough(inner) => FfiValue::Passthrough(Box::new(duplicate(inner))), - } -} - -impl Plaintext for Value { - fn visit(self, sink: S) -> Result { - self.0.visit(sink) - } -} - -impl Plaintext for &Value { - fn visit(self, sink: S) -> Result { - (&self.0).visit(sink) - } -} - -impl Decrypt for Value { - type Sink = ValueSink; - fn sink() -> Self::Sink { - ValueSink::new(AsValue) - } -} - -/// Where [`Value`]'s [`Decrypt`] sink puts what it builds: the -/// [`FfiValue`] sink's value, wrapped. -pub struct AsValue; - -impl Put for AsValue { - type Ok = Value; - fn put(self, value: FfiValue) -> Value { - Value(value) - } -} - -/// A string value read as a plan's context: what a data plan's -/// `"context_field"` holds (`"tenants/acme"`), parsed as a Rust chain's -/// `String` context field is ([`Label::parse`]). Any other variant is -/// [`LabelError::NotText`], which the plan reports as -/// `PlanError::ContextLabel` before any key is requested. -impl IntoLabel for Value { - fn into_label(self) -> Result { - match self.0 { - FfiValue::String(s) => match std::str::from_utf8(s.risky_ref()) { - Ok(text) => Label::parse(text), - Err(_) => Err(LabelError::NotText), - }, - _ => Err(LabelError::NotText), - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn s(value: &str) -> FfiValue { - FfiValue::String(value.into()) - } - - /// Structural equality, which `FfiValue` offers only inside vitaminc's - /// own tests. - fn same(a: &FfiValue, b: &FfiValue) -> bool { - match (a, b) { - (FfiValue::Null, FfiValue::Null) | (FfiValue::Undefined, FfiValue::Undefined) => true, - (FfiValue::Bool(x), FfiValue::Bool(y)) => x == y, - (FfiValue::Int32(x), FfiValue::Int32(y)) => x == y, - (FfiValue::Int64(x), FfiValue::Int64(y)) => x == y, - (FfiValue::UInt32(x), FfiValue::UInt32(y)) => x == y, - (FfiValue::UInt64(x), FfiValue::UInt64(y)) => x == y, - (FfiValue::Float32(x), FfiValue::Float32(y)) => x.to_bits() == y.to_bits(), - (FfiValue::Float64(x), FfiValue::Float64(y)) => x.to_bits() == y.to_bits(), - (FfiValue::String(x), FfiValue::String(y)) => x.risky_ref() == y.risky_ref(), - (FfiValue::Bytes(x), FfiValue::Bytes(y)) => x.risky_ref() == y.risky_ref(), - (FfiValue::Array(x), FfiValue::Array(y)) => { - x.len() == y.len() && x.iter().zip(y).all(|(a, b)| same(a, b)) - } - (FfiValue::Object(x), FfiValue::Object(y)) => { - x.len() == y.len() - && x.iter() - .zip(y) - .all(|((ka, va), (kb, vb))| ka == kb && same(va, vb)) - } - (FfiValue::Passthrough(x), FfiValue::Passthrough(y)) => same(x, y), - _ => false, - } - } - - /// A clone is the same value, leaf for leaf, at every depth and for - /// every variant, and shares no payload with its original. - #[test] - fn a_clone_is_the_same_value_in_fresh_payloads() { - let original = Value::new(FfiValue::Object(vec![ - ("n".to_string(), FfiValue::Null), - ("u".to_string(), FfiValue::Undefined), - ("b".to_string(), FfiValue::Bool(true)), - ("i32".to_string(), FfiValue::Int32(-3)), - ("i64".to_string(), FfiValue::Int64(-4)), - ("u32".to_string(), FfiValue::UInt32(34)), - ("u64".to_string(), FfiValue::UInt64(35)), - ("f32".to_string(), FfiValue::Float32(1.5)), - ("f64".to_string(), FfiValue::Float64(2.5)), - ("s".to_string(), s("alice")), - ( - "bytes".to_string(), - FfiValue::Bytes(Protected::new(vec![1, 2, 3])), - ), - ( - "list".to_string(), - FfiValue::Array(vec![s("a"), FfiValue::Passthrough(Box::new(s("p")))]), - ), - ])); - let copy = original.clone(); - assert!(same(copy.get(), original.get()), "the same value"); - let (FfiValue::Object(a), FfiValue::Object(b)) = (original.get(), copy.get()) else { - panic!("objects"); - }; - let (FfiValue::String(x), FfiValue::String(y)) = (&a[9].1, &b[9].1) else { - panic!("strings"); - }; - assert_ne!( - x.risky_ref().as_ptr(), - y.risky_ref().as_ptr(), - "the text was copied, not shared" - ); - assert_eq!(format!("{copy:?}"), "Value", "debug shows no plaintext"); - assert!(matches!(copy.into_inner(), FfiValue::Object(_))); - } -} diff --git a/packages/stack-encrypt/src/op/mod.rs b/packages/stack-encrypt/src/op/mod.rs index aa3106539..d71559db0 100644 --- a/packages/stack-encrypt/src/op/mod.rs +++ b/packages/stack-encrypt/src/op/mod.rs @@ -606,7 +606,7 @@ impl QueryToken { } } -impl Supports for Equality { +impl Supports for Equality { type Query = EqualityTerm; type Carry = (); fn check(&self, _: &S) -> Result<(), Error> { @@ -641,13 +641,17 @@ where // A query with no tokens would match every row. The stored side // keeps such a value as the empty filter; the query side refuses it // here, before any term is derived. - if QueryTokens::of::(value.as_ref()).is_empty() { + // Text the index cannot normalize is refused here too. + if QueryTokens::of::(value.as_ref())?.is_empty() { return Err(TermError::EmptyTermText.into()); } Ok(()) } fn carry(&self, value: &S) -> QueryTokens { - QueryTokens::of::(value.as_ref()) + // `check` has refused text the index cannot normalize, so this is + // the query's tokens; a query that got here otherwise verifies + // nothing. + QueryTokens::of::(value.as_ref()).unwrap_or_default() } fn query(tokens: QueryTokens, term: MatchTerms, _: QueryToken) -> MatchQuery { MatchQuery::new(term, tokens) @@ -734,7 +738,7 @@ pub trait Answers> { type Via: Supports + Default; } -impl Answers for EqualityTerm { +impl Answers for EqualityTerm { type Via = Equality; } @@ -782,7 +786,7 @@ pub trait DefaultIndex>: Operator { type Via: Supports + Default; } -impl DefaultIndex for Eq { +impl DefaultIndex for Eq { type Via = Equality; } diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 83a97578f..47a70e37c 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -138,7 +138,9 @@ use std::marker::PhantomData; // wrapper over the term APIs has to be able to name them without depending // on `cllw-ore` directly. pub use cllw_ore::{CllwOpeEncrypt, CllwOreEncrypt}; +use vitaminc_aead::Plaintext; use vitaminc_hmac::HmacSha256Prf; +use vitaminc_normalize::{terms, NormalizeError, Normalized}; use vitaminc_prf::{ BlockVisitor, Context, IntoPrfContext, MapAccess, PrfError, PrfValue, PrfVisitor, PrfVisitorError, SeqAccess, @@ -152,16 +154,22 @@ use crate::target::core::Term; use crate::target::{DecryptField, Decryptable, Decryption}; use crate::{Error, KeysetCipher, Pending}; -// The `/v1` suffix versions the *derivation* (domain + input framing), not the -// crate. Any change to the bytes a term derives from must bump it: a changed +// The version suffix (`/v1`, `/v2`) versions the *derivation* (domain + +// input framing), not the crate. Any change to the bytes a term derives from must bump it: a changed // derivation under an unchanged domain makes every existing term silently // unfindable, with no error to notice. The byte-level pins in // `tests/term_bytes.rs` are what force that bump to be deliberate. /// PAE domain for equality (exact-match) terms. -const EQUALITY_DOMAIN: &[u8] = b"stack-encrypt/sem/equality/v1"; +/// +/// `v2`: the PRF input is the value's `vitaminc-normalize` encoding under +/// its domain, where `v1` was the value's own PRF encoding. +const EQUALITY_DOMAIN: &[u8] = b"stack-encrypt/sem/equality/v2"; /// PAE domain for match (full-text) token terms. -const MATCH_DOMAIN: &[u8] = b"stack-encrypt/sem/match/v1"; +/// +/// `v2`: the text is tokenized in Unicode 16 NFC, where `v1` tokenized it as +/// written. +const MATCH_DOMAIN: &[u8] = b"stack-encrypt/sem/match/v2"; /// PAE domain for ORE key derivation. const ORE_KEY_DOMAIN: &[u8] = b"stack-encrypt/sem/ore-key/v1"; /// PAE domain for OPE key derivation (distinct from ORE: OPE ciphertexts are @@ -214,6 +222,26 @@ pub enum TermError { help("Give `op::Between` its bounds as `(lo, hi)` with `lo <= hi`. The range is inclusive at both ends, so one value is `(v, v)`.") )] InvertedRange, + /// The value is not one scalar an equality term is defined for: a + /// null, a boolean, a container, or a value of a logical type vitaminc + /// gives no equality domain. Equality derives over one scalar, in the + /// domain `vitaminc-normalize` gives its kind (every integer width one + /// domain, both float widths one, text, bytes, and the dates, + /// timestamps, decimals and uuids that opt in); there is no term for + /// anything else. + #[error("no equality term is defined for this value")] + #[diagnostic( + code(stack_encrypt::no_equality_term), + help("An equality term is defined for one integer, float, text, bytes, date, timestamp, decimal or uuid value, not for a boolean, a null or a container.") + )] + NoEqualityTerm, + /// The value cannot be normalized for its term: text with a code point + /// unassigned in Unicode 16 (an equality or match term), or a value not + /// valid for its logical type. The reason is vitaminc's, and names no + /// part of the value. + #[error("the value cannot be normalized for its term: {0}")] + #[diagnostic(code(stack_encrypt::term_normalize))] + Normalize(#[from] NormalizeError), /// Term bytes do not decode under the term kind's frozen encoding — see /// [`TermBytesError`]. #[error(transparent)] @@ -229,7 +257,9 @@ impl crate::ErrorPayload for TermError { | Self::Ore(_) | Self::InvalidOptions(_) | Self::EmptyTermText - | Self::InvertedRange => serde_json::Map::new(), + | Self::InvertedRange + | Self::NoEqualityTerm + | Self::Normalize(_) => serde_json::Map::new(), } } } @@ -311,7 +341,8 @@ impl TermError { // Equality // ============================================================================= -/// An equality (exact-match) index term: one PRF block over the whole value. +/// An equality (exact-match) index term: one PRF block over the value's +/// normalized bytes (see [`KeysetCipher::equality_term`]). /// /// Terms are pseudorandom under the index key; they are stored server-side and /// are not secret key material. @@ -385,27 +416,68 @@ impl PrfVisitor<[u8; 32], P> for EqualityVisitor { /// deferred backend the same visitor runs after the round-trip instead). /// `context` is the encoding of a [`NonEmpty`] — every caller holds one — /// so there is nothing left to validate here. +/// +/// The value is normalized first, by `vitaminc-normalize`'s [`terms`] sink: +/// it must be one scalar with an equality domain, and the PRF derives over +/// that scalar's [`Normalized`] bytes under its domain. So every integer +/// width of one number is one term, both float widths of one number are +/// one term, and text is compared in Unicode 16 NFC: `café` written with a +/// precomposed `é` and with `e` and a combining accent is one term. fn equality( prf: &HmacSha256Prf, value: T, context: Context<'_>, ) -> Result where - T: PrfValue, + T: Plaintext, { + equality_of_normalized(prf, normalized(value)?, context) +} + +/// The equality term of a scalar already normalized: the PRF over its +/// bytes, under its `vitaminc-normalize` domain and the PAE-framed +/// `[EQUALITY_DOMAIN, context]`. [`equality`] and the `dynamic` path, which +/// normalizes the one scalar it walks to, both end here. +fn equality_of_normalized( + prf: &HmacSha256Prf, + normalized: Normalized, + context: Context<'_>, +) -> Result { let context = Context::pae(&[EQUALITY_DOMAIN, context.as_bytes()]); - value + normalized .prf_visit_with_context(prf, context, EqualityVisitor) .into_result() .map_err(TermError::from_prf) } -/// An equality term of any [`PrfValue`] source. Under the local HMAC -/// backend, derived during the synchronous build — the returned [`Pending`] -/// carries no requests. +/// The one [`Normalized`] scalar of a value, as `vitaminc-normalize`'s +/// [`terms`] sink derives it: [`TermError::NoEqualityTerm`] unless the value +/// is one scalar with an equality domain, and [`TermError::Normalize`] for a +/// scalar the sink refuses (text with a code point unassigned in Unicode 16, +/// a value not valid for its logical type). +fn normalized(value: T) -> Result { + let mut terms = terms(value).map_err(|error| match error.term() { + Some(refused) => TermError::Normalize(refused.error()), + // A shape refusal or a source failure: not one scalar the sink could + // read, so there is no term. + None => TermError::NoEqualityTerm, + })?; + match (terms.pop(), terms.is_empty()) { + // One term, and at the root: the value is that scalar. A container + // with one indexable scalar inside is not a scalar, and a null or a + // boolean produces no term at all. + (Some(term), true) if term.path.steps().is_empty() => Ok(term.normalized), + _ => Err(TermError::NoEqualityTerm), + } +} + +/// An equality term of any [`Plaintext`] source that is one scalar with an +/// equality domain (see [`equality`]). Under the local HMAC backend, +/// derived during the synchronous build — the returned [`Pending`] carries +/// no requests. impl<'c, S, K, T> Term> for EqualityTerm where - S: PrfValue, + S: Plaintext, T: IntoPrfContext<'c>, { // Derived locally: no data key, no descriptor. @@ -727,11 +799,11 @@ impl crate::op::Probabilistic for MatchQuery { fn verify(&self, value: &str) -> bool { let options = O::options(); - let tokens = Zeroizing::new(tokenize::tokenize( - value, - options.tokenizer, - options.downcase, - )); + // A value the index cannot normalize holds no token of any query. + let Ok(tokens) = tokenize::tokens(value, options.tokenizer, options.downcase) else { + return false; + }; + let tokens = Zeroizing::new(tokens); let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); !self.tokens.0.is_empty() && self.tokens.0.iter().all(|t| have.contains(t.as_str())) } @@ -835,22 +907,31 @@ impl RangeQuery { /// ([`Supports::query`](crate::op::Supports::query) takes a token no caller /// can make), so one query's tokens never verify rows for another value's /// filter. -#[derive(Clone)] +#[derive(Clone, Default)] pub struct QueryTokens(Zeroizing>); impl QueryTokens { /// The tokens of `text` under `O`'s tokenizer and case folding. - pub(crate) fn of(text: &str) -> Self { + /// + /// # Errors + /// + /// [`TermError::Normalize`] for text the index cannot normalize (a code + /// point unassigned in Unicode 16). + pub(crate) fn of(text: &str) -> Result { Self::under(text, &O::options()) } /// The tokens of `text` under `options`' tokenizer and case folding. - pub(crate) fn under(text: &str, options: &MatchOptions) -> Self { - Self(Zeroizing::new(tokenize::tokenize( + /// + /// # Errors + /// + /// As for [`of`](Self::of). + pub(crate) fn under(text: &str, options: &MatchOptions) -> Result { + Ok(Self(Zeroizing::new(tokenize::tokens( text, options.tokenizer, options.downcase, - ))) + )?))) } /// Whether the text cut no token at all: the one query the library @@ -923,7 +1004,7 @@ fn match_term( options: MatchOptions, ) -> Result, TermError> { let mask = options.validate()?; - let tokens = tokenize::tokenize(text, options.tokenizer, options.downcase); + let tokens = tokenize::tokens(text, options.tokenizer, options.downcase)?; if tokens.is_empty() { return Ok(MatchTerms::normalised(Vec::new())); } @@ -1327,13 +1408,22 @@ impl KeysetCipher<'_, K> { /// the same term, at write time and at query time. Byte-identical to /// `value.encrypt_into_with_context(&keyset, descriptor)` into an /// `EqualityTerm`. + /// + /// The value is one scalar, normalized by `vitaminc-normalize` before the + /// PRF sees it: every integer width of one number is one term + /// (`34u32` and `34i64` match), both float widths of one number are one + /// term, text is compared in Unicode 16 NFC, and a date, timestamp, + /// decimal or uuid derives in a domain of its own. A boolean, a null or + /// a container has no term ([`TermError::NoEqualityTerm`]), and text with + /// a code point unassigned in Unicode 16 is refused + /// ([`TermError::Normalize`]). pub fn equality_term<'c, T>( &self, value: T, descriptor: NonEmpty>, ) -> Pending<'_, EqualityTerm, K> where - T: PrfValue, + T: Plaintext, { let term = equality(self.prf(), value, descriptor.into_prf_context()); Pending::ready(self, term.map_err(Error::from)) @@ -1370,6 +1460,20 @@ impl KeysetCipher<'_, K> { self.match_terms_under(text, descriptor, O::options()) } + /// [`equality_term`](Self::equality_term) of one scalar the caller has + /// normalized already, for the `dynamic` path, which walks a value to + /// its one scalar and normalizes it there, with the same + /// `vitaminc_normalize::normalize` the typed path's sink applies. + #[cfg_attr(not(feature = "dynamic"), allow(dead_code))] + pub(crate) fn equality_term_normalized<'c>( + &self, + normalized: Normalized, + descriptor: NonEmpty>, + ) -> Pending<'_, EqualityTerm, K> { + let term = equality_of_normalized(self.prf(), normalized, descriptor.into_prf_context()); + Pending::ready(self, term.map_err(Error::from)) + } + /// [`match_terms`](Self::match_terms) under runtime options, for the /// `dynamic` path, whose plan carries a match index's options as data /// (an `IndexSpec::Match`). Not public: the typed API keeps the options @@ -1454,6 +1558,45 @@ mod tests { assert!(!crate::op::Probabilistic::verify(&query, "")); } + /// An equality term is one scalar's: a container, even one holding one + /// scalar, and a scalar with no equality domain have none, and text with + /// no normal form is refused for that reason. + #[test] + fn equality_is_defined_for_one_scalar_with_a_domain() { + let prf = HmacSha256Prf::new(Protected::new([7; 32])); + let derive = |value| equality(&prf, value, Context::empty()); + assert!(equality(&prf, 34u32, Context::empty()).is_ok()); + for (what, refused) in [ + ("a boolean", equality(&prf, true, Context::empty())), + ("a null", equality(&prf, None::, Context::empty())), + ( + "a sequence of one", + equality(&prf, vec![34u32], Context::empty()), + ), + ( + "a map of one", + equality(&prf, BTreeMap::from([("a", 34u32)]), Context::empty()), + ), + ( + "a map of two", + equality( + &prf, + BTreeMap::from([("a", 34u32), ("b", 35u32)]), + Context::empty(), + ), + ), + ] { + assert!( + matches!(refused, Err(TermError::NoEqualityTerm)), + "{what}: {refused:?}" + ); + } + assert!(matches!( + derive("a\u{378}b"), + Err(TermError::Normalize(NormalizeError::UnassignedCodePoint)) + )); + } + /// The Bloom fold reads a token *stream*: handed a map-shaped PRF input /// it refuses, rather than folding the entries' blocks as if they were /// tokens. diff --git a/packages/stack-encrypt/src/sem/tokenize.rs b/packages/stack-encrypt/src/sem/tokenize.rs index 735a218a5..92085bbe4 100644 --- a/packages/stack-encrypt/src/sem/tokenize.rs +++ b/packages/stack-encrypt/src/sem/tokenize.rs @@ -10,6 +10,8 @@ //! is that empty tokens are dropped (v1 keeps the empty strings its separator //! split produces, which only add noise bits to every filter). +use vitaminc_normalize::NormalizeError; +use vitaminc_protected::Controlled; use zeroize::Zeroizing; /// How text is split into tokens before each token is run through the PRF. @@ -40,6 +42,28 @@ impl Default for Tokenizer { } } +/// Split `text` into tokens as a match index does: in Unicode 16 NFC, the +/// form `vitaminc-normalize` gives text, then [`tokenize`]d. So text written +/// with a precomposed `é` and with `e` and a combining accent cuts the same +/// tokens, as it derives one equality term. Text holding a code point +/// unassigned in Unicode 16 is refused, as equality refuses it: its NFC +/// form could change when the code point is assigned. +/// +/// The normalized copy is held in [`Protected`](vitaminc_protected::Protected) +/// and wiped on drop. +pub(crate) fn tokens( + text: &str, + tokenizer: Tokenizer, + downcase: bool, +) -> Result, NormalizeError> { + let normalized = vitaminc_normalize::text(text)?; + // `text` returns the UTF-8 of a `str`, so this does not fail; a failure + // would be a bug there, refused rather than tokenized. + let normalized = + std::str::from_utf8(normalized.risky_ref()).map_err(|_| NormalizeError::Invalid)?; + Ok(tokenize(normalized, tokenizer, downcase)) +} + /// Split `text` into tokens. `downcase` lower-cases the text first so matches /// are case-insensitive. /// @@ -94,6 +118,25 @@ fn gram(chars: &[char]) -> String { mod tests { use super::*; + #[test] + fn tokens_are_cut_from_the_nfc_form() { + let standard = |text| tokens(text, Tokenizer::Standard, false).expect("tokens"); + assert_eq!( + standard("cafe\u{301} au lait"), + standard("caf\u{e9} au lait"), + "a combining accent and the precomposed letter cut the same tokens" + ); + assert_eq!(standard("caf\u{e9}"), vec!["caf\u{e9}"], "the token is NFC"); + } + + #[test] + fn tokens_refuse_a_code_point_unassigned_in_unicode_16() { + assert_eq!( + tokens("a\u{378}b", Tokenizer::Standard, true), + Err(NormalizeError::UnassignedCodePoint) + ); + } + #[test] fn ngram_tokenizes_sliding_windows() { assert_eq!( diff --git a/packages/stack-encrypt/src/target/index.rs b/packages/stack-encrypt/src/target/index.rs index 63db8d400..28ed57269 100644 --- a/packages/stack-encrypt/src/target/index.rs +++ b/packages/stack-encrypt/src/target/index.rs @@ -203,7 +203,7 @@ impl fmt::Debug for Match { } } -impl Index for Equality { +impl Index for Equality { type Term = EqualityTerm; fn spec(&self) -> IndexSpec { IndexSpec::Equality @@ -553,7 +553,7 @@ pub trait TermSet: Sized + 'static { /// Those indexes, as a value. const INDEXES: Self::Indexes; } -impl TermSet for EqualityTerm { +impl TermSet for EqualityTerm { type Indexes = Equality; const INDEXES: Equality = Equality; } diff --git a/packages/stack-encrypt/src/target/operations.rs b/packages/stack-encrypt/src/target/operations.rs index eae1d175c..6e668091a 100644 --- a/packages/stack-encrypt/src/target/operations.rs +++ b/packages/stack-encrypt/src/target/operations.rs @@ -535,8 +535,9 @@ macro_rules! term_operation { } term_operation!( /// The equality term of `S` under the context the tree hands it. Requires - /// only `S`'s PRF contract, not recoverable encryption. - equality, crate::sem::EqualityTerm, consume, [], [S: vitaminc_prf::PrfValue] + /// only that `S` describes itself (`Plaintext`), as one scalar with an + /// equality domain; not recoverable encryption. + equality, crate::sem::EqualityTerm, consume, [], [S: crate::Plaintext] ); term_operation!( /// The match term of any text `S` under the context the tree hands it, @@ -893,7 +894,7 @@ impl DecryptInto

for StackCipherText { open(self, context) } } -impl EncryptFrom for crate::sem::EqualityTerm { +impl EncryptFrom for crate::sem::EqualityTerm { type Context = CallerContext; fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where diff --git a/packages/stack-encrypt/tests/fixtures/record_lowering.json b/packages/stack-encrypt/tests/fixtures/record_lowering.json index b0744958c..868b695f7 100644 --- a/packages/stack-encrypt/tests/fixtures/record_lowering.json +++ b/packages/stack-encrypt/tests/fixtures/record_lowering.json @@ -47,8 +47,8 @@ "passthrough": "tenants/acme" }, "text": { - "c": "0100000000000000000000000000000000cf1a45d1517a08c45d6ae1c989ae99de2000da620d1dbdbba9372d1439b6d44f7cc4169e9a255387aa180d6a95fb17926d70027eff9bae3dcbbf74666b87b43afcdeeb5d90fe4c4e316142e647677cb9597d6cf9c0", - "eq": "0d84fbd44d82b5150351556d6d25a038c70cf0c696e2499a71b659a7732a9d08" + "c": "0100000000000000000000000000000000cf1a45d1517a08c45d6ae1c989ae99de2000da620d1dbdbba9372d1439b6d44f7cc4169e9a255387aa180d6a95fb17926d7002bc0ceebce867d0c4942a2cf544f1195a02327f01117b05892d35d19348b1018e2dc2", + "eq": "d07ccccb55205dbb5c69c80bd88c907496a18bbed9317f39d96a83f6fd308a85" } }, "typed_chain": { @@ -59,8 +59,8 @@ "passthrough": "tenants/acme" }, "text": { - "c": "010000000000000000000000000000000048a99626f697c02001f07da2edb19f4220009868b833b6220fa0d13b612af03af1f38fee082c7555b6e7fbf8e13ccb70f80702c1591a387d3f6fd6c83d4b547b857ec46a3ef86919ace434e5ca69581ce699cf0765", - "eq": "0d84fbd44d82b5150351556d6d25a038c70cf0c696e2499a71b659a7732a9d08" + "c": "010000000000000000000000000000000048a99626f697c02001f07da2edb19f4220009868b833b6220fa0d13b612af03af1f38fee082c7555b6e7fbf8e13ccb70f807027e9874c89aa2b6dc1f7a073c6c41e7679b8380880c5daa0aff707d91b85142f688c2", + "eq": "d07ccccb55205dbb5c69c80bd88c907496a18bbed9317f39d96a83f6fd308a85" } } } @@ -125,38 +125,38 @@ "records": { "lowering": { "age": { - "c": "01000000000000000000000000000000004d5626148f77c4c6d9da0ddbdf7663a2200021162641133485216f389886cbd7c6e9b25afa0f5f166f985b23384e81ffcfac02b8821c190a544161b6ac13825768cd6b494515bdd96c26715ffecd7321277d13d6", - "eq": "5433335e609685a33b29c98369e68cd5d92e838aa566a3ff1df5c2d251b63885", + "c": "01000000000000000000000000000000004d5626148f77c4c6d9da0ddbdf7663a2200021162641133485216f389886cbd7c6e9b25afa0f5f166f985b23384e81ffcfac0228dd18e6432418924ee25f51a3859d9a22895f62ee52705e57ea1491ef1d88a5bf", + "eq": "cc854fd5c7e3f4bb28cb15cae47d866b65b7012c06df43b4b9a6c51adff4710a", "ore": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a605cbf3f1" }, "email": { - "c": "0100000000000000000000000000000000b2afa77357a61488146c0edaa0246f65200043956c4f4bba857028c28165dfce9c63d904b738bbb039b6e11c805112d4a1ee02d3ce6ecc6cc31f82d07a829142cf39b2fcd9a742f89627f316622416f49df065d381337c5da03a8f3ab5b84d", - "eq": "ec534cfbf2336785378c3761ff88623c46361b2ba7c5fa83930ebacae49cf646", - "match": "06000d0013001d002200260029002a003000320043005a006000660068006c0077008f00960098009e00c200c500d600d900de00e300e700e900eb00ec00f200fa00" + "c": "0100000000000000000000000000000000b2afa77357a61488146c0edaa0246f65200043956c4f4bba857028c28165dfce9c63d904b738bbb039b6e11c805112d4a1ee028903df249ec7db270d3e5f25a72060fd12bc5174ed5776c9477df43aa7dc6421105f6090f8fd54e0287081c1", + "eq": "0cf708c07acaaa169e84e4adf7081f0433d87760bd7e9c63bd7a58c000b8d33b", + "match": "030005000900100012001a001c0027002a002e003000340042004b0068006d007100750083009200a800b200b400b700ba00c200c500dc00e500ea00f200f300f900fe00" }, "id": { "passthrough": 42 }, "notes": { - "c": "0100000000000000000000000000000000723f7a7f14b07197168aa13bc52010b3200072a73194b45af604cc6201b5a57554b05cc7ed6ad22b9102cbcb71bd90678496020f72b5054a1a4467aa441215b4af28502c55a2590aae2a844ea40392f5fd64e679614a072b8736" + "c": "0100000000000000000000000000000000723f7a7f14b07197168aa13bc52010b3200072a73194b45af604cc6201b5a57554b05cc7ed6ad22b9102cbcb71bd90678496025763fabba65839aab92f515ccb053a4bfbea672c516be6a6c51f97e00cd0071ab8e7b337fd8055" } }, "typed_chain": { "age": { - "c": "010000000000000000000000000000000092e427c5e176b60261add6ce80fd02df20008d231453c97fdec97a0b6559ac8f509129d20163f262d40ce5d77cf1ebe0cad502c51f3fa24db5e34fadc3553b401d674928b00126a41cb7f30da889259f472223e8", - "eq": "5433335e609685a33b29c98369e68cd5d92e838aa566a3ff1df5c2d251b63885", + "c": "010000000000000000000000000000000092e427c5e176b60261add6ce80fd02df20008d231453c97fdec97a0b6559ac8f509129d20163f262d40ce5d77cf1ebe0cad502f71ddc78cf13e463d4af42f9f837778bf5a0793bb6794a2f3fa406a8d3aa20e772", + "eq": "cc854fd5c7e3f4bb28cb15cae47d866b65b7012c06df43b4b9a6c51adff4710a", "ore": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a605cbf3f1" }, "email": { - "c": "0100000000000000000000000000000000cf92191950409f9fe82a2b11888ffc1d2000ca4d5bbafbff919af4d3a95c7cc046d632fd4c28785b238f4b6f7ea3aa08482902a44da283fdefa835c68377a191ccf172943720b07e7779cc903619869844eb9cfed80a8a7f9034e69385ddb1", - "eq": "ec534cfbf2336785378c3761ff88623c46361b2ba7c5fa83930ebacae49cf646", - "match": "06000d0013001d002200260029002a003000320043005a006000660068006c0077008f00960098009e00c200c500d600d900de00e300e700e900eb00ec00f200fa00" + "c": "0100000000000000000000000000000000cf92191950409f9fe82a2b11888ffc1d2000ca4d5bbafbff919af4d3a95c7cc046d632fd4c28785b238f4b6f7ea3aa08482902053217eb2c4844e6700106487ec76d10b0052c05089848a1e18a003a8fc94a4b0b9b90db118ddf73eba753d2", + "eq": "0cf708c07acaaa169e84e4adf7081f0433d87760bd7e9c63bd7a58c000b8d33b", + "match": "030005000900100012001a001c0027002a002e003000340042004b0068006d007100750083009200a800b200b400b700ba00c200c500dc00e500ea00f200f300f900fe00" }, "id": { "passthrough": 42 }, "notes": { - "c": "0100000000000000000000000000000000562a41e032316ff2d1264a9e9f5c3b842000de33c8d1f2caaab8b90f691161a57eeb1328f7f1c9b5be76dec83e8a53765f1102c76bf0a61684b320c316ad94b01eb76bb52b3c8493f21a73012de6289580bf355b7192c61500e0" + "c": "0100000000000000000000000000000000562a41e032316ff2d1264a9e9f5c3b842000de33c8d1f2caaab8b90f691161a57eeb1328f7f1c9b5be76dec83e8a53765f1102bfdd18635fde4d7d775297d5e47c5b9a3ade28f51212c0bc5a2b64925f0225c006685f8db07cee" } } } diff --git a/packages/stack-encrypt/tests/frozen_bytes.rs b/packages/stack-encrypt/tests/frozen_bytes.rs index f930043a0..12efc2303 100644 --- a/packages/stack-encrypt/tests/frozen_bytes.rs +++ b/packages/stack-encrypt/tests/frozen_bytes.rs @@ -378,7 +378,7 @@ async fn match_term_bytes_are_pinned() { let bytes = term.to_bytes(); assert_eq!( hex(&bytes), - "040005000a000d000e001e002700350038003d0055005e005f0064006f007d007f009c00ad00bc00bd00ca00d000e000e500ef00" + "06000f001c0028003300360037003d0040005c0073009400970099009b00a100a300ac00b100b300ba00bb00c500d800f600fb00fe00" ); assert_eq!( MatchTerms::::from_bytes(&bytes).expect("decode match term"), diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 6d99f4bb4..b58cd7fbd 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -531,6 +531,49 @@ async fn verify_keeps_every_real_match_and_folds_case_as_the_index_does() { ); } +/// A match query is cut in Unicode 16 NFC, as the stored value is, so a +/// query and a row that spell one string in its two normal forms match and +/// verify. Text holding a code point unassigned in Unicode 16 has no normal +/// form: the query is refused, and such a row verifies against nothing. +#[tokio::test] +async fn a_match_query_is_cut_in_nfc_and_refuses_text_with_no_normal_form() { + let cipher = stack_cipher().await; + let plan = people_plan(); + let email_plan = plan.field("email").unwrap(); + + let query = cipher + .query(op::Matches, "Rene\u{301}e") + .on(&email_plan) + .await + .unwrap(); + let precomposed = cipher + .query(op::Matches, "Ren\u{e9}e") + .on(&email_plan) + .await + .unwrap(); + assert_eq!(query, precomposed, "one string, one filter"); + assert!(query.verify("Ren\u{e9}e Martin")); + assert!(precomposed.verify("Rene\u{301}e Martin")); + assert!( + !query.verify("Ren\u{e9}e \u{378}"), + "a value with no normal form verifies against nothing" + ); + + let refused = cipher + .query(op::Matches, "Ren\u{378}e") + .on(&email_plan) + .await; + assert!( + matches!( + refused, + Err(Error::Term(TermError::Normalize( + vitaminc_normalize::NormalizeError::UnassignedCodePoint + ))) + ), + "{refused:?}" + ); +} + #[tokio::test] async fn verify_drops_a_row_the_filter_matched_by_chance() { let cipher = stack_cipher().await; diff --git a/packages/stack-encrypt/tests/term_bytes.rs b/packages/stack-encrypt/tests/term_bytes.rs index ebe5f04cb..164eb66af 100644 --- a/packages/stack-encrypt/tests/term_bytes.rs +++ b/packages/stack-encrypt/tests/term_bytes.rs @@ -5,7 +5,7 @@ //! them changes every stored term — and because terms are compared for //! equality server-side, the failure mode is not an error but a query that //! silently stops matching. Breaking one of these tests means the derivation -//! moved, and the `/v1` suffix in the domain labels has to move with it. +//! moved, and the version suffix in the domain labels has to move with it. //! //! Keyed by `FakeDataKeySource`'s deterministic index key, so the expected //! bytes are stable without ZeroKMS. @@ -16,6 +16,12 @@ //! labels and framing here did not. That was a prerelease wire break, taken //! deliberately (CIP-4036); the `/v1` suffixes stayed because nothing of //! this crate's own moved. +//! +//! The equality and match pins moved again with the labels, to `/v2`, when +//! both began deriving through `vitaminc-normalize`: an equality term is the +//! PRF over the value's normalized bytes under vitaminc's domain for its +//! kind, and match text is tokenized in Unicode 16 NFC. No stored term +//! existed to keep. The ORE and OPE derivations did not move. use stack_encrypt::nonempty; use stack_encrypt::sem::{DefaultMatch, MatchConfig, MatchOptions}; @@ -45,7 +51,33 @@ async fn equality_term_bytes_are_pinned() { assert_eq!( hex(term.as_bytes()), - "c101ef066547cb33003c352dcd02a42bfe4a24e2d0609caca0189d9f1af8262a" + "0aa0df4adabbad408580770b6e156aecca21a1d45f84e554ad4a1a0f76006fef" + ); +} + +/// Equality compares text in Unicode 16 NFC and numbers whatever their +/// width: two spellings of one string, and one number at two widths, are +/// one term each. +#[tokio::test] +async fn equality_terms_are_one_per_normal_form() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let term = |value: &'static str| keyset.equality_term(value, nonempty!("users/email")); + assert_eq!( + term("cafe\u{301}").await.unwrap(), + term("caf\u{e9}").await.unwrap(), + "a combining accent and the precomposed letter are one string" + ); + assert_eq!( + keyset + .equality_term(34u32, nonempty!("users/age")) + .await + .unwrap(), + keyset + .equality_term(34i64, nonempty!("users/age")) + .await + .unwrap(), + "34 is one number at every width" ); } @@ -63,8 +95,8 @@ async fn match_term_positions_are_pinned() { assert_eq!( term.positions(), [ - 4, 5, 10, 13, 14, 30, 39, 53, 56, 61, 85, 94, 95, 100, 111, 125, 127, 156, 173, 188, - 189, 202, 208, 224, 229, 239 + 6, 15, 28, 40, 51, 54, 55, 61, 64, 92, 115, 148, 151, 153, 155, 161, 163, 172, 177, + 179, 186, 187, 197, 216, 246, 251, 254 ] ); } @@ -130,9 +162,9 @@ async fn match_term_positions_are_pinned_for_a_wide_filter() { assert_eq!( term.positions(), [ - 2287, 2398, 2404, 2829, 5150, 5181, 5293, 9255, 11964, 14175, 16080, 24350, 25354, - 28362, 31748, 33647, 35845, 39141, 39480, 41998, 42621, 45365, 50303, 60896, 64668, - 64957, 65365 + 6398, 8087, 11318, 13096, 14587, 16924, 19131, 21007, 22427, 22518, 24792, 25004, + 34675, 35505, 35770, 36919, 38067, 41990, 42949, 45716, 48448, 55869, 56883, 57753, + 58019, 60065, 65116 ], "positions for a 65536-bit filter are frozen: both bytes of each slice are in play" ); diff --git a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr index 24ed43ab0..3a9ac194a 100644 --- a/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr +++ b/packages/stack-encrypt/tests/ui/op_target_does_not_answer.stderr @@ -9,7 +9,7 @@ help: the trait `Answers` is no but trait `Answers` is implemented for it --> src/op/mod.rs | - | impl Answers for EqualityTerm { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + | impl Answers for EqualityTerm { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = help: for that trait implementation, expected `stack_encrypt::op::Eq`, found `stack_encrypt::op::Matches` = note: required for `ValuePlan>` to implement `Chooses` diff --git a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs index 2573c44ca..99df4f560 100644 --- a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs +++ b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.rs @@ -6,7 +6,7 @@ use stack_encrypt::target::{ciphertext, equality, CallerContext, Owned}; use stack_encrypt::{nonempty, KeysetCipher, Plaintext}; use stack_kms::FakeDataKeySource; -fn seal_and_index( +fn seal_and_index( keyset: &KeysetCipher<'_, FakeDataKeySource>, value: S, ) { diff --git a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr index ca42a4fa4..367597e11 100644 --- a/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr +++ b/packages/stack-encrypt/tests/ui/owned_fan_out_needs_clone.stderr @@ -15,5 +15,5 @@ note: required by a bound in `Encryption::<'s, S, T, K, Ctx, M>::zip` | ^^^^^^^^^^^^^^^^^^ required by this bound in `Encryption::<'s, S, T, K, Ctx, M>::zip` help: consider further restricting type parameter `S` with trait `Clone` | - 9 | fn seal_and_index( - | +++++++++++++++++++ + 9 | fn seal_and_index( + | +++++++++++++++++++ diff --git a/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs b/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs index f15dcfedd..ea721c5b3 100644 --- a/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs +++ b/packages/stack-encrypt/tests/ui/pass/owned_without_clone.rs @@ -14,7 +14,7 @@ fn seal<'a, S: Plaintext>( ) -> Pending<'a, StackCipherText, FakeDataKeySource> { keyset.run(ciphertext::<_, _, Owned>(), value, nonempty!("column").into()) } -fn index<'a, S: vitaminc_prf::PrfValue>( +fn index<'a, S: Plaintext>( keyset: &'a KeysetCipher<'_, FakeDataKeySource>, value: S, ) -> Pending<'a, EqualityTerm, FakeDataKeySource> { diff --git a/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs b/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs index 78f5c82f6..27faba642 100644 --- a/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs +++ b/packages/stack-encrypt/tests/ui/pass/query_only_and_borrowed.rs @@ -1,12 +1,13 @@ -use stack_encrypt::{EncryptFrom, StackCipher, StackCipherText, nonempty}; +use stack_encrypt::{EncryptFrom, Plaintext, StackCipher, StackCipherText, nonempty}; use stack_encrypt::sem::EqualityTerm; use stack_kms::FakeDataKeySource; #[derive(EncryptFrom)] struct Probe { hm: EqualityTerm } -// No Plaintext bound: producing a term requires only the PRF capability. -fn query(cipher:&StackCipher, value:&S) { +// No `Decrypt` bound: producing a term requires only that the value +// describes itself, not that it can be read back. +fn query(cipher:&StackCipher, value:&S) { let keyset=cipher.default_keyset(); let _ = keyset.encrypt_as::<_,Probe>(value,nonempty!("column").into()); } From 886fd41f726d0147fc04b532abba9581e9f8f035 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 02:28:31 +1100 Subject: [PATCH 29/38] feat(stack-encrypt)!: walk a dynamic value to its scalar as vitaminc Plaintext and give a boolean no order term dynamic::Value and dynamic::Scalar restated vitaminc's vocabulary: one to make an FfiValue Clone, the other to pick a Rust type per variant. FfiValue is now Clone itself and is a plan field's plaintext, and dynamic::term takes any Plaintext, so a guest walks the frame it was handed in place and the term derives from the scalar vitaminc hands the sink. IndexSpec::defines replaces supports, and admits reads the same table over kinds: equality is vitaminc's (floats now have a domain, booleans none), match takes text, and ORE and OPE refuse a boolean, whose order term would tell anyone holding the column which rows are true. Part of #1140 --- packages/stack-encrypt/CHANGELOG.md | 29 + .../stack-encrypt/src/dynamic/ffi_value.rs | 99 ++ packages/stack-encrypt/src/dynamic/kind.rs | 96 +- packages/stack-encrypt/src/dynamic/mod.rs | 41 +- packages/stack-encrypt/src/dynamic/record.rs | 142 ++- packages/stack-encrypt/src/dynamic/term.rs | 952 +++++++++++------- .../stack-encrypt/tests/fixtures/README.md | 2 +- packages/stack-encrypt/tests/index.rs | 10 +- .../stack-encrypt/tests/record_lowering.rs | 41 +- 9 files changed, 888 insertions(+), 524 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 0a65e7e85..9a92842d1 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -135,6 +135,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 same way. Storing the empty filter tells a reader of the database that the value is empty, separator-only or shorter than the n-gram length (ADR-0009). +- **Equality terms derive through `vitaminc-normalize`, so `TextEq` compares + text in Unicode 16 NFC.** An equality term is the PRF over the value's + normalized bytes under vitaminc's domain for its kind: `café` written with + a precomposed `é` and with `e` and a combining accent is one term, every + integer width of one number is one term (`34u32` and `34i64` match), both + float widths of one number are one term, and a date, timestamp, decimal + or uuid derives in a domain of its own. `KeysetCipher::equality_term`, + `target::equality`, the `Equality` index and `op::Eq` take a `Plaintext` + that is one scalar where they took a `PrfValue`. A boolean, a null or a + container has no equality term (`TermError::NoEqualityTerm`), and text + holding a code point unassigned in Unicode 16 is refused + (`TermError::Normalize`). Match text is tokenized in the same normal + form, and refused the same way. Every equality and match term changes + bytes (the domains move to `/v2`); ORE and OPE terms do not. No stored + term existed to keep; the record fixture is regenerated. +- **`dynamic::Value` and `dynamic::Scalar` are gone.** A field lowered from + data holds an `FfiValue`, which is now `Clone` (a deep copy into fresh + `Protected` payloads, as `Value`'s was) and reads as a context label. + `dynamic::term` takes any `Plaintext` (an `&FfiValue`, a transport frame + walked in place, a Rust value) and derives at the one scalar it holds; + `IndexSpec::defines(value)` replaces `IndexSpec::supports(&scalar)`, and + `IndexSpec` is an `Index`. +- **A boolean has no order term**, through a plan or `dynamic::term`: with + two values, an ORE or OPE term would tell anyone holding the column which + rows are `true`. `dynamic::admits` now refuses ORE and OPE on `bool`, and + admits equality on `float32` and `float64`, which vitaminc gives an + equality domain. A 128-bit integer has an equality term and no order + term. + - **Values describe themselves with vitaminc's `Plaintext` and read back with `Decrypt` (vitaminc ADR 0001).** `KeysetCipher::encrypt`, `StackCipher::decrypt` and every target take `Plaintext` where they took diff --git a/packages/stack-encrypt/src/dynamic/ffi_value.rs b/packages/stack-encrypt/src/dynamic/ffi_value.rs index 231d5db48..a34611973 100644 --- a/packages/stack-encrypt/src/dynamic/ffi_value.rs +++ b/packages/stack-encrypt/src/dynamic/ffi_value.rs @@ -23,6 +23,9 @@ use vitaminc_aead::{to_frame, DecodeError, Decrypt, Frame, Plaintext}; use vitaminc_protected::{Controlled, Protected}; use zeroize::Zeroize; +use crate::plan::IntoLabel; +use crate::{Label, LabelError}; + /// An owned, `Send`, language-neutral representation of a dynamically typed /// host value: what a binding converts a host value into on the host's /// thread, so encryption and decryption can run on any thread. @@ -127,6 +130,62 @@ impl TryFrom>> for Utf8String { } } +/// A deep copy, each text and bytes leaf into a fresh [`Protected`], which +/// wipes on drop as the original does, so a copy is under the same custody +/// as the value it was made from. +/// +/// The engine hands a borrowed field to every operation that consumes it and +/// clones it on the way in, as it does a `String` field of a derived record, +/// so a field's plaintext type is `Clone`; this is what lets an `FfiValue` +/// be that type for every field lowered from data. A record of a few fields +/// clones each once per operation that consumes it, as the derive's does. +impl Clone for FfiValue { + fn clone(&self) -> Self { + // Exhaustive, so a variant added later has to say how it is copied. + match self { + FfiValue::Null => FfiValue::Null, + FfiValue::Undefined => FfiValue::Undefined, + FfiValue::Bool(v) => FfiValue::Bool(*v), + FfiValue::Int32(v) => FfiValue::Int32(*v), + FfiValue::Int64(v) => FfiValue::Int64(*v), + FfiValue::UInt32(v) => FfiValue::UInt32(*v), + FfiValue::UInt64(v) => FfiValue::UInt64(*v), + FfiValue::Float32(v) => FfiValue::Float32(*v), + FfiValue::Float64(v) => FfiValue::Float64(*v), + FfiValue::String(s) => FfiValue::String(s.clone()), + FfiValue::Bytes(b) => FfiValue::Bytes(Protected::new(b.risky_ref().to_vec())), + FfiValue::Array(items) => FfiValue::Array(items.clone()), + FfiValue::Object(entries) => FfiValue::Object(entries.clone()), + FfiValue::Passthrough(inner) => FfiValue::Passthrough(inner.clone()), + } + } +} + +/// A copy of the text in a fresh [`Protected`]: valid UTF-8 because the +/// original is. +impl Clone for Utf8String { + fn clone(&self) -> Self { + Self(Protected::new(self.0.risky_ref().to_vec())) + } +} + +/// A string value read as a plan's context: what a data plan's +/// `"context_field"` holds (`"tenants/acme"`), parsed as a Rust chain's +/// `String` context field is ([`Label::parse`]). Any other variant is +/// [`LabelError::NotText`], which the plan reports as +/// `PlanError::ContextLabel` before any key is requested. +impl IntoLabel for FfiValue { + fn into_label(self) -> Result { + match &self { + FfiValue::String(s) => match s.as_str() { + Some(text) => Label::parse(text), + None => Err(LabelError::NotText), + }, + _ => Err(LabelError::NotText), + } + } +} + impl FfiValue { /// This value as a transport frame: what a binding hands its host. /// @@ -571,6 +630,46 @@ mod tests { ]) } + /// A clone is the same value, leaf for leaf, at every depth and for + /// every variant, and shares no payload with its original. + #[test] + fn a_clone_is_the_same_value_in_fresh_payloads() { + let original = every_variant(); + let copy = original.clone(); + assert_eq!(copy, original, "the same value"); + let (FfiValue::Object(a), FfiValue::Object(b)) = (&original, ©) else { + panic!("objects"); + }; + let (FfiValue::String(x), FfiValue::String(y)) = (&a[9].1, &b[9].1) else { + panic!("strings"); + }; + assert_ne!( + x.risky_ref().as_ptr(), + y.risky_ref().as_ptr(), + "the text was copied, not shared" + ); + let (FfiValue::Bytes(x), FfiValue::Bytes(y)) = (&a[10].1, &b[10].1) else { + panic!("bytes"); + }; + assert_ne!( + x.risky_ref().as_ptr(), + y.risky_ref().as_ptr(), + "the bytes were copied, not shared" + ); + } + + #[test] + fn a_string_reads_as_a_label_and_nothing_else_does() { + assert_eq!( + s("tenants/acme").into_label().expect("a label"), + Label::parse("tenants/acme").expect("a label") + ); + assert!(matches!( + FfiValue::UInt32(7).into_label(), + Err(LabelError::NotText) + )); + } + #[test] fn every_variant_survives_a_frame_round_trip() { let frame = every_variant().to_frame().expect("frames"); diff --git a/packages/stack-encrypt/src/dynamic/kind.rs b/packages/stack-encrypt/src/dynamic/kind.rs index 4d8ba4ade..875821ebc 100644 --- a/packages/stack-encrypt/src/dynamic/kind.rs +++ b/packages/stack-encrypt/src/dynamic/kind.rs @@ -13,7 +13,8 @@ //! of a kind. Beside it, this module adds two rules: //! //! - [`admits`]: which indexes a kind is defined for, so a plan asking for -//! match on an integer or equality on a float is refused when it is built; +//! match on an integer or equality on a boolean is refused when it is +//! built; //! - [`read`]: how a query value is read as a field's kind, converting a //! number only when the conversion is exact. //! @@ -35,6 +36,8 @@ //! field has none is refused when it is built //! ([`Error::UntypedIndex`], naming the field). A field that only seals, or //! only carries its value through, may leave it out: no term derives from it. +use vitaminc_aead::plaintext::{Float, Int, Physical}; + use super::{FfiValue, ValueKind}; use super::{Error, Reason}; @@ -42,28 +45,45 @@ use crate::target::IndexSpec; /// Whether the scheme defines an `index` term for values of `kind`. /// -/// The same table as [`IndexSpec::supports`], stated over kinds instead of -/// values, so a plan can be refused when it is built rather than when its -/// first value arrives: equality over every integer, text and bytes (no -/// floats, no booleans); match over text alone; ORE and OPE over every -/// scalar. A composite (`array`, `object`) has no term. Only the index's -/// kind matters: a match index's options do not change what it applies to. +/// The table [`IndexSpec::defines`] reads for a value, stated over kinds +/// instead, so a plan can be refused when it is built rather than when its +/// first value arrives. Equality is vitaminc's: every kind +/// `vitaminc-normalize` gives an equality domain, which is every integer, +/// both floats, text and bytes, and not a boolean. The rest are this +/// crate's: match over text alone, and ORE and OPE over every scalar but a +/// boolean, whose order term would reveal which rows are `true`. A +/// composite (`array`, `object`) has no term. Only the index's kind +/// matters: a match index's options do not change what it applies to. /// /// ``` /// use stack_encrypt::dynamic::{admits, ValueKind}; /// use stack_encrypt::target::IndexSpec; /// /// assert!(admits(ValueKind::UInt64, &IndexSpec::Equality)); -/// assert!(!admits(ValueKind::Float64, &IndexSpec::Equality)); +/// assert!(!admits(ValueKind::Bool, &IndexSpec::Equality)); +/// assert!(!admits(ValueKind::Bool, &IndexSpec::Ore)); /// assert!(!admits(ValueKind::Object, &IndexSpec::Ore)); /// ``` pub fn admits(kind: ValueKind, index: &IndexSpec) -> bool { - use ValueKind::*; - match index { - IndexSpec::Equality => matches!(kind, Int32 | Int64 | UInt32 | UInt64 | String | Bytes), - IndexSpec::Match(_) => kind == String, - IndexSpec::Ore | IndexSpec::Ope => !matches!(kind, Array | Object), - } + scalar_of(kind).is_some_and(|scalar| index.takes(None, &scalar)) +} + +/// A scalar of `kind`, for asking the table about the kind: any value of a +/// kind is taken by an index exactly when every other is. `None` for a +/// composite, which is no scalar. +fn scalar_of(kind: ValueKind) -> Option> { + Some(match kind { + ValueKind::Bool => Physical::Bool(false), + ValueKind::Int32 => Physical::Int(Int::I32(0)), + ValueKind::Int64 => Physical::Int(Int::I64(0)), + ValueKind::UInt32 => Physical::Int(Int::U32(0)), + ValueKind::UInt64 => Physical::Int(Int::U64(0)), + ValueKind::Float32 => Physical::Float(Float::F32(0.0)), + ValueKind::Float64 => Physical::Float(Float::F64(0.0)), + ValueKind::String => Physical::Text(""), + ValueKind::Bytes => Physical::Bytes(&[]), + ValueKind::Array | ValueKind::Object => return None, + }) } /// Read a query value as `kind`. @@ -82,8 +102,7 @@ pub fn admits(kind: ValueKind, index: &IndexSpec) -> bool { /// and is refused otherwise. /// /// **The engine does not call this.** [`term`](super::term()) takes a -/// [`Scalar`](super::Scalar) and an [`IndexSpec`] and never sees the field's -/// declared kind, so a binding deriving a query term for a typed field must +/// value and an [`IndexSpec`] and never sees the field's declared kind, so a binding deriving a query term for a typed field must /// call `read` first, with [`FieldPlan::field_type`](super::FieldPlan::field_type). /// One that skips it hands `Float64(34.0)` to a `uint64` field, derives a /// float term where the stored one is the `u64` term, and matches nothing, @@ -190,7 +209,6 @@ impl Number { #[cfg(test)] mod tests { use super::*; - use crate::dynamic::Scalar; use crate::sem::{MatchOptions, Tokenizer}; use vitaminc_protected::Protected; @@ -224,27 +242,23 @@ mod tests { ] } - /// `admits` is `IndexSpec::supports` stated over kinds: the two tables - /// cannot disagree about any scalar, and a composite admits nothing. + /// `admits` is `IndexSpec::defines` stated over kinds: the two cannot + /// disagree about a value of any kind, and a composite admits nothing. #[test] - fn admits_agrees_with_supports_for_every_scalar_kind() { + fn admits_agrees_with_defines_for_every_kind() { for kind in ValueKind::ALL { for index in indexes() { - match Scalar::of(&sample(kind), &index) { - Ok(scalar) => { - assert_eq!( - admits(kind, &index), - index.supports(&scalar), - "{kind} and {index}" - ) - } - Err(_) => assert!(!admits(kind, &index), "{kind} is not a scalar"), - } + assert_eq!( + admits(kind, &index), + index.defines(sample(kind)), + "{kind} and {index}" + ); } } // Spelled out, so the table reads without the cross-check. assert!(admits(ValueKind::UInt64, &IndexSpec::Equality)); - assert!(!admits(ValueKind::Float64, &IndexSpec::Equality)); + assert!(admits(ValueKind::Float64, &IndexSpec::Equality)); + assert!(admits(ValueKind::Bytes, &IndexSpec::Equality)); assert!(!admits(ValueKind::Bool, &IndexSpec::Equality)); assert!(admits( ValueKind::String, @@ -255,10 +269,30 @@ mod tests { &IndexSpec::Match(MatchOptions::default()) )); assert!(admits(ValueKind::Float64, &IndexSpec::Ore)); + assert!(admits(ValueKind::String, &IndexSpec::Ope)); + assert!(!admits(ValueKind::Bool, &IndexSpec::Ore)); + assert!(!admits(ValueKind::Bool, &IndexSpec::Ope)); assert!(!admits(ValueKind::Object, &IndexSpec::Ope)); assert!(!admits(ValueKind::Array, &IndexSpec::Ore)); } + /// Equality is vitaminc's table: a kind admits it exactly when + /// `vitaminc-normalize` gives the kind's values an equality domain. + #[test] + fn admits_equality_exactly_where_vitaminc_gives_a_domain() { + for kind in ValueKind::ALL { + let has_domain = matches!( + vitaminc_normalize::terms(sample(kind)).as_deref(), + Ok([term]) if term.path.steps().is_empty() + ); + assert_eq!( + admits(kind, &IndexSpec::Equality), + has_domain, + "{kind} admits equality exactly when it has a domain" + ); + } + } + /// A match index's options are not part of what it applies to: text /// admits it under any options, and nothing else does. #[test] diff --git a/packages/stack-encrypt/src/dynamic/mod.rs b/packages/stack-encrypt/src/dynamic/mod.rs index 225b41a7e..9e9397b60 100644 --- a/packages/stack-encrypt/src/dynamic/mod.rs +++ b/packages/stack-encrypt/src/dynamic/mod.rs @@ -2,8 +2,7 @@ //! //! Everything else in this crate is typed: a target names its source type, //! and the operations it composes are reached through bounds — `Plaintext` for -//! a ciphertext, [`PrfValue`](vitaminc_prf::PrfValue) for an equality term, -//! `AsRef` for a match term. That is what makes a term a cross-language +//! a ciphertext or an equality term, `AsRef` for a match term. That is what makes a term a cross-language //! contract: `equality_term(34u32)` derives the same bytes wherever it is //! called from, because `34u32` is the same value everywhere. //! @@ -13,13 +12,19 @@ //! the typed operation. This module is that something, written once here //! rather than once per language binding. //! -//! The runtime value is [`FfiValue`], a language-neutral value tree and -//! the type every binding already funnels through. +//! A term is derived from anything that describes itself to vitaminc +//! ([`Plaintext`](crate::Plaintext)), walked to its one scalar: a guest +//! walks the transport frame it was handed in place, with no tree built. +//! What the engine holds is [`FfiValue`], a language-neutral value tree: a +//! plan, a record source and a field's plaintext are values a binding +//! decoded, which the engine reads, splits into fields and clones, and +//! which outlive the buffer they arrived in. //! //! # What is here //! //! * [`context`](context()) — an [`FfiValue`] read as an encryption context. -//! * [`term`](term()) — one index term for a value, dispatched on its variant. +//! * [`term`](term()) — one index term for a value, derived at the one +//! scalar it is walked to. //! * [`record`] — a fields plan spelled as data, *lowered* into the plan //! builder ([`Plan`](crate::Plan)) and run by the engine: the same code a //! Rust chain and the derive run, so a record written from any language @@ -30,9 +35,8 @@ //! that links them, so a plan field may name one as its target and the //! lowering runs that type's own plan in the same request; [`NoTargets`] //! is the build without them. -//! * [`Value`] — an [`FfiValue`] as a plan field's plaintext, the type of -//! every field lowered from data; [`TermBytes`] — the term such a field -//! derives, as its frozen bytes. +//! * [`TermBytes`] — the term a field lowered from data derives, as its +//! frozen bytes. Such a field's plaintext is an [`FfiValue`]. //! * [`Scope`] — which cipher an opening operation decrypts through. //! //! # What is not here @@ -55,11 +59,10 @@ //! //! A plan field's `"type"` names (`"int64"`, `"string"`, …) are wire format //! in the same way: a binding spells them, and a stored row opens only under -//! the type it was sealed as. They are not this crate's: a declared type is -//! vitaminc's [`ValueKind`], re-exported here, whose names vitaminc freezes -//! beside its tag table. This crate adds only what a kind means to an index -//! ([`admits`]) and to a query value ([`read`]); it decides nothing about -//! the bytes ([`record`]). Every field with a term output declares one; +//! the type it was sealed as. A declared type is a [`ValueKind`], whose +//! names are frozen. What a kind means to an index ([`admits`]) and to a +//! query value ([`read`]) is stated over it; the bytes are vitaminc's, from +//! the tagged leaf a value of the kind seals as ([`record`]). Every field with a term output declares one; //! a plan whose indexed field has none is refused when it is built //! ([`Error::UntypedIndex`], naming the field). //! @@ -75,7 +78,6 @@ mod kind; pub mod record; mod target; mod term; -mod value; use std::fmt; @@ -94,8 +96,7 @@ pub use ffi_value::{ pub use kind::{admits, read}; pub use record::{FieldPlan, Output, Plan}; pub use target::{NoTargets, TargetDescriptor, TargetError, TargetResolver}; -pub use term::{term, Scalar, TermBytes}; -pub use value::{AsValue, Value}; +pub use term::{term, TermBytes}; use crate::target::IndexSpec; use crate::{KeysetCipher, StackCipher}; @@ -190,9 +191,9 @@ pub enum Error { /// A term was asked for a value the scheme defines no such term for: a /// container, null or passthrough (which have no term semantics at all), - /// or a scalar outside the kind's domain — equality over a float or a - /// boolean, match over anything but text. See - /// [`IndexSpec::supports`]. + /// or a scalar outside the kind's domain — equality over a boolean, + /// match over anything but text, an order term over a boolean. See + /// [`IndexSpec::defines`]. #[error("no {kind} term is defined for this value{}", in_field_text(field))] #[diagnostic(code(stack_encrypt::dynamic_term))] Term { @@ -504,7 +505,7 @@ reasons! { /// `"type"` is not a string naming a value type. UnknownType => "unknown_type", r#""type" does not name a value type"#; /// The field's declared type has no such index: match on an integer, - /// equality on a float, any index on a composite. + /// equality or an order index on a boolean, any index on a composite. IndexNotAdmitted => "index_not_admitted", "the declared type has no such index"; /// Two fields are keyed under one identity. SharedIdentity => "shared_identity", "two fields are keyed under one identity"; diff --git a/packages/stack-encrypt/src/dynamic/record.rs b/packages/stack-encrypt/src/dynamic/record.rs index 91c6e8758..d32cb0e35 100644 --- a/packages/stack-encrypt/src/dynamic/record.rs +++ b/packages/stack-encrypt/src/dynamic/record.rs @@ -86,7 +86,7 @@ //! //! # What a field's `"type"` decides, and what it does not //! -//! Every field lowered from data is a [`Value`]: its plaintext type is the +//! Every field lowered from data is an [`FfiValue`]: its plaintext type is the //! runtime value itself, and it seals in vitaminc's self-describing tagged //! leaf encoding (`[tag] ++ payload`) whatever its declared `"type"`. The //! type decides which indexes the field admits ([`admits`]), which values it @@ -99,13 +99,13 @@ //! whatever tag each value arrived with (`34` sent once as a float and once //! as an integer would otherwise store two terms under one field). A field //! that only seals, or only carries its value through, may leave the type -//! out. A field's terms dispatch on the value's variant — the declared kind, -//! once checked — to the typed term operation, so they are the bytes a Rust -//! `u32` or `String` field derives under the same label. +//! out. A field's terms derive at the value's one scalar — of the declared +//! kind, once checked — by the typed path's derivation, so they are the +//! bytes a Rust `u32` or `String` field derives under the same label. //! //! The ciphertext is where a data plan and a Rust chain meet too. Since //! vitaminc ADR 0001 every leaf is tagged, a typed Rust value's as much as -//! a [`Value`]'s, so a Rust `u32` field and a `uint32` data field seal the +//! an [`FfiValue`]'s, so a Rust `u32` field and a `uint32` data field seal the //! same leaf under the same context, and each opens the other's. A reader //! of another type refuses a leaf rather than reading it as changed //! plaintext: the tag is authenticated. The lowering still never chooses an @@ -141,8 +141,7 @@ use super::{FfiValue, ValueKind}; use vitaminc_protected::Controlled; use super::{ - admits, utf8, Error, NoTargets, Reason, Scalar, Scope, TargetError, TargetResolver, TermBytes, - Value, + admits, utf8, Error, NoTargets, Reason, Scope, TargetError, TargetResolver, TermBytes, }; use crate::plan::{FieldValues, FieldsBuilder, Opens, Runs}; use crate::target::{CallerContext, DeclaredContext, Decryption, Encrypted, IndexSpec}; @@ -341,8 +340,8 @@ impl FieldPlan { /// # Errors /// /// [`Error::Plan`] if the field asks for an index the kind is not - /// defined for ([`admits`]): match on an integer, equality on a float, - /// any index on a composite. + /// defined for ([`admits`]): match on an integer, equality or + /// an order index on a boolean, any index on a composite. pub fn with_type(mut self, field_type: ValueKind) -> Result { for output in &self.outputs { if let Output::Term(index) = output { @@ -844,7 +843,7 @@ impl Plan { /// /// Built afresh per call: a built plan is bound to its key source type, /// and a declaration is not. Every field is declared by name, as a - /// [`Value`], read out of the [`FieldValues`] the source is converted + /// [`FfiValue`], read out of the [`FieldValues`] the source is converted /// into. A context field is declared through the builder's own /// `context_field`, which carries it as a passthrough and reads it as /// the context; the loop then skips it. @@ -852,7 +851,7 @@ impl Plan { let mut builder = match &self.context { Context::Label(label) => crate::Plan::context(label.clone()).fields::(), Context::Field(name) => { - crate::Plan::fields::().context_field::(name.as_str()) + crate::Plan::fields::().context_field::(name.as_str()) } }; for field in &self.fields { @@ -890,18 +889,18 @@ impl Plan { } } -/// One field's verb, over a [`Value`]; its indexes are the [`IndexSpec`]s -/// the plan named, each an [`Index`] of a `Value`. +/// One field's verb, over an [`FfiValue`]; its indexes are the [`IndexSpec`]s +/// the plan named, each an [`Index`] of a `FfiValue`. fn declare( builder: FieldsBuilder, field: &FieldPlan, ) -> FieldsBuilder { let name = field.name.as_str(); match field.verb() { - Verb::Encrypt => builder.encrypt::(name), - Verb::EncryptIndex => builder.encrypt_index::(name, field.indexes()), - Verb::Index => builder.index::(name, field.indexes()), - Verb::Passthrough => builder.passthrough::(name), + Verb::Encrypt => builder.encrypt::(name), + Verb::EncryptIndex => builder.encrypt_index::(name, field.indexes()), + Verb::Index => builder.index::(name, field.indexes()), + Verb::Passthrough => builder.passthrough::(name), // `lower` never hands a target field here: the resolver runs it. Verb::Target => builder, } @@ -1792,7 +1791,7 @@ fn source_row(mut row: Vec<(String, FfiValue)>, plan: &Plan) -> Result, plan: &Plan) -> Result Result<(), Error> { @@ -1823,9 +1822,7 @@ fn check_field(value: &FfiValue, field: &FieldPlan) -> Result<(), Error> { match output { Output::Ciphertext => check_tree(value, &field.name)?, Output::Term(kind) => { - let scalar = - Scalar::of(value, kind).map_err(|error| error.in_field(&field.name))?; - if !kind.supports(&scalar) { + if !kind.defines(value) { return Err(Error::Term { field: Some(field.name.clone()), kind: kind.clone(), @@ -1944,7 +1941,7 @@ fn keyed_terms( /// The field `name` out of the record, as the value it holds. fn take_leaf(values: &mut FieldValues, name: &str) -> Result { - Ok(slot::(values, name)?.into_inner()) + slot::(values, name) } /// The record the plan opened, as a value: the fields that come back, in @@ -2052,7 +2049,7 @@ fn record_row(mut row: Vec<(String, StackCipherText)>, plan: &Plan) -> Result { let ciphertext = node("c", Reason::NoCiphertextNode)?; @@ -3103,33 +3100,41 @@ mod tests { ); } - /// A float cannot reach an equality index through a plan: a field - /// declared `float64` is refused when the plan is built, and a field - /// declared an integer refuses the float value as the wrong kind, - /// before any key request. There is no untyped indexed field for a - /// float to slip through. + /// A boolean cannot reach an equality or order index through a + /// plan: a field declared `bool` is refused when the plan is built, + /// and a field declared an integer refuses the boolean value as the + /// wrong kind, before any key request. There is no untyped indexed + /// field for a boolean to slip through. A float can: vitaminc gives + /// floats an equality domain. #[tokio::test] - async fn a_float_asked_for_equality_is_refused_at_build_or_as_the_wrong_kind() { + async fn a_boolean_asked_for_a_term_is_refused_at_build_or_as_the_wrong_kind() { let cipher = cipher().await; let keyset = cipher.default_keyset(); - let as_float = plan(obj(vec![( - "score", - typed(label("score"), &["c", "eq"], "float64"), - )])); - assert!( - matches!(as_float, Err(Error::Plan { .. })), - "no PRF encoding exists for a float: {as_float:?}" - ); + for outputs in [&["c", "eq"], &["c", "ore"], &["c", "ope"]] { + let as_bool = plan(obj(vec![("flag", typed(label("flag"), outputs, "bool"))])); + assert!( + matches!(as_bool, Err(Error::Plan { .. })), + "a boolean has no {outputs:?} term: {as_bool:?}" + ); + } let as_u32 = plan(obj(vec![( - "score", - typed(label("score"), &["c", "eq"], "uint32"), + "flag", + typed(label("flag"), &["c", "eq"], "uint32"), )])) .expect("plan"); - let source = obj(vec![("score", FfiValue::Float64(1.5))]); + let source = obj(vec![("flag", FfiValue::Bool(true))]); let err = encrypt(&keyset, source, &as_u32).err(); assert!( matches!(err, Some(Error::Source { .. })), - "a float is not the declared kind: {err:?}" + "a boolean is not the declared kind: {err:?}" + ); + assert!( + plan(obj(vec![( + "score", + typed(label("score"), &["c", "eq"], "float64"), + )])) + .is_ok(), + "a float has an equality term" ); assert_eq!(generates(&cipher), 0, "refused before any key request"); } @@ -3260,7 +3265,7 @@ mod tests { let eq = term( &keyset, - Scalar::U32(34), + &FfiValue::UInt32(34), &IndexSpec::Equality, age_ctx.clone(), ) @@ -3271,7 +3276,7 @@ mod tests { eq, "the equality term is the standalone derivation under the field label" ); - let ore = term(&keyset, Scalar::U32(34), &IndexSpec::Ore, age_ctx) + let ore = term(&keyset, &FfiValue::UInt32(34), &IndexSpec::Ore, age_ctx) .await .expect("standalone ore term"); assert_eq!( @@ -3279,14 +3284,9 @@ mod tests { ore, "the ore term is the standalone derivation under the field label" ); - let scalar = Scalar::of( - &s("al smith"), - &IndexSpec::Match(crate::sem::MatchOptions::default()), - ) - .expect("text"); let matched = term( &keyset, - scalar, + &s("al smith"), &IndexSpec::Match(crate::sem::MatchOptions::default()), nick_ctx, ) @@ -3329,13 +3329,15 @@ mod tests { let stored = term_bytes(&node(&mut nick, "match")); let kind = IndexSpec::Match(options); - let scalar = Scalar::of(&s("al smith"), &kind).expect("text"); - let expected = term(&keyset, scalar.clone(), &kind, ctx.clone()) + let scalar = s("al smith"); + let expected = term(&keyset, &scalar, &kind, ctx.clone()) .await .expect("standalone"); assert_eq!(stored, expected, "the term is the plan options' derivation"); let default = IndexSpec::Match(crate::sem::MatchOptions::default()); - let under_default = term(&keyset, scalar, &default, ctx).await.expect("default"); + let under_default = term(&keyset, &scalar, &default, ctx) + .await + .expect("default"); assert_ne!(stored, under_default, "and not the defaults'"); } @@ -3411,22 +3413,22 @@ mod tests { /// The lowering is the plan builder: the record a data plan seals is /// the record the chain seals under the same declaration over the - /// same plaintext type, a [`Value`] per field — the same terms, and + /// same plaintext type, an [`FfiValue`] per field — the same terms, and /// ciphertexts each side opens. #[tokio::test] async fn seals_what_the_chain_seals_under_the_same_declaration() { struct User { - age: Value, - email: Value, - nick: Value, + age: FfiValue, + email: FfiValue, + nick: FfiValue, id: u64, } let cipher = cipher().await; let keyset = cipher.default_keyset(); let user = User { - age: Value::new(FfiValue::UInt32(34)), - email: Value::new(s("a@x")), - nick: Value::new(s("al smith")), + age: FfiValue::UInt32(34), + email: s("a@x"), + nick: s("al smith"), id: 7, }; let default_match = IndexSpec::Match(crate::sem::MatchOptions::default()); @@ -4640,16 +4642,16 @@ mod tests { parsed.fields()[0].outputs(), [Output::Ciphertext, Output::Term(IndexSpec::Equality)] ); - let refused = plan(obj(vec![("age", early("float64"))])); + let refused = plan(obj(vec![("age", early("bool"))])); assert!( matches!(refused, Err(Error::Plan { .. })), - "equality on a float is refused whatever the key order: {refused:?}" + "equality on a boolean is refused whatever the key order: {refused:?}" ); } #[test] fn the_plan_refuses_an_unresolvable_type_or_one_that_does_not_admit_an_index() { - let refused: [(&str, FfiValue); 8] = [ + let refused: [(&str, FfiValue); 9] = [ ("an unknown type", typed(label("x"), &["c"], "u64")), ( "a type in the wrong case", @@ -4676,11 +4678,9 @@ mod tests { "match on an integer", typed(label("x"), &["c", "match"], "int64"), ), - ( - "equality on a float", - typed(label("x"), &["c", "eq"], "float64"), - ), ("equality on a bool", typed(label("x"), &["eq"], "bool")), + ("order on a bool", typed(label("x"), &["c", "ore"], "bool")), + ("ope on a bool", typed(label("x"), &["ope"], "bool")), ( "order on a composite", typed(label("x"), &["c", "ore"], "object"), @@ -4710,7 +4710,7 @@ mod tests { .expect("equality admits a u32"); assert_eq!(typed.field_type(), Some(ValueKind::UInt32)); assert!(matches!( - field.with_type(ValueKind::Float32), + field.with_type(ValueKind::Bool), Err(Error::Plan { .. }) )); let sealed_only = FieldPlan::new( @@ -4932,10 +4932,9 @@ mod tests { let declared = field.field_type().expect("typed"); let value = crate::dynamic::read(declared, FfiValue::Float64(34.0)).expect("an exact 34"); - let scalar = Scalar::of(&value, &IndexSpec::Equality).expect("scalar"); let query = term( &keyset, - scalar, + &value, &IndexSpec::Equality, field.context().clone(), ) @@ -5065,8 +5064,7 @@ mod tests { let mut fields = map(seal(&keyset, obj(vec![("f", value())]), &plan).await); let mut f = map(node(&mut fields, "f")); let stored = term_bytes(&node(&mut f, kind.key())); - let scalar = Scalar::of(&value(), &kind).expect("scalar"); - let probe = term(&keyset, scalar, &kind, plan.fields()[0].context().clone()) + let probe = term(&keyset, &value(), &kind, plan.fields()[0].context().clone()) .await .expect("probe"); assert_eq!(stored, probe, "{ty} {kind}"); diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index d456ac511..8fa97faea 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -1,11 +1,27 @@ //! One index term for a runtime value. //! -//! The dispatch here is the whole point of the module: which arm a value -//! takes decides the term's PRF/CLLW *input encoding*, and that encoding is -//! part of the cross-language contract. An equality term for -//! `FfiValue::UInt32(34)` must equal the term the typed path derives for -//! `34u32`, so each variant is handed to the same typed operation a Rust -//! caller would have named. +//! A value arrives here as anything that describes itself to vitaminc +//! ([`Plaintext`]): a transport frame a guest walks in place, an +//! [`FfiValue`] a binding decoded, a Rust value. It is walked to the one +//! scalar it must be, and the term derives there, from the scalar vitaminc +//! hands the sink, so this module restates none of vitaminc's vocabulary. +//! The derivation is the typed path's, which is what makes a term a +//! cross-language contract: an equality term for a frame holding `34` is the +//! term `equality_term(34u32)` derives. +//! +//! - **Equality** normalizes the scalar with `vitaminc-normalize` and runs +//! the PRF over the result, as the typed path does through that crate's +//! `Terms` sink. Every integer width of one number is one term, text is +//! compared in Unicode 16 NFC, and a date, timestamp, decimal or uuid has +//! a domain of its own. A boolean has none. +//! - **Match** takes text only, cut in Unicode 16 NFC. +//! - **ORE and OPE** take an integer of at most 64 bits, a float, text or +//! bytes, each as the Rust type the typed path would have named. An 8- or +//! 16-bit integer is read as the 32-bit integer of its sign, as an +//! [`FfiValue`] reads it, so a frame from a host with narrower integers +//! derives the term of the width a plan field declares. A boolean has no +//! order term: with two values, its order term would tell anyone holding +//! the column which rows are `true`. //! //! # These are the query-probe path //! @@ -14,17 +30,23 @@ //! going to be **stored** should come from the record path, where it shares //! one context with the ciphertext beside it (ADR-0004). +use std::convert::Infallible; +use std::marker::PhantomData; + use super::FfiValue; use stack_kms::DataKeySource; -use vitaminc_protected::{Controlled, OpaqueDebug, Protected}; +use vitaminc_aead::plaintext::{ + Error as SinkError, Float, Int, Logical, MapSink, Physical, SeqSink, Sink, SourceError, +}; +use vitaminc_normalize::{domain_of, normalize, NormalizeError}; use zeroize::Zeroizing; -use super::{utf8, Error, Reason, Value}; +use super::{utf8, Error, Reason}; use crate::sem::{ CllwOpeEncrypt, CllwOreEncrypt, DefaultMatch, MatchOptions, QueryTokens, TermError, Tokenizer, }; use crate::target::{chosen, CallerContext, ConsumeSource, Encryption, Index, IndexSpec, Pending}; -use crate::{IntoPrfContext, KeysetCipher, NonEmpty}; +use crate::{IntoPrfContext, KeysetCipher, NonEmpty, Plaintext}; /// The runtime half of [`IndexSpec`]: the domain table, and the index's wire /// form in a record plan. @@ -116,26 +138,65 @@ impl IndexSpec { } } - /// Whether the scheme defines this index's term for `scalar`. + /// Whether the scheme defines this index's term for `value`: whether it + /// is one scalar the index takes. Equality takes every scalar vitaminc + /// gives an equality domain (an integer, a float, text, bytes, and a + /// date, timestamp, decimal or uuid), not a boolean or a null. Match + /// takes text. ORE and OPE take an integer of at most 64 bits, a float, + /// text or bytes, not a boolean. A match index's options do not change + /// what it takes. + /// + /// This is the one table: [`term`] refuses exactly what it refuses, and + /// checking it first is how a binding refuses a bad request at its + /// boundary, before a keyset is resolved or anything is derived. It says + /// nothing about whether the value's text normalizes; that is found when + /// the term is derived. /// - /// No PRF encoding exists for floats (equality on IEEE-754 values is a - /// modelling error) or booleans; match is text-only; the ordering - /// schemes take every scalar. This is the one table — [`term`]'s arms - /// mirror it and are unreachable for a pair it refuses — and it is - /// consulted before any cipher work, so a binding can reject a bad - /// request at its boundary without minting anything. A match index's - /// options do not change its domain. - pub fn supports(&self, scalar: &Scalar) -> bool { + /// ``` + /// use stack_encrypt::target::IndexSpec; + /// + /// assert!(IndexSpec::Equality.defines(34u32)); + /// assert!(IndexSpec::Equality.defines(1.5f64)); + /// assert!(!IndexSpec::Equality.defines(true)); + /// assert!(!IndexSpec::Ore.defines(true)); + /// assert!(!IndexSpec::Ope.defines(vec![1u32, 2])); + /// ``` + pub fn defines(&self, value: P) -> bool { + value + .visit(OneScalar::new( + |ty: Option, value: Physical<'_>| { + if self.takes(ty, &value) { + Ok(()) + } else { + Err(None) + } + }, + )) + .is_ok() + } + + /// The table [`defines`](Self::defines) reads, over one scalar. + pub(super) fn takes(&self, ty: Option, value: &Physical<'_>) -> bool { match self { - IndexSpec::Equality => { - !matches!(scalar, Scalar::Bool(_) | Scalar::F32(_) | Scalar::F64(_)) - } - IndexSpec::Match(_) => matches!(scalar, Scalar::Text(_)), - IndexSpec::Ore | IndexSpec::Ope => true, + // vitaminc's table: a kind with an equality domain. + IndexSpec::Equality => domain_of(ty, value).is_some(), + IndexSpec::Match(_) => ty.is_none() && matches!(value, Physical::Text(_)), + IndexSpec::Ore | IndexSpec::Ope => ty.is_none() && ordered(value), } } } +/// Whether CLLW ORE and OPE order this scalar: an integer of at most 64 +/// bits, a float, text or bytes. Not a boolean, which would reveal which +/// rows hold `true`, and not a null. +fn ordered(value: &Physical<'_>) -> bool { + match value { + Physical::Int(int) => !matches!(int, Int::I128(_) | Int::U128(_)), + Physical::Float(_) | Physical::Text(_) | Physical::Bytes(_) => true, + Physical::Null | Physical::Bool(_) => false, + } +} + /// The options object of a match index's wire form: each key at most once, /// each defaulting, and the whole checked against the scheme's bounds. fn match_options(entries: &[(String, FfiValue)]) -> Result { @@ -185,75 +246,6 @@ fn integer>(value: &FfiValue) -> Result { .ok_or(Error::bad_plan(Reason::UnknownOutput)) } -/// A term-able scalar lifted out of an [`FfiValue`] leaf. -/// -/// Lifting is a copy, so the value it came from stays movable into the -/// ciphertext path beside it. The owned text and bytes copies wipe on drop; -/// the PRF and CLLW layers move them into [`Protected`] internally. -/// -/// It is plaintext, so its `Debug` is opaque: the variant is named, the -/// value is masked. -#[derive(Clone, OpaqueDebug)] -#[non_exhaustive] -pub enum Scalar { - /// From [`FfiValue::Bool`]. - Bool(bool), - /// From [`FfiValue::Int32`]. - I32(i32), - /// From [`FfiValue::Int64`]. - I64(i64), - /// From [`FfiValue::UInt32`]. - U32(u32), - /// From [`FfiValue::UInt64`]. - U64(u64), - /// From [`FfiValue::Float32`]. - F32(f32), - /// From [`FfiValue::Float64`]. - F64(f64), - /// From [`FfiValue::String`]. - Text(Zeroizing), - /// From [`FfiValue::Bytes`]. - Bytes(Zeroizing>), -} - -impl Scalar { - /// Lift the scalar out of a value leaf. - /// - /// # Errors - /// - /// [`Error::Term`] for a container, null, undefined or passthrough: - /// those have no term semantics at all, whatever the kind. `kind` names - /// the term the caller was after, for the error only — whether that kind - /// is defined for the scalar is [`IndexSpec::supports`]. - pub fn of(value: &FfiValue, kind: &IndexSpec) -> Result { - Ok(match value { - FfiValue::Bool(v) => Scalar::Bool(*v), - FfiValue::Int32(v) => Scalar::I32(*v), - FfiValue::Int64(v) => Scalar::I64(*v), - FfiValue::UInt32(v) => Scalar::U32(*v), - FfiValue::UInt64(v) => Scalar::U64(*v), - FfiValue::Float32(v) => Scalar::F32(*v), - FfiValue::Float64(v) => Scalar::F64(*v), - FfiValue::String(s) => Scalar::Text(Zeroizing::new( - utf8(s) - .ok_or_else(|| Error::Term { - field: None, - kind: kind.clone(), - })? - .to_string(), - )), - FfiValue::Bytes(b) => Scalar::Bytes(Zeroizing::new(b.risky_ref().to_vec())), - // Containers, nulls and passthroughs have no term semantics. - _ => { - return Err(Error::Term { - field: None, - kind: kind.clone(), - }) - } - }) - } -} - /// Derive one index term's frozen byte encoding. /// /// # Examples @@ -262,7 +254,7 @@ impl Scalar { /// derives for the same value under the same context: /// /// ``` -/// use stack_encrypt::dynamic::{context, term, FfiValue, Scalar}; +/// use stack_encrypt::dynamic::{context, term, FfiValue}; /// use stack_encrypt::target::IndexSpec; /// use stack_encrypt::StackCipher; /// use stack_encrypt::kms::FakeDataKeySource; @@ -278,49 +270,40 @@ impl Scalar { /// FfiValue::String("users".into()), /// FfiValue::String("age".into()), /// ]))?; -/// let probe = term(&keyset, Scalar::U32(34), &IndexSpec::Equality, ctx.clone()).await?; +/// let probe = term(&keyset, &FfiValue::UInt32(34), &IndexSpec::Equality, ctx.clone()).await?; /// let typed = keyset.equality_term(34u32, ctx).await?; /// assert_eq!(probe, typed.into_bytes().to_vec()); /// # Ok::<(), stack_encrypt::dynamic::Error>(()) /// # }).unwrap(); /// ``` /// -/// The derivation is `scalar_term`'s, the one dispatch from a runtime -/// scalar to the typed term operation; the record path runs the same -/// dispatch as an [`Index`] of a [`Value`] field. It is local to the keyset -/// cipher, so the await settles a pending that has nothing to request. +/// The record path runs the same derivation as an [`Index`] of an +/// [`FfiValue`] field. It is local to the keyset cipher, so the await +/// settles a pending that has nothing to request. /// /// # Errors /// -/// [`Error::Term`] if the scheme defines no such term for the scalar -/// ([`IndexSpec::supports`] is the table, and checking it first is how a +/// [`Error::Term`] if the value is not one scalar the index takes +/// ([`IndexSpec::defines`] is the table, and checking it first is how a /// binding turns this into a boundary rejection). [`Error::Cipher`] if the -/// derivation itself fails, or for the one query value the engine refuses: +/// derivation itself fails: text with a code point unassigned in Unicode 16 +/// ([`TermError::Normalize`]), or the one query value the engine refuses, /// match text that cuts no token under the index's options /// ([`TermError::EmptyTermText`]), whose filter would match every row. That /// refusal is the query path's alone; the record path stores such a value /// as the empty filter, which no query matches. -pub async fn term<'c, K, D>( +pub async fn term<'c, K, P, D>( cipher: &KeysetCipher<'_, K>, - scalar: Scalar, + value: P, kind: &IndexSpec, context: NonEmpty, ) -> Result, Error> where K: DataKeySource + Sync + 'static, + P: Plaintext, D: IntoPrfContext<'c>, { - if let (IndexSpec::Match(options), Scalar::Text(text)) = (kind, &scalar) { - // Options the index cannot use are refused before the text is - // tokenised, as the record path refuses them. - let _mask = options - .validate() - .map_err(|error| Error::Cipher(error.into()))?; - if QueryTokens::under(text, options).is_empty() { - return Err(Error::Cipher(TermError::EmptyTermText.into())); - } - } - scalar_term(cipher, scalar, kind, context)? + derive(cipher, value, kind, context, Use::Query)? .await .map(TermBytes::into_bytes) .map_err(Error::Cipher) @@ -357,139 +340,160 @@ impl AsRef<[u8]> for TermBytes { } } -/// The one dispatch from a runtime scalar to a typed term operation: which -/// arm a scalar takes decides the term's input encoding, and each arm is -/// the operation a Rust caller would have named for that type, so the -/// bytes are the typed path's. Nothing is requested: a term is derived by -/// the keyset cipher's own PRF, so the pending is ready when it is built. +/// What a derived term is for: a query refuses match text that cuts no +/// token, which a stored value keeps as the empty filter. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Use { + Query, + Store, +} + +/// Walk `value` to its one scalar and derive `kind`'s term there. Nothing +/// is requested: a term is derived by the keyset cipher's own PRF, so the +/// pending is ready when it is built. /// /// # Errors /// -/// [`Error::Term`] if the scheme defines no such term for the scalar. -pub(super) fn scalar_term<'a, 'c, K: 'static, D>( +/// [`Error::Term`] if the value is not one scalar `kind` takes, and +/// [`Error::Cipher`] for a query's match text with no token or text the +/// index cannot normalize, as for [`term`]. +fn derive<'a, 'c, K: 'static, P, D>( cipher: &'a KeysetCipher<'_, K>, - scalar: Scalar, + value: P, kind: &IndexSpec, context: NonEmpty, + purpose: Use, ) -> Result, Error> where + P: Plaintext, D: IntoPrfContext<'c>, { - Ok(match kind { - IndexSpec::Equality => equality_of(cipher, scalar, context)?, - IndexSpec::Match(options) => match_of(cipher, scalar, options, context)?, - IndexSpec::Ore => ore_of(cipher, scalar, context), - IndexSpec::Ope => ope_of(cipher, scalar, context), - }) -} - -fn equality_bytes(term: crate::sem::EqualityTerm) -> TermBytes { - TermBytes(term.into_bytes().to_vec()) -} - -/// [`IndexSpec::Equality`] per scalar: one PRF block over the value, for -/// every integer width, text and bytes. -fn equality_of<'a, 'c, K: 'static, D>( - cipher: &'a KeysetCipher<'_, K>, - scalar: Scalar, - context: NonEmpty, -) -> Result, Error> -where - D: IntoPrfContext<'c>, -{ - let term = match scalar { - Scalar::I32(v) => cipher.equality_term(v, context), - Scalar::I64(v) => cipher.equality_term(v, context), - Scalar::U32(v) => cipher.equality_term(v, context), - Scalar::U64(v) => cipher.equality_term(v, context), - Scalar::Text(t) => cipher.equality_term(String::clone(&t), context), - Scalar::Bytes(b) => cipher.equality_term(Protected::new(Vec::clone(&b)), context), - // No PRF encoding is defined for floats (equality on IEEE-754 - // values is a modelling error) or booleans. - Scalar::Bool(_) | Scalar::F32(_) | Scalar::F64(_) => { - return Err(Error::Term { + value + .visit(OneScalar::new( + |ty: Option, value: Physical<'_>| { + scalar_term(cipher, kind, ty, value, context, purpose) + }, + )) + .map_err(|refused| { + refused.0.unwrap_or(Error::Term { field: None, - kind: IndexSpec::Equality, + kind: kind.clone(), }) - } - }; - Ok(term.map(equality_bytes)) + }) } -/// [`IndexSpec::Match`] per scalar: text only, under the index's options. -/// Under the default options the bytes are the typed -/// `match_terms::`'s; under others, those of a -/// [`MatchConfig`](crate::sem::MatchConfig) returning the same options. -fn match_of<'a, 'c, K: 'static, D>( +/// The term of one scalar: [`IndexSpec::takes`] decides whether there is +/// one, and each arm is the operation a Rust caller would have named for +/// the scalar's type, so the bytes are the typed path's. +/// +/// `None` is a scalar the index does not take; the caller names the index. +fn scalar_term<'a, 'c, K: 'static, D>( cipher: &'a KeysetCipher<'_, K>, - scalar: Scalar, - options: &MatchOptions, + kind: &IndexSpec, + ty: Option, + value: Physical<'_>, context: NonEmpty, -) -> Result, Error> + purpose: Use, +) -> Result, Option> where D: IntoPrfContext<'c>, { - match scalar { - Scalar::Text(t) => Ok(cipher - .match_terms_under::(&t, context, options.clone()) - .map(|terms| TermBytes(terms.to_bytes()))), - _ => Err(Error::Term { - field: None, - kind: IndexSpec::Match(options.clone()), - }), + if !kind.takes(ty, &value) { + return Err(None); + } + let unnormalized = |error: NormalizeError| Some(Error::Cipher(TermError::from(error).into())); + match kind { + IndexSpec::Equality => { + let normalized = normalize(ty, value).map_err(unnormalized)?; + Ok(cipher + .equality_term_normalized(normalized, context) + .map(|term| TermBytes(term.into_bytes().to_vec()))) + } + IndexSpec::Match(options) => { + let Physical::Text(text) = value else { + return Err(None); + }; + if purpose == Use::Query { + // Options the index cannot use are refused before the text + // is tokenised, as the record path refuses them. + let _mask = options + .validate() + .map_err(|error| Some(Error::Cipher(error.into())))?; + } + if purpose == Use::Query + && QueryTokens::under(text, options) + .map_err(|error| Some(Error::Cipher(error.into())))? + .is_empty() + { + return Err(Some(Error::Cipher(TermError::EmptyTermText.into()))); + } + Ok(cipher + .match_terms_under::(text, context, options.clone()) + .map(|terms| TermBytes(terms.to_bytes()))) + } + IndexSpec::Ore => ore_of(cipher, value, context).ok_or(None), + IndexSpec::Ope => ope_of(cipher, value, context).ok_or(None), } } -/// [`IndexSpec::Ore`] per scalar: every scalar has an ORE encoding. +/// [`IndexSpec::Ore`] per scalar, each handed to the CLLW encryptor as the +/// Rust type the typed path would have named; `None` for a scalar it does +/// not order ([`ordered`]). /// -/// The text and bytes arms hand the encryptor the `Zeroizing` operand -/// itself, not a bare clone of its contents: the CLLW encryptors take -/// their value by `'static` ownership (the visitor carries it), so a -/// cloned-out `String`/`Vec` would be freed with the plaintext -/// still in it — in a guest's linear memory, where the host can read -/// it. Keeping the wrapper costs nothing and saves the copy as well. +/// Text and bytes are copied out of the walk into a `Zeroizing` wrapper, +/// and the encryptor is handed the wrapper itself: the CLLW encryptors take +/// their value by `'static` ownership (the visitor carries it), so a bare +/// `String` or `Vec` would be freed with the plaintext still in it, in +/// a guest's linear memory, where the host can read it. fn ore_of<'a, 'c, K: 'static, D>( cipher: &'a KeysetCipher<'_, K>, - scalar: Scalar, + value: Physical<'_>, context: NonEmpty, -) -> Pending<'a, TermBytes, K> +) -> Option> where D: IntoPrfContext<'c>, { - match scalar { - Scalar::Bool(v) => ore(cipher, v, context), - Scalar::I32(v) => ore(cipher, v, context), - Scalar::I64(v) => ore(cipher, v, context), - Scalar::U32(v) => ore(cipher, v, context), - Scalar::U64(v) => ore(cipher, v, context), - Scalar::F32(v) => ore(cipher, v, context), - Scalar::F64(v) => ore(cipher, v, context), - Scalar::Text(t) => ore(cipher, t, context), - Scalar::Bytes(b) => ore(cipher, b, context), - } + Some(match value { + Physical::Int(Int::I8(v)) => ore(cipher, i32::from(v), context), + Physical::Int(Int::I16(v)) => ore(cipher, i32::from(v), context), + Physical::Int(Int::I32(v)) => ore(cipher, v, context), + Physical::Int(Int::I64(v)) => ore(cipher, v, context), + Physical::Int(Int::U8(v)) => ore(cipher, u32::from(v), context), + Physical::Int(Int::U16(v)) => ore(cipher, u32::from(v), context), + Physical::Int(Int::U32(v)) => ore(cipher, v, context), + Physical::Int(Int::U64(v)) => ore(cipher, v, context), + Physical::Float(Float::F32(v)) => ore(cipher, v, context), + Physical::Float(Float::F64(v)) => ore(cipher, v, context), + Physical::Text(t) => ore(cipher, Zeroizing::new(t.to_owned()), context), + Physical::Bytes(b) => ore(cipher, Zeroizing::new(b.to_vec()), context), + _ => return None, + }) } -/// [`IndexSpec::Ope`] per scalar; see [`ore_of`] for why the text and bytes -/// arms pass the wrapper. +/// [`IndexSpec::Ope`] per scalar; see [`ore_of`]. fn ope_of<'a, 'c, K: 'static, D>( cipher: &'a KeysetCipher<'_, K>, - scalar: Scalar, + value: Physical<'_>, context: NonEmpty, -) -> Pending<'a, TermBytes, K> +) -> Option> where D: IntoPrfContext<'c>, { - match scalar { - Scalar::Bool(v) => ope(cipher, v, context), - Scalar::I32(v) => ope(cipher, v, context), - Scalar::I64(v) => ope(cipher, v, context), - Scalar::U32(v) => ope(cipher, v, context), - Scalar::U64(v) => ope(cipher, v, context), - Scalar::F32(v) => ope(cipher, v, context), - Scalar::F64(v) => ope(cipher, v, context), - Scalar::Text(t) => ope(cipher, t, context), - Scalar::Bytes(b) => ope(cipher, b, context), - } + Some(match value { + Physical::Int(Int::I8(v)) => ope(cipher, i32::from(v), context), + Physical::Int(Int::I16(v)) => ope(cipher, i32::from(v), context), + Physical::Int(Int::I32(v)) => ope(cipher, v, context), + Physical::Int(Int::I64(v)) => ope(cipher, v, context), + Physical::Int(Int::U8(v)) => ope(cipher, u32::from(v), context), + Physical::Int(Int::U16(v)) => ope(cipher, u32::from(v), context), + Physical::Int(Int::U32(v)) => ope(cipher, v, context), + Physical::Int(Int::U64(v)) => ope(cipher, v, context), + Physical::Float(Float::F32(v)) => ope(cipher, v, context), + Physical::Float(Float::F64(v)) => ope(cipher, v, context), + Physical::Text(t) => ope(cipher, Zeroizing::new(t.to_owned()), context), + Physical::Bytes(b) => ope(cipher, Zeroizing::new(b.to_vec()), context), + _ => return None, + }) } /// The `AsRef<[u8]>` on the output is what turns the typed CLLW ciphertext @@ -525,42 +529,163 @@ where .map(|term| TermBytes(term.as_ref().to_vec())) } +/// A sink that takes one scalar and hands it to `F`, and refuses anything +/// else: a sequence, a map or a passthrough has no term. +/// +/// It refuses with [`Refused`], whose `None` is "not a scalar this index +/// takes" (the caller names the index) and whose `Some` is the error `F` +/// returned or the walk failed with. +struct OneScalar(F); + +impl OneScalar { + fn new(f: F) -> Self { + Self(f) + } +} + +/// Why a [`OneScalar`] walk stopped. A source that fails (an [`FfiValue`] +/// string that is not UTF-8, which its construction rules out) is `None`, +/// as a value with no term is. +struct Refused(Option); + +impl SinkError for Refused { + fn source(_: SourceError) -> Self { + Self(None) + } +} + +impl Sink for OneScalar +where + F: FnOnce(Option, Physical<'_>) -> Result>, +{ + type Ok = T; + type Error = Refused; + type Seq = NoContainer; + type Map = NoContainer; + + fn scalar(self, ty: Option, value: Physical<'_>) -> Result { + (self.0)(ty, value).map_err(Refused) + } + + fn seq(self, _: Option) -> Result, Refused> { + Err(Refused(None)) + } + + fn map(self, _: Option) -> Result, Refused> { + Err(Refused(None)) + } + + fn passthrough(self, _: P) -> Result { + Err(Refused(None)) + } +} + +/// The container sinks a [`OneScalar`] never makes: it refuses a sequence +/// or a map before one exists, so this type has no values and its methods +/// cannot be called. +enum NoContainer { + #[allow(dead_code)] + Never(Infallible, PhantomData), +} + +impl NoContainer { + fn unreachable(&self) -> ! { + match *self { + NoContainer::Never(never, _) => match never {}, + } + } +} + +impl Sink for NoContainer { + type Ok = T; + type Error = Refused; + type Seq = Self; + type Map = Self; + + fn scalar(self, _: Option, _: Physical<'_>) -> Result { + self.unreachable() + } + + fn seq(self, _: Option) -> Result { + self.unreachable() + } + + fn map(self, _: Option) -> Result { + self.unreachable() + } + + fn passthrough(self, _: P) -> Result { + self.unreachable() + } +} + +impl SeqSink for NoContainer { + type Ok = T; + type Error = Refused; + type Element<'s> + = Self + where + Self: 's; + + fn element(&mut self) -> Result { + self.unreachable() + } + + fn end(self) -> Result { + self.unreachable() + } +} + +impl MapSink for NoContainer { + type Ok = T; + type Error = Refused; + type Entry<'s> + = Self + where + Self: 's; + + fn entry(&mut self, _: &str) -> Result { + self.unreachable() + } + + fn end(self) -> Result { + self.unreachable() + } +} + /// A dynamic error, where an operation's pending can only carry the crate's. fn lifted(error: Error) -> crate::Error { crate::Error::Other(Box::new(error)) } -/// An [`IndexSpec`] is an [`Index`] of a [`Value`]: the index named as data, -/// over a plaintext whose type is known only when it arrives. Its -/// `operation` is `scalar_term`'s dispatch — the one step of the dynamic -/// path that stays dynamic — wrapped as a description, so a field lowered -/// from data runs through the same `indexed()` and `zip` every other field -/// does, and its term is a [`TermBytes`]. A Rust chain over `Value` fields -/// names its indexes this way too (`(IndexSpec::Equality, IndexSpec::Ore)`), -/// and is then the same declaration as a data plan's. +/// An [`IndexSpec`] is an [`Index`] of an [`FfiValue`]: the index named as +/// data, over a plaintext whose type is known only when it arrives. Its +/// `operation` is [`term`]'s derivation wrapped as a description, so a +/// field lowered from data runs through the same `indexed()` and `zip` +/// every other field does, and its term is a [`TermBytes`]. A Rust chain +/// over `FfiValue` fields names its indexes this way too +/// (`(IndexSpec::Equality, IndexSpec::Ore)`), and is then the same +/// declaration as a data plan's. /// -/// A value the scheme defines no such term for (a container, a float under -/// equality) fails the description when it runs; a plan lowered from data -/// refuses it before that, at its boundary ([`IndexSpec::supports`]). -impl Index for IndexSpec { +/// A value the index does not take (a container, a boolean under ORE) +/// fails the description when it runs, with the dynamic [`Error::Term`] +/// inside [`crate::Error::Other`]; a plan lowered from data refuses it +/// before that, at its boundary ([`admits`](super::admits)). +impl Index for IndexSpec { type Term = TermBytes; fn spec(&self) -> IndexSpec { self.clone() } - fn operation<'s, K: 'static, M: ConsumeSource<'s, Value>>( + fn operation<'s, K: 'static, M: ConsumeSource<'s, FfiValue>>( &self, - ) -> Encryption<'s, Value, TermBytes, K, CallerContext, M> { + ) -> Encryption<'s, FfiValue, TermBytes, K, CallerContext, M> { let spec = self.clone(); chosen(move |source: M::Source, cipher, cx: CallerContext| { - let scalar = match Scalar::of(M::view(&source).get(), &spec) { - Ok(scalar) => scalar, - Err(error) => return Pending::failed(cipher, lifted(error)), - }; let context = match cx.validated() { Ok(context) => context, Err(error) => return Pending::failed(cipher, error), }; - match scalar_term(cipher, scalar, &spec, context) { + match derive(cipher, M::view(&source), &spec, context, Use::Store) { Ok(pending) => pending, Err(error) => Pending::failed(cipher, lifted(error)), } @@ -574,6 +699,7 @@ mod tests { use crate::dynamic::{context, Output}; use crate::{nonempty, StackCipher}; use stack_kms::FakeDataKeySource; + use vitaminc_protected::Controlled; async fn cipher() -> StackCipher { StackCipher::builder() @@ -584,9 +710,9 @@ mod tests { } /// A pair the scheme refuses fails the run of `IndexSpec`'s public - /// `Index` with the dynamic `Error::Term` inside the crate error, - /// and requests nothing: the record path refuses such a pair at its - /// boundary, a Rust caller running the index directly meets it here. + /// `Index` with the dynamic `Error::Term` inside the crate + /// error, and requests nothing: the record path refuses such a pair at + /// its boundary, a Rust caller running the index directly meets it here. #[tokio::test] async fn a_refused_pair_fails_the_index_when_it_runs() { let cipher = cipher().await; @@ -594,8 +720,8 @@ mod tests { let ctx = || CallerContext::from(nonempty!("users/age")); for (what, value, kind) in [ ( - "a float under equality", - FfiValue::Float64(1.5), + "a boolean under equality", + FfiValue::Bool(true), IndexSpec::Equality, ), ( @@ -603,16 +729,17 @@ mod tests { FfiValue::UInt32(34), default_match(), ), + ("a boolean under ore", FfiValue::Bool(true), IndexSpec::Ore), ( - "a container under ore", + "a container under ope", FfiValue::Array(vec![FfiValue::UInt32(1)]), - IndexSpec::Ore, + IndexSpec::Ope, ), ] { let refused = keyset .run( - Index::::operation::(&kind), - Value::new(value), + Index::::operation::(&kind), + value, ctx(), ) .await; @@ -630,18 +757,17 @@ mod tests { } /// The query path refuses match text with no tokens; the record path, - /// which runs the same dispatch as an index of a `Value`, stores it as - /// the empty filter. Same scalar, same options, two answers, because a - /// query with no tokens would match every row and a stored value with - /// none is matched by no query. + /// which runs the same derivation as an index of an `FfiValue`, stores + /// it as the empty filter. Same value, same options, two answers, + /// because a query with no tokens would match every row and a stored + /// value with none is matched by no query. #[tokio::test] async fn match_text_with_no_tokens_is_refused_as_a_query_and_stored_empty() { let cipher = cipher().await; let keyset = cipher.default_keyset(); let ctx = context(s("users/bio")).expect("context"); for text in ["", " ", "hi"] { - let scalar = Scalar::of(&s(text), &default_match()).expect("a scalar"); - let refused = term(&keyset, scalar, &default_match(), ctx.clone()).await; + let refused = term(&keyset, &s(text), &default_match(), ctx.clone()).await; assert!( matches!( refused, @@ -652,10 +778,10 @@ mod tests { let stored: TermBytes = keyset .run( - Index::::operation::( + Index::::operation::( &default_match(), ), - Value::new(s(text)), + s(text), CallerContext::from(nonempty!("users/bio")), ) .await @@ -663,8 +789,9 @@ mod tests { assert!(stored.as_bytes().is_empty(), "{text:?} is the empty filter"); } // Text with a token is a query like any other. - let scalar = Scalar::of(&s("hello"), &default_match()).expect("a scalar"); - assert!(term(&keyset, scalar, &default_match(), ctx).await.is_ok()); + assert!(term(&keyset, &s("hello"), &default_match(), ctx) + .await + .is_ok()); } /// Options the index cannot use are refused before the query's text is @@ -679,8 +806,7 @@ mod tests { tokenizer: Tokenizer::Ngram { length: 0 }, ..MatchOptions::default() }); - let scalar = Scalar::of(&s("hello"), &zero).expect("a scalar"); - let refused = term(&keyset, scalar, &zero, ctx).await; + let refused = term(&keyset, &s("hello"), &zero, ctx).await; assert!( matches!( refused, @@ -695,7 +821,7 @@ mod tests { /// A `TermBytes` reads as the typed term's bytes, through every /// accessor: `as_bytes`, `as_ref` and `into_bytes` give the same /// 32 PRF bytes an `EqualityTerm` holds, for a term derived through - /// `IndexSpec`'s `Index`. + /// `IndexSpec`'s `Index`. #[tokio::test] async fn term_bytes_read_as_the_typed_terms_bytes() { let cipher = cipher().await; @@ -710,10 +836,10 @@ mod tests { let through_value: TermBytes = keyset .run( - Index::::operation::( + Index::::operation::( &IndexSpec::Equality, ), - Value::new(FfiValue::UInt32(34)), + FfiValue::UInt32(34), CallerContext::from(nonempty!("users/age")), ) .await @@ -734,10 +860,10 @@ mod tests { } fn bytes(value: &[u8]) -> FfiValue { - FfiValue::Bytes(Protected::new(value.to_vec())) + FfiValue::Bytes(vitaminc_protected::Protected::new(value.to_vec())) } - /// Every scalar variant, from the leaf it lifts out of. + /// One value of every scalar variant. fn every_scalar() -> Vec<(&'static str, FfiValue)> { vec![ ("a bool", FfiValue::Bool(true)), @@ -755,7 +881,7 @@ mod tests { mod given_a_scalar_the_scheme_defines_the_term_for { use super::*; - /// The contract the dispatch exists for: the bytes are the typed + /// The contract the derivation exists for: the bytes are the typed /// path's, so a probe from any language finds a Rust-written term. #[tokio::test] async fn derives_the_bytes_the_typed_path_derives() { @@ -763,64 +889,31 @@ mod tests { let keyset = cipher.default_keyset(); let ctx = context(s("users/x")).expect("context"); let ks = &keyset; - let dynamic = |value: &FfiValue, kind: IndexSpec| { - let scalar = Scalar::of(value, &kind).expect("a scalar"); + let dynamic = |value: FfiValue, kind: IndexSpec| { let ctx = ctx.clone(); - async move { term(ks, scalar, &kind, ctx).await } + async move { term(ks, &value, &kind, ctx).await } }; let eq = |t: crate::sem::EqualityTerm| t.into_bytes().to_vec(); - // Equality, per PRF-encodable variant. - let typed = keyset.equality_term(-3i32, nonempty!("users/x")).await; - assert_eq!( - dynamic(&FfiValue::Int32(-3), IndexSpec::Equality) - .await - .expect("eq"), - eq(typed.expect("typed")), - "i32 equality" - ); - let typed = keyset.equality_term(-4i64, nonempty!("users/x")).await; - assert_eq!( - dynamic(&FfiValue::Int64(-4), IndexSpec::Equality) - .await - .expect("eq"), - eq(typed.expect("typed")), - "i64 equality" - ); - let typed = keyset.equality_term(34u32, nonempty!("users/x")).await; - assert_eq!( - dynamic(&FfiValue::UInt32(34), IndexSpec::Equality) - .await - .expect("eq"), - eq(typed.expect("typed")), - "u32 equality" - ); - let typed = keyset.equality_term(35u64, nonempty!("users/x")).await; - assert_eq!( - dynamic(&FfiValue::UInt64(35), IndexSpec::Equality) - .await - .expect("eq"), - eq(typed.expect("typed")), - "u64 equality" - ); - let typed = keyset - .equality_term("alice".to_string(), nonempty!("users/x")) - .await; - assert_eq!( - dynamic(&s("alice"), IndexSpec::Equality).await.expect("eq"), - eq(typed.expect("typed")), - "text equality" - ); - let typed = keyset - .equality_term(Protected::new(b"ab".to_vec()), nonempty!("users/x")) - .await; - assert_eq!( - dynamic(&bytes(b"ab"), IndexSpec::Equality) - .await - .expect("eq"), - eq(typed.expect("typed")), - "bytes equality" - ); + // Equality, per variant with an equality domain. + macro_rules! equal { + ($value:expr, $leaf:expr, $label:literal) => { + let typed = keyset.equality_term($value, nonempty!("users/x")).await; + assert_eq!( + dynamic($leaf, IndexSpec::Equality).await.expect("eq"), + eq(typed.expect("typed")), + concat!($label, " equality") + ); + }; + } + equal!(-3i32, FfiValue::Int32(-3), "i32"); + equal!(-4i64, FfiValue::Int64(-4), "i64"); + equal!(34u32, FfiValue::UInt32(34), "u32"); + equal!(35u64, FfiValue::UInt64(35), "u64"); + equal!(1.5f32, FfiValue::Float32(1.5), "f32"); + equal!(2.5f64, FfiValue::Float64(2.5), "f64"); + equal!("alice".to_string(), s("alice"), "text"); + equal!(b"ab".to_vec(), bytes(b"ab"), "bytes"); // Match, text only. let typed = keyset @@ -828,32 +921,32 @@ mod tests { .await .expect("typed"); assert_eq!( - dynamic(&s("alice smith"), default_match()) + dynamic(s("alice smith"), default_match()) .await .expect("match"), typed.to_bytes(), "text match" ); - // The ordering schemes take every scalar; the typed side is - // spelled once per variant because each is its own type. + // The ordering schemes take every scalar but a boolean; the + // typed side is spelled once per variant because each is its + // own type. macro_rules! ordered { ($value:expr, $leaf:expr, $label:literal) => { let typed = keyset.ore_term($value, nonempty!("users/x")).await; assert_eq!( - dynamic(&$leaf, IndexSpec::Ore).await.expect("ore"), + dynamic($leaf, IndexSpec::Ore).await.expect("ore"), typed.expect("typed").as_ref().to_vec(), concat!($label, " ore") ); let typed = keyset.ope_term($value, nonempty!("users/x")).await; assert_eq!( - dynamic(&$leaf, IndexSpec::Ope).await.expect("ope"), + dynamic($leaf, IndexSpec::Ope).await.expect("ope"), typed.expect("typed").as_ref().to_vec(), concat!($label, " ope") ); }; } - ordered!(true, FfiValue::Bool(true), "bool"); ordered!(-3i32, FfiValue::Int32(-3), "i32"); ordered!(-4i64, FfiValue::Int64(-4), "i64"); ordered!(34u32, FfiValue::UInt32(34), "u32"); @@ -864,32 +957,125 @@ mod tests { ordered!(b"ab".to_vec(), bytes(b"ab"), "bytes"); } - #[test] - fn supports_is_true() { - for (label, leaf) in every_scalar() { - let scalar = Scalar::of(&leaf, &IndexSpec::Ore).expect("a scalar"); - assert!( - IndexSpec::Ore.supports(&scalar), - "{label} takes an ore term" - ); - assert!( - IndexSpec::Ope.supports(&scalar), - "{label} takes an ope term" + /// Equality normalizes: every integer width of one number is one + /// term, and text in two Unicode forms of one string is one term, + /// whichever path derives it. + #[tokio::test] + async fn equality_is_one_term_per_number_and_per_nfc_string() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/x")).expect("context"); + let eq = |value: FfiValue| { + let ctx = ctx.clone(); + let keyset = &keyset; + async move { + term(keyset, &value, &IndexSpec::Equality, ctx) + .await + .expect("eq") + } + }; + let thirty_four = eq(FfiValue::UInt32(34)).await; + for width in [ + FfiValue::Int32(34), + FfiValue::Int64(34), + FfiValue::UInt64(34), + ] { + assert_eq!(eq(width).await, thirty_four, "34 at every width"); + } + assert_ne!( + eq(FfiValue::Int64(-34)).await, + thirty_four, + "the sign counts" + ); + assert_ne!( + eq(FfiValue::Float64(34.0)).await, + thirty_four, + "a float is not the integer it equals" + ); + let composed = eq(s("cafe\u{301}")).await; + assert_eq!(composed, eq(s("caf\u{e9}")).await, "one string in NFC"); + assert_eq!( + composed, + keyset + .equality_term("caf\u{e9}", nonempty!("users/x")) + .await + .expect("typed") + .into_bytes() + .to_vec(), + "the typed path normalizes too" + ); + } + + /// A transport frame is walked in place to the same term an + /// `FfiValue` derives, at every width a host sends: an 8- or 16-bit + /// integer is read as the 32-bit integer of its sign, as an + /// `FfiValue` reads it. + #[tokio::test] + async fn a_frame_derives_the_terms_its_value_does() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/x")).expect("context"); + let derive = |frame: Vec, kind: IndexSpec| { + let ctx = ctx.clone(); + let keyset = &keyset; + async move { + let frame = vitaminc_aead::Frame::parse(&frame).expect("a frame"); + term(keyset, frame, &kind, ctx).await.expect("derives") + } + }; + let of_value = |value: FfiValue, kind: IndexSpec| { + let ctx = ctx.clone(); + let keyset = &keyset; + async move { term(keyset, &value, &kind, ctx).await.expect("derives") } + }; + for kind in [IndexSpec::Equality, IndexSpec::Ore, IndexSpec::Ope] { + for (frame, value) in [ + (vitaminc_aead::to_frame(-8i8), FfiValue::Int32(-8)), + (vitaminc_aead::to_frame(-300i16), FfiValue::Int32(-300)), + (vitaminc_aead::to_frame(300u16), FfiValue::UInt32(300)), + (vitaminc_aead::to_frame(34u64), FfiValue::UInt64(34)), + (vitaminc_aead::to_frame("alice"), s("alice")), + ] { + let frame = frame.expect("frames"); + assert_eq!( + derive(frame.risky_ref().to_vec(), kind.clone()).await, + of_value(value, kind.clone()).await, + "{kind}" + ); + } + } + // The `UINT8` tag and its byte: no Rust source walks a `u8` as an + // int, but a host's frame can carry one. + for kind in [IndexSpec::Equality, IndexSpec::Ore, IndexSpec::Ope] { + assert_eq!( + derive(vec![vitaminc_aead::tags::UINT8, 200], kind.clone()).await, + of_value(FfiValue::UInt32(200), kind.clone()).await, + "{kind}" ); } + } + + #[test] + fn defines_is_true() { for (label, leaf) in every_scalar() { - let scalar = Scalar::of(&leaf, &IndexSpec::Equality).expect("a scalar"); - let prf_encodable = !matches!( - leaf, - FfiValue::Bool(_) | FfiValue::Float32(_) | FfiValue::Float64(_) + let not_bool = !matches!(leaf, FfiValue::Bool(_)); + assert_eq!( + IndexSpec::Ore.defines(&leaf), + not_bool, + "{label} takes an ore term exactly when it is not a boolean" + ); + assert_eq!( + IndexSpec::Ope.defines(&leaf), + not_bool, + "{label} takes an ope term exactly when it is not a boolean" ); assert_eq!( - IndexSpec::Equality.supports(&scalar), - prf_encodable, - "{label} takes an equality term exactly when it has a PRF encoding" + IndexSpec::Equality.defines(&leaf), + not_bool, + "{label} takes an equality term exactly when it has a domain" ); assert_eq!( - default_match().supports(&scalar), + default_match().defines(&leaf), matches!(leaf, FfiValue::String(_)), "{label} takes a match term exactly when it is text" ); @@ -900,9 +1086,8 @@ mod tests { mod given_a_pair_the_scheme_refuses { use super::*; - /// The arms of `term` are unreachable for a pair `supports` refuses, - /// and they say so with the same error the table would have let a - /// binding raise at its boundary. + /// `term` refuses exactly what `defines` refuses, with the error the + /// table would have let a binding raise at its boundary. #[tokio::test] async fn term_is_error_term_naming_the_kind() { let cipher = cipher().await; @@ -910,32 +1095,74 @@ mod tests { let ctx = context(s("users/x")).expect("context"); let refused = [ ("a bool", FfiValue::Bool(true), IndexSpec::Equality), - ("an f32", FfiValue::Float32(1.5), IndexSpec::Equality), - ("an f64", FfiValue::Float64(2.5), IndexSpec::Equality), ("a bool", FfiValue::Bool(true), default_match()), ("a u32", FfiValue::UInt32(34), default_match()), ("bytes", bytes(b"ab"), default_match()), + ("a bool", FfiValue::Bool(true), IndexSpec::Ore), + ("a bool", FfiValue::Bool(false), IndexSpec::Ope), ]; for (label, leaf, kind) in refused { - let scalar = Scalar::of(&leaf, &kind).expect("a scalar"); - assert!( - !kind.supports(&scalar), - "{label} must not take a {kind} term" - ); - let result = term(&keyset, scalar, &kind, ctx.clone()).await; + assert!(!kind.defines(&leaf), "{label} must not take a {kind} term"); + let result = term(&keyset, &leaf, &kind, ctx.clone()).await; assert!( matches!(&result, Err(Error::Term { kind: k, .. }) if *k == kind), "{label} asked for a {kind} term must be refused as that kind: {result:?}" ); } } + + /// A 128-bit integer has an equality term, from vitaminc, and no + /// order term: the order schemes take what an `FfiValue` holds. + #[tokio::test] + async fn a_128_bit_integer_has_no_order_term() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/x")).expect("context"); + for kind in [IndexSpec::Ore, IndexSpec::Ope] { + assert!(!kind.defines(7u128)); + assert!(matches!( + term(&keyset, 7u128, &kind, ctx.clone()).await, + Err(Error::Term { .. }) + )); + } + assert!(IndexSpec::Equality.defines(7u128)); + assert!(term(&keyset, 7u128, &IndexSpec::Equality, ctx) + .await + .is_ok()); + } + + /// Text with a code point unassigned in Unicode 16 has a domain but + /// no normal form, so equality and match refuse it when they derive. + #[tokio::test] + async fn text_that_does_not_normalize_is_refused_when_derived() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/x")).expect("context"); + for kind in [IndexSpec::Equality, default_match()] { + let unassigned = s("a\u{378}b"); + assert!(kind.defines(&unassigned), "{kind} takes text"); + let result = term(&keyset, &unassigned, &kind, ctx.clone()).await; + assert!( + matches!( + result, + Err(Error::Cipher(crate::Error::Term(TermError::Normalize( + NormalizeError::UnassignedCodePoint + )))) + ), + "{kind}: {result:?}" + ); + } + } } mod given_a_value_that_is_not_a_scalar { use super::*; - #[test] - fn lifting_is_error_term_naming_the_kind() { + #[tokio::test] + async fn term_is_error_term_naming_the_kind() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let ctx = context(s("users/x")).expect("context"); let not_scalars = [ ("null", FfiValue::Null), ("undefined", FfiValue::Undefined), @@ -956,7 +1183,8 @@ mod tests { IndexSpec::Ore, IndexSpec::Ope, ] { - let result = Scalar::of(&value, &kind); + assert!(!kind.defines(&value), "{label} has no {kind} term"); + let result = term(&keyset, &value, &kind, ctx.clone()).await; assert!( matches!(&result, Err(Error::Term { kind: k, .. }) if *k == kind), "{label} has no {kind} term: {result:?}" @@ -1218,10 +1446,10 @@ mod tests { let cipher = cipher().await; let keyset = cipher.default_keyset(); let ctx = context(s("users/x")).expect("context"); - let text = || Scalar::of(&s("Alice Smith"), &default_match()).expect("text"); + let text = || s("Alice Smith"); let wide = term( &keyset, - text(), + &text(), &IndexSpec::Match(non_default_match()), ctx.clone(), ) @@ -1232,32 +1460,12 @@ mod tests { .await .expect("typed"); assert_eq!(wide, typed.to_bytes(), "the options are the typed config's"); - let default = term(&keyset, text(), &default_match(), ctx) + let default = term(&keyset, &text(), &default_match(), ctx) .await .expect("default"); assert_ne!(wide, default, "other options derive other terms"); } } - - mod given_a_scalar_holding_plaintext { - use super::*; - - #[test] - fn debug_prints_none_of_it() { - let rendered = format!( - "{:?}", - Scalar::of(&s("hunter2"), &IndexSpec::Equality).expect("a scalar") - ); - assert!( - !rendered.contains("hunter2"), - "a scalar's Debug must not print its plaintext: {rendered}" - ); - assert!( - rendered.contains("Text"), - "a scalar's Debug names the variant, which is not secret: {rendered}" - ); - } - } } /// The typed side's lowering meets the wire form: an `Index`'s `spec()` is diff --git a/packages/stack-encrypt/tests/fixtures/README.md b/packages/stack-encrypt/tests/fixtures/README.md index c7a90d045..c48fd835f 100644 --- a/packages/stack-encrypt/tests/fixtures/README.md +++ b/packages/stack-encrypt/tests/fixtures/README.md @@ -2,7 +2,7 @@ One declaration, two authors, one engine. A Rust chain (`cipher.encrypt(&user).context("users").fields()…`, over fields of type -`stack_encrypt::dynamic::Value`, the plaintext type a binding's values have) +`stack_encrypt::dynamic::FfiValue`, the plaintext type a binding's values have) and the data-plan lowering (`stack_encrypt::dynamic::record`, what a language binding calls) each sealed the same plaintext under the same declaration. The fixture holds both records, with their term bytes, sealed diff --git a/packages/stack-encrypt/tests/index.rs b/packages/stack-encrypt/tests/index.rs index d5a89ec5b..d3e37fbb3 100644 --- a/packages/stack-encrypt/tests/index.rs +++ b/packages/stack-encrypt/tests/index.rs @@ -38,19 +38,19 @@ fn caller() -> CallerContext { #[cfg(feature = "dynamic")] #[tokio::test] async fn a_vec_of_indexes_is_an_index_set_in_order_and_never_empty() { - use stack_encrypt::dynamic::{FfiValue, Value}; + use stack_encrypt::dynamic::FfiValue; let (cipher, generates, _) = counting_cipher().await; let keyset = cipher.default_keyset(); let list: Vec = vec![IndexSpec::Ore, IndexSpec::Equality]; assert_eq!( - Indexes::::specs(&list), + Indexes::::specs(&list), [IndexSpec::Ore, IndexSpec::Equality], "the specs are the indexes', in the order given" ); assert_eq!( - Indexes::::specs(&vec![IndexSpec::Equality]), + Indexes::::specs(&vec![IndexSpec::Equality]), [IndexSpec::Equality] ); assert_eq!( @@ -61,8 +61,8 @@ async fn a_vec_of_indexes_is_an_index_set_in_order_and_never_empty() { let terms: Vec = keyset .run( - Indexes::::operations::<_, Borrowed>(&list), - &Value::new(FfiValue::UInt32(34)), + Indexes::::operations::<_, Borrowed>(&list), + &FfiValue::UInt32(34), caller(), ) .await diff --git a/packages/stack-encrypt/tests/record_lowering.rs b/packages/stack-encrypt/tests/record_lowering.rs index 7f84e5999..fef703d28 100644 --- a/packages/stack-encrypt/tests/record_lowering.rs +++ b/packages/stack-encrypt/tests/record_lowering.rs @@ -1,7 +1,7 @@ //! The record fixture is the proof of the lowering (ADR-0007, amended //! 2026-10-06): a Rust chain and the data-plan lowering each open the //! records the other sealed, and both derive the same bytes for each term. -//! The chain's fields are `dynamic::Value`s, the plaintext type a binding's +//! The chain's fields are `dynamic::FfiValue`s, the plaintext type a binding's //! values have, so the two authors write one declaration over one type; a //! chain over bare `u32`/`String` fields seals the same tagged leaves (see //! `dynamic::record`). `tests/fixtures/record_lowering.json` holds one @@ -29,7 +29,7 @@ use std::path::PathBuf; use common::{deterministic_cipher, DeterministicSource}; use serde_json::{json, Map, Value as Json}; use stack_encrypt::dynamic::record::{self, Plan as DataPlan}; -use stack_encrypt::dynamic::{FfiValue, Scope, TermBytes, Value}; +use stack_encrypt::dynamic::{FfiValue, Scope, TermBytes}; use stack_encrypt::plan::{pick, FieldValues}; use stack_encrypt::sem::MatchOptions; use stack_encrypt::target::{Encrypted, IndexSpec}; @@ -39,22 +39,22 @@ use vitaminc_protected::Controlled; const SEED: [u8; 32] = *b"stack-encrypt record fixture v1 "; struct User { - age: Value, - email: Value, - notes: Value, + age: FfiValue, + email: FfiValue, + notes: FfiValue, id: u32, } fn user() -> User { User { - age: Value::new(FfiValue::UInt32(34)), - email: Value::new(FfiValue::String("bob@example.com".into())), - notes: Value::new(FfiValue::String("likes cats".into())), + age: FfiValue::UInt32(34), + email: FfiValue::String("bob@example.com".into()), + notes: FfiValue::String("likes cats".into()), id: 42, } } -/// The one declaration, as the Rust chain writes it over `Value` fields. +/// The one declaration, as the Rust chain writes it over `FfiValue` fields. fn typed_plan() -> Plan { Plan::context("users") .fields() @@ -365,8 +365,7 @@ async fn open_typed(cipher: &StackCipher, tree: StackCipher .using(&typed_plan()) .await .expect("the typed chain opens it"); - let mut value = - |name: &str| json_of_opened(opened.take::(name).expect(name).into_inner()); + let mut value = |name: &str| json_of_opened(opened.take::(name).expect(name)); let (age, email, notes) = (value("age"), value("email"), value("notes")); json!({ "age": age, @@ -523,15 +522,15 @@ fn write_fixture(path: &PathBuf, fixture: &Json) { /// context, `text` is sealed and indexed under `/text`, `id` is /// carried through. struct Note { - tenant: Value, - text: Value, + tenant: FfiValue, + text: FfiValue, id: u32, } fn note() -> Note { Note { - tenant: Value::new(FfiValue::String("tenants/acme".into())), - text: Value::new(FfiValue::String("hello".into())), + tenant: FfiValue::String("tenants/acme".into()), + text: FfiValue::String("hello".into()), id: 42, } } @@ -572,16 +571,13 @@ fn globex() -> Label { /// The typed chain's note record, shaped as the stored tree. fn record_of_note(mut values: FieldValues) -> StackCipherText { - let tenant: Value = values.take("tenant").expect("tenant"); + let tenant: FfiValue = values.take("tenant").expect("tenant"); let text: Encrypted = values.take("text").expect("text"); let id: u32 = values.take("id").expect("id"); CipherText::Map(vec![ ( "tenant".into(), - CipherText::Map(vec![( - "passthrough".into(), - passthrough(tenant.into_inner()), - )]), + CipherText::Map(vec![("passthrough".into(), passthrough(tenant))]), ), ( "text".into(), @@ -629,7 +625,7 @@ fn note_of_record(tree: StackCipherText) -> FieldValues { let _ = values.insert(&name, id); } text => { - let _ = values.insert(&name, Value::new(text)); + let _ = values.insert(&name, text); } } } @@ -675,8 +671,7 @@ async fn open_note_typed( Some(expected) => open.context(expected).using(¬e_plan()).await?, None => open.using(¬e_plan()).await?, }; - let mut value = - |name: &str| json_of_opened(opened.take::(name).expect(name).into_inner()); + let mut value = |name: &str| json_of_opened(opened.take::(name).expect(name)); let (tenant, text) = (value("tenant"), value("text")); Ok(json!({ "tenant": tenant, "text": text, "id": opened.take::("id").expect("id") })) } From 6cfba912f1e5abf1a6e5963e8019abc61adec17d Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 02:28:31 +1100 Subject: [PATCH 30/38] feat(golang)!: walk a term's frame in place in the encrypt guest and refuse order terms on a boolean se_term parses the value's frame and hands it to dynamic::term without decoding it into a value tree. The stashgen fake engine and the Go tests follow the engine's new table: a float takes an equality term, a boolean takes no equality or order term. Part of #1140 --- languages/golang/encrypt/guest/Cargo.lock | 79 ++++++++++++++++++- languages/golang/encrypt/guest/Cargo.toml | 1 + languages/golang/encrypt/guest/src/ops.rs | 54 ++++++++----- .../golang/encrypt/guest/tests/native_ops.rs | 15 ++-- languages/golang/encrypt/guest_test.go | 7 +- .../golang/stashgen/enginetest/enginetest.go | 11 +-- languages/golang/stashgen/refusal_test.go | 5 +- 7 files changed, 134 insertions(+), 38 deletions(-) diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 9d0d082b1..56ae2e946 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -218,6 +218,30 @@ dependencies = [ "zeroize", ] +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.4", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -290,6 +314,12 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + [[package]] name = "chacha20" version = "0.10.2" @@ -709,7 +739,6 @@ dependencies = [ "stack-encrypt", "thiserror 2.0.20", "ts-rs", - "vitaminc-prf", "zeroize", ] @@ -1398,6 +1427,14 @@ dependencies = [ "libc", ] +[[package]] +name = "orderable-bytes" +version = "0.1.1" +source = "git+https://github.com/cipherstash/ore.rs?rev=b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5#b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5" +dependencies = [ + "rust_decimal", +] + [[package]] name = "parking_lot" version = "0.12.5" @@ -1782,6 +1819,23 @@ dependencies = [ "serde", ] +[[package]] +name = "rust_decimal" +version = "1.43.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a" +dependencies = [ + "arrayvec", + "borsh", + "bytes", + "num-traits", + "rand 0.8.8", + "rand 0.9.5", + "serde", + "serde_json", + "wasm-bindgen", +] + [[package]] name = "rustc_version" version = "0.4.1" @@ -2087,6 +2141,7 @@ dependencies = [ "vitaminc-aead-frame", "vitaminc-encrypt", "vitaminc-hmac", + "vitaminc-normalize", "vitaminc-prf", "vitaminc-protected", "zeroize", @@ -2463,6 +2518,12 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.24" @@ -2687,6 +2748,20 @@ dependencies = [ "zeroize", ] +[[package]] +name = "vitaminc-normalize" +version = "0.5.1" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" +dependencies = [ + "orderable-bytes", + "rust_decimal", + "thiserror 2.0.20", + "unicode-general-category", + "unicode-normalization", + "vitaminc-aead", + "vitaminc-protected", +] + [[package]] name = "vitaminc-prf" version = "0.5.1" @@ -2694,7 +2769,9 @@ source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove- dependencies = [ "mutants", "thiserror 2.0.20", + "vitaminc-aead", "vitaminc-context", + "vitaminc-normalize", "vitaminc-protected", ] diff --git a/languages/golang/encrypt/guest/Cargo.toml b/languages/golang/encrypt/guest/Cargo.toml index 4cef89332..4e3012d19 100644 --- a/languages/golang/encrypt/guest/Cargo.toml +++ b/languages/golang/encrypt/guest/Cargo.toml @@ -26,6 +26,7 @@ vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "fea vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-normalize = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index ed08a9236..d7bc64d23 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -35,8 +35,15 @@ //! directions, buffers sized before a byte of plaintext is written, the //! ABI's numeric term kinds, and the mapping from a library error to a //! status code. - -use stack_encrypt::dynamic::{self, FfiValue, Scalar, Scope}; +//! +//! A term's value is not decoded at all: the frame is parsed and walked in +//! place to the scalar it holds, the way vitaminc's own guest walks a frame +//! into encryption, so no copy of the plaintext is built to derive it. A +//! record's source, a plan and an options object are decoded into an +//! [`FfiValue`]: the engine reads them, splits a source into fields and +//! keeps each field's value past this call. + +use stack_encrypt::dynamic::{self, FfiValue, Scope}; use stack_encrypt::sem::MatchOptions; use stack_encrypt::target::IndexSpec; use stack_encrypt::ErrorPayload; @@ -44,7 +51,7 @@ use stack_encrypt::{ CipherText, KeysetCipher, Label, Node, Plaintext, SealedValue, StackCipherText, }; use stack_kms::DataKeySource; -use vitaminc_aead_frame::{decode_ciphertext, encode_ciphertext, FrameWriter}; +use vitaminc_aead_frame::{decode_ciphertext, encode_ciphertext, Frame, FrameWriter}; use vitaminc_protected::Controlled; use stack_guest_abi::last_error; @@ -72,7 +79,7 @@ type BytesTree = Node>; /// Derive one index term: a codec-encoded scalar and a codec-encoded /// context in, the term's frozen byte encoding out (see `stack-encrypt`'s -/// `sem` module docs). Under the local HMAC backend the derivation is one +/// `sem` module docs). The scalar's frame is walked in place. Under the local HMAC backend the derivation is one /// PRF/CLLW computation with no ZeroKMS I/O; that is the backend's /// property, not this operation's contract — the term API is a `Pending` /// so a backend that derives terms at ZeroKMS settles the same way. @@ -99,22 +106,22 @@ where // The same proof every stack-encrypt leaf demands: an empty context is // `STATUS_ENCODING` here, before any derivation. let context = dynamic::context(decode_value(context)?).map_err(|e| fail_dynamic(&e))?; - let (scalar, kind) = parse_term(decode_value(value)?, kind)?; - dynamic::term(cipher, scalar, &kind, context) + let value = parse_frame(value)?; + let kind = parse_term(value, kind)?; + dynamic::term(cipher, value, &kind, context) .await .map_err(|e| fail_dynamic(&e)) } -/// The static half of a term: the kind is one of the ABI's table, the value -/// is a scalar, and the scheme defines the pair -/// ([`IndexSpec::supports`]). Shared by [`term`] and [`validate::term`] so -/// the ABI refuses exactly what the operation would, before any keyset is -/// resolved. +/// The static half of a term: the kind is one of the ABI's table, and the +/// value is one scalar the index takes ([`IndexSpec::defines`]). Shared by +/// [`term`] and [`validate::term`] so the ABI refuses exactly what the +/// operation would, before any keyset is resolved. /// /// The ABI's kind codes carry no options, so [`TERM_MATCH`] is the match /// index under the default options, the same index a plan's bare `"match"` /// names. -fn parse_term(value: FfiValue, kind: u32) -> Result<(Scalar, IndexSpec), u32> { +fn parse_term(value: Frame<'_>, kind: u32) -> Result { let kind = match kind { TERM_EQUALITY => IndexSpec::Equality, TERM_MATCH => IndexSpec::Match(MatchOptions::default()), @@ -126,11 +133,10 @@ fn parse_term(value: FfiValue, kind: u32) -> Result<(Scalar, IndexSpec), u32> { )) } }; - let scalar = Scalar::of(&value, &kind).map_err(|e| fail_dynamic(&e))?; - if !kind.supports(&scalar) { + if !kind.defines(value) { return Err(fail_dynamic(&dynamic::Error::Term { field: None, kind })); } - Ok((scalar, kind)) + Ok(kind) } // ============================================================================= @@ -309,13 +315,12 @@ pub mod validate { /// A term's inputs, as [`term`] takes them: the context decodes and is /// non-empty, the kind is one of [`TERM_EQUALITY`] .. [`TERM_OPE`], and - /// the value is a scalar the scheme defines that term for - /// ([`IndexSpec::supports`]). + /// the value is one scalar the index takes ([`IndexSpec::defines`]). pub fn term(value: &[u8], context: &[u8], kind: u32) -> Result<(), u32> { dynamic::context(decode_value(context)?) .map(drop) .map_err(|e| fail_dynamic(&e))?; - parse_term(decode_value(value)?, kind).map(drop) + parse_term(parse_frame(value)?, kind).map(drop) } /// A record source against its plan, as [`encrypt_record`] takes them: @@ -375,8 +380,17 @@ pub mod validate { // ============================================================================= fn decode_value(bytes: &[u8]) -> Result { - FfiValue::from_frame(bytes) - .map_err(|_| last_error::malformed("an input is not a value in the transport frame")) + FfiValue::from_frame(bytes).map_err(|_| not_a_frame()) +} + +/// One frame, parsed and checked but not decoded: what a term's value is +/// walked from, in place. +fn parse_frame(bytes: &[u8]) -> Result, u32> { + Frame::parse(bytes).map_err(|_| not_a_frame()) +} + +fn not_a_frame() -> u32 { + last_error::malformed("an input is not a value in the transport frame") } /// Encode a value tree into a buffer sized **before** the first byte is diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 8cafb754d..d002ad70a 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -347,12 +347,13 @@ fn unsupported_term_inputs_are_encoding_errors() { let cipher = cipher(); let ctx = encode(s("f")); - // Floats and bools have no equality encoding; match is text-only; + // Bools have no equality domain and no order term; match is text-only; // containers have no term semantics; kinds outside the table and empty // contexts are rejected. for (value, kind) in [ - (FfiValue::Float64(1.5), TERM_EQUALITY), (FfiValue::Bool(true), TERM_EQUALITY), + (FfiValue::Bool(true), TERM_ORE), + (FfiValue::Bool(false), TERM_OPE), (FfiValue::UInt32(1), TERM_MATCH), (FfiValue::Array(vec![]), TERM_ORE), (FfiValue::Null, TERM_EQUALITY), @@ -1055,12 +1056,9 @@ fn term_validation_refuses_what_the_term_op_refuses() { let ctx = encode(s("f")); for (label, value, kind) in [ - ( - "a float under equality", - FfiValue::Float64(1.5), - TERM_EQUALITY, - ), ("a bool under equality", FfiValue::Bool(true), TERM_EQUALITY), + ("a bool under ore", FfiValue::Bool(true), TERM_ORE), + ("a bool under ope", FfiValue::Bool(true), TERM_OPE), ("an integer under match", FfiValue::UInt32(1), TERM_MATCH), ("a container", FfiValue::Array(vec![]), TERM_ORE), ("an object", obj(vec![("k", s("v"))]), TERM_EQUALITY), @@ -1101,8 +1099,9 @@ fn term_validation_refuses_what_the_term_op_refuses() { (s("x"), TERM_EQUALITY), (FfiValue::Bytes(Protected::new(vec![1])), TERM_EQUALITY), (s("x y"), TERM_MATCH), + (FfiValue::Float64(1.5), TERM_EQUALITY), (FfiValue::Float64(1.5), TERM_ORE), - (FfiValue::Bool(true), TERM_OPE), + (FfiValue::Int32(-1), TERM_OPE), (s("x"), TERM_ORE), ] { assert_eq!(ops::validate::term(&encode(value), &ctx, kind), Ok(())); diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index f6863f62a..11a21dfdb 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -734,10 +734,10 @@ func TestGuestRefusesMalformedInputsBeforeState(t *testing.T) { c := rawInstance(t) def := c.DefaultKeyset() plan := usersPlan() - floatAge := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Ciphertext, record.Equality}}}} + boolFlag := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "flag", Kind: record.Bool, Outputs: []record.Output{record.Ciphertext, record.Equality}}}} matchInt := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Match}}}} calls := map[string]func() error{ - "float under equality": func() error { _, err := def.Derive(ctx, floatAge, "age", record.Equality, 1.5); return err }, + "bool under equality": func() error { _, err := def.Derive(ctx, boolFlag, "flag", record.Equality, true); return err }, "integer under match": func() error { _, err := def.Derive(ctx, matchInt, "age", record.Match, uint32(1)); return err }, "container as term value": func() error { _, err := def.Derive(ctx, plan, "age", record.Equality, []any{1}); return err }, "a kind the type refuses": func() error { _, err := def.Seal(ctx, matchInt, []record.Source{usersRow(1, "x")}); return err }, @@ -837,7 +837,8 @@ func TestPlanCheckAnswersWithoutACipher(t *testing.T) { } refused := []*record.Plan{ {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Match}}}}, - {Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Float64, Outputs: []record.Output{record.Equality}}}}, + {Context: []string{"users"}, Fields: []record.Field{{Name: "flag", Kind: record.Bool, Outputs: []record.Output{record.Equality}}}}, + {Context: []string{"users"}, Fields: []record.Field{{Name: "flag", Kind: record.Bool, Outputs: []record.Output{record.Ore}}}}, } for i, p := range refused { if err := k.Check(ctx, p); !errors.Is(err, ErrEncoding) { diff --git a/languages/golang/stashgen/enginetest/enginetest.go b/languages/golang/stashgen/enginetest/enginetest.go index 5a817c87a..efb759584 100644 --- a/languages/golang/stashgen/enginetest/enginetest.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -65,14 +65,15 @@ func (e Static) Check(ctx context.Context, d stashgen.Declaration) error { } // indexApplies mirrors dynamic::admits in stack-encrypt: equality over -// integers, text and bytes (no floats, no booleans: they have no PRF -// encoding), match over text alone, ORE and OPE over every scalar. The JSON -// index is not in the engine yet. +// integers, floats, text and bytes (the kinds vitaminc gives an equality +// domain; a boolean has none), match over text alone, ORE and OPE over +// every scalar but a boolean, whose order term would reveal which rows are +// true. The JSON index is not in the engine yet. func indexApplies(name stashgen.IndexName, t stashgen.GoType) error { switch name { case stashgen.IndexEquality: switch t.Kind { - case stashgen.KindString, stashgen.KindBytes, stashgen.KindInt, stashgen.KindUint: + case stashgen.KindString, stashgen.KindBytes, stashgen.KindInt, stashgen.KindUint, stashgen.KindFloat: return nil } case stashgen.IndexMatch: @@ -81,7 +82,7 @@ func indexApplies(name stashgen.IndexName, t stashgen.GoType) error { } return fmt.Errorf("match applies to a string, not to %s", t) case stashgen.IndexOre, stashgen.IndexOpe: - if t.Kind.Scalar() { + if t.Kind.Scalar() && t.Kind != stashgen.KindBool { return nil } case stashgen.IndexJSON: diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index b962c0084..3b5bcba0d 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -265,8 +265,9 @@ func TestRefusalsHoldAgainstTheEmbeddedEngine(t *testing.T) { {"equality on a map", "Attrs", "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`", "seals only as part of an opaque struct"}, {"a sealed []string", "Tags", "\tTags []string `stash:\"tags,encrypt\"`", "seals only as part of an opaque struct"}, {"a sealed map[string]int64", "Counts", "\tCounts map[string]int64 `stash:\"counts,encrypt\"`", "seals only as part of an opaque struct"}, - {"equality on a float", "Score", "\tScore float64 `stash:\"score,encrypt,index=equality\"`", "the engine refuses the declaration"}, {"equality on a bool", "Done", "\tDone bool `stash:\"done,encrypt,index=equality\"`", "the engine refuses the declaration"}, + {"order on a bool", "Done", "\tDone bool `stash:\"done,encrypt,index=ore\"`", "the engine refuses the declaration"}, + {"ope on a bool", "Done", "\tDone bool `stash:\"done,encrypt,index=ope\"`", "the engine refuses the declaration"}, {"match on an integer", "Age", "\tAge int32 `stash:\"age,encrypt,index=match\"`", "the engine refuses the declaration"}, {"encrypt_into, no EQL type in this build", "Email", "\tEmail string `stash:\"email,encrypt_into=TextEq\"`", "EQL types are not available yet"}, } { @@ -284,6 +285,8 @@ func TestRefusalsHoldAgainstTheEmbeddedEngine(t *testing.T) { } // What the fake refuses, the engine refuses: the same cases against both. for _, c := range []struct{ name, line string }{ + {"equality on a bool", "\tDone bool `stash:\"done,encrypt,index=equality\"`"}, + {"order on a bool", "\tDone bool `stash:\"done,encrypt,index=ore\"`"}, {"equality on a float", "\tScore float64 `stash:\"score,encrypt,index=equality\"`"}, {"a declaration the engine runs", "\tAge uint32 `stash:\"age,encrypt,index=equality;ore\"`\n\tName string `stash:\"name,encrypt,index=match\"`"}, } { From 98ba142170d42a234e1d99e897500975dfc6b532 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 02:28:32 +1100 Subject: [PATCH 31/38] feat(eql)!: compare TextEq text in Unicode 16 NFC through Stack Encrypt's normalized equality term Hmac256 encrypts from any Plaintext, eql-bindings no longer depends on vitaminc-prf, the EQL workspace patches vitaminc-normalize, and the TextEqQuery fixture the Go SDK reads is regenerated. Part of #1140 --- .changeset/eql-text-equality-transcoding.md | 2 +- packages/eql/Cargo.lock | 32 ++++++++++++++++--- packages/eql/Cargo.toml | 1 + packages/eql/crates/eql-bindings/CHANGELOG.md | 9 ++++++ packages/eql/crates/eql-bindings/Cargo.toml | 6 ---- .../eql/crates/eql-bindings/src/encryption.rs | 9 +++--- .../encryption/fixtures/text_eq_query.json | 2 +- .../eql/tests/encryption/tests/text_eq.rs | 27 ++++++++++++---- 8 files changed, 65 insertions(+), 23 deletions(-) diff --git a/.changeset/eql-text-equality-transcoding.md b/.changeset/eql-text-equality-transcoding.md index 969aa9653..5dc9bba90 100644 --- a/.changeset/eql-text-equality-transcoding.md +++ b/.changeset/eql-text-equality-transcoding.md @@ -2,4 +2,4 @@ '@cipherstash/eql': minor --- -**Rust `TextEq` and `TextEqQuery` support Stack Encrypt's target-directed API behind the `stack-encrypt` feature.** The stored table and column identifier supplies the encryption context; native ciphertext and equality terms are transcoded into EQL payloads while Vitamin C owns plaintext encoding. This is a new producer profile with exact string equality, independent of existing cipherstash-client ciphertext and terms. Query operands carry no recoverable ciphertext. +**Rust `TextEq` and `TextEqQuery` support Stack Encrypt's target-directed API behind the `stack-encrypt` feature.** The stored table and column identifier supplies the encryption context; native ciphertext and equality terms are transcoded into EQL payloads while Vitamin C owns plaintext encoding. This is a new producer profile whose string equality compares text in Unicode 16 NFC without folding case, independent of existing cipherstash-client ciphertext and terms. Query operands carry no recoverable ciphertext. diff --git a/packages/eql/Cargo.lock b/packages/eql/Cargo.lock index c4451d716..fc561d76f 100644 --- a/packages/eql/Cargo.lock +++ b/packages/eql/Cargo.lock @@ -581,7 +581,7 @@ dependencies = [ "log", "miette", "opaque-debug", - "orderable-bytes", + "orderable-bytes 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", "ore-rs", "percent-encoding", "rand 0.8.6", @@ -649,7 +649,7 @@ dependencies = [ "blake3", "chrono", "hex", - "orderable-bytes", + "orderable-bytes 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", "rust_decimal", "subtle", "thiserror 1.0.69", @@ -1204,7 +1204,6 @@ dependencies = [ "stack-encrypt", "thiserror 2.0.20", "ts-rs", - "vitaminc-prf", "zeroize", ] @@ -2660,6 +2659,14 @@ dependencies = [ "rust_decimal", ] +[[package]] +name = "orderable-bytes" +version = "0.1.1" +source = "git+https://github.com/cipherstash/ore.rs?rev=b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5#b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5" +dependencies = [ + "rust_decimal", +] + [[package]] name = "ore-rs" version = "0.8.3" @@ -2673,7 +2680,7 @@ dependencies = [ "hex", "lazy_static", "num", - "orderable-bytes", + "orderable-bytes 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", "rand 0.8.6", "rand_chacha 0.3.1", "rust_decimal", @@ -4242,6 +4249,7 @@ dependencies = [ "vitaminc-aead-frame", "vitaminc-encrypt 0.5.1", "vitaminc-hmac", + "vitaminc-normalize", "vitaminc-prf", "vitaminc-protected 0.5.1", "zeroize", @@ -5136,6 +5144,20 @@ dependencies = [ "zeroize", ] +[[package]] +name = "vitaminc-normalize" +version = "0.5.1" +source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove-legacy-traits#2e895cb9caedeb04d1adc7d98fc3b067d24fd35d" +dependencies = [ + "orderable-bytes 0.1.1 (git+https://github.com/cipherstash/ore.rs?rev=b3b7800b9c0cd8f7c53c5382f8e018a4e8bcbbd5)", + "rust_decimal", + "thiserror 2.0.20", + "unicode-general-category", + "unicode-normalization", + "vitaminc-aead 0.5.1", + "vitaminc-protected 0.5.1", +] + [[package]] name = "vitaminc-prf" version = "0.5.1" @@ -5143,7 +5165,9 @@ source = "git+https://github.com/cipherstash/vitaminc?branch=feat%2Faead-remove- dependencies = [ "mutants", "thiserror 2.0.20", + "vitaminc-aead 0.5.1", "vitaminc-context", + "vitaminc-normalize", "vitaminc-protected 0.5.1", ] diff --git a/packages/eql/Cargo.toml b/packages/eql/Cargo.toml index e12e2ec0c..805f00553 100644 --- a/packages/eql/Cargo.toml +++ b/packages/eql/Cargo.toml @@ -47,5 +47,6 @@ vitaminc-aead = { git = "https://github.com/cipherstash/vitaminc", branch = "fea vitaminc-aead-frame = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-encrypt = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-hmac = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } +vitaminc-normalize = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-prf = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } vitaminc-protected = { git = "https://github.com/cipherstash/vitaminc", branch = "feat/aead-remove-legacy-traits" } diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index c4ea4cf1e..b9fc603ee 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -35,6 +35,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **`TextEq` and `TextEqQuery` compare text in Unicode 16 NFC** + (`stack-encrypt` feature). Their `hm` is Stack Encrypt's equality term, + which now derives through `vitaminc-normalize`: `café` written with a + precomposed `é` and with `e` and a combining accent is one value, and + text holding a code point unassigned in Unicode 16 is refused. Case and + whitespace are kept. Every `hm` changes bytes; no stored value existed to + keep. `Hmac256` encrypts from any `Plaintext` where it took a + `PrfValue`, and the feature no longer depends on `vitaminc-prf`. + - **The `stack-encrypt` feature builds on vitaminc ADR 0001.** `encryption::targets` takes `FfiValue` and `ValueKind` from `stack_encrypt::dynamic` (the feature turns on stack-encrypt's `dynamic` diff --git a/packages/eql/crates/eql-bindings/Cargo.toml b/packages/eql/crates/eql-bindings/Cargo.toml index e4b2023b7..e5c7f90ce 100644 --- a/packages/eql/crates/eql-bindings/Cargo.toml +++ b/packages/eql/crates/eql-bindings/Cargo.toml @@ -47,11 +47,6 @@ schemars = "1" # target takes and the kind names a plan's `"type"` key spells # (`encryption::targets`). stack-encrypt = { path = "../../../stack-encrypt", version = "0.2.0", optional = true, default-features = false, features = ["dynamic"] } -# The vitaminc line stack-encrypt is built on. `PrfValue` is named from this -# crate directly (stack-encrypt re-exports only `IntoPrfContext` and -# `PrfContext`), and two vitaminc lines in one graph do not interoperate, so -# the requirement is stack-encrypt's. -vitaminc-prf = { version = "0.5.0", optional = true } # The plaintext a target reads out of its `Protected` runtime value is held # in `Zeroizing` until it is sealed, so the copy is wiped like the original. zeroize = { version = "1", optional = true } @@ -63,7 +58,6 @@ hex = { version = "0.4", optional = true } default = [] stack-encrypt = [ "dep:stack-encrypt", - "dep:vitaminc-prf", "dep:zeroize", "dep:thiserror", "dep:base64", diff --git a/packages/eql/crates/eql-bindings/src/encryption.rs b/packages/eql/crates/eql-bindings/src/encryption.rs index 356be754e..48820d7c0 100644 --- a/packages/eql/crates/eql-bindings/src/encryption.rs +++ b/packages/eql/crates/eql-bindings/src/encryption.rs @@ -2,8 +2,10 @@ //! //! The target declares its operations; Stack Encrypt invokes Vitamin C's //! plaintext implementations. These adapters only encode encrypted outputs. -//! Equality uses Vitamin C's exact string semantics: no normalization or case -//! folding. The stored identifier supplies context for ciphertext and terms. +//! Equality compares text in Unicode 16 NFC, the normal form +//! `vitaminc-normalize` gives it, and folds no case: `café` written with a +//! precomposed `é` and with `e` and a combining accent is one value, `Café` +//! another. The stored identifier supplies context for ciphertext and terms. //! //! This is a new producer profile, independent of existing cipherstash-client //! ciphertext and terms. The EQL envelope and SQL domains remain unchanged. @@ -88,7 +90,6 @@ use stack_encrypt::{ Describe, Description, EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty, NonEmpty, Plaintext, SealedValue, StackCipherText, }; -use vitaminc_prf::PrfValue; use crate::{ v3::terms::{Ciphertext, Hmac256}, @@ -232,7 +233,7 @@ impl Transcode for Hmac256 { EqualityVisitor } } -impl EncryptFrom for Hmac256 { +impl EncryptFrom for Hmac256 { type Context = CallerContext; fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> where diff --git a/packages/eql/tests/encryption/fixtures/text_eq_query.json b/packages/eql/tests/encryption/fixtures/text_eq_query.json index ed65759f2..9f4c67044 100644 --- a/packages/eql/tests/encryption/fixtures/text_eq_query.json +++ b/packages/eql/tests/encryption/fixtures/text_eq_query.json @@ -2,6 +2,6 @@ "_comment": "The TextEqQuery eql-bindings derives for the plaintext under FakeDataKeySource's index key (keyset nil), read by the Go SDK's hermetic test; regenerate with EQL_UPDATE_FIXTURES=1, do not edit.", "column": "email", "plaintext": "bob@example.com", - "query": "{\"v\":3,\"i\":{\"t\":\"users\",\"c\":\"email\"},\"hm\":\"ec534cfbf2336785378c3761ff88623c46361b2ba7c5fa83930ebacae49cf646\"}", + "query": "{\"v\":3,\"i\":{\"t\":\"users\",\"c\":\"email\"},\"hm\":\"0cf708c07acaaa169e84e4adf7081f0433d87760bd7e9c63bd7a58c000b8d33b\"}", "table": "users" } diff --git a/packages/eql/tests/encryption/tests/text_eq.rs b/packages/eql/tests/encryption/tests/text_eq.rs index 0691d5a57..a20ee6908 100644 --- a/packages/eql/tests/encryption/tests/text_eq.rs +++ b/packages/eql/tests/encryption/tests/text_eq.rs @@ -119,13 +119,26 @@ async fn independent_queries_match_native_terms_without_requesting_data_keys() { let json = serde_json::to_value(&query).unwrap(); assert!(json.get("c").is_none()); assert_eq!(serde_json::from_value::(json).unwrap(), query); - for different in ["CAFÉ", "cafe\u{301}", "café "] { - let probe: TextEqQuery = reader - .default_keyset() - .encrypt_as(&different.to_owned(), column()) - .await - .unwrap(); - assert_ne!(query.hm, probe.hm, "no added normalization or case folding"); + let probe = |text: &str| { + let text = text.to_owned(); + let keyset = reader.default_keyset(); + async move { + let probe: TextEqQuery = keyset.encrypt_as(&text, column()).await.unwrap(); + probe.hm + } + }; + // Unicode 16 NFC: the same text in its other normal form is one value. + assert_eq!( + probe("cafe\u{301}").await, + probe("caf\u{e9}").await, + "text is compared in NFC" + ); + for different in ["CAFÉ", "café "] { + assert_ne!( + query.hm, + probe(different).await, + "no case folding or trimming" + ); } let calls = calls.lock().unwrap(); assert!(calls.generate.is_empty()); From 0f58c9d087e685368d150e0579e94ade5d77c9fc Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 03:13:19 +1100 Subject: [PATCH 32/38] feat(stack-encrypt): let a binding query a plan field by operator, and verify and size a probabilistic query with no key A binding names a query by field, operator and input, as ADR-0009 has a Rust caller name it by operator type: dynamic::query_term reads the input as the field's declared type, finds the index that answers by the typed path's default table (OpSpec::answered_by), and derives the bytes the typed query chain derives. An operator the field does not answer is refused as a mistyped value is, naming the field. dynamic::verify and dynamic::target_limit run the typed MatchQuery's code under the field's own match options, and the catalog entry of each EQL type lists the operators the engine answers for it with their fidelity. tests/fixtures/query_operators.json records each term, a verified page and fetch limits, checked against the typed chain, for the Go SDK's tests. Part of #1188 --- .../eql-bindings/src/encryption/targets.rs | 5 + packages/stack-encrypt/CHANGELOG.md | 19 + packages/stack-encrypt/src/dynamic/mod.rs | 40 +- packages/stack-encrypt/src/dynamic/query.rs | 1828 +++++++++++++++++ packages/stack-encrypt/src/dynamic/record.rs | 2 +- packages/stack-encrypt/src/dynamic/target.rs | 56 +- packages/stack-encrypt/src/dynamic/term.rs | 4 +- packages/stack-encrypt/src/sem/mod.rs | 53 +- .../stack-encrypt/tests/fixtures/README.md | 52 + .../tests/fixtures/query_operators.json | 283 +++ .../stack-encrypt/tests/query_operators.rs | 354 ++++ .../ui/op_at_path_inherits_fidelity.stderr | 4 +- .../tests/ui/op_between_takes_a_pair.stderr | 8 +- ...nexact_operator_needs_probabilistic.stderr | 4 +- .../tests/ui/op_json_not_answered.stderr | 10 +- .../tests/ui/op_order_on_text.stderr | 4 +- 16 files changed, 2688 insertions(+), 38 deletions(-) create mode 100644 packages/stack-encrypt/src/dynamic/query.rs create mode 100644 packages/stack-encrypt/tests/fixtures/query_operators.json create mode 100644 packages/stack-encrypt/tests/query_operators.rs diff --git a/packages/eql/crates/eql-bindings/src/encryption/targets.rs b/packages/eql/crates/eql-bindings/src/encryption/targets.rs index 83fded6d9..e56704577 100644 --- a/packages/eql/crates/eql-bindings/src/encryption/targets.rs +++ b/packages/eql/crates/eql-bindings/src/encryption/targets.rs @@ -52,6 +52,11 @@ //! | `producible` | Whether the engine can produce this type today. [`encrypt`] and [`query`] refuse a type that is not, and a generator should too. | //! | `reason` | Why not, when `producible` is `false`; `null` when it is. | //! +//! The engine adds an eleventh key, `operators`, when it lists the table: +//! the operators it answers for the type, computed from the row by its own +//! rules (`stack_encrypt::dynamic::TargetDescriptor`), so this table holds +//! none of them. +//! //! The table has one row per stored domain in catalog order; query twins are //! not rows (each row names its own). Adding a field is a wire change for //! every reader of `se_targets`; renaming or removing one is a breaking one. diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 9a92842d1..f58686bce 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -9,6 +9,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **A binding queries by operator** (`dynamic` feature; ADR-0009, ADR-0007 + amended 2026-10-16). `dynamic::query_term(keyset, &plan, field, op, + input, &resolver)` derives the query term for a field of a data plan, an + operator (`dynamic::OpSpec`, named `eq`, `lt`, `lte`, `gt`, `gte`, + `between` or `matches`) and its input, reading the input as the field's + declared type first. The index that answers is the typed path's default + (`OpSpec::answered_by`), and the term is the bytes the typed query chain + derives for the same query. A field that names an EQL type answers through + the type's query twin. An operator the field does not answer is refused + as a mistyped value is, `dynamic::Error::Source` naming the field with + `Reason::OperatorNotAnswered`; `check_query` refuses the same without a + cipher. `dynamic::verify` checks a page of decrypted values against a + probabilistic query and `dynamic::target_limit` sizes its fetch from the + term's bytes and `Stats` (`Stats::from_value` reads their wire form), by + the code the typed `MatchQuery` runs, with the field's own match options; + neither uses a key. The catalog entry of an EQL type + (`TargetDescriptor::to_value`) lists the operators the engine answers for + it, each with its fidelity (`dynamic::Fidelity`), arity and index + (`TargetDescriptor::operators`). - **A query names an operator, not an index.** `cipher.query(op::Eq, "bob@example.com").on(&plan)` and `cipher.query(op::Matches, "ali").on(&plan)`: the operator comes first, as diff --git a/packages/stack-encrypt/src/dynamic/mod.rs b/packages/stack-encrypt/src/dynamic/mod.rs index 9e9397b60..2f7752b47 100644 --- a/packages/stack-encrypt/src/dynamic/mod.rs +++ b/packages/stack-encrypt/src/dynamic/mod.rs @@ -37,6 +37,10 @@ //! is the build without them. //! * [`TermBytes`] — the term a field lowered from data derives, as its //! frozen bytes. Such a field's plaintext is an [`FfiValue`]. +//! * [`query_term`], [`verify`] and [`target_limit`] — a query named by +//! operator ([`OpSpec`]) on a plan field, and a probabilistic term's two +//! questions, over a page of decrypted values and over [`Stats`](crate::op::Stats) +//! (ADR-0009; ADR-0007, amended 2026-10-16). //! * [`Scope`] — which cipher an opening operation decrypts through. //! //! # What is not here @@ -75,6 +79,7 @@ mod context; mod ffi_value; mod kind; +mod query; pub mod record; mod target; mod term; @@ -94,6 +99,10 @@ pub use ffi_value::{ ValueKind, ValueMap, ValueSeq, ValueSink, }; pub use kind::{admits, read}; +pub use query::{ + check_query, limit_to_value, query_term, target_limit, verify, Fidelity, OpSpec, Operand, + QueryTerm, +}; pub use record::{FieldPlan, Output, Plan}; pub use target::{NoTargets, TargetDescriptor, TargetError, TargetResolver}; pub use term::{term, TermBytes}; @@ -240,8 +249,12 @@ pub enum Error { /// them), a field the plan does not name, a plan field the source does /// not carry or carries twice, or a passthrough or a repeated map key /// under a field the plan seals, or a value of another type than its - /// field declares. Also a query value that cannot be read as its field's - /// type ([`read`]). + /// field declares. Also a query its field does not answer: an operator + /// the field's type does not answer, an input that cannot be read as the + /// field's type ([`read`]) or is not the pair a range takes, a page of + /// values to verify that holds a value of another type, or statistics + /// or a term that are not what [`target_limit`] reads — the same shape of + /// refusal as a mistyped field, naming the field. #[error( "record source does not fit the plan: {reason}{}", in_field_text(field) @@ -444,8 +457,8 @@ macro_rules! reasons { } reasons! { - /// What was wrong with a context, plan, source or stored record: the reason - /// a dynamic input error carries beside the field it names. + /// What was wrong with a context, plan, source, stored record or query: the + /// reason a dynamic input error carries beside the field it names. /// /// One vocabulary for all four, since several reasons apply to more than /// one (a field given twice is a misfit in a source and in a stored @@ -541,6 +554,25 @@ reasons! { /// A stored `"passthrough"` or `"eql"` node is not a passthrough /// carrying a value of the kind it holds. NotPassthrough => "not_passthrough", "a node does not carry a value of the kind it holds"; + /// A plan takes its context from a field of each record, so a query on + /// it has no context: the caller names the context, and sends the plan + /// under it. + QueryContextField => "query_context_field", "the plan takes its context from a field of each record, so a query has none"; + /// An operator name is not one the engine knows ([`OpSpec::parse`]). + UnknownOperator => "unknown_operator", "the operator is not one the engine knows"; + /// The field's type does not answer the operator: no index it declares + /// answers it, or the EQL type it names does not ([`OpSpec::answered_by`]). + OperatorNotAnswered => "operator_not_answered", "the field's type does not answer the operator"; + /// A range's input is not a list of two values. + NotARange => "not_a_range", "a range is not a list of two values"; + /// The operator's answer is exact, so there is nothing to verify and no + /// fetch to size. + ExactOperator => "exact_operator", "the operator's answer is exact, so there is nothing to verify or size"; + /// The term is not one the field's index derives for the operator. + NotAQueryTerm => "not_a_query_term", "the term is not one the field's index derives for the operator"; + /// Statistics are not a row count with an optional histogram and + /// confidence ([`Stats::from_value`](crate::op::Stats::from_value)). + NotStats => "not_stats", "the statistics are not a row count with an optional histogram and confidence"; } } diff --git a/packages/stack-encrypt/src/dynamic/query.rs b/packages/stack-encrypt/src/dynamic/query.rs new file mode 100644 index 000000000..8386ee73d --- /dev/null +++ b/packages/stack-encrypt/src/dynamic/query.rs @@ -0,0 +1,1828 @@ +//! Queries named by operator, for a binding. +//! +//! A Rust caller names an operator as a type, `cipher.query(op::Matches, +//! "ali").on(&plan)`, and the compiler finds the index that answers it +//! (ADR-0009). A binding has no type to name: its plan arrives as data and +//! its operator as a string. This module is the runtime form of that query, +//! written once here rather than once per language binding (ADR-0007, +//! amended 2026-10-16): +//! +//! - [`OpSpec`] names an operator, as [`IndexSpec`] names an index, and +//! [`OpSpec::answered_by`] is the table the typed path's `DefaultIndex` +//! impls state: which declared index answers the operator for a field of +//! a kind, if any does. +//! - [`query_term`] derives the query term for a plan field, an operator and +//! its input: the field's own index term for the input, or, for a field +//! that names an EQL type, that type's query value. It refuses an operator +//! the field does not answer as it refuses a value of the wrong type, with +//! [`Error::Source`] naming the field. +//! - [`verify`] and [`target_limit`] are a probabilistic term's two +//! questions, [`Probabilistic::verify`] and +//! [`Probabilistic::target_limit`], over a page of decrypted values and +//! over [`Stats`]. They run the same code the typed `MatchQuery` runs, with +//! the match index's options read from the plan, and need no key. +//! +//! An exact term has neither question: [`verify`] and [`target_limit`] +//! refuse an exact operator ([`Reason::ExactOperator`]) rather than answer +//! `true` for every row, as the typed path does not compile the call. +//! +//! # Which operators a field answers +//! +//! A field with indexes answers what [`OpSpec::answered_by`] says for its +//! declared kind: `eq` through an equality index, `matches` through a match +//! index over text, and the orderings (`lt`, `lte`, `gt`, `gte`, +//! `between`) through an ORE index over an integer, or an OPE index when +//! the field declares no ORE. Those are the typed path's defaults: an +//! ordering over text, bytes or a float is not answered, as `op::Ordered` +//! covers the integers only. +//! +//! A field that names an EQL type answers what its +//! [`TargetDescriptor::operators`] lists: the exact operators of a single +//! value whose index the type carries, for a type the engine produces and +//! that has a query twin. The term is the twin, which carries every term of +//! the type; one query domain per operator is EQL v4's (ADR-0009, decision +//! 8). A probabilistic operator is not answered through a target yet: the +//! descriptor names the type's match index but not its options, and +//! `verify` must cut tokens as the stored filter did. +//! +//! # Wire format +//! +//! An operator is its [`name`](OpSpec::name). A query term is the object +//! [`QueryTerm::to_value`] writes, the statistics the object +//! [`Stats::from_value`] reads, and a limit the value +//! [`limit_to_value`] writes; each is documented where it is written. +//! +//! [`Probabilistic::verify`]: crate::op::Probabilistic::verify +//! [`Probabilistic::target_limit`]: crate::op::Probabilistic::target_limit + +use std::cmp::Ordering; +use std::fmt; + +use stack_kms::DataKeySource; + +use super::record::{FieldPlan, Plan}; +use super::term::{derive, Use}; +use super::ValueKind; +use super::{admits, read, utf8, Error, FfiValue, Reason, TargetDescriptor, TargetResolver}; +use crate::op::{Limit, Stats}; +use crate::sem::{match_limit, MatchOptions, QueryTokens, TermError}; +use crate::target::IndexSpec; +use crate::{KeysetCipher, Pending}; + +/// An operator, named at run time: the runtime form of the +/// [`op`](crate::op) module's operator types, as [`IndexSpec`] is of the +/// index types. +/// +/// Not `#[non_exhaustive]`, for the reason the [module docs](super#stability) +/// give for [`IndexSpec`]: the names are wire format, and a new operator is +/// one every binding has to be taught. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum OpSpec { + /// `"eq"`: [`op::Eq`](struct@crate::op::Eq). + Eq, + /// `"lt"`: [`op::Lt`](crate::op::Lt). + Lt, + /// `"lte"`: [`op::Lte`](crate::op::Lte). + Lte, + /// `"gt"`: [`op::Gt`](crate::op::Gt). + Gt, + /// `"gte"`: [`op::Gte`](crate::op::Gte). + Gte, + /// `"between"`: [`op::Between`](crate::op::Between), whose input is a + /// pair of bounds, low then high, inclusive. + Between, + /// `"matches"`: [`op::Matches`](struct@crate::op::Matches), in the case + /// the field's match index fixes. + Matches, +} + +impl OpSpec { + /// Every operator, in declaration order: the order a catalog lists them + /// in. + pub const ALL: [OpSpec; 7] = [ + OpSpec::Eq, + OpSpec::Lt, + OpSpec::Lte, + OpSpec::Gt, + OpSpec::Gte, + OpSpec::Between, + OpSpec::Matches, + ]; + + /// The operator a name names, or `None` for a name that is not one. + /// Names match exactly: `"eq"`, not `"Eq"` or `"="`. + pub fn parse(name: &str) -> Option { + Self::ALL.into_iter().find(|op| op.name() == name) + } + + /// The operator's wire name. + pub fn name(self) -> &'static str { + match self { + OpSpec::Eq => "eq", + OpSpec::Lt => "lt", + OpSpec::Lte => "lte", + OpSpec::Gt => "gt", + OpSpec::Gte => "gte", + OpSpec::Between => "between", + OpSpec::Matches => "matches", + } + } + + /// How far the server's answer to this operator can be trusted: what + /// the typed path says by the trait the term implements. + pub fn fidelity(self) -> Fidelity { + match self { + OpSpec::Matches => Fidelity::Probabilistic, + _ => Fidelity::Exact, + } + } + + /// Whether the operator takes a pair of values, low then high, rather + /// than one: `between` does. + pub fn takes_pair(self) -> bool { + self == OpSpec::Between + } + + /// The index among `indexes` that answers this operator for a field of + /// `kind`, or `None` when none does: the typed path's `DefaultIndex` + /// table, stated over a kind and the indexes a field declares. + /// + /// ``` + /// use stack_encrypt::dynamic::{OpSpec, ValueKind}; + /// use stack_encrypt::target::IndexSpec; + /// + /// let both = [IndexSpec::Ope, IndexSpec::Ore]; + /// assert_eq!(OpSpec::Lt.answered_by(ValueKind::UInt32, &both), Some(&IndexSpec::Ore)); + /// assert_eq!(OpSpec::Lt.answered_by(ValueKind::String, &both), None); + /// assert_eq!(OpSpec::Eq.answered_by(ValueKind::UInt32, &both), None); + /// ``` + pub fn answered_by(self, kind: ValueKind, indexes: &[IndexSpec]) -> Option<&IndexSpec> { + let declared = |wanted: fn(&IndexSpec) -> bool| indexes.iter().find(|index| wanted(index)); + match self { + OpSpec::Eq => declared(|index| matches!(index, IndexSpec::Equality)), + OpSpec::Matches => declared(|index| matches!(index, IndexSpec::Match(_))), + OpSpec::Lt | OpSpec::Lte | OpSpec::Gt | OpSpec::Gte | OpSpec::Between => { + if !ordered(kind) { + return None; + } + declared(|index| matches!(index, IndexSpec::Ore)) + .or_else(|| declared(|index| matches!(index, IndexSpec::Ope))) + } + } + .filter(|index| admits(kind, index)) + } +} + +impl fmt::Display for OpSpec { + /// Writes [`name`](Self::name). + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// The kinds the ordering operators are answered over: the integers, as +/// [`op::Ordered`](crate::op::Ordered) covers them in the typed path. +fn ordered(kind: ValueKind) -> bool { + matches!( + kind, + ValueKind::Int32 | ValueKind::Int64 | ValueKind::UInt32 | ValueKind::UInt64 + ) +} + +/// How far the server's answer to an operator can be trusted (ADR-0009, +/// decision 4), as a catalog reports it. +/// +/// The typed path says this with traits, not a value: a term that +/// implements [`Probabilistic`](crate::op::Probabilistic) can be verified, +/// and one that does not cannot. A binding's generator asks before the +/// program runs, so it needs the answer as data. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Fidelity { + /// `"exact"`: the server's answer is the answer. + Exact, + /// `"probabilistic"`: the answer can hold false positives and no false + /// negatives; [`verify`] decides each row, and [`target_limit`] sizes + /// the fetch. + Probabilistic, +} + +impl Fidelity { + /// The fidelity's wire name. + pub fn name(self) -> &'static str { + match self { + Fidelity::Exact => "exact", + Fidelity::Probabilistic => "probabilistic", + } + } +} + +/// The operators an EQL type answers, as [`TargetDescriptor::operators`] +/// lists them: none for a type the engine does not produce or that has no +/// query twin, and otherwise every exact operator of one value that one of +/// the type's indexes answers for its plaintext kind. See the +/// [module docs](self#which-operators-a-field-answers) for why a +/// probabilistic operator, and `between`, wait. +pub(super) fn target_operators(target: &TargetDescriptor) -> Vec<(OpSpec, IndexSpec)> { + let Some(kind) = target + .plaintext + .filter(|_| target.producible && target.query.is_some()) + else { + return Vec::new(); + }; + let indexes: Vec = target + .indexes + .iter() + .filter_map(|key| IndexSpec::parse(key)) + .collect(); + OpSpec::ALL + .into_iter() + .filter(|op| op.fidelity() == Fidelity::Exact && !op.takes_pair()) + .filter_map(|op| { + op.answered_by(kind, &indexes) + .map(|index| (op, index.clone())) + }) + .collect() +} + +/// A query term a binding asked for by operator: the operand to bind in +/// SQL, and what it is. +/// +/// The bytes are not plaintext: an index term, or an EQL query value, which +/// is what the server compares. The query's own tokens, which +/// [`verify`] needs, are plaintext and are not here; [`verify`] cuts them +/// again from the query's input. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct QueryTerm { + operator: OpSpec, + index: IndexSpec, + domain: Option, + operand: Operand, +} + +/// What a [`QueryTerm`] binds. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Operand { + /// One term: an index term's frozen bytes ([`TermBytes`](super::TermBytes)), or an EQL + /// query value's JSON bytes. + One(Vec), + /// A range's two terms, each the index term of one bound, inclusive. + Range { + /// The low bound's term. + lo: Vec, + /// The high bound's term. + hi: Vec, + }, +} + +impl QueryTerm { + /// The operator the term answers. + pub fn operator(&self) -> OpSpec { + self.operator + } + + /// How far the server's answer to it can be trusted. + pub fn fidelity(&self) -> Fidelity { + self.operator.fidelity() + } + + /// The index that answers the operator for the field. + pub fn index(&self) -> &IndexSpec { + &self.index + } + + /// The PostgreSQL domain the operand is cast to, for a field that names + /// an EQL type; `None` for a field with indexes, whose terms are bytes. + pub fn domain(&self) -> Option<&str> { + self.domain.as_deref() + } + + /// What the term binds. + pub fn operand(&self) -> &Operand { + &self.operand + } + + /// The term in its wire form: an object with every key present, an + /// absent value null. + /// + /// | key | value | + /// |---|---| + /// | `operator` | string: the [operator's name](OpSpec::name), `"matches"` | + /// | `fidelity` | string: `"exact"` or `"probabilistic"` ([`Fidelity::name`]) | + /// | `index` | string: the [key](IndexSpec::key) of the index that answers, `"match"` | + /// | `domain` | string or null: the query value's PostgreSQL domain, for a field that names an EQL type | + /// | `term` | bytes: the term, or for `between` a list of two byte strings, the low bound's term then the high's | + pub fn to_value(&self) -> FfiValue { + let bytes = |b: &[u8]| FfiValue::Bytes(vitaminc_protected::Protected::new(b.to_vec())); + let term = match &self.operand { + Operand::One(term) => bytes(term), + Operand::Range { lo, hi } => FfiValue::Array(vec![bytes(lo), bytes(hi)]), + }; + FfiValue::Object(vec![ + ( + "operator".to_owned(), + FfiValue::String(self.operator.name().into()), + ), + ( + "fidelity".to_owned(), + FfiValue::String(self.fidelity().name().into()), + ), + ( + "index".to_owned(), + FfiValue::String(self.index.key().into()), + ), + ( + "domain".to_owned(), + self.domain + .as_deref() + .map_or(FfiValue::Null, |domain| FfiValue::String(domain.into())), + ), + ("term".to_owned(), term), + ]) + } +} + +/// The field a query is about, and the index that answers its operator. +struct Answer<'p> { + field: &'p FieldPlan, + kind: ValueKind, + index: IndexSpec, + target: Option, +} + +/// The input of a query, read as its field's kind. +enum Input { + One(FfiValue), + Pair(FfiValue, FfiValue), +} + +/// Find `field` in `plan` and the index that answers `operator` for it. +/// +/// # Errors +/// +/// [`Error::Plan`] for a field the plan does not have +/// ([`Reason::NoSuchField`]), [`Error::Target`] for a target the resolver +/// refuses, and [`Error::Source`] +/// ([`Reason::OperatorNotAnswered`]) for an operator the field does not +/// answer. +fn answer<'p>( + plan: &'p Plan, + field: &str, + operator: OpSpec, + resolver: &(impl TargetResolver + ?Sized), +) -> Result, Error> { + let found = plan + .fields() + .iter() + .find(|candidate| candidate.name() == field) + .ok_or_else(|| Error::Plan { + field: Some(field.to_owned()), + reason: Reason::NoSuchField, + })?; + let not_answered = || Error::Source { + field: Some(field.to_owned()), + reason: Reason::OperatorNotAnswered, + }; + let kind = found.field_type().ok_or_else(not_answered)?; + let (index, target) = match found.target() { + Some(name) => { + let target = resolver.resolve(name).map_err(Error::Target)?; + let index = target_operators(&target) + .into_iter() + .find_map(|(op, index)| (op == operator).then_some(index)); + (index, Some(target)) + } + None => (operator.answered_by(kind, &found.indexes()).cloned(), None), + }; + Ok(Answer { + field: found, + kind, + index: index.ok_or_else(not_answered)?, + target, + }) +} + +/// Refuse a plan whose context is a field of each record: a term derives +/// under a context, and such a plan has none until a record names it, so a +/// binding sends the plan under the context its query names. +/// [`verify`] and [`target_limit`] derive nothing and take such a plan. +fn has_context(plan: &Plan, field: &str) -> Result<(), Error> { + match plan.context_field() { + Some(_) => Err(Error::Plan { + field: Some(field.to_owned()), + reason: Reason::QueryContextField, + }), + None => Ok(()), + } +} + +/// Read a query's input as the field's kind ([`read`]): one value, or for +/// `between` a list of two, low then high. +/// +/// # Errors +/// +/// [`Error::Source`] naming the field: [`Reason::FieldType`] for a value +/// that cannot be read as the kind, [`Reason::NotARange`] for a `between` +/// input that is not a list of two values. +fn read_input(answer: &Answer<'_>, operator: OpSpec, input: FfiValue) -> Result { + let name = answer.field.name(); + let one = |value| read(answer.kind, value).map_err(|error| error.in_field(name)); + if !operator.takes_pair() { + return one(input).map(Input::One); + } + match input { + FfiValue::Array(bounds) => match <[FfiValue; 2]>::try_from(bounds) { + Ok([lo, hi]) => Ok(Input::Pair(one(lo)?, one(hi)?)), + Err(_) => Err(not_a_range(name)), + }, + _ => Err(not_a_range(name)), + } +} + +fn not_a_range(field: &str) -> Error { + Error::Source { + field: Some(field.to_owned()), + reason: Reason::NotARange, + } +} + +/// The refusals the typed path makes when a query is awaited, before any +/// term exists ([`Supports::check`](crate::op::Supports::check)): a range +/// whose low bound is above its high bound, and match text the index +/// cannot normalize or that cuts no token. The match index's own +/// derivation refuses the last two again; asking here lets a binding +/// refuse at its boundary. +fn check_input(index: &IndexSpec, input: &Input) -> Result<(), Error> { + let refused = |error: TermError| Error::Cipher(error.into()); + match (index, input) { + (_, Input::Pair(lo, hi)) => { + if integer_order(lo, hi) == Some(Ordering::Greater) { + return Err(refused(TermError::InvertedRange)); + } + } + (IndexSpec::Match(options), Input::One(value)) => { + query_tokens(value, options).map(drop)?; + } + _ => {} + } + Ok(()) +} + +/// The order of two integers of one kind, or `None` for anything else. +fn integer_order(lo: &FfiValue, hi: &FfiValue) -> Option { + match (lo, hi) { + (FfiValue::Int32(lo), FfiValue::Int32(hi)) => Some(lo.cmp(hi)), + (FfiValue::Int64(lo), FfiValue::Int64(hi)) => Some(lo.cmp(hi)), + (FfiValue::UInt32(lo), FfiValue::UInt32(hi)) => Some(lo.cmp(hi)), + (FfiValue::UInt64(lo), FfiValue::UInt64(hi)) => Some(lo.cmp(hi)), + _ => None, + } +} + +/// The tokens a match query of `value` keeps, under the index's options. +/// +/// # Errors +/// +/// [`Error::Cipher`] with [`TermError::Normalize`] for text the index cannot +/// normalize, and [`TermError::EmptyTermText`] for text that cuts no token, +/// whose filter would match every row. [`Error::Internal`] for a value that +/// is not text, which [`read_input`] has ruled out. +fn query_tokens(value: &FfiValue, options: &MatchOptions) -> Result { + let FfiValue::String(text) = value else { + return Err(Error::Internal); + }; + let text = utf8(text).ok_or(Error::Internal)?; + let tokens = QueryTokens::under(text, options).map_err(|error| Error::Cipher(error.into()))?; + if tokens.is_empty() { + return Err(Error::Cipher(TermError::EmptyTermText.into())); + } + Ok(tokens) +} + +/// Check a query without a cipher: everything [`query_term`] refuses before +/// it derives anything. A binding calls this at its boundary, so a bad +/// query is refused before a keyset is resolved. +/// +/// # Errors +/// +/// As [`query_term`]. +pub fn check_query( + plan: &Plan, + field: &str, + operator: OpSpec, + input: FfiValue, + resolver: &(impl TargetResolver + ?Sized), +) -> Result<(), Error> { + let answer = answer(plan, field, operator, resolver)?; + has_context(plan, field)?; + let input = read_input(&answer, operator, input)?; + check_input(&answer.index, &input) +} + +/// Derive the query term for one field of a plan, an operator and its +/// input: one value for most operators, a list of two for +/// [`between`](OpSpec::Between). +/// +/// The input is read as the field's declared kind first ([`read`]), so a +/// host's `34` as a float queries a `uint64` field as the `u64` it stored. +/// A field with indexes derives the term of the index that answers the +/// operator ([`OpSpec::answered_by`]) under the field's context, by the +/// derivation its stored terms came from; a range derives one term per +/// bound under that one context. A field that names an EQL type runs the +/// type's query twin through `resolver`. Nothing is requested: the pending +/// settles without I/O, as a typed query's does. +/// +/// # Examples +/// +/// ``` +/// use stack_encrypt::dynamic::{query_term, record, FfiValue, NoTargets, OpSpec, Operand}; +/// use stack_encrypt::kms::FakeDataKeySource; +/// use stack_encrypt::{op, Ore, Plan, StackCipher}; +/// +/// # tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(async { +/// let cipher = StackCipher::builder().kms(FakeDataKeySource::new()).init().await?; +/// let keyset = cipher.default_keyset(); +/// let text = |s: &str| FfiValue::String(s.into()); +/// let plan = record::plan(FfiValue::Object(vec![( +/// "age".into(), +/// FfiValue::Object(vec![ +/// ("context".into(), FfiValue::Array(vec![text("users"), text("age")])), +/// ("outputs".into(), FfiValue::Array(vec![text("c"), text("ore")])), +/// ("type".into(), text("uint32")), +/// ]), +/// )]))?; +/// +/// let term = query_term(&keyset, &plan, "age", OpSpec::Lt, FfiValue::UInt32(40), &NoTargets)?.await?; +/// +/// // The typed path's term for the same query. +/// let typed_plan = Plan::context("users/age").with::(Ore).build()?; +/// let typed = cipher.query(op::Lt, &40u32).on(&typed_plan).await?; +/// assert_eq!(term.operand(), &Operand::One(typed.to_bytes())); +/// # Ok::<(), Box>(()) +/// # }).unwrap(); +/// ``` +/// +/// # Errors +/// +/// [`Error::Plan`] for a field the plan does not have +/// ([`Reason::NoSuchField`]), or a plan that takes its context from a field +/// of each record ([`Reason::QueryContextField`]): a query names the +/// context such a plan's records hold, so a binding sends the plan under +/// that context. [`Error::Source`] naming the field for an operator the +/// field does not answer ([`Reason::OperatorNotAnswered`]), an input that +/// cannot be read as the field's kind ([`Reason::FieldType`]) and a +/// `between` input that is not two values ([`Reason::NotARange`]). +/// [`Error::Cipher`] for the query refusals the typed path makes: a range +/// whose low bound is above its high ([`TermError::InvertedRange`]), and +/// match text that cuts no token ([`TermError::EmptyTermText`]) or that +/// the index cannot normalize ([`TermError::Normalize`]). +/// [`Error::Target`] for what the resolver refuses. +pub fn query_term<'a, K>( + cipher: &'a KeysetCipher<'_, K>, + plan: &Plan, + field: &str, + operator: OpSpec, + input: FfiValue, + resolver: &(impl TargetResolver + ?Sized), +) -> Result, Error> +where + K: DataKeySource + Sync + 'static, +{ + let answer = answer(plan, field, operator, resolver)?; + has_context(plan, field)?; + let input = read_input(&answer, operator, input)?; + check_input(&answer.index, &input)?; + let index = answer.index.clone(); + let made = move |domain: Option, operand| QueryTerm { + operator, + index, + domain, + operand, + }; + if let Some(target) = answer.target { + let Input::One(value) = input else { + return Err(Error::Internal); + }; + let domain = target.query_sql_domain; + return Ok(super::record::query(cipher, plan, field, value, resolver)? + .map(move |json| made(domain, Operand::One(json)))); + } + let context = answer.field.context(); + let term = |value: &FfiValue| { + derive(cipher, value, &answer.index, context.clone(), Use::Query) + .map_err(|error| error.in_field(field)) + }; + Ok(match input { + Input::One(value) => { + term(&value)?.map(move |term| made(None, Operand::One(term.into_bytes()))) + } + Input::Pair(lo, hi) => term(&lo)?.zip(term(&hi)?).map(move |(lo, hi)| { + made( + None, + Operand::Range { + lo: lo.into_bytes(), + hi: hi.into_bytes(), + }, + ) + }), + }) +} + +/// The match index that answers a probabilistic operator for the field, +/// or the refusal for an exact one. +fn probabilistic( + plan: &Plan, + field: &str, + operator: OpSpec, + resolver: &(impl TargetResolver + ?Sized), +) -> Result { + let answer = answer(plan, field, operator, resolver)?; + // A match index answers `matches` alone, and `matches` is the one + // probabilistic operator, so the index says which this is. + match answer.index { + IndexSpec::Match(options) => Ok(options), + _ => Err(Error::Source { + field: Some(field.to_owned()), + reason: Reason::ExactOperator, + }), + } +} + +/// Whether each of a page of decrypted values really matches a +/// probabilistic query, in order: [`Probabilistic::verify`] for every +/// value, run by the code the typed `MatchQuery` runs, under the options of +/// the field's match index. +/// +/// The query is named as it was derived, by the plan, the field, the +/// operator and the input, and the input's tokens are cut again here: they +/// are plaintext, so no query term carries them across a binding. The +/// result needs no key. +/// +/// A value that does not hold the query's tokens is `false`, and so is a +/// null (a row whose field is empty holds no token), as +/// [`Probabilistic::verify`] is `false` for every outcome but a match. +/// +/// [`Probabilistic::verify`]: crate::op::Probabilistic::verify +/// +/// # Errors +/// +/// As [`query_term`] for the field, the operator and the input, before any +/// value is read, save that a plan whose context is a field of each record +/// is taken, since verifying derives nothing; [`Error::Source`] with +/// [`Reason::ExactOperator`] for an exact operator, which has nothing to +/// verify; and [`Error::Source`] with [`Reason::FieldType`] for a value +/// that is neither a string nor null. +pub fn verify( + plan: &Plan, + field: &str, + operator: OpSpec, + input: FfiValue, + values: &[FfiValue], + resolver: &(impl TargetResolver + ?Sized), +) -> Result, Error> { + let options = probabilistic(plan, field, operator, resolver)?; + let input = read(ValueKind::String, input).map_err(|error| error.in_field(field))?; + let tokens = query_tokens(&input, &options)?; + values + .iter() + .map(|value| match value { + FfiValue::String(text) => { + Ok(utf8(text).is_some_and(|text| tokens.verify(text, &options))) + } + FfiValue::Null => Ok(false), + _ => Err(Error::Source { + field: Some(field.to_owned()), + reason: Reason::FieldType, + }), + }) + .collect() +} + +/// How many rows to fetch so that, at the confidence `stats` names, `want` +/// of them [`verify`]: [`Probabilistic::target_limit`] for the term +/// [`query_term`] derived, run by the code the typed `MatchQuery` runs. +/// +/// `term` is the term's bytes as [`query_term`] returned them, from which +/// the number of bits the query sets is read. The field's match index +/// declares no `expected_tokens` prior in a data plan, so a [`Stats`] with +/// no measured histogram answers [`Limit::Unbounded`]. The result needs no +/// key. +/// +/// [`Probabilistic::target_limit`]: crate::op::Probabilistic::target_limit +/// +/// # Errors +/// +/// As [`verify`] for the plan, the field and the operator; and +/// [`Error::Source`] with [`Reason::NotAQueryTerm`] for bytes that are not +/// a match query term under the field's options: not whole positions, a +/// position outside the filter, positions out of order or repeated, or +/// none at all. +pub fn target_limit( + plan: &Plan, + field: &str, + operator: OpSpec, + term: &[u8], + want: u64, + stats: &Stats, + resolver: &(impl TargetResolver + ?Sized), +) -> Result { + let options = probabilistic(plan, field, operator, resolver)?; + let bits = query_bits(term, &options).ok_or_else(|| Error::Source { + field: Some(field.to_owned()), + reason: Reason::NotAQueryTerm, + })?; + Ok(match_limit(bits, &options, None, want, stats)) +} + +/// The number of positions a match query term sets, if `term` is one the +/// index derives under `options`: little-endian `u16` positions, each in +/// the filter, strictly ascending, at least one. +fn query_bits(term: &[u8], options: &MatchOptions) -> Option { + if term.is_empty() || !term.len().is_multiple_of(2) { + return None; + } + let positions: Vec = term + .chunks_exact(2) + .map(|pair| u32::from(u16::from_le_bytes([pair[0], pair[1]]))) + .collect(); + let in_filter = positions.iter().all(|&position| position < options.m); + let ascending = positions.windows(2).all(|pair| pair[0] < pair[1]); + (in_filter && ascending).then_some(positions.len()) +} + +impl Stats { + /// Read statistics from their wire form, an object: + /// + /// | key | value | + /// |---|---| + /// | `rows` | integer, required: the rows the query scans | + /// | `histogram` | optional, or null: a list of `[bits, rows]` pairs of integers, a measured histogram of the stored filters' set-bit counts ([`Stats::measured`]); absent, the declared prior ([`Stats::declared`]) | + /// | `confidence` | optional, or null: a float, the confidence to target instead of 99% ([`Stats::with_confidence`]) | + /// + /// Integers are read at any width and must not be negative. + /// + /// # Errors + /// + /// [`Error::Source`] with [`Reason::NotStats`] for anything else: not an + /// object, no `rows`, a key given twice or not one of the three, or a + /// value of another type. + pub fn from_value(value: &FfiValue) -> Result { + let refused = || Error::bad_source(Reason::NotStats); + let FfiValue::Object(entries) = value else { + return Err(refused()); + }; + let mut seen: Vec<&str> = Vec::with_capacity(entries.len()); + let (mut rows, mut histogram, mut confidence) = (None, None, None); + for (key, value) in entries { + if seen.contains(&key.as_str()) { + return Err(refused()); + } + seen.push(key); + match (key.as_str(), value) { + ("histogram" | "confidence", FfiValue::Null) => {} + ("rows", rows_value) => rows = Some(unsigned(rows_value).ok_or_else(refused)?), + ("histogram", FfiValue::Array(buckets)) => { + histogram = Some( + buckets + .iter() + .map(|bucket| match bucket { + FfiValue::Array(pair) => match pair.as_slice() { + [bits, rows] => Some(( + u32::try_from(unsigned(bits)?).ok()?, + unsigned(rows)?, + )), + _ => None, + }, + _ => None, + }) + .collect::>>() + .ok_or_else(refused)?, + ); + } + ("confidence", FfiValue::Float64(c)) => confidence = Some(*c), + ("confidence", FfiValue::Float32(c)) => confidence = Some(f64::from(*c)), + _ => return Err(refused()), + } + } + let rows = rows.ok_or_else(refused)?; + let stats = match histogram { + Some(histogram) => Stats::measured(rows, histogram), + None => Stats::declared(rows), + }; + Ok(match confidence { + Some(confidence) => stats.with_confidence(confidence), + None => stats, + }) + } +} + +/// A non-negative integer of any width. +fn unsigned(value: &FfiValue) -> Option { + match value { + FfiValue::Int32(v) => u64::try_from(*v).ok(), + FfiValue::Int64(v) => u64::try_from(*v).ok(), + FfiValue::UInt32(v) => Some(u64::from(*v)), + FfiValue::UInt64(v) => Some(*v), + _ => None, + } +} + +/// A limit in its wire form: the rows to fetch as an unsigned 64-bit +/// integer, or null for [`Limit::Unbounded`]. +pub fn limit_to_value(limit: Limit) -> FfiValue { + limit.rows().map_or(FfiValue::Null, FfiValue::UInt64) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::dynamic::{record, NoTargets, TargetError}; + use crate::op::{self, Probabilistic as _}; + use crate::sem::MatchTerms; + use crate::{Equality, Label, Match, Ope, Ore, StackCipher}; + use stack_kms::FakeDataKeySource; + use vitaminc_protected::Controlled; + + async fn cipher() -> StackCipher { + StackCipher::builder() + .kms(FakeDataKeySource::new()) + .init() + .await + .expect("build cipher") + } + + fn text(s: &str) -> FfiValue { + FfiValue::String(s.into()) + } + + /// One field of `users`, as a binding spells it. + fn field(name: &str, kind: &str, outputs: Vec) -> (String, FfiValue) { + ( + name.to_owned(), + FfiValue::Object(vec![ + ( + "context".into(), + FfiValue::Array(vec![text("users"), text(name)]), + ), + ("outputs".into(), FfiValue::Array(outputs)), + ("type".into(), text(kind)), + ]), + ) + } + + /// The plan every test queries: an email with equality, a name with + /// match, an age with ORE and OPE, a score with OPE alone, a nickname + /// with ORE over text, and a case-sensitive handle. + fn users() -> Plan { + let case_sensitive = FfiValue::Object(vec![( + "match".into(), + FfiValue::Object(vec![("downcase".into(), FfiValue::Bool(false))]), + )]); + record::plan(FfiValue::Object(vec![ + field("email", "string", vec![text("c"), text("eq")]), + field("name", "string", vec![text("c"), text("match")]), + field("age", "uint32", vec![text("c"), text("ope"), text("ore")]), + field("score", "int64", vec![text("ope")]), + field("nickname", "string", vec![text("c"), text("ore")]), + field("handle", "string", vec![text("c"), case_sensitive]), + ])) + .expect("the plan parses") + } + + async fn query( + keyset: &KeysetCipher<'_, FakeDataKeySource>, + name: &str, + operator: OpSpec, + input: FfiValue, + ) -> Result { + query_term(keyset, &users(), name, operator, input, &NoTargets)? + .await + .map_err(Error::Cipher) + } + + fn reason_of(error: &Error) -> (Option<&str>, Option) { + (error.field(), error.reason()) + } + + fn term_error(error: &Error) -> Option<&TermError> { + match error { + Error::Cipher(crate::Error::Term(term)) => Some(term), + _ => None, + } + } + + #[test] + fn an_operator_reads_back_from_its_name_and_no_other() { + for op in OpSpec::ALL { + assert_eq!(OpSpec::parse(op.name()), Some(op), "{op}"); + assert_eq!(op.to_string(), op.name()); + } + let names: std::collections::BTreeSet<&str> = + OpSpec::ALL.iter().map(|op| op.name()).collect(); + assert_eq!(names.len(), OpSpec::ALL.len(), "names are distinct"); + for not_one in ["Eq", "=", "", "contains", "at_path", "matches "] { + assert_eq!(OpSpec::parse(not_one), None, "{not_one:?}"); + } + } + + #[test] + fn only_matches_is_probabilistic_and_only_between_takes_a_pair() { + for op in OpSpec::ALL { + assert_eq!( + op.fidelity() == Fidelity::Probabilistic, + op == OpSpec::Matches, + "{op}" + ); + assert_eq!(op.takes_pair(), op == OpSpec::Between, "{op}"); + } + assert_eq!(Fidelity::Exact.name(), "exact"); + assert_eq!(Fidelity::Probabilistic.name(), "probabilistic"); + } + + /// The typed path's `DefaultIndex` table, over kinds: equality answers + /// `eq`, match answers `matches` over text, and the orderings go to ORE + /// over an integer, or to OPE when there is no ORE. + #[test] + fn the_answering_index_is_the_typed_paths_default() { + let all = [ + IndexSpec::Equality, + IndexSpec::Match(MatchOptions::default()), + IndexSpec::Ope, + IndexSpec::Ore, + ]; + let answered = + |op: OpSpec, kind, indexes: &[IndexSpec]| op.answered_by(kind, indexes).cloned(); + assert_eq!( + answered(OpSpec::Eq, ValueKind::String, &all), + Some(IndexSpec::Equality) + ); + assert_eq!( + answered(OpSpec::Eq, ValueKind::Bool, &all), + None, + "a boolean has no equality term" + ); + assert_eq!( + answered(OpSpec::Matches, ValueKind::String, &all), + Some(IndexSpec::Match(MatchOptions::default())) + ); + assert_eq!(answered(OpSpec::Matches, ValueKind::UInt64, &all), None); + for op in [ + OpSpec::Lt, + OpSpec::Lte, + OpSpec::Gt, + OpSpec::Gte, + OpSpec::Between, + ] { + for kind in [ + ValueKind::Int32, + ValueKind::Int64, + ValueKind::UInt32, + ValueKind::UInt64, + ] { + assert_eq!( + answered(op, kind, &all), + Some(IndexSpec::Ore), + "{op} {kind}" + ); + assert_eq!( + answered(op, kind, &[IndexSpec::Ope]), + Some(IndexSpec::Ope), + "{op} {kind}" + ); + assert_eq!( + answered(op, kind, &[IndexSpec::Equality]), + None, + "{op} {kind}" + ); + } + for kind in [ + ValueKind::String, + ValueKind::Bytes, + ValueKind::Float64, + ValueKind::Bool, + ] { + assert_eq!( + answered(op, kind, &all), + None, + "{op} is answered over integers only: {kind}" + ); + } + } + for op in OpSpec::ALL { + assert_eq!( + answered(op, ValueKind::UInt32, &[]), + None, + "no index answers nothing" + ); + } + } + + #[tokio::test] + async fn eq_derives_the_typed_paths_equality_term() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let term = query(&keyset, "email", OpSpec::Eq, text("bob@example.com")) + .await + .unwrap(); + let plan = crate::Plan::context("users/email") + .with::(Equality) + .build() + .unwrap(); + let typed = cipher + .query(op::Eq, "bob@example.com") + .on(&plan) + .await + .unwrap(); + assert_eq!(term.operand(), &Operand::One(typed.to_bytes())); + assert_eq!(term.operator(), OpSpec::Eq); + assert_eq!(term.fidelity(), Fidelity::Exact); + assert_eq!(term.index(), &IndexSpec::Equality); + assert_eq!(term.domain(), None, "a field with indexes binds bytes"); + } + + #[tokio::test] + async fn an_ordering_derives_the_typed_paths_ore_term_and_reads_the_input_as_the_fields_kind() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let plan = crate::Plan::context("users/age") + .with::(Ore) + .build() + .unwrap(); + let typed = cipher.query(op::Lt, &40u32).on(&plan).await.unwrap(); + for input in [ + FfiValue::UInt32(40), + FfiValue::Float64(40.0), + FfiValue::Int64(40), + ] { + let term = query(&keyset, "age", OpSpec::Lt, input).await.unwrap(); + assert_eq!(term.operand(), &Operand::One(typed.to_bytes())); + assert_eq!(term.index(), &IndexSpec::Ore, "ORE before OPE"); + } + } + + #[tokio::test] + async fn between_derives_both_bounds_under_one_context_as_the_typed_range() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let plan = crate::Plan::context("users/age") + .with::(Ore) + .build() + .unwrap(); + let typed = cipher + .query(op::Between, &(18u32, 65)) + .on(&plan) + .await + .unwrap(); + let term = query( + &keyset, + "age", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::UInt32(18), FfiValue::UInt32(65)]), + ) + .await + .unwrap(); + assert_eq!( + term.operand(), + &Operand::Range { + lo: typed.lo().to_bytes(), + hi: typed.hi().to_bytes() + } + ); + let one = query( + &keyset, + "age", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::UInt32(7), FfiValue::UInt32(7)]), + ) + .await; + assert!(one.is_ok(), "(v, v) is one value: {one:?}"); + + let plan = crate::Plan::context("users/score") + .with::(Ope) + .build() + .unwrap(); + let typed = cipher + .query(op::Between, &(-5i64, 5)) + .via(Ope) + .on(&plan) + .await + .unwrap(); + let term = query( + &keyset, + "score", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::Int64(-5), FfiValue::Int64(5)]), + ) + .await + .unwrap(); + assert_eq!( + term.index(), + &IndexSpec::Ope, + "OPE when the field declares no ORE" + ); + assert_eq!( + term.operand(), + &Operand::Range { + lo: typed.lo().to_bytes(), + hi: typed.hi().to_bytes() + } + ); + } + + #[tokio::test] + async fn matches_derives_the_typed_paths_filter() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let plan = crate::Plan::context("users/name") + .with::(Match::default()) + .build() + .unwrap(); + let typed = cipher.query(op::Matches, "alice").on(&plan).await.unwrap(); + let term = query(&keyset, "name", OpSpec::Matches, text("alice")) + .await + .unwrap(); + assert_eq!(term.operand(), &Operand::One(typed.to_bytes())); + assert_eq!(term.fidelity(), Fidelity::Probabilistic); + } + + #[tokio::test] + async fn a_query_the_field_does_not_answer_is_refused_naming_the_field() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let cases = [ + ("name", OpSpec::Eq, text("x"), Reason::OperatorNotAnswered), + ( + "email", + OpSpec::Matches, + text("bob"), + Reason::OperatorNotAnswered, + ), + ( + "nickname", + OpSpec::Lt, + text("bob"), + Reason::OperatorNotAnswered, + ), + ( + "age", + OpSpec::Eq, + FfiValue::UInt32(1), + Reason::OperatorNotAnswered, + ), + ("age", OpSpec::Lt, text("40"), Reason::FieldType), + ( + "age", + OpSpec::Lt, + FfiValue::Float64(40.5), + Reason::FieldType, + ), + ( + "age", + OpSpec::Between, + FfiValue::UInt32(40), + Reason::NotARange, + ), + ( + "age", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::UInt32(1)]), + Reason::NotARange, + ), + ( + "age", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::UInt32(1), text("2")]), + Reason::FieldType, + ), + ]; + for (name, op, input, reason) in cases { + let what = format!("{op} on {name}"); + let error = query(&keyset, name, op, input.clone()).await.unwrap_err(); + assert!(matches!(error, Error::Source { .. }), "{what}: {error:?}"); + assert_eq!(reason_of(&error), (Some(name), Some(reason)), "{what}"); + let checked = check_query(&users(), name, op, input, &NoTargets).unwrap_err(); + assert_eq!( + reason_of(&checked), + (Some(name), Some(reason)), + "{what}: checked alike" + ); + } + let error = query(&keyset, "phone", OpSpec::Eq, text("1")) + .await + .unwrap_err(); + assert!(matches!(error, Error::Plan { .. })); + assert_eq!( + reason_of(&error), + (Some("phone"), Some(Reason::NoSuchField)) + ); + } + + #[tokio::test] + async fn the_typed_paths_query_refusals_are_made_before_any_term() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let pair = |lo, hi| FfiValue::Array(vec![FfiValue::UInt32(lo), FfiValue::UInt32(hi)]); + let cases = [ + ("age", OpSpec::Between, pair(65, 18), "inverted"), + ("name", OpSpec::Matches, text("al"), "empty"), + ("name", OpSpec::Matches, text("\u{0378}abc"), "normalize"), + ]; + for (name, op, input, what) in cases { + for error in [ + query(&keyset, name, op, input.clone()).await.unwrap_err(), + check_query(&users(), name, op, input, &NoTargets).unwrap_err(), + ] { + let term = term_error(&error); + let expected = match what { + "inverted" => matches!(term, Some(TermError::InvertedRange)), + "empty" => matches!(term, Some(TermError::EmptyTermText)), + _ => matches!(term, Some(TermError::Normalize(_))), + }; + assert!(expected, "{what}: {error:?}"); + } + } + } + + /// A plan whose context is a field of each record has no context a + /// query could derive under; the binding sends the plan under the + /// context it names. + #[tokio::test] + async fn a_plan_whose_context_is_a_record_field_is_refused() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let one = |name: &str, kind: &str, output: &str| { + ( + name.to_owned(), + FfiValue::Object(vec![ + ("context".into(), FfiValue::Array(vec![text(name)])), + ("outputs".into(), FfiValue::Array(vec![text(output)])), + ("type".into(), text(kind)), + ]), + ) + }; + let plan = record::plan(FfiValue::Object(vec![ + ("context_field".into(), text("tenant")), + one("tenant", "string", "passthrough"), + one("email", "string", "eq"), + ])) + .unwrap(); + let error = query_term(&keyset, &plan, "email", OpSpec::Eq, text("a"), &NoTargets) + .map(drop) + .unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::QueryContextField)) + ); + let checked = check_query(&plan, "email", OpSpec::Eq, text("a"), &NoTargets).unwrap_err(); + assert_eq!( + reason_of(&checked), + (Some("email"), Some(Reason::QueryContextField)) + ); + // Verifying derives nothing, so it needs no context. + let error = verify(&plan, "email", OpSpec::Eq, text("a"), &[], &NoTargets).unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::ExactOperator)), + "refused for its operator, not its context" + ); + } + + /// `verify` is the typed `MatchQuery::verify`, value for value, under + /// the options the field declares, and needs no key. + #[tokio::test] + async fn verify_answers_as_the_typed_match_query() { + let cipher = cipher().await; + let plan = crate::Plan::context("users/name") + .with::(Match::default()) + .build() + .unwrap(); + let typed = cipher.query(op::Matches, "alice").on(&plan).await.unwrap(); + let page = ["Alice Smith", "alicia", "Bob", "MALICE", "", "ali ce"]; + let values: Vec = page.iter().map(|v| text(v)).collect(); + let verified = verify( + &users(), + "name", + OpSpec::Matches, + text("alice"), + &values, + &NoTargets, + ) + .unwrap(); + let expected: Vec = page.iter().map(|v| typed.verify(v)).collect(); + assert_eq!(verified, expected); + assert!( + verified.contains(&true) && verified.contains(&false), + "{verified:?}" + ); + + let nulls = verify( + &users(), + "name", + OpSpec::Matches, + text("alice"), + &[FfiValue::Null], + &NoTargets, + ) + .unwrap(); + assert_eq!(nulls, [false], "a null holds no token"); + let empty = verify( + &users(), + "name", + OpSpec::Matches, + text("alice"), + &[], + &NoTargets, + ) + .unwrap(); + assert!(empty.is_empty()); + } + + #[test] + fn verify_cuts_tokens_under_the_fields_own_options() { + let sensitive = |input: &str, value: &str| { + verify( + &users(), + "handle", + OpSpec::Matches, + text(input), + &[text(value)], + &NoTargets, + ) + .unwrap()[0] + }; + assert!(sensitive("Alice", "Alice")); + assert!( + !sensitive("Alice", "alice"), + "the handle's index does not fold case" + ); + let folding = verify( + &users(), + "name", + OpSpec::Matches, + text("Alice"), + &[text("alice")], + &NoTargets, + ) + .unwrap(); + assert_eq!(folding, [true], "the name's index folds case"); + } + + #[test] + fn verify_refuses_what_it_cannot_answer() { + let refused = |name: &str, op: OpSpec, input: FfiValue, values: &[FfiValue]| { + verify(&users(), name, op, input, values, &NoTargets).unwrap_err() + }; + let error = refused("email", OpSpec::Eq, text("a"), &[text("a")]); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::ExactOperator)) + ); + let error = refused("age", OpSpec::Lt, FfiValue::UInt32(1), &[]); + assert_eq!( + reason_of(&error), + (Some("age"), Some(Reason::ExactOperator)) + ); + let error = refused("email", OpSpec::Matches, text("abc"), &[]); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::OperatorNotAnswered)) + ); + let error = refused( + "name", + OpSpec::Matches, + text("alice"), + &[FfiValue::UInt32(1)], + ); + assert_eq!(reason_of(&error), (Some("name"), Some(Reason::FieldType))); + let error = refused("name", OpSpec::Matches, FfiValue::UInt32(1), &[]); + assert_eq!(reason_of(&error), (Some("name"), Some(Reason::FieldType))); + let error = refused("name", OpSpec::Matches, text("al"), &[text("al")]); + assert!( + matches!(term_error(&error), Some(TermError::EmptyTermText)), + "{error:?}" + ); + } + + /// `target_limit` is the typed `MatchQuery::target_limit` for the term + /// `query_term` returned, read from its bytes. + #[tokio::test] + async fn target_limit_answers_as_the_typed_match_query() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let plan = crate::Plan::context("users/name") + .with::(Match::default()) + .build() + .unwrap(); + let typed = cipher.query(op::Matches, "ali").on(&plan).await.unwrap(); + let term = query(&keyset, "name", OpSpec::Matches, text("ali")) + .await + .unwrap(); + let Operand::One(bytes) = term.operand() else { + panic!("one term"); + }; + let measured = Stats::measured(1_000_000, [(40, 500_000), (120, 500_000)]); + for stats in [ + measured.clone(), + measured.clone().with_confidence(0.5), + Stats::declared(1_000_000), + ] { + let limit = target_limit( + &users(), + "name", + OpSpec::Matches, + bytes, + 10, + &stats, + &NoTargets, + ) + .unwrap(); + assert_eq!(limit, typed.target_limit(10, &stats), "{stats:?}"); + } + assert!(matches!( + target_limit(&users(), "name", OpSpec::Matches, bytes, 10, &measured, &NoTargets), + Ok(Limit::Rows(rows)) if rows > 10 + )); + assert_eq!( + target_limit( + &users(), + "name", + OpSpec::Matches, + bytes, + 10, + &Stats::declared(1_000_000), + &NoTargets + ) + .unwrap(), + Limit::Unbounded, + "a data plan's match index declares no prior" + ); + } + + #[test] + fn target_limit_refuses_a_term_its_index_would_not_derive() { + let stats = Stats::measured(100, [(10, 100)]); + let limit = |name: &str, op: OpSpec, term: &[u8]| { + target_limit(&users(), name, op, term, 1, &stats, &NoTargets) + }; + let positions = |p: &[u16]| p.iter().flat_map(|p| p.to_le_bytes()).collect::>(); + assert!(limit("name", OpSpec::Matches, &positions(&[1, 2, 255])).is_ok()); + for (what, bytes) in [ + ("none", positions(&[])), + ("half a position", vec![1, 0, 2]), + ("outside the filter", positions(&[1, 256])), + ("out of order", positions(&[2, 1])), + ("repeated", positions(&[2, 2])), + ] { + let error = limit("name", OpSpec::Matches, &bytes).unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("name"), Some(Reason::NotAQueryTerm)), + "{what}" + ); + } + let error = limit("email", OpSpec::Eq, &[0; 32]).unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::ExactOperator)) + ); + // A stored filter under the same options reads as a term too: the + // bytes say nothing of which side derived them, and the limit is a + // number, not a query. + let stored = MatchTerms::::from_positions(vec![3, 9]).unwrap(); + assert!(limit("name", OpSpec::Matches, &stored.to_bytes()).is_ok()); + } + + #[test] + fn statistics_read_from_their_wire_form() { + let object = |entries: Vec<(&str, FfiValue)>| { + FfiValue::Object( + entries + .into_iter() + .map(|(k, v)| (k.to_owned(), v)) + .collect(), + ) + }; + let pair = |bits: u32, rows: u64| { + FfiValue::Array(vec![FfiValue::UInt32(bits), FfiValue::UInt64(rows)]) + }; + assert_eq!( + Stats::from_value(&object(vec![("rows", FfiValue::UInt64(9))])).unwrap(), + Stats::declared(9) + ); + assert_eq!( + Stats::from_value(&object(vec![ + ("rows", FfiValue::Int64(9)), + ("histogram", FfiValue::Null), + ("confidence", FfiValue::Null), + ])) + .unwrap(), + Stats::declared(9) + ); + assert_eq!( + Stats::from_value(&object(vec![ + ("histogram", FfiValue::Array(vec![pair(3, 4), pair(5, 6)])), + ("confidence", FfiValue::Float64(0.9)), + ("rows", FfiValue::UInt32(10)), + ])) + .unwrap(), + Stats::measured(10, [(3, 4), (5, 6)]).with_confidence(0.9) + ); + for (what, bad) in [ + ("not an object", FfiValue::UInt64(9)), + ("no rows", object(vec![])), + ("null rows", object(vec![("rows", FfiValue::Null)])), + ("negative rows", object(vec![("rows", FfiValue::Int64(-1))])), + ("float rows", object(vec![("rows", FfiValue::Float64(9.0))])), + ( + "rows twice", + object(vec![ + ("rows", FfiValue::UInt64(1)), + ("rows", FfiValue::UInt64(1)), + ]), + ), + ( + "unknown key", + object(vec![ + ("rows", FfiValue::UInt64(1)), + ("want", FfiValue::Null), + ]), + ), + ( + "bucket not a pair", + object(vec![ + ("rows", FfiValue::UInt64(1)), + ("histogram", FfiValue::Array(vec![FfiValue::UInt32(3)])), + ]), + ), + ( + "bits too wide", + object(vec![ + ("rows", FfiValue::UInt64(1)), + ( + "histogram", + FfiValue::Array(vec![FfiValue::Array(vec![ + FfiValue::UInt64(u64::from(u32::MAX) + 1), + FfiValue::UInt64(1), + ])]), + ), + ]), + ), + ( + "confidence as text", + object(vec![ + ("rows", FfiValue::UInt64(1)), + ("confidence", text("high")), + ]), + ), + ] { + let error = Stats::from_value(&bad).unwrap_err(); + assert_eq!(reason_of(&error), (None, Some(Reason::NotStats)), "{what}"); + } + } + + /// A range is refused the wrong way round for every integer kind a + /// field can declare, and two values of different kinds have no order + /// here: the input was read as one kind first. + #[test] + fn a_range_is_ordered_within_each_integer_kind() { + let cases = [ + (FfiValue::Int32(-1), FfiValue::Int32(1)), + (FfiValue::Int64(-1), FfiValue::Int64(1)), + (FfiValue::UInt32(1), FfiValue::UInt32(2)), + (FfiValue::UInt64(1), FfiValue::UInt64(2)), + ]; + for (lo, hi) in cases { + assert_eq!(integer_order(&lo, &hi), Some(Ordering::Less), "{lo:?}"); + assert_eq!(integer_order(&hi, &lo), Some(Ordering::Greater), "{lo:?}"); + assert_eq!(integer_order(&lo, &lo), Some(Ordering::Equal), "{lo:?}"); + } + assert_eq!( + integer_order(&FfiValue::UInt32(1), &FfiValue::UInt64(2)), + None + ); + assert_eq!(integer_order(&text("a"), &text("b")), None); + } + + #[test] + fn a_count_is_read_at_any_width_and_never_negative() { + assert_eq!(unsigned(&FfiValue::Int32(7)), Some(7)); + assert_eq!(unsigned(&FfiValue::Int64(7)), Some(7)); + assert_eq!(unsigned(&FfiValue::UInt32(7)), Some(7)); + assert_eq!(unsigned(&FfiValue::UInt64(7)), Some(7)); + assert_eq!(unsigned(&FfiValue::Int32(-7)), None); + assert_eq!(unsigned(&FfiValue::Int64(-7)), None); + assert_eq!(unsigned(&FfiValue::Float64(7.0)), None); + let confidence = Stats::from_value(&FfiValue::Object(vec![ + ("rows".into(), FfiValue::Int32(3)), + ("confidence".into(), FfiValue::Float32(0.5)), + ])) + .unwrap(); + assert_eq!(confidence, Stats::declared(3).with_confidence(0.5)); + } + + #[test] + fn a_limit_is_its_rows_or_null() { + assert!(matches!( + limit_to_value(Limit::Rows(11)), + FfiValue::UInt64(11) + )); + assert!(matches!(limit_to_value(Limit::Unbounded), FfiValue::Null)); + } + + #[tokio::test] + async fn a_query_term_writes_every_key_in_order() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let term = query( + &keyset, + "age", + OpSpec::Between, + FfiValue::Array(vec![FfiValue::UInt32(1), FfiValue::UInt32(2)]), + ) + .await + .unwrap(); + let FfiValue::Object(entries) = term.to_value() else { + panic!("an object"); + }; + let keys: Vec<&str> = entries.iter().map(|(k, _)| k.as_str()).collect(); + assert_eq!(keys, ["operator", "fidelity", "index", "domain", "term"]); + assert!(matches!(&entries[0].1, FfiValue::String(s) if s.risky_ref() == b"between")); + assert!(matches!(&entries[1].1, FfiValue::String(s) if s.risky_ref() == b"exact")); + assert!(matches!(&entries[2].1, FfiValue::String(s) if s.risky_ref() == b"ore")); + assert!(matches!(&entries[3].1, FfiValue::Null)); + let Operand::Range { lo, hi } = term.operand() else { + panic!("a range"); + }; + assert!(matches!( + &entries[4].1, + FfiValue::Array(bounds) if matches!( + bounds.as_slice(), + [FfiValue::Bytes(l), FfiValue::Bytes(h)] if l.risky_ref() == lo && h.risky_ref() == hi + ) + )); + } + + fn descriptor( + kind: Option, + indexes: &[&str], + query: bool, + producible: bool, + ) -> TargetDescriptor { + TargetDescriptor::new( + "Some", + "some", + "", + kind, + "public.some", + indexes.iter().map(|i| (*i).to_owned()).collect(), + query.then(|| "SomeQuery".to_owned()), + query.then(|| "eql_v3.query_some".to_owned()), + producible, + (!producible).then(|| "not yet".to_owned()), + ) + } + + /// A type answers the exact operators of one value its indexes answer, + /// and only once the engine produces it and it has a query twin. + #[test] + fn a_target_answers_the_exact_single_value_operators_of_its_indexes() { + let string = Some(ValueKind::String); + let int = Some(ValueKind::Int64); + assert_eq!( + descriptor(string, &["eq"], true, true).operators(), + [OpSpec::Eq] + ); + assert_eq!( + descriptor(int, &["eq", "ore"], true, true).operators(), + [OpSpec::Eq, OpSpec::Lt, OpSpec::Lte, OpSpec::Gt, OpSpec::Gte], + "no between: the twin is one value" + ); + assert!( + descriptor(string, &["match"], true, true) + .operators() + .is_empty(), + "no probabilistic operator until the descriptor carries the index's options" + ); + assert!(descriptor(string, &["eq"], true, false) + .operators() + .is_empty()); + assert!(descriptor(string, &["eq"], false, true) + .operators() + .is_empty()); + assert!(descriptor(None, &["eq"], true, true).operators().is_empty()); + assert!(descriptor(Some(ValueKind::Object), &["json"], true, true) + .operators() + .is_empty()); + } + + #[test] + fn a_targets_operators_are_listed_in_its_wire_form() { + let FfiValue::Object(entries) = + descriptor(Some(ValueKind::Int64), &["ore"], true, true).to_value() + else { + panic!("an object"); + }; + let (key, FfiValue::Array(operators)) = &entries[10] else { + panic!("a list"); + }; + assert_eq!(key, "operators"); + assert_eq!(operators.len(), 4); + let FfiValue::Object(first) = &operators[0] else { + panic!("an object"); + }; + let first: Vec<(&str, &[u8])> = first + .iter() + .map(|(k, v)| match v { + FfiValue::String(s) => (k.as_str(), s.risky_ref()), + _ => panic!("{k} is a string"), + }) + .collect(); + assert_eq!( + first, + [ + ("operator", b"lt".as_slice()), + ("fidelity", b"exact".as_slice()), + ("arity", b"one".as_slice()), + ("index", b"ore".as_slice()) + ] + ); + } + + /// A resolver holding one type that answers `eq`, whose query is a + /// fixed JSON value: what a target field's query term carries. + struct OneType; + + impl TargetResolver for OneType { + fn targets(&self) -> Vec { + vec![descriptor(Some(ValueKind::String), &["eq"], true, true)] + } + fn encrypt<'a, K: 'static>( + &self, + _: &str, + _: &'a KeysetCipher<'_, K>, + _: &Label, + _: FfiValue, + ) -> Result, K>, TargetError> { + unreachable!("a query encrypts nothing") + } + fn decrypt<'a, K: 'static>( + &self, + _: &str, + _: &'a crate::StackCipher, + _: &Label, + _: &[u8], + ) -> Result, TargetError> { + unreachable!("a query decrypts nothing") + } + fn query<'a, K: 'static>( + &self, + _: &str, + keyset: &'a KeysetCipher<'_, K>, + _: &Label, + _: FfiValue, + ) -> Result, K>, TargetError> { + Ok(Pending::ready(keyset, Ok(br#"{"hm":"00"}"#.to_vec()))) + } + } + + #[tokio::test] + async fn a_target_field_answers_with_its_types_query_and_domain() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let plan = record::plan_with( + FfiValue::Object(vec![( + "email".into(), + FfiValue::Object(vec![ + ( + "context".into(), + FfiValue::Array(vec![text("users"), text("email")]), + ), + ("target".into(), text("Some")), + ]), + )]), + &OneType, + ) + .unwrap(); + let term = query_term(&keyset, &plan, "email", OpSpec::Eq, text("a"), &OneType) + .unwrap() + .await + .unwrap(); + assert_eq!(term.operand(), &Operand::One(br#"{"hm":"00"}"#.to_vec())); + assert_eq!(term.domain(), Some("eql_v3.query_some")); + assert_eq!(term.index(), &IndexSpec::Equality); + for op in [OpSpec::Lt, OpSpec::Matches, OpSpec::Between] { + let error = query_term(&keyset, &plan, "email", op, text("a"), &OneType) + .map(drop) + .unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::OperatorNotAnswered)), + "{op}" + ); + } + let error = verify(&plan, "email", OpSpec::Eq, text("a"), &[], &OneType).unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("email"), Some(Reason::ExactOperator)) + ); + } +} diff --git a/packages/stack-encrypt/src/dynamic/record.rs b/packages/stack-encrypt/src/dynamic/record.rs index d32cb0e35..762352ab0 100644 --- a/packages/stack-encrypt/src/dynamic/record.rs +++ b/packages/stack-encrypt/src/dynamic/record.rs @@ -429,7 +429,7 @@ impl FieldPlan { } /// The indexes the field declares, in output order. - fn indexes(&self) -> Vec { + pub(super) fn indexes(&self) -> Vec { self.outputs .iter() .filter_map(|output| match output { diff --git a/packages/stack-encrypt/src/dynamic/target.rs b/packages/stack-encrypt/src/dynamic/target.rs index 5f211e6ab..757c536b1 100644 --- a/packages/stack-encrypt/src/dynamic/target.rs +++ b/packages/stack-encrypt/src/dynamic/target.rs @@ -27,7 +27,8 @@ use std::fmt; -use super::{FfiValue, ValueKind}; +use super::query::target_operators; +use super::{FfiValue, OpSpec, ValueKind}; use crate::{KeysetCipher, Label, Pending, StackCipher}; @@ -52,6 +53,21 @@ use crate::{KeysetCipher, Label, Pending, StackCipher}; /// | `query_sql_domain` | string or null: the query twin's PostgreSQL domain | /// | `producible` | bool: whether the engine produces the type today | /// | `reason` | string or null: why not, when it does not | +/// | `operators` | list of objects: the operators the engine answers for the type ([`operators`](Self::operators)), each `{"operator", "fidelity", "arity", "index"}`; empty for a type it does not produce | +/// +/// Each `operators` entry has every key: +/// +/// | key | value | +/// |---|---| +/// | `operator` | string: the operator's name, `eq` ([`OpSpec::name`]) | +/// | `fidelity` | string: `exact` or `probabilistic` ([`Fidelity::name`](super::Fidelity::name)) | +/// | `arity` | string: `one`, or `pair` for an operator whose input is two values, low then high | +/// | `index` | string: the key of the type's index that answers it, `eq` | +/// +/// The first ten keys are the row `eql-bindings` serializes for the type; +/// `operators` is the engine's, computed from the row by the table every +/// plan field is held to ([`OpSpec::answered_by`]), so a generator reads the +/// operators from the engine it embeds rather than from a copy of the rules. #[derive(Clone, Debug, PartialEq, Eq)] #[non_exhaustive] pub struct TargetDescriptor { @@ -133,8 +149,41 @@ impl TargetDescriptor { ), ("producible".to_string(), FfiValue::Bool(self.producible)), ("reason".to_string(), optional(&self.reason)), + ( + "operators".to_string(), + FfiValue::Array( + target_operators(self) + .into_iter() + .map(|(op, index)| { + FfiValue::Object(vec![ + ("operator".to_string(), text(op.name())), + ("fidelity".to_string(), text(op.fidelity().name())), + ( + "arity".to_string(), + text(if op.takes_pair() { "pair" } else { "one" }), + ), + ("index".to_string(), text(index.key())), + ]) + }) + .collect(), + ), + ), ]) } + + /// The operators the engine answers for a field that names this type, + /// in [`OpSpec::ALL`]'s order: none for a type it does not produce or + /// that has no query twin, and otherwise each exact operator of one value + /// that one of the type's indexes answers for its plaintext kind + /// ([`OpSpec::answered_by`]). The query for each is the type's query + /// twin. A probabilistic operator and `between` are not answered through + /// a type yet: see [`query_term`](super::query_term)'s module. + pub fn operators(&self) -> Vec { + target_operators(self) + .into_iter() + .map(|(op, _)| op) + .collect() + } } /// Why a target field could not be resolved or run. @@ -604,7 +653,8 @@ mod tests { "query", "query_sql_domain", "producible", - "reason" + "reason", + "operators" ], "the se_targets entry's keys, in order" ); @@ -624,7 +674,7 @@ mod tests { assert!(matches!(&entries[6].1, FfiValue::Null)); assert_eq!( entries.len(), - 10, + 11, "an absent value is null, never a missing key" ); } diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index 8fa97faea..95cd9f36d 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -343,7 +343,7 @@ impl AsRef<[u8]> for TermBytes { /// What a derived term is for: a query refuses match text that cuts no /// token, which a stored value keeps as the empty filter. #[derive(Clone, Copy, PartialEq, Eq)] -enum Use { +pub(super) enum Use { Query, Store, } @@ -357,7 +357,7 @@ enum Use { /// [`Error::Term`] if the value is not one scalar `kind` takes, and /// [`Error::Cipher`] for a query's match text with no token or text the /// index cannot normalize, as for [`term`]. -fn derive<'a, 'c, K: 'static, P, D>( +pub(super) fn derive<'a, 'c, K: 'static, P, D>( cipher: &'a KeysetCipher<'_, K>, value: P, kind: &IndexSpec, diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index 47a70e37c..7e1997d7d 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -798,14 +798,7 @@ impl crate::op::Probabilistic for MatchQuery { type Plaintext = str; fn verify(&self, value: &str) -> bool { - let options = O::options(); - // A value the index cannot normalize holds no token of any query. - let Ok(tokens) = tokenize::tokens(value, options.tokenizer, options.downcase) else { - return false; - }; - let tokens = Zeroizing::new(tokens); - let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); - !self.tokens.0.is_empty() && self.tokens.0.iter().all(|t| have.contains(t.as_str())) + self.tokens.verify(value, &O::options()) } /// `want` plus the false positives expected among the rows `stats` @@ -817,11 +810,13 @@ impl crate::op::Probabilistic for MatchQuery { /// neither, the answer is /// [`Limit::Unbounded`](crate::op::Limit::Unbounded). fn target_limit(&self, want: u64, stats: &crate::op::Stats) -> crate::op::Limit { - let options = O::options(); - let declared = O::expected_tokens() - .map(|tokens| crate::op::stats::expected_fill(tokens, options.k, options.m)); - let bits = u32::try_from(self.positions().len()).unwrap_or(u32::MAX); - stats.limit(want, stats.pass_rate(bits, options.m, declared)) + match_limit( + self.positions().len(), + &O::options(), + O::expected_tokens(), + want, + stats, + ) } } @@ -939,6 +934,38 @@ impl QueryTokens { pub(crate) fn is_empty(&self) -> bool { self.0.is_empty() } + + /// Whether `value` holds every one of these tokens, cut under the same + /// `options` the tokens were: what [`MatchQuery`]'s `verify` asks, with + /// the options a value rather than a type, so a match index declared as + /// data verifies by the same code. No tokens verify nothing, and a value + /// the index cannot normalize holds no token of any query. + pub(crate) fn verify(&self, value: &str, options: &MatchOptions) -> bool { + let Ok(tokens) = tokenize::tokens(value, options.tokenizer, options.downcase) else { + return false; + }; + let tokens = Zeroizing::new(tokens); + let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); + !self.0.is_empty() && self.0.iter().all(|t| have.contains(t.as_str())) + } +} + +/// The fetch for `want` real matches of a match query whose filter sets +/// `bits` positions, under `options` and the index's `expected_tokens` +/// prior: what [`MatchQuery`]'s `target_limit` answers, with the options a +/// value rather than a type, so a match index declared as data sizes its +/// fetch by the same code. +pub(crate) fn match_limit( + bits: usize, + options: &MatchOptions, + expected_tokens: Option, + want: u64, + stats: &crate::op::Stats, +) -> crate::op::Limit { + let declared = + expected_tokens.map(|tokens| crate::op::stats::expected_fill(tokens, options.k, options.m)); + let bits = u32::try_from(bits).unwrap_or(u32::MAX); + stats.limit(want, stats.pass_rate(bits, options.m, declared)) } /// A [`PrfVisitor`] that folds a sequence of per-token PRF blocks into diff --git a/packages/stack-encrypt/tests/fixtures/README.md b/packages/stack-encrypt/tests/fixtures/README.md index c48fd835f..440e4c962 100644 --- a/packages/stack-encrypt/tests/fixtures/README.md +++ b/packages/stack-encrypt/tests/fixtures/README.md @@ -86,3 +86,55 @@ by hand. `label_segments.json` beside this file is a different fixture: the one rule for a plain label segment, read by the Rust, derive and Go tests. + +# Query fixture: `query_operators.json` + +Queries named by operator through the data path +(`stack_encrypt::dynamic::query_term`, what a language binding calls), each +term equal to the one the typed query chain (`cipher.query(op::Lt, +&40u32).on(&plan)`) derives for the same query, with a page `verify` +answered and fetch limits `target_limit` sized (ADR-0009; ADR-0007, amended +2026-10-16). `tests/query_operators.rs` derives every entry and checks it +against the typed path; the Go SDK's `encrypt` tests derive, verify and size +the same queries through the guest's `se_query_term`, `se_verify` and +`se_target_limit` exports and compare. No entry depends on a nonce, so the +file changes only when a derivation does: regenerate it with +`STACK_ENCRYPT_UPDATE_FIXTURES=1 cargo test -p stack-encrypt --all-features +--test query_operators`, and do not edit it by hand. + +```json +{ + "_comment": "what this file is", + "key_source": { "kind": "deterministic-sha256", "seed": "<32 bytes, hex>" }, + "keyset_id": "", + "plan": { "": { "context": [...], "outputs": [...], "type": "" }, ... }, + "queries": [ + { "field": "", "operator": "", "input": , + "term": { "operator": "", "fidelity": "exact" | "probabilistic", + "index": "", "domain": null, "term": "" | ["", ""] } } + ], + "verify": { "field": "", "operator": "matches", "input": "", + "values": [, ...], "verified": [, ...] }, + "target_limit": [ + { "field": "", "operator": "matches", "input": "", "term": "", + "want": , "stats": { "rows": , "histogram": [[, ], ...], + "confidence": }, "limit": } + ] +} +``` + +- **`key_source`**, **`keyset_id`.** As in the record fixture: terms derive + under `FakeDataKeySource`'s index key for the nil keyset, whatever the + seed, so the Go tests run the deterministic guest built from the same + source. +- **`queries`.** Each query as a binding sends it, and its term in the wire + form `QueryTerm::to_value` writes, the bytes in hex: one term, or for + `between` the low bound's then the high's. An input is JSON, read as the + field's declared type, so `40` queries the `uint32` field `age`. +- **`verify`.** A page of decrypted values, null for an empty field, and the + boolean `verify` answers for each, which is what the typed `MatchQuery` + answers. +- **`target_limit`.** A match term's bytes, the statistics in the wire form + `Stats::from_value` reads (an absent histogram is the declared prior), and + the limit, null when no finite fetch can be promised, which is what the + typed `MatchQuery` answers. diff --git a/packages/stack-encrypt/tests/fixtures/query_operators.json b/packages/stack-encrypt/tests/fixtures/query_operators.json new file mode 100644 index 000000000..26c80d601 --- /dev/null +++ b/packages/stack-encrypt/tests/fixtures/query_operators.json @@ -0,0 +1,283 @@ +{ + "_comment": "Query terms derived by operator through the data path (stack_encrypt::dynamic::query_term), each equal to the typed query chain's, with a verified page and fetch limits. Read by tests/query_operators.rs and the Go SDK's tests; the schema is in README.md beside this file. Regenerate with STACK_ENCRYPT_UPDATE_FIXTURES=1; do not edit by hand.", + "key_source": { + "kind": "deterministic-sha256", + "seed": "737461636b2d656e6372797074207265636f7264206669787475726520763120" + }, + "keyset_id": "00000000-0000-0000-0000-000000000000", + "plan": { + "age": { + "context": [ + "users", + "age" + ], + "outputs": [ + "c", + "eq", + "ore" + ], + "type": "uint32" + }, + "email": { + "context": [ + "users", + "email" + ], + "outputs": [ + "c", + "eq" + ], + "type": "string" + }, + "name": { + "context": [ + "users", + "name" + ], + "outputs": [ + "c", + "match" + ], + "type": "string" + } + }, + "queries": [ + { + "field": "email", + "input": "bob@example.com", + "operator": "eq", + "term": { + "domain": null, + "fidelity": "exact", + "index": "eq", + "operator": "eq", + "term": "0cf708c07acaaa169e84e4adf7081f0433d87760bd7e9c63bd7a58c000b8d33b" + } + }, + { + "field": "age", + "input": 34, + "operator": "eq", + "term": { + "domain": null, + "fidelity": "exact", + "index": "eq", + "operator": "eq", + "term": "cc854fd5c7e3f4bb28cb15cae47d866b65b7012c06df43b4b9a6c51adff4710a" + } + }, + { + "field": "age", + "input": 40, + "operator": "lt", + "term": { + "domain": null, + "fidelity": "exact", + "index": "ore", + "operator": "lt", + "term": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a606d6d336" + } + }, + { + "field": "age", + "input": 40, + "operator": "lte", + "term": { + "domain": null, + "fidelity": "exact", + "index": "ore", + "operator": "lte", + "term": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a126a606d6d336" + } + }, + { + "field": "age", + "input": 18, + "operator": "gt", + "term": { + "domain": null, + "fidelity": "exact", + "index": "ore", + "operator": "gt", + "term": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a1256421f2eba6" + } + }, + { + "field": "age", + "input": 18, + "operator": "gte", + "term": { + "domain": null, + "fidelity": "exact", + "index": "ore", + "operator": "gte", + "term": "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a1256421f2eba6" + } + }, + { + "field": "age", + "input": [ + 18, + 65 + ], + "operator": "between", + "term": { + "domain": null, + "fidelity": "exact", + "index": "ore", + "operator": "between", + "term": [ + "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a1256421f2eba6", + "7ca54e16d7944c0a9d8577d87f3864fdf79e99fea5956ba956a20bc4002878be" + ] + } + }, + { + "field": "name", + "input": "alice", + "operator": "matches", + "term": { + "domain": null, + "fidelity": "probabilistic", + "index": "match", + "operator": "matches", + "term": "2800300053006d00890095009d00c100fe00" + } + }, + { + "field": "name", + "input": "ali", + "operator": "matches", + "term": { + "domain": null, + "fidelity": "probabilistic", + "index": "match", + "operator": "matches", + "term": "30009d00c100" + } + } + ], + "target_limit": [ + { + "field": "name", + "input": "ali", + "limit": 53954, + "operator": "matches", + "stats": { + "histogram": [ + [ + 40, + 500000 + ], + [ + 120, + 500000 + ] + ], + "rows": 1000000 + }, + "term": "30009d00c100", + "want": 10 + }, + { + "field": "name", + "input": "ali", + "limit": 3825, + "operator": "matches", + "stats": { + "confidence": 0.5, + "histogram": [ + [ + 40, + 1000000 + ] + ], + "rows": 1000000 + }, + "term": "30009d00c100", + "want": 10 + }, + { + "field": "name", + "input": "ali", + "limit": null, + "operator": "matches", + "stats": { + "rows": 1000000 + }, + "term": "30009d00c100", + "want": 10 + }, + { + "field": "name", + "input": "alice", + "limit": 611, + "operator": "matches", + "stats": { + "histogram": [ + [ + 40, + 500000 + ], + [ + 120, + 500000 + ] + ], + "rows": 1000000 + }, + "term": "2800300053006d00890095009d00c100fe00", + "want": 10 + }, + { + "field": "name", + "input": "alice", + "limit": 10, + "operator": "matches", + "stats": { + "confidence": 0.5, + "histogram": [ + [ + 40, + 1000000 + ] + ], + "rows": 1000000 + }, + "term": "2800300053006d00890095009d00c100fe00", + "want": 10 + }, + { + "field": "name", + "input": "alice", + "limit": null, + "operator": "matches", + "stats": { + "rows": 1000000 + }, + "term": "2800300053006d00890095009d00c100fe00", + "want": 10 + } + ], + "verify": { + "field": "name", + "input": "alice", + "operator": "matches", + "values": [ + "Alice Smith", + "alicia", + "Bob", + "malice", + "ALICE", + null + ], + "verified": [ + true, + false, + false, + true, + true, + false + ] + } +} diff --git a/packages/stack-encrypt/tests/query_operators.rs b/packages/stack-encrypt/tests/query_operators.rs new file mode 100644 index 000000000..d8e133690 --- /dev/null +++ b/packages/stack-encrypt/tests/query_operators.rs @@ -0,0 +1,354 @@ +//! The query fixture: a query a binding names by operator derives the term +//! the typed query chain derives, and a probabilistic term's two questions +//! answer as the typed `MatchQuery` answers them (ADR-0009; ADR-0007, +//! amended 2026-10-16). `tests/fixtures/query_operators.json` records each +//! query's term, a page of verified values and a few fetch limits, so the +//! Go SDK's tests can hold the guest's `se_query_term`, `se_verify` and +//! `se_target_limit` exports to the same results; its README gives the +//! schema. +//! +//! Every query derives under the deterministic key source the record +//! fixture uses, whose index key is `FakeDataKeySource`'s, so nothing in the +//! file depends on a nonce: a run that derives other bytes fails, and +//! `STACK_ENCRYPT_UPDATE_FIXTURES=1 cargo test --test query_operators +//! --all-features` rewrites the file when a derivation changes on purpose. +#![cfg(feature = "dynamic")] + +mod common; + +use std::path::PathBuf; + +use common::{deterministic_cipher, DeterministicSource}; +use serde_json::{json, Value as Json}; +use stack_encrypt::dynamic::{self, record, FfiValue, NoTargets, OpSpec, Operand, QueryTerm}; +use stack_encrypt::op::{self, Limit, Probabilistic, Stats}; +use stack_encrypt::{Equality, KeysetCipher, Match, Ore, Plan, StackCipher}; + +const SEED: [u8; 32] = *b"stack-encrypt record fixture v1 "; + +/// The declaration every query runs on, in the data grammar. +fn plan_json() -> Json { + json!({ + "email": { "context": ["users", "email"], "outputs": ["c", "eq"], "type": "string" }, + "age": { "context": ["users", "age"], "outputs": ["c", "eq", "ore"], "type": "uint32" }, + "name": { "context": ["users", "name"], "outputs": ["c", "match"], "type": "string" } + }) +} + +/// Each query: a field, an operator and its input, as a binding sends them. +fn queries() -> Vec<(&'static str, OpSpec, Json)> { + vec![ + ("email", OpSpec::Eq, json!("bob@example.com")), + ("age", OpSpec::Eq, json!(34)), + ("age", OpSpec::Lt, json!(40)), + ("age", OpSpec::Lte, json!(40)), + ("age", OpSpec::Gt, json!(18)), + ("age", OpSpec::Gte, json!(18)), + ("age", OpSpec::Between, json!([18, 65])), + ("name", OpSpec::Matches, json!("alice")), + ("name", OpSpec::Matches, json!("ali")), + ] +} + +/// The page `verify` answers for `matches "alice"` on `name`. +fn page() -> Json { + json!(["Alice Smith", "alicia", "Bob", "malice", "ALICE", null]) +} + +/// The statistics each recorded limit is sized under. +fn stats_cases() -> Vec<(Json, Stats)> { + vec![ + ( + json!({ "rows": 1_000_000, "histogram": [[40, 500_000], [120, 500_000]] }), + Stats::measured(1_000_000, [(40, 500_000), (120, 500_000)]), + ), + ( + json!({ "rows": 1_000_000, "histogram": [[40, 1_000_000]], "confidence": 0.5 }), + Stats::measured(1_000_000, [(40, 1_000_000)]).with_confidence(0.5), + ), + (json!({ "rows": 1_000_000 }), Stats::declared(1_000_000)), + ] +} + +fn ffi_of_json(value: &Json) -> FfiValue { + match value { + Json::Null => FfiValue::Null, + Json::String(s) => FfiValue::String(s.as_str().into()), + Json::Array(items) => FfiValue::Array(items.iter().map(ffi_of_json).collect()), + Json::Object(entries) => FfiValue::Object( + entries + .iter() + .map(|(k, v)| (k.clone(), ffi_of_json(v))) + .collect(), + ), + Json::Number(n) => match n.as_u64() { + Some(n) => FfiValue::UInt64(n), + None => FfiValue::Float64(n.as_f64().expect("a number")), + }, + Json::Bool(b) => FfiValue::Bool(*b), + } +} + +fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|b| format!("{b:02x}")).collect() +} + +/// A query term as the fixture holds it: the wire form's keys, the bytes in +/// hex. +fn json_of_term(term: &QueryTerm) -> Json { + json!({ + "operator": term.operator().name(), + "fidelity": term.fidelity().name(), + "index": term.index().key(), + "domain": term.domain(), + "term": match term.operand() { + Operand::One(bytes) => json!(hex(bytes)), + Operand::Range { lo, hi } => json!([hex(lo), hex(hi)]), + }, + }) +} + +fn limit_json(limit: Limit) -> Json { + limit.rows().map_or(Json::Null, |rows| json!(rows)) +} + +/// The typed chain's term for the same query, as the operand's bytes: the +/// proof that the data path derives what a Rust caller does. +async fn typed_operand( + cipher: &StackCipher, + field: &str, + operator: OpSpec, + input: &Json, +) -> Operand { + let int = |value: &Json| u32::try_from(value.as_u64().expect("an integer")).expect("a u32"); + let text = |value: &Json| value.as_str().expect("text").to_owned(); + let age = Plan::context("users/age") + .with::((Equality, Ore)) + .build() + .expect("plan"); + let one = Operand::One; + match (field, operator) { + ("email", OpSpec::Eq) => { + let plan = Plan::context("users/email") + .with::(Equality) + .build() + .expect("plan"); + one(cipher + .query(op::Eq, &text(input)) + .on(&plan) + .await + .expect("eq") + .to_bytes()) + } + ("age", OpSpec::Eq) => one(cipher + .query(op::Eq, &int(input)) + .on(&age) + .await + .expect("eq") + .to_bytes()), + ("age", OpSpec::Lt) => one(cipher + .query(op::Lt, &int(input)) + .on(&age) + .await + .expect("lt") + .to_bytes()), + ("age", OpSpec::Lte) => one(cipher + .query(op::Lte, &int(input)) + .on(&age) + .await + .expect("lte") + .to_bytes()), + ("age", OpSpec::Gt) => one(cipher + .query(op::Gt, &int(input)) + .on(&age) + .await + .expect("gt") + .to_bytes()), + ("age", OpSpec::Gte) => one(cipher + .query(op::Gte, &int(input)) + .on(&age) + .await + .expect("gte") + .to_bytes()), + ("age", OpSpec::Between) => { + let range = cipher + .query(op::Between, &(int(&input[0]), int(&input[1]))) + .on(&age) + .await + .expect("between"); + Operand::Range { + lo: range.lo().to_bytes(), + hi: range.hi().to_bytes(), + } + } + ("name", OpSpec::Matches) => one(typed_match(cipher, &text(input)).await.to_bytes()), + other => panic!("no typed query for {other:?}"), + } +} + +async fn typed_match( + cipher: &StackCipher, + input: &str, +) -> stack_encrypt::sem::MatchQuery { + let plan = Plan::context("users/name") + .with::(Match::default()) + .build() + .expect("plan"); + cipher + .query(op::Matches, input) + .on(&plan) + .await + .expect("matches") +} + +async fn data_term( + keyset: &KeysetCipher<'_, DeterministicSource>, + plan: &record::Plan, + field: &str, + operator: OpSpec, + input: &Json, +) -> QueryTerm { + dynamic::query_term( + keyset, + plan, + field, + operator, + ffi_of_json(input), + &NoTargets, + ) + .expect("the query is answered") + .await + .expect("the term derives") +} + +/// The fixture as this run derives it, checked against the typed path as +/// it is built. +async fn derived() -> Json { + let cipher = deterministic_cipher(SEED).await; + let keyset = cipher.default_keyset(); + let plan = record::plan(ffi_of_json(&plan_json())).expect("the plan parses"); + + let mut terms = Vec::new(); + for (field, operator, input) in queries() { + let term = data_term(&keyset, &plan, field, operator, &input).await; + assert_eq!( + term.operand(), + &typed_operand(&cipher, field, operator, &input).await, + "{operator} on {field}: the data path derives the typed chain's term" + ); + terms.push(json!({ + "field": field, + "operator": operator.name(), + "input": input, + "term": json_of_term(&term), + })); + } + + let typed = typed_match(&cipher, "alice").await; + let values: Vec = page() + .as_array() + .expect("a list") + .iter() + .map(ffi_of_json) + .collect(); + let verified = dynamic::verify( + &plan, + "name", + OpSpec::Matches, + ffi_of_json(&json!("alice")), + &values, + &NoTargets, + ) + .expect("verify"); + let typed_verified: Vec = page() + .as_array() + .expect("a list") + .iter() + .map(|value| value.as_str().is_some_and(|text| typed.verify(text))) + .collect(); + assert_eq!(verified, typed_verified, "verify is the typed MatchQuery's"); + + let mut limits = Vec::new(); + for input in ["ali", "alice"] { + let term = data_term(&keyset, &plan, "name", OpSpec::Matches, &json!(input)).await; + let Operand::One(bytes) = term.operand() else { + panic!("a match term is one term"); + }; + let typed = typed_match(&cipher, input).await; + for (stats_json, stats) in stats_cases() { + let read = Stats::from_value(&ffi_of_json(&stats_json)).expect("stats read"); + assert_eq!(read, stats, "the wire form reads as the typed stats"); + let limit = + dynamic::target_limit(&plan, "name", OpSpec::Matches, bytes, 10, &read, &NoTargets) + .expect("a limit"); + assert_eq!( + limit, + typed.target_limit(10, &stats), + "target_limit is the typed MatchQuery's" + ); + limits.push(json!({ + "field": "name", + "operator": "matches", + "input": input, + "term": hex(bytes), + "want": 10, + "stats": stats_json, + "limit": limit_json(limit), + })); + } + } + + json!({ + "_comment": "Query terms derived by operator through the data path (stack_encrypt::dynamic::query_term), each equal to the typed query chain's, with a verified page and fetch limits. Read by tests/query_operators.rs and the Go SDK's tests; the schema is in README.md beside this file. Regenerate with STACK_ENCRYPT_UPDATE_FIXTURES=1; do not edit by hand.", + "key_source": { "kind": "deterministic-sha256", "seed": hex(&SEED) }, + "keyset_id": keyset.keyset_id().to_string(), + "plan": plan_json(), + "queries": terms, + "verify": { + "field": "name", + "operator": "matches", + "input": "alice", + "values": page(), + "verified": verified, + }, + "target_limit": limits, + }) +} + +fn fixture_path() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/query_operators.json") +} + +/// The committed fixture is what this run derives. With +/// `STACK_ENCRYPT_UPDATE_FIXTURES=1` it is written first. +#[tokio::test] +async fn the_committed_query_fixture_is_what_the_engine_derives() { + let now = derived().await; + let path = fixture_path(); + if std::env::var_os("STACK_ENCRYPT_UPDATE_FIXTURES").is_some() { + let mut text = serde_json::to_string_pretty(&now).expect("serialise"); + text.push('\n'); + std::fs::write(&path, text).expect("write the fixture"); + } + let committed: Json = + serde_json::from_str(&std::fs::read_to_string(&path).expect("the fixture is committed")) + .expect("the fixture is JSON"); + assert_eq!(committed, now, "the committed query fixture is today's"); +} + +/// The fixture pins more than one answer of each kind: a verified page +/// with both outcomes, and limits both finite and unbounded. +#[tokio::test] +async fn the_fixture_pins_both_outcomes() { + let now = derived().await; + let verified = now["verify"]["verified"].as_array().expect("a list"); + assert!(verified.contains(&json!(true)) && verified.contains(&json!(false))); + let limits: Vec<&Json> = now["target_limit"] + .as_array() + .expect("a list") + .iter() + .map(|case| &case["limit"]) + .collect(); + assert!(limits.iter().any(|limit| limit.is_u64())); + assert!(limits.iter().any(|limit| limit.is_null())); +} diff --git a/packages/stack-encrypt/tests/ui/op_at_path_inherits_fidelity.stderr b/packages/stack-encrypt/tests/ui/op_at_path_inherits_fidelity.stderr index 64fc20e2f..8463a62f4 100644 --- a/packages/stack-encrypt/tests/ui/op_at_path_inherits_fidelity.stderr +++ b/packages/stack-encrypt/tests/ui/op_at_path_inherits_fidelity.stderr @@ -4,9 +4,9 @@ error[E0277]: `EqualityTerm` cannot answer `AtPath` 9 | answers::, EqualityTerm>(); | ^^^^^^^^^^^^ this query term does not meet the operator's requirement | - = help: the trait `Probabilistic` is not implemented for `EqualityTerm` + = help: the trait `stack_encrypt::op::Probabilistic` is not implemented for `EqualityTerm` = note: a term that answers `op::Matches` must implement `op::Probabilistic` -help: the trait `Probabilistic` is implemented for `MatchQuery` +help: the trait `stack_encrypt::op::Probabilistic` is implemented for `MatchQuery` --> src/sem/mod.rs | | impl crate::op::Probabilistic for MatchQuery { diff --git a/packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr b/packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr index af0a38a17..dff82a9e1 100644 --- a/packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr +++ b/packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr @@ -1,4 +1,4 @@ -error[E0599]: the method `on` exists for struct `QueryBuilder<'_, Between, u32, FakeDataKeySource>`, but its trait bounds were not satisfied +error[E0599]: the method `on` exists for struct `QueryBuilder<'_, stack_encrypt::op::Between, u32, FakeDataKeySource>`, but its trait bounds were not satisfied --> tests/ui/op_between_takes_a_pair.rs:12:47 | 12 | let _ = cipher.query(op::Between, &40u32).on(&age_plan); @@ -15,7 +15,7 @@ error[E0271]: type mismatch resolving `<(u32, {integer}) as Input>::Plainte | = note: expected type `u32` found tuple `(u32, {integer})` - = note: required for `ValuePlan>` to implement `Chooses` + = note: required for `ValuePlan>` to implement `Chooses` error[E0277]: the trait bound `(u32, {integer}): Ordered` is not satisfied --> tests/ui/op_between_takes_a_pair.rs:13:48 @@ -33,5 +33,5 @@ error[E0277]: the trait bound `(u32, {integer}): Ordered` is not satisfied u16 u32 and $N others - = note: required for `Lt` to implement `DefaultIndex<(u32, {integer})>` - = note: required for `ValuePlan>` to implement `Chooses` + = note: required for `stack_encrypt::op::Lt` to implement `DefaultIndex<(u32, {integer})>` + = note: required for `ValuePlan>` to implement `Chooses` diff --git a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr index ceaa770be..248900eee 100644 --- a/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr +++ b/packages/stack-encrypt/tests/ui/op_inexact_operator_needs_probabilistic.stderr @@ -4,9 +4,9 @@ error[E0277]: `EqualityTerm` cannot answer `stack_encrypt::op::Matches` 24 | type Query = EqualityTerm; | ^^^^^^^^^^^^ this query term does not meet the operator's requirement | - = help: the trait `Probabilistic` is not implemented for `EqualityTerm` + = help: the trait `stack_encrypt::op::Probabilistic` is not implemented for `EqualityTerm` = note: a term that answers `op::Matches` must implement `op::Probabilistic` -help: the trait `Probabilistic` is implemented for `MatchQuery` +help: the trait `stack_encrypt::op::Probabilistic` is implemented for `MatchQuery` --> src/sem/mod.rs | | impl crate::op::Probabilistic for MatchQuery { diff --git a/packages/stack-encrypt/tests/ui/op_json_not_answered.stderr b/packages/stack-encrypt/tests/ui/op_json_not_answered.stderr index 1105d61e6..b2235d87a 100644 --- a/packages/stack-encrypt/tests/ui/op_json_not_answered.stderr +++ b/packages/stack-encrypt/tests/ui/op_json_not_answered.stderr @@ -7,12 +7,12 @@ error[E0277]: `Contains` has no default index given `std::string::String` = help: the trait `DefaultIndex` is not implemented for `Contains` = note: `op::Eq` defaults to `Equality`; `op::Matches` defaults to `Match` with the default configuration, which is case-insensitive and over text only; `op::Lt`, `Lte`, `Gt`, `Gte` and `Between` default to `Ore` over an integer, and `Between` takes a pair; name another index, such as `Ope`, with `.via(index)` = help: the following other types implement trait `DefaultIndex`: - `Between` implements `DefaultIndex<(S, S)>` - `Gt` implements `DefaultIndex` - `Gte` implements `DefaultIndex` - `Lt` implements `DefaultIndex` - `Lte` implements `DefaultIndex` + `stack_encrypt::op::Between` implements `DefaultIndex<(S, S)>` `stack_encrypt::op::Eq` implements `DefaultIndex` + `stack_encrypt::op::Gt` implements `DefaultIndex` + `stack_encrypt::op::Gte` implements `DefaultIndex` + `stack_encrypt::op::Lt` implements `DefaultIndex` + `stack_encrypt::op::Lte` implements `DefaultIndex` `stack_encrypt::op::Matches<::Case>` implements `DefaultIndex` = note: required for `ValuePlan>` to implement `Chooses` diff --git a/packages/stack-encrypt/tests/ui/op_order_on_text.stderr b/packages/stack-encrypt/tests/ui/op_order_on_text.stderr index 9d2a9bacc..fe5d7aa30 100644 --- a/packages/stack-encrypt/tests/ui/op_order_on_text.stderr +++ b/packages/stack-encrypt/tests/ui/op_order_on_text.stderr @@ -14,5 +14,5 @@ error[E0277]: the trait bound `std::string::String: Ordered` is not satisfied u16 u32 and $N others - = note: required for `Lt` to implement `DefaultIndex` - = note: required for `ValuePlan>` to implement `Chooses` + = note: required for `stack_encrypt::op::Lt` to implement `DefaultIndex` + = note: required for `ValuePlan>` to implement `Chooses` From 7dd62ca8c8a536b7df566f03d67ed375edfe4565 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 03:13:20 +1100 Subject: [PATCH 33/38] feat(golang): export se_query_term, se_verify and se_target_limit so the Go SDK can query by operator without a tokenizer of its own The encrypt guest derives a query term from a plan, a field, an operator name and its input, verifies a page of decrypted values in one call, and sizes a probabilistic query's fetch from its term and the table's statistics; the last two use no key and run with no cipher. se_targets entries carry the operators each EQL type answers, and the Go reader requires them. Cipher.QueryTerm, Cipher.Verify and Cipher.TargetLimit wrap the exports for generated code, and the Go tests hold them to the Rust query fixture through the deterministic guest. Part of #1188 --- languages/golang/encrypt/checker.go | 64 ++- languages/golang/encrypt/guest.go | 4 + languages/golang/encrypt/guest/src/abi.rs | 124 +++++- languages/golang/encrypt/guest/src/lib.rs | 4 +- languages/golang/encrypt/guest/src/ops.rs | 149 ++++++- languages/golang/encrypt/guest/tests/leak.rs | 97 +++++ .../golang/encrypt/guest/tests/native_ops.rs | 377 +++++++++++++++++- languages/golang/encrypt/guest_test.go | 2 +- languages/golang/encrypt/query.go | 302 ++++++++++++++ languages/golang/encrypt/query_test.go | 362 +++++++++++++++++ languages/golang/encrypt/records.go | 45 ++- languages/golang/encrypt/targets_test.go | 68 +++- languages/golang/internal/record/record.go | 36 +- 13 files changed, 1593 insertions(+), 41 deletions(-) create mode 100644 languages/golang/encrypt/query.go create mode 100644 languages/golang/encrypt/query_test.go diff --git a/languages/golang/encrypt/checker.go b/languages/golang/encrypt/checker.go index 7f3a99e25..807967325 100644 --- a/languages/golang/encrypt/checker.go +++ b/languages/golang/encrypt/checker.go @@ -165,6 +165,8 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { t.Producible = b case "reason": t.Reason, err = targetOptionalString(f) + case "operators": + t.Operators, err = targetOperators(f.Value) default: return t, fmt.Errorf("unknown key %q", f.Key) } @@ -175,7 +177,7 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { if t.Name == "" { return t, errors.New("no name") } - for _, key := range []string{"indexes", "producible"} { + for _, key := range []string{"indexes", "producible", "operators"} { if !seen[key] { return t, fmt.Errorf("no %s", key) } @@ -183,6 +185,66 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { return t, nil } +// targetOperators reads an entry's operators: a list of objects, each with +// exactly the keys operator, fidelity, arity and index, every value one the +// engine writes. +func targetOperators(value any) ([]record.TargetOperator, error) { + list, ok := value.([]any) + if !ok { + return nil, fmt.Errorf("operators is %T, not a list", value) + } + operators := make([]record.TargetOperator, 0, len(list)) + for i, item := range list { + entry, ok := item.(vcvalue.Object) + if !ok { + return nil, fmt.Errorf("operator %d is %T, not an object", i, item) + } + var op record.TargetOperator + seen := map[string]bool{} + for _, f := range entry { + if seen[f.Key] { + return nil, fmt.Errorf("operator %d: key %q twice", i, f.Key) + } + seen[f.Key] = true + s, err := targetString(f) + if err != nil { + return nil, fmt.Errorf("operator %d: %v", i, err) + } + switch f.Key { + case "operator": + op.Name = s + case "fidelity": + op.Fidelity = record.Fidelity(s) + if op.Fidelity != record.Exact && op.Fidelity != record.Probabilistic { + return nil, fmt.Errorf("operator %d: unknown fidelity %q", i, s) + } + case "arity": + switch s { + case "one": + case "pair": + op.Pair = true + default: + return nil, fmt.Errorf("operator %d: unknown arity %q", i, s) + } + case "index": + switch o := record.Output(s); o { + case record.Equality, record.Match, record.Ore, record.Ope: + op.Index = o + default: + return nil, fmt.Errorf("operator %d: unknown index %q", i, s) + } + default: + return nil, fmt.Errorf("operator %d: unknown key %q", i, f.Key) + } + } + if len(seen) != 4 || op.Name == "" { + return nil, fmt.Errorf("operator %d: not every key, or no name", i) + } + operators = append(operators, op) + } + return operators, nil +} + func targetString(f vcvalue.Field) (string, error) { s, ok := f.Value.(string) if !ok { diff --git a/languages/golang/encrypt/guest.go b/languages/golang/encrypt/guest.go index c5bbecbd4..78ef91c4d 100644 --- a/languages/golang/encrypt/guest.go +++ b/languages/golang/encrypt/guest.go @@ -85,6 +85,7 @@ type instance struct { exports guest.Exports cipherInit, shutdown, keyset api.Function term, query api.Function + queryTerm, verify, limit api.Function encryptRecord, decryptRecord api.Function planCheck, targets api.Function } @@ -172,6 +173,9 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo "se_keyset": &inst.keyset, "se_term": &inst.term, "se_query": &inst.query, + "se_query_term": &inst.queryTerm, + "se_verify": &inst.verify, + "se_target_limit": &inst.limit, "se_encrypt_record": &inst.encryptRecord, "se_decrypt_record": &inst.decryptRecord, "se_plan_check": &inst.planCheck, diff --git a/languages/golang/encrypt/guest/src/abi.rs b/languages/golang/encrypt/guest/src/abi.rs index 7abbdd660..e608bb90b 100644 --- a/languages/golang/encrypt/guest/src/abi.rs +++ b/languages/golang/encrypt/guest/src/abi.rs @@ -6,8 +6,10 @@ //! //! - **Every export handed plaintext wipes that buffer in place before it //! returns**, rather than leaving it for `se_dealloc`: [`se_cipher_init`] -//! (the config carries the client key), and [`se_term`], [`se_query`] and -//! [`se_encrypt_record`] (their value/source buffers). The host's plaintext therefore lives no longer +//! (the config carries the client key), and [`se_term`], [`se_query`], +//! [`se_query_term`], [`se_verify`] and [`se_encrypt_record`] (their +//! value, source and input buffers, and [`se_verify`]'s page of decrypted +//! values). The host's plaintext therefore lives no longer //! than the call, instead of until the host gets round to releasing it. //! **A host must not read a plaintext input buffer back after the call, or //! pass the same buffer to two calls** — it will be zeros. Option, context, @@ -53,7 +55,8 @@ //! otherwise): each is a [`stack_encrypt::NonEmpty`] from the moment it is //! parsed, and the sealing and opening sides bind that one value. //! [`se_plan_check`] and [`se_targets`] answer the Go generator's questions -//! and need no cipher. +//! and need no cipher; nor do [`se_verify`] and [`se_target_limit`], a +//! probabilistic query's two questions, which use no key. //! //! Every export decodes and validates *all* of its inputs — the operation //! payload, the plan or context, the term kind, the value against the @@ -500,6 +503,121 @@ pub unsafe extern "C" fn se_query( }) } +/// Derive the query term for one plan field, an operator and its input +/// under the keyset `opts` selects ([`ops::query_term`]): a codec-encoded +/// input (one value, or for `between` a list of two), the codec-encoded +/// plan, and the field's and the operator's names (UTF-8 bytes) in; the +/// term's codec-encoded object out. An operator the field does not answer +/// is `STATUS_ENCODING` naming the field, as a mistyped value is, before +/// the cipher is consulted; the refusals the typed path makes before a term +/// exists (a range the wrong way round, match text that cuts no token or +/// does not normalize) are `STATUS_TERM`, also before the cipher. The input +/// buffer is wiped in place. +/// +/// # Safety +/// +/// As for [`se_term`]. +#[no_mangle] +pub unsafe extern "C" fn se_query_term( + val_ptr: *mut u8, + val_len: u32, + plan_ptr: *const u8, + plan_len: u32, + field_ptr: *const u8, + field_len: u32, + op_ptr: *const u8, + op_len: u32, + opt_ptr: *const u8, + opt_len: u32, +) -> u64 { + export(|| { + let value = unsafe { take_plaintext(val_ptr, val_len)? }; + let value = value.as_slice(); + // SAFETY: host-owned ranges the export was handed; the borrows end + // before it returns and before any wipe of an overlapping range. + let plan = unsafe { input(plan_ptr, plan_len)? }; + let field = unsafe { input(field_ptr, field_len)? }; + let operator = unsafe { input(op_ptr, op_len)? }; + let opts = unsafe { input(opt_ptr, opt_len)? }; + ops::validate::query_term(value, plan, field, operator)?; + with_keyset(opts, |keyset| { + block_on(ops::query_term(keyset, value, plan, field, operator)) + }) + }) +} + +/// Verify a page of decrypted values against a probabilistic query +/// ([`ops::verify`]): the query's codec-encoded input and a codec-encoded +/// list of values, the codec-encoded plan, and the field's and the +/// operator's names in; a codec-encoded list of booleans out, one per value. +/// Both plaintext buffers, the input and the values, are wiped in place. An +/// exact operator has nothing to verify and is `STATUS_ENCODING`. Needs no +/// cipher: it works in every instance state, as [`se_plan_check`] does. +/// +/// # Safety +/// +/// As for [`se_term`]. +#[no_mangle] +pub unsafe extern "C" fn se_verify( + in_ptr: *mut u8, + in_len: u32, + vals_ptr: *mut u8, + vals_len: u32, + plan_ptr: *const u8, + plan_len: u32, + field_ptr: *const u8, + field_len: u32, + op_ptr: *const u8, + op_len: u32, +) -> u64 { + export(|| { + let query = unsafe { take_plaintext(in_ptr, in_len)? }; + let values = unsafe { take_plaintext(vals_ptr, vals_len)? }; + // SAFETY: host-owned ranges the export was handed; the borrows end + // before it returns. + let plan = unsafe { input(plan_ptr, plan_len)? }; + let field = unsafe { input(field_ptr, field_len)? }; + let operator = unsafe { input(op_ptr, op_len)? }; + ops::verify(query.as_slice(), values.as_slice(), plan, field, operator) + }) +} + +/// How many rows a probabilistic query should fetch for `want` of them to +/// verify ([`ops::target_limit`]): the term's bytes as [`se_query_term`] +/// returned them, the codec-encoded statistics, the codec-encoded plan, +/// and the field's and the operator's names in, with `want`; the +/// codec-encoded limit out, an unsigned integer or null. An exact operator +/// has no fetch to size and is `STATUS_ENCODING`. Needs no cipher. +/// +/// # Safety +/// +/// As for [`se_term`]. +#[no_mangle] +pub unsafe extern "C" fn se_target_limit( + term_ptr: *const u8, + term_len: u32, + stats_ptr: *const u8, + stats_len: u32, + plan_ptr: *const u8, + plan_len: u32, + field_ptr: *const u8, + field_len: u32, + op_ptr: *const u8, + op_len: u32, + want: u64, +) -> u64 { + export(|| { + // SAFETY: host-owned ranges the export was handed; the borrows end + // before it returns. + let term = unsafe { input(term_ptr, term_len)? }; + let stats = unsafe { input(stats_ptr, stats_len)? }; + let plan = unsafe { input(plan_ptr, plan_len)? }; + let field = unsafe { input(field_ptr, field_len)? }; + let operator = unsafe { input(op_ptr, op_len)? }; + ops::target_limit(term, stats, plan, field, operator, want) + }) +} + /// The EQL types this build knows, as a codec-encoded /// `{"targets": [...]}` ([`ops::targets`]): the catalog in the `eql` build, /// nothing in the other. Needs no cipher. diff --git a/languages/golang/encrypt/guest/src/lib.rs b/languages/golang/encrypt/guest/src/lib.rs index e402e14d0..c22d3a2dc 100644 --- a/languages/golang/encrypt/guest/src/lib.rs +++ b/languages/golang/encrypt/guest/src/lib.rs @@ -46,7 +46,9 @@ //! //! The exports are the record path (`se_encrypt_record`, `se_decrypt_record`), //! per-field term derivation (`se_term`), a target field's EQL query value -//! (`se_query`), keyset resolution (`se_keyset`), the generator's two +//! (`se_query`), a query named by operator and a probabilistic query's two +//! questions (`se_query_term`, `se_verify`, `se_target_limit`; ADR-0007, +//! amended 2026-10-16), keyset resolution (`se_keyset`), the generator's two //! questions (`se_plan_check`, `se_targets`) and the lifetime pair //! (`se_cipher_init`, `se_shutdown`). There is no whole-value export: the //! Go SDK seals every value under a declaration (ADR-0007). diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index d7bc64d23..be09d95d9 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -238,6 +238,132 @@ where .map_err(|e| fail_error(&e)) } +// ============================================================================= +// Queries by operator +// ============================================================================= + +/// Derive the query term for one plan field, an operator and its input +/// (ADR-0009; ADR-0007, amended 2026-10-16): a codec-encoded input, the +/// codec-encoded plan, the field's name and the operator's name (both UTF-8) +/// in, the term's codec-encoded object out ([`dynamic::QueryTerm::to_value`]): +/// +/// ```text +/// { "operator": "matches", "fidelity": "probabilistic", "index": "match", +/// "domain": null, "term": } +/// ``` +/// +/// The input is one value for every operator but `between`, whose input is +/// a list of two, low then high. It is read as the field's declared kind +/// first, so a host's float `34.0` queries a `uint64` field. A field with +/// indexes derives the term of the index that answers the operator, as +/// bytes; `between` derives one per bound, as a list of two. A field that +/// names an EQL type runs the type's query twin through this build's +/// resolver, and `term` holds the twin's JSON bytes and `domain` its +/// PostgreSQL domain. No data key is minted. +pub async fn query_term( + cipher: &KeysetCipher<'_, K>, + input: &[u8], + plan: &[u8], + field: &[u8], + operator: &[u8], +) -> Result, u32> +where + K: DataKeySource + Sync + 'static, +{ + let (plan, field, operator) = parse_query(plan, field, operator)?; + let term = dynamic::query_term( + cipher, + &plan, + field, + operator, + decode_value(input)?, + &resolver(), + ) + .map_err(|e| fail_dynamic(&e))? + .await + .map_err(|e| fail_error(&e))?; + encode_value(term.to_value()) +} + +/// Verify a page of decrypted values against a probabilistic query: the +/// query's codec-encoded input and a codec-encoded list of values, with the +/// plan, the field's name and the operator's name, in; a codec-encoded list +/// of booleans out, one per value in order ([`dynamic::verify`]). The query +/// is named as it was derived, and its tokens are cut again from the input: +/// they are plaintext, so no term carries them. A null value is `false`. +/// Needs no cipher and no key. +pub fn verify( + input: &[u8], + values: &[u8], + plan: &[u8], + field: &[u8], + operator: &[u8], +) -> Result, u32> { + let (plan, field, operator) = parse_query(plan, field, operator)?; + let FfiValue::Array(values) = decode_value(values)? else { + return Err(last_error::malformed("the values to verify are not a list")); + }; + let verified = dynamic::verify( + &plan, + field, + operator, + decode_value(input)?, + &values, + &resolver(), + ) + .map_err(|e| fail_dynamic(&e))?; + encode_value(FfiValue::Array( + verified.into_iter().map(FfiValue::Bool).collect(), + )) +} + +/// How many rows a probabilistic query should fetch for `want` of them to +/// verify: the term's bytes as [`query_term`] returned them (not a frame), +/// the codec-encoded statistics +/// ([`Stats::from_value`](stack_encrypt::op::Stats::from_value)), the plan, +/// the field's name and the operator's name in; the codec-encoded limit out, +/// an unsigned 64-bit integer or null for no finite fetch +/// ([`dynamic::limit_to_value`]). Needs no cipher and no key. +pub fn target_limit( + term: &[u8], + stats: &[u8], + plan: &[u8], + field: &[u8], + operator: &[u8], + want: u64, +) -> Result, u32> { + let (plan, field, operator) = parse_query(plan, field, operator)?; + let stats = stack_encrypt::op::Stats::from_value(&decode_value(stats)?) + .map_err(|e| fail_dynamic(&e))?; + let limit = dynamic::target_limit(&plan, field, operator, term, want, &stats, &resolver()) + .map_err(|e| fail_dynamic(&e))?; + encode_value(dynamic::limit_to_value(limit)) +} + +/// The plan, field name and operator a query names: the plan parsed against +/// this build's resolver, the field's name UTF-8, and the operator one the +/// engine knows ([`dynamic::OpSpec::parse`]); an unknown operator is +/// refused naming the field, as an operator the field does not answer is. +fn parse_query<'f>( + plan: &[u8], + field: &'f [u8], + operator: &[u8], +) -> Result<(dynamic::record::Plan, &'f str, dynamic::OpSpec), u32> { + let plan = parse_plan(plan)?; + let field = std::str::from_utf8(field) + .map_err(|_| last_error::malformed("the field name is not UTF-8"))?; + let operator = std::str::from_utf8(operator) + .ok() + .and_then(dynamic::OpSpec::parse) + .ok_or_else(|| { + fail_dynamic(&dynamic::Error::Source { + field: Some(field.to_owned()), + reason: dynamic::Reason::UnknownOperator, + }) + })?; + Ok((plan, field, operator)) +} + /// A codec-encoded plan, parsed against this build's resolver: a target /// name this build cannot run is refused here, the same way at every /// export that takes a plan. @@ -285,6 +411,11 @@ pub fn plan_check(plan: &[u8]) -> Result<(), u32> { /// | `query_sql_domain` | string or null | the Postgres domain of the query value | /// | `producible` | bool | whether this build produces the type | /// | `reason` | string or null | why not, when `producible` is false | +/// | `operators` | list of objects | the operators the engine answers for the type, each `{"operator", "fidelity", "arity", "index"}` ([`TargetDescriptor`](stack_encrypt::dynamic::TargetDescriptor#wire-format)) | +/// +/// The first ten keys are eql-bindings' row; `operators` is the engine's, +/// computed from the row, so a generator asks the engine which operators a +/// type answers and how far each answer can be trusted. pub fn targets() -> Result, u32> { use stack_encrypt::dynamic::TargetResolver as _; encode_value(FfiValue::Object(vec![( @@ -363,6 +494,17 @@ pub mod validate { Ok(()) } + /// A query's inputs, as [`query_term`](super::query_term) takes them: + /// the plan parses, the field is one of its fields, the operator is one + /// the engine knows and the field answers, and the input reads as the + /// field's kind and passes the refusals the typed path makes before a + /// term exists ([`dynamic::check_query`]). + pub fn query_term(input: &[u8], plan: &[u8], field: &[u8], operator: &[u8]) -> Result<(), u32> { + let (plan, field, operator) = parse_query(plan, field, operator)?; + dynamic::check_query(&plan, field, operator, decode_value(input)?, &resolver()) + .map_err(|e| fail_dynamic(&e)) + } + /// A record tree against its plan, as [`decrypt_record`] takes them: /// the plan parses, the tree decodes with well-formed leaves, every /// ciphertext-bearing field has a `"c"` node that is not a passthrough, @@ -686,10 +828,15 @@ mod tests { "query", "query_sql_domain", "producible", - "reason" + "reason", + "operators" ] ); assert!(matches!(text_eq[8].1, FfiValue::Bool(true))); + assert!( + matches!(&text_eq[10].1, FfiValue::Array(operators) if operators.len() == 1), + "TextEq answers one operator" + ); } #[test] diff --git a/languages/golang/encrypt/guest/tests/leak.rs b/languages/golang/encrypt/guest/tests/leak.rs index 46cb0816d..64dd85a31 100644 --- a/languages/golang/encrypt/guest/tests/leak.rs +++ b/languages/golang/encrypt/guest/tests/leak.rs @@ -646,6 +646,103 @@ fn scenarios() -> Vec<(&'static str, FfiValue)> { ), ), ]; + // -- queries by operator ----------------------------------------------- + let marker = || FfiValue::Bytes(vitaminc_protected::Protected::new(PLAINTEXT.into())); + let match_plan = || { + encode(obj(vec![( + "name", + field(label("name"), &["c", "match"], Some("string")), + )])) + }; + let source = &["stack_encrypt::dynamic_source"]; + out.extend([ + ( + "an operator the field does not answer", + failure("not answered", source, || { + block_on(ops::query_term( + &keyset, + &encode(s(PLAINTEXT)), + &plan_under(label("email")), + b"email", + b"lt", + )) + }), + ), + ( + "a query input of another type", + failure("query input", source, || { + block_on(ops::query_term( + &keyset, + &encode(marker()), + &plan_under(label("email")), + b"email", + b"eq", + )) + }), + ), + ( + "an operator name the engine does not know", + failure("unknown operator", source, || { + block_on(ops::query_term( + &keyset, + &encode(s(PLAINTEXT)), + &plan_under(label("email")), + b"email", + PLAINTEXT.as_bytes(), + )) + }), + ), + ( + "verifying an exact operator", + failure("verify exact", source, || { + ops::verify( + &encode(s(PLAINTEXT)), + &encode(FfiValue::Array(vec![s(PLAINTEXT)])), + &plan_under(label("email")), + b"email", + b"eq", + ) + }), + ), + ( + "verifying a value of another type", + failure("verify value", source, || { + ops::verify( + &encode(s(PLAINTEXT)), + &encode(FfiValue::Array(vec![marker()])), + &match_plan(), + b"name", + b"matches", + ) + }), + ), + ( + "a fetch sized from bytes that are not a term", + failure("limit term", source, || { + ops::target_limit( + PLAINTEXT.as_bytes(), + &encode(obj(vec![("rows", FfiValue::UInt64(1))])), + &match_plan(), + b"name", + b"matches", + 10, + ) + }), + ), + ( + "statistics that are not statistics", + failure("limit stats", source, || { + ops::target_limit( + &[1, 0], + &encode(obj(vec![("rows", s(PLAINTEXT))])), + &match_plan(), + b"name", + b"matches", + 10, + ) + }), + ), + ]); // An EQL value whose stored bytes are a JSON string where an object // belongs: serde_json's message would quote the string. #[cfg(feature = "eql")] diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index d002ad70a..4cb971ac3 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -26,7 +26,7 @@ use stack_encrypt::{ }; use stack_encrypt_guest::ops::{self, TERM_EQUALITY, TERM_MATCH, TERM_OPE, TERM_ORE}; use stack_encrypt_guest::status::{ - STATUS_AUTH, STATUS_CONTEXT_MISMATCH, STATUS_ENCODING, STATUS_FOREIGN_KEYSET, + STATUS_AUTH, STATUS_CONTEXT_MISMATCH, STATUS_ENCODING, STATUS_FOREIGN_KEYSET, STATUS_TERM, }; use stack_kms::{ DataKey, DataKeySource, DataKeyWithTag, FakeDataKeySource, GenerateKeyPayload, IndexKeySource, @@ -1370,6 +1370,381 @@ fn targets_lists_what_the_build_holds() { assert_eq!(producible, ["TextEq"], "the engine produces TextEq only"); } +// ============================================================================= +// Queries by operator +// ============================================================================= + +/// The entries of a query term's object, by key. +fn query_term_entries(out: &[u8]) -> Vec<(String, FfiValue)> { + let FfiValue::Object(entries) = decode(out) else { + panic!("a query term is an object"); + }; + entries +} + +fn entry<'a>(entries: &'a [(String, FfiValue)], key: &str) -> &'a FfiValue { + &entries + .iter() + .find(|(k, _)| k == key) + .unwrap_or_else(|| panic!("no {key}")) + .1 +} + +fn bytes_of(value: &FfiValue) -> Vec { + match value { + FfiValue::Bytes(b) => b.risky_ref().to_vec(), + other => panic!("expected bytes, got {}", kind(other)), + } +} + +/// A query by operator derives, through the guest, the bytes the term +/// export derives for the index that answers it, under the field's +/// context: the plan picks the index, and the derivation is the one a +/// stored term came from. +#[test] +fn a_query_term_is_the_term_of_the_index_that_answers_its_operator() { + let cipher = cipher(); + let keyset = cipher.default_keyset(); + let term_of = |value: FfiValue, field: &str, kind: u32| { + block_on(ops::term( + &keyset, + &encode(value), + &encode(label(field)), + kind, + )) + .expect("term") + }; + let query = |input: FfiValue, field: &str, operator: &str| { + let input = encode(input); + assert_eq!( + ops::validate::query_term(&input, &plan(), field.as_bytes(), operator.as_bytes()), + Ok(()), + "{operator} on {field} validates" + ); + query_term_entries( + &block_on(ops::query_term( + &keyset, + &input, + &plan(), + field.as_bytes(), + operator.as_bytes(), + )) + .expect("query term"), + ) + }; + + let eq = query(FfiValue::UInt32(34), "age", "eq"); + assert_eq!(text(entry(&eq, "operator")), "eq"); + assert_eq!(text(entry(&eq, "fidelity")), "exact"); + assert_eq!(text(entry(&eq, "index")), "eq"); + assert!(matches!(entry(&eq, "domain"), FfiValue::Null)); + assert_eq!( + bytes_of(entry(&eq, "term")), + term_of(FfiValue::UInt32(34), "age", TERM_EQUALITY) + ); + + // A host's float reads as the field's integer. + let lt = query(FfiValue::Float64(40.0), "age", "lt"); + assert_eq!(text(entry(<, "index")), "ore"); + assert_eq!( + bytes_of(entry(<, "term")), + term_of(FfiValue::UInt32(40), "age", TERM_ORE) + ); + + let range = query( + FfiValue::Array(vec![FfiValue::UInt32(18), FfiValue::UInt32(65)]), + "age", + "between", + ); + let FfiValue::Array(bounds) = entry(&range, "term") else { + panic!("a range is a list of two terms"); + }; + assert_eq!(bounds.len(), 2); + assert_eq!( + bytes_of(&bounds[0]), + term_of(FfiValue::UInt32(18), "age", TERM_ORE) + ); + assert_eq!( + bytes_of(&bounds[1]), + term_of(FfiValue::UInt32(65), "age", TERM_ORE) + ); + + let matches = query(s("alice"), "name", "matches"); + assert_eq!(text(entry(&matches, "fidelity")), "probabilistic"); + assert_eq!( + bytes_of(entry(&matches, "term")), + term_of(s("alice"), "name", TERM_MATCH) + ); +} + +/// The validator refuses what the operation refuses, with the same status, +/// before a keyset is resolved: an operator the field does not answer, or +/// one the engine does not know, as a mistyped value is refused; the typed +/// path's own query refusals as `STATUS_TERM`. +#[test] +fn query_term_validation_refuses_what_the_query_op_refuses() { + let cipher = cipher(); + let keyset = cipher.default_keyset(); + for (what, input, field, operator, status) in [ + ( + "unknown operator", + FfiValue::UInt32(1), + "age", + "like", + STATUS_ENCODING, + ), + ("not answered", s("ali"), "age", "matches", STATUS_ENCODING), + ( + "not answered by a match index", + s("ali"), + "name", + "eq", + STATUS_ENCODING, + ), + ( + "no such field", + FfiValue::UInt32(1), + "height", + "eq", + STATUS_ENCODING, + ), + ("mistyped input", s("34"), "age", "eq", STATUS_ENCODING), + ( + "not a range", + FfiValue::UInt32(1), + "age", + "between", + STATUS_ENCODING, + ), + ( + "inverted range", + FfiValue::Array(vec![FfiValue::UInt32(65), FfiValue::UInt32(18)]), + "age", + "between", + STATUS_TERM, + ), + ("no token", s("al"), "name", "matches", STATUS_TERM), + ] { + let input = encode(input); + assert_eq!( + ops::validate::query_term(&input, &plan(), field.as_bytes(), operator.as_bytes()), + Err(status), + "{what}: validation" + ); + assert_eq!( + block_on(ops::query_term( + &keyset, + &input, + &plan(), + field.as_bytes(), + operator.as_bytes() + )), + Err(status), + "{what}: the op" + ); + } + assert_eq!( + ops::validate::query_term(&encode(FfiValue::UInt32(1)), &plan(), b"\xff", b"eq"), + Err(STATUS_ENCODING), + "a field name that is not UTF-8" + ); +} + +/// `verify` answers one boolean per value, in order, with no cipher; an +/// exact operator has nothing to verify. +#[test] +fn verify_answers_a_page_in_one_call() { + let values = encode(FfiValue::Array(vec![ + s("Alice Smith"), + s("Bob"), + FfiValue::Null, + s("malice"), + ])); + let out = + ops::verify(&encode(s("alice")), &values, &plan(), b"name", b"matches").expect("verify"); + let FfiValue::Array(verified) = decode(&out) else { + panic!("a list"); + }; + let verified: Vec = verified + .iter() + .map(|v| match v { + FfiValue::Bool(b) => *b, + other => panic!("a boolean, not {}", kind(other)), + }) + .collect(); + assert_eq!( + verified, + [true, false, false, true], + "malice holds every 3-gram of alice" + ); + + for (what, input, values, field, operator) in [ + ( + "an exact operator", + FfiValue::UInt32(1), + FfiValue::Array(vec![]), + "age", + "eq", + ), + ( + "values not a list", + s("alice"), + s("Alice"), + "name", + "matches", + ), + ( + "a value of another type", + s("alice"), + FfiValue::Array(vec![FfiValue::UInt32(1)]), + "name", + "matches", + ), + ( + "an unknown operator", + s("alice"), + FfiValue::Array(vec![]), + "name", + "match", + ), + ] { + assert_eq!( + ops::verify( + &encode(input), + &encode(values), + &plan(), + field.as_bytes(), + operator.as_bytes() + ), + Err(STATUS_ENCODING), + "{what}" + ); + } + assert_eq!( + ops::verify( + &encode(s("al")), + &encode(FfiValue::Array(vec![])), + &plan(), + b"name", + b"matches" + ), + Err(STATUS_TERM), + "a query that cuts no token verifies nothing" + ); +} + +/// `target_limit` reads the term's bytes and the statistics, with no +/// cipher: a measured histogram sizes a fetch, the declared prior a data +/// plan has none of answers null. +#[test] +fn target_limit_sizes_a_fetch_from_the_terms_bytes() { + let cipher = cipher(); + let keyset = cipher.default_keyset(); + let out = block_on(ops::query_term( + &keyset, + &encode(s("ali")), + &plan(), + b"name", + b"matches", + )) + .expect("query term"); + let term = bytes_of(entry(&query_term_entries(&out), "term")); + let pair = |bits: u32, rows: u64| { + FfiValue::Array(vec![FfiValue::UInt32(bits), FfiValue::UInt64(rows)]) + }; + let measured = encode(obj(vec![ + ("rows", FfiValue::UInt64(1_000_000)), + ( + "histogram", + FfiValue::Array(vec![pair(40, 500_000), pair(120, 500_000)]), + ), + ])); + let limit = |stats: &[u8], term: &[u8], field: &str, operator: &str| { + ops::target_limit( + term, + stats, + &plan(), + field.as_bytes(), + operator.as_bytes(), + 10, + ) + }; + let rows = decode(&limit(&measured, &term, "name", "matches").expect("a limit")); + assert!( + matches!(rows, FfiValue::UInt64(n) if n > 10), + "{}", + kind(&rows) + ); + let declared = encode(obj(vec![("rows", FfiValue::UInt64(1_000_000))])); + assert!(matches!( + decode(&limit(&declared, &term, "name", "matches").expect("a limit")), + FfiValue::Null + )); + + assert_eq!( + limit(&measured, &[1], "name", "matches"), + Err(STATUS_ENCODING), + "half a position" + ); + assert_eq!( + limit(&encode(s("x")), &term, "name", "matches"), + Err(STATUS_ENCODING), + "not stats" + ); + assert_eq!( + limit(&measured, &term, "age", "eq"), + Err(STATUS_ENCODING), + "an exact operator" + ); +} + +/// The `eql` build's catalog lists, per type, the operators the engine +/// answers: `TextEq` answers `eq`, exactly, and a type the engine does not +/// produce answers nothing. +#[test] +fn the_catalog_lists_each_types_operators() { + if !stack_encrypt_guest::targets::HOLDS_EQL { + return; + } + let FfiValue::Object(entries) = decode(&ops::targets().expect("targets")) else { + panic!("an object"); + }; + let FfiValue::Array(items) = &entries[0].1 else { + panic!("a list"); + }; + for item in items { + let FfiValue::Object(fields) = item else { + panic!("an object"); + }; + let name = text(entry(fields, "name")).to_owned(); + let FfiValue::Array(operators) = entry(fields, "operators") else { + panic!("{name}: operators is a list"); + }; + let producible = matches!(entry(fields, "producible"), FfiValue::Bool(true)); + if name == "TextEq" { + assert_eq!(operators.len(), 1, "TextEq answers eq alone"); + let FfiValue::Object(eq) = &operators[0] else { + panic!("an object"); + }; + let eq: Vec<(&str, &str)> = eq.iter().map(|(k, v)| (k.as_str(), text(v))).collect(); + assert_eq!( + eq, + [ + ("operator", "eq"), + ("fidelity", "exact"), + ("arity", "one"), + ("index", "eq") + ] + ); + } else if !producible { + assert!( + operators.is_empty(), + "{name} is not produced, so answers nothing" + ); + } + } +} + // ============================================================================= // A plan whose context is a field of the record // ============================================================================= diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index 11a21dfdb..166d6d57d 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -145,7 +145,7 @@ func TestImportSurfaceIsWASIPlusTransport(t *testing.T) { if !reflect.DeepEqual(transportImports, want) { t.Fatalf("transport imports = %v, want %v", transportImports, want) } - for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_term": true, "se_encrypt_record": true, "se_decrypt_record": true, "se_plan_check": true, "se_targets": true} { + for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_term": true, "se_query": true, "se_query_term": true, "se_verify": true, "se_target_limit": true, "se_encrypt_record": true, "se_decrypt_record": true, "se_plan_check": true, "se_targets": true} { if _, ok := compiled.ExportedFunctions()[name]; !ok { t.Errorf("guest does not export %s", name) } diff --git a/languages/golang/encrypt/query.go b/languages/golang/encrypt/query.go new file mode 100644 index 000000000..21c96acfa --- /dev/null +++ b/languages/golang/encrypt/query.go @@ -0,0 +1,302 @@ +package encrypt + +import ( + "context" + "fmt" + + "github.com/cipherstash/stack/languages/golang/internal/record" + "github.com/cipherstash/vitaminc/bindings/go/vcffi" + "github.com/cipherstash/vitaminc/bindings/go/vcvalue" +) + +// Queries named by operator (ADR-0009; ADR-0007, amended 2026-10-16): the +// guest's se_query_term, se_verify and se_target_limit exports, for +// generated code. A query names a field of a declaration and an operator, +// and the engine picks the index that answers it, or refuses an operator +// the field does not answer as it refuses a mistyped value. + +// QueryTerm is the query term the engine derives for one field, an operator +// and its input. +type QueryTerm struct { + // Operator is the operator the term answers: eq, lt, lte, gt, gte, + // between or matches. + Operator string + // Fidelity is how far the server's answer can be trusted. A + // [record.Probabilistic] term's rows are checked with [Cipher.Verify] + // after decrypt, and its fetch sized with [Cipher.TargetLimit]. + Fidelity record.Fidelity + // Index is the field's index that answers the operator. + Index record.Output + // Domain is the PostgreSQL domain the term is cast to, for a field + // that names an EQL type; "" for a field with indexes, whose terms are + // bytes. + Domain string + // Terms is the term: one, or for between two, the low bound's then the + // high's. For a field that names an EQL type it is the query value's + // JSON. + Terms [][]byte +} + +// Bucket is one bar of a histogram of the stored match filters' set-bit +// counts: Rows rows whose filter sets Bits bits. +type Bucket struct { + Bits uint32 + Rows uint64 +} + +// Stats is what [Cipher.TargetLimit] knows of the table: how many rows the +// query scans, how full their stored filters are, and the confidence to +// size the fetch at. None of it is secret: the server holds the filters and +// the catalog holds the row count. +type Stats struct { + // Rows is the rows the query scans, such as greatest(reltuples, 0) + // from pg_class. + Rows uint64 + // Histogram is a measured histogram of the stored filters' set-bit + // counts. With none, the engine uses the index's declared prior, and + // a field declared as data has none, so the limit is unbounded. + Histogram []Bucket + // Confidence is the probability that the fetch holds the rows wanted + // when the table has that many; 0 is the engine's default, 99%. + Confidence float64 +} + +// wire is the statistics as se_target_limit reads them. +func (s Stats) wire() vcvalue.Object { + out := vcvalue.Object{{Key: "rows", Value: s.Rows}} + if s.Histogram != nil { + buckets := make([]any, len(s.Histogram)) + for i, b := range s.Histogram { + buckets[i] = []any{b.Bits, b.Rows} + } + out = append(out, vcvalue.Field{Key: "histogram", Value: buckets}) + } + if s.Confidence != 0 { + out = append(out, vcvalue.Field{Key: "confidence", Value: s.Confidence}) + } + return out +} + +// Limit is how many rows a probabilistic query should fetch: a number, or +// unbounded when the statistics say nothing of the stored filters or the +// confidence is not below 1, in which case a caller pages with a cursor. +type Limit struct { + rows uint64 + bounded bool +} + +// Rows is the rows to fetch, and false when no finite fetch can be +// promised. +func (l Limit) Rows() (uint64, bool) { return l.rows, l.bounded } + +// FetchOr is the rows to fetch, or page when no finite fetch can be +// promised, for a caller that pages either way. +func (l Limit) FetchOr(page uint64) uint64 { + if l.bounded { + return l.rows + } + return page +} + +// QueryTerm derives the query term for one field of the plan, an operator +// and its input under this cipher's keyset: one value for every operator +// but between, whose input is a two-element slice, low then high. The +// engine reads the input as the field's declared type, finds the index +// that answers the operator, and derives that index's term under the +// field's context; for a field that names an EQL type it runs the type's +// query. An operator the field does not answer, or one the engine does not +// know, is [ErrEncoding] naming the field, as a mistyped value is; a range +// the wrong way round, and match text that cuts no token, are [ErrTerm]. +// No data key is minted. For generated code. +func (cph *Cipher) QueryTerm(ctx context.Context, plan *record.Plan, field, operator string, input any) (QueryTerm, error) { + p, err := cph.queryPlan(plan, field) + if err != nil { + return QueryTerm{}, err + } + encodedInput, err := vcffi.Marshal(input) + if err != nil { + return QueryTerm{}, fmt.Errorf("%w: %v", ErrEncoding, err) + } + // The input is plaintext: its transport copy is wiped once it is in the + // guest. + defer wipe(encodedInput) + encodedPlan, err := vcffi.Marshal(p.Wire()) + if err != nil { + return QueryTerm{}, fmt.Errorf("%w: %v", ErrEncoding, err) + } + opts, err := vcffi.Marshal(options(cph.keyset, "")) + if err != nil { + return QueryTerm{}, err + } + out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.queryTerm, buf(encodedInput), buf(encodedPlan), buf([]byte(field)), buf([]byte(operator)), buf(opts)) + }) + if err != nil { + return QueryTerm{}, err + } + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return QueryTerm{}, fmt.Errorf("%w: %v", ErrInternal, err) + } + term, err := parseQueryTerm(decoded) + if err != nil { + return QueryTerm{}, fmt.Errorf("%w: se_query_term: %v", ErrInternal, err) + } + return term, nil +} + +// parseQueryTerm reads se_query_term's object: exactly the keys operator, +// fidelity, index, domain (a string or null) and term (bytes, or a list of +// two for between). +func parseQueryTerm(decoded any) (QueryTerm, error) { + obj, ok := decoded.(vcvalue.Object) + if !ok { + return QueryTerm{}, fmt.Errorf("came back as %T, not an object", decoded) + } + var t QueryTerm + seen := map[string]bool{} + for _, f := range obj { + if seen[f.Key] { + return QueryTerm{}, fmt.Errorf("key %q twice", f.Key) + } + seen[f.Key] = true + var err error + switch f.Key { + case "operator": + t.Operator, err = targetString(f) + case "fidelity": + var s string + s, err = targetString(f) + t.Fidelity = record.Fidelity(s) + if err == nil && t.Fidelity != record.Exact && t.Fidelity != record.Probabilistic { + err = fmt.Errorf("unknown fidelity %q", s) + } + case "index": + var s string + s, err = targetString(f) + t.Index = record.Output(s) + case "domain": + t.Domain, err = targetOptionalString(f) + case "term": + switch v := f.Value.(type) { + case []byte: + t.Terms = [][]byte{v} + case []any: + if len(v) != 2 { + return QueryTerm{}, fmt.Errorf("a range of %d terms", len(v)) + } + for _, bound := range v { + b, ok := bound.([]byte) + if !ok { + return QueryTerm{}, fmt.Errorf("a bound is %T, not bytes", bound) + } + t.Terms = append(t.Terms, b) + } + default: + return QueryTerm{}, fmt.Errorf("term is %T", f.Value) + } + default: + return QueryTerm{}, fmt.Errorf("unknown key %q", f.Key) + } + if err != nil { + return QueryTerm{}, err + } + } + if len(seen) != 5 { + return QueryTerm{}, fmt.Errorf("%d keys, not 5", len(seen)) + } + return t, nil +} + +// Verify reports, for each of a page of decrypted values, whether it really +// matches a probabilistic query: one call into the guest for the page. The +// query is named as it was derived, by the field, the operator and the +// input; the engine cuts the input's tokens again with the field's own +// tokenizer, so nothing here holds a copy of it. A nil value is false. An +// exact operator has nothing to verify and is [ErrEncoding]. Uses no key. +// For generated code. +func (cph *Cipher) Verify(ctx context.Context, plan *record.Plan, field, operator string, input any, values []any) ([]bool, error) { + p, err := cph.plan(plan) + if err != nil { + return nil, err + } + encodedInput, err := vcffi.Marshal(input) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + defer wipe(encodedInput) + if values == nil { + values = []any{} + } + encodedValues, err := vcffi.Marshal(values) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + // The values are decrypted plaintext: wiped once they are in the guest. + defer wipe(encodedValues) + encodedPlan, err := vcffi.Marshal(p.Wire()) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.verify, buf(encodedInput), buf(encodedValues), buf(encodedPlan), buf([]byte(field)), buf([]byte(operator))) + }) + if err != nil { + return nil, err + } + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrInternal, err) + } + list, ok := decoded.([]any) + if !ok || len(list) != len(values) { + return nil, fmt.Errorf("%w: se_verify returned %T for %d values", ErrInternal, decoded, len(values)) + } + verified := make([]bool, len(list)) + for i, v := range list { + b, ok := v.(bool) + if !ok { + return nil, fmt.Errorf("%w: se_verify returned %T for value %d", ErrInternal, v, i) + } + verified[i] = b + } + return verified, nil +} + +// TargetLimit is how many rows to fetch so that, at the confidence stats +// names, want of them verify: term is the probabilistic term's bytes as +// [Cipher.QueryTerm] returned them. An exact operator has no fetch to size +// and is [ErrEncoding], as are bytes the field's index would not derive. +// Uses no key. For generated code. +func (cph *Cipher) TargetLimit(ctx context.Context, plan *record.Plan, field, operator string, term []byte, want uint64, stats Stats) (Limit, error) { + p, err := cph.plan(plan) + if err != nil { + return Limit{}, err + } + encodedStats, err := vcffi.Marshal(stats.wire()) + if err != nil { + return Limit{}, fmt.Errorf("%w: %v", ErrEncoding, err) + } + encodedPlan, err := vcffi.Marshal(p.Wire()) + if err != nil { + return Limit{}, fmt.Errorf("%w: %v", ErrEncoding, err) + } + out, err := cph.client.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.limit, buf(term), buf(encodedStats), buf(encodedPlan), buf([]byte(field)), buf([]byte(operator)), scalar(want)) + }) + if err != nil { + return Limit{}, err + } + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return Limit{}, fmt.Errorf("%w: %v", ErrInternal, err) + } + switch v := decoded.(type) { + case nil: + return Limit{}, nil + case uint64: + return Limit{rows: v, bounded: true}, nil + default: + return Limit{}, fmt.Errorf("%w: se_target_limit returned %T", ErrInternal, decoded) + } +} diff --git a/languages/golang/encrypt/query_test.go b/languages/golang/encrypt/query_test.go new file mode 100644 index 000000000..2deb0cd8f --- /dev/null +++ b/languages/golang/encrypt/query_test.go @@ -0,0 +1,362 @@ +package encrypt_test + +import ( + "context" + "encoding/hex" + "encoding/json" + "errors" + "os" + "path/filepath" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/internal/record" +) + +// The query fixture holds the guest's query exports to the Rust results: +// packages/stack-encrypt/tests/fixtures/query_operators.json records, for one +// declaration, the term each query by operator derives (each equal to the +// typed Rust query chain's), a page of verified values and a few fetch +// limits. These tests derive, verify and size the same queries through the +// deterministic test build of the guest and compare. Skipped when that build +// is absent. + +type queryFixture struct { + KeySource struct { + Seed string `json:"seed"` + } `json:"key_source"` + Queries []struct { + Field string `json:"field"` + Operator string `json:"operator"` + Input json.RawMessage `json:"input"` + Term struct { + Operator string `json:"operator"` + Fidelity string `json:"fidelity"` + Index string `json:"index"` + Domain *string `json:"domain"` + Term json.RawMessage `json:"term"` + } `json:"term"` + } `json:"queries"` + Verify struct { + Field string `json:"field"` + Operator string `json:"operator"` + Input string `json:"input"` + Values []*string `json:"values"` + Verified []bool `json:"verified"` + } `json:"verify"` + TargetLimit []struct { + Field string `json:"field"` + Operator string `json:"operator"` + Term string `json:"term"` + Want uint64 `json:"want"` + Stats struct { + Rows uint64 `json:"rows"` + Histogram [][2]uint64 `json:"histogram"` + Confidence float64 `json:"confidence"` + } `json:"stats"` + Limit *uint64 `json:"limit"` + } `json:"target_limit"` +} + +func readQueryFixture(t *testing.T) queryFixture { + t.Helper() + raw, err := os.ReadFile(filepath.Join("..", "..", "..", "packages", "stack-encrypt", "tests", "fixtures", "query_operators.json")) + if err != nil { + t.Fatal(err) + } + var f queryFixture + if err := json.Unmarshal(raw, &f); err != nil { + t.Fatal(err) + } + return f +} + +// queryPlan is the fixture's declaration: an email with equality, an age +// with equality and ORE, a name with match, all under users. +var queryPlan = &record.Plan{Context: []string{"users"}, Fields: []record.Field{ + {Name: "email", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Equality}}, + {Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Equality, record.Ore}}, + {Name: "name", Kind: record.String, Outputs: []record.Output{record.Ciphertext, record.Match}}, +}} + +func queryFixtureClient(t *testing.T, f queryFixture) *encrypt.Client { + t.Helper() + seed, err := hex.DecodeString(f.KeySource.Seed) + if err != nil || len(seed) != 32 { + t.Fatalf("seed: %v (%d bytes)", err, len(seed)) + } + c, err := encrypt.NewDeterministicClient(context.Background(), [32]byte(seed)) + if errors.Is(err, encrypt.ErrDeterministicGuestNotBuilt) || errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = c.Close() }) + return c +} + +// goInput is a fixture input as Go code passes it: text as a string, an +// age as the field's uint32, a range as two of them. +func goInput(t *testing.T, field string, raw json.RawMessage) any { + t.Helper() + if field != "age" { + var s string + if err := json.Unmarshal(raw, &s); err != nil { + t.Fatal(err) + } + return s + } + var n uint32 + if err := json.Unmarshal(raw, &n); err == nil { + return n + } + var pair [2]uint32 + if err := json.Unmarshal(raw, &pair); err != nil { + t.Fatal(err) + } + return []any{pair[0], pair[1]} +} + +func fixtureTerms(t *testing.T, raw json.RawMessage) [][]byte { + t.Helper() + var one string + if err := json.Unmarshal(raw, &one); err == nil { + b, err := hex.DecodeString(one) + if err != nil { + t.Fatal(err) + } + return [][]byte{b} + } + var pair []string + if err := json.Unmarshal(raw, &pair); err != nil { + t.Fatal(err) + } + out := make([][]byte, len(pair)) + for i, s := range pair { + b, err := hex.DecodeString(s) + if err != nil { + t.Fatal(err) + } + out[i] = b + } + return out +} + +func TestQueryTermsAreTheRustFixtures(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + ctx := context.Background() + cipher := c.DefaultKeyset() + if len(f.Queries) == 0 { + t.Fatal("the fixture holds no query") + } + for _, q := range f.Queries { + got, err := cipher.QueryTerm(ctx, queryPlan, q.Field, q.Operator, goInput(t, q.Field, q.Input)) + if err != nil { + t.Fatalf("%s on %s: %v", q.Operator, q.Field, err) + } + want := encrypt.QueryTerm{ + Operator: q.Term.Operator, + Fidelity: record.Fidelity(q.Term.Fidelity), + Index: record.Output(q.Term.Index), + Terms: fixtureTerms(t, q.Term.Term), + } + if q.Term.Domain != nil { + want.Domain = *q.Term.Domain + } + if got.Operator != want.Operator || got.Fidelity != want.Fidelity || got.Index != want.Index || got.Domain != want.Domain || len(got.Terms) != len(want.Terms) { + t.Fatalf("%s on %s: %+v, want %+v", q.Operator, q.Field, got, want) + } + for i := range want.Terms { + if hex.EncodeToString(got.Terms[i]) != hex.EncodeToString(want.Terms[i]) { + t.Fatalf("%s on %s: term %d = %x, want %x", q.Operator, q.Field, i, got.Terms[i], want.Terms[i]) + } + } + } + + // The engine reads the input as the field's type: a Go int queries a + // uint32 field as the uint32 it stored. + asUint32, err := cipher.QueryTerm(ctx, queryPlan, "age", "lt", uint32(40)) + if err != nil { + t.Fatal(err) + } + asInt, err := cipher.QueryTerm(ctx, queryPlan, "age", "lt", 40) + if err != nil { + t.Fatal(err) + } + if hex.EncodeToString(asInt.Terms[0]) != hex.EncodeToString(asUint32.Terms[0]) { + t.Fatal("an int input derived another term than the field's uint32") + } +} + +func TestVerifyIsTheRustFixtures(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + values := make([]any, len(f.Verify.Values)) + for i, v := range f.Verify.Values { + if v != nil { + values[i] = *v + } + } + verified, err := c.DefaultKeyset().Verify(context.Background(), queryPlan, f.Verify.Field, f.Verify.Operator, f.Verify.Input, values) + if err != nil { + t.Fatal(err) + } + if len(verified) != len(f.Verify.Verified) { + t.Fatalf("Verify = %v, want %v", verified, f.Verify.Verified) + } + for i := range verified { + if verified[i] != f.Verify.Verified[i] { + t.Fatalf("Verify = %v, want %v", verified, f.Verify.Verified) + } + } + empty, err := c.DefaultKeyset().Verify(context.Background(), queryPlan, "name", "matches", "alice", nil) + if err != nil || len(empty) != 0 { + t.Fatalf("an empty page: %v %v", empty, err) + } +} + +func TestTargetLimitIsTheRustFixtures(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + if len(f.TargetLimit) == 0 { + t.Fatal("the fixture holds no limit") + } + for i, l := range f.TargetLimit { + term, err := hex.DecodeString(l.Term) + if err != nil { + t.Fatal(err) + } + stats := encrypt.Stats{Rows: l.Stats.Rows, Confidence: l.Stats.Confidence} + for _, b := range l.Stats.Histogram { + stats.Histogram = append(stats.Histogram, encrypt.Bucket{Bits: uint32(b[0]), Rows: b[1]}) + } + got, err := c.DefaultKeyset().TargetLimit(context.Background(), queryPlan, l.Field, l.Operator, term, l.Want, stats) + if err != nil { + t.Fatalf("case %d: %v", i, err) + } + rows, bounded := got.Rows() + switch { + case l.Limit == nil && bounded: + t.Fatalf("case %d: %d rows, want unbounded", i, rows) + case l.Limit != nil && (!bounded || rows != *l.Limit): + t.Fatalf("case %d: %d (bounded %v), want %d", i, rows, bounded, *l.Limit) + } + if l.Limit == nil && got.FetchOr(50) != 50 { + t.Fatalf("case %d: an unbounded limit fetches the page", i) + } + } +} + +// A query the field does not answer is refused as a mistyped value is: an +// encoding error that names the field and the rule. The typed path's own +// refusals are term errors. +func TestAQueryTheFieldDoesNotAnswerIsRefused(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + ctx := context.Background() + cipher := c.DefaultKeyset() + for _, tc := range []struct { + field, operator string + input any + reason string + }{ + {"age", "matches", "ali", "operator_not_answered"}, + {"name", "lt", "ali", "operator_not_answered"}, + {"age", "like", uint32(1), "unknown_operator"}, + {"age", "between", uint32(1), "not_a_range"}, + {"age", "eq", "34", "field_type"}, + } { + _, err := cipher.QueryTerm(ctx, queryPlan, tc.field, tc.operator, tc.input) + if !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("%s on %s: %v, want ErrEncoding", tc.operator, tc.field, err) + } + var d *encrypt.Diagnostic + if !errors.As(err, &d) || d.Field() != tc.field || d.Reason() != tc.reason { + t.Fatalf("%s on %s: %v, want field %q and reason %q", tc.operator, tc.field, err, tc.field, tc.reason) + } + } + if _, err := cipher.QueryTerm(ctx, queryPlan, "age", "between", []any{uint32(65), uint32(18)}); !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("an inverted range: %v, want ErrTerm", err) + } + if _, err := cipher.QueryTerm(ctx, queryPlan, "name", "matches", "al"); !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("text that cuts no token: %v, want ErrTerm", err) + } + + _, err := cipher.Verify(ctx, queryPlan, "email", "eq", "bob@example.com", []any{"bob@example.com"}) + var d *encrypt.Diagnostic + if !errors.Is(err, encrypt.ErrEncoding) || !errors.As(err, &d) || d.Reason() != "exact_operator" { + t.Fatalf("Verify on an exact operator: %v", err) + } + _, err = cipher.TargetLimit(ctx, queryPlan, "age", "lt", []byte{1, 0}, 10, encrypt.Stats{Rows: 1}) + if !errors.Is(err, encrypt.ErrEncoding) || !errors.As(err, &d) || d.Reason() != "exact_operator" { + t.Fatalf("TargetLimit on an exact operator: %v", err) + } + _, err = cipher.TargetLimit(ctx, queryPlan, "name", "matches", []byte{1}, 10, encrypt.Stats{Rows: 1}) + if !errors.Is(err, encrypt.ErrEncoding) || !errors.As(err, &d) || d.Reason() != "not_a_query_term" { + t.Fatalf("TargetLimit on half a position: %v", err) + } +} + +// Verifying and sizing use no key, so a guest never given one answers them, +// as it answers the generator's questions. +func TestVerifyAndTargetLimitNeedNoCipher(t *testing.T) { + wasm, err := encrypt.EmbeddedGuest() + if errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + c := encrypt.RawClient(t, wasm) + ctx := context.Background() + cipher := c.DefaultKeyset() + verified, err := cipher.Verify(ctx, queryPlan, "name", "matches", "alice", []any{"Alice Smith", "Bob"}) + if err != nil || len(verified) != 2 || !verified[0] || verified[1] { + t.Fatalf("Verify = %v, %v", verified, err) + } + limit, err := cipher.TargetLimit(ctx, queryPlan, "name", "matches", []byte{1, 0, 2, 0}, 10, encrypt.Stats{Rows: 100, Histogram: []encrypt.Bucket{{Bits: 10, Rows: 100}}}) + if rows, ok := limit.Rows(); err != nil || !ok || rows < 10 { + t.Fatalf("TargetLimit = %d %v, %v", rows, ok, err) + } + if _, err := cipher.QueryTerm(ctx, queryPlan, "age", "eq", uint32(1)); !errors.Is(err, encrypt.ErrState) { + t.Fatalf("a query term needs the cipher: %v, want ErrState", err) + } +} + +// The catalog reports, per EQL type, the operators the engine answers: +// TextEq answers eq, exactly, through its equality index. +func TestTheCatalogListsEachTypesOperators(t *testing.T) { + wasm, err := encrypt.EmbeddedGuest() + if errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + checker, err := encrypt.NewCheckerOver(ctx, wasm) + if err != nil { + t.Fatal(err) + } + defer func() { _ = checker.Close() }() + targets, err := checker.Targets(ctx) + if err != nil { + t.Fatal(err) + } + if len(targets) == 0 { + t.Skip("this build holds no EQL types: import encrypt/eql") + } + for _, target := range targets { + switch { + case target.Name == "TextEq": + want := []record.TargetOperator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}} + if len(target.Operators) != 1 || target.Operators[0] != want[0] { + t.Fatalf("TextEq answers %+v, want %+v", target.Operators, want) + } + case !target.Producible && len(target.Operators) != 0: + t.Fatalf("%s is not produced and answers %+v", target.Name, target.Operators) + } + } +} diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index 39094109e..a12b1cf4c 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -118,25 +118,10 @@ func (cph *Cipher) Open(ctx context.Context, plan *record.Plan, records []record // with [Cipher.Context], and a cipher that names none is refused as // [ErrEncoding]. For generated code. func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, output record.Output, value any) ([]byte, error) { - p, err := cph.plan(plan) + p, err := cph.queryPlan(plan, field) if err != nil { return nil, err } - if p.ContextField != "" { - if cph.context == "" { - return nil, fmt.Errorf("%w: field %q: the type takes its context from its field %q, so a query names it with Cipher.Context", ErrEncoding, field, p.ContextField) - } - // The label a query compares against is /, as - // the record stored it: the plan under the named context, with no - // context field of its own. Validate has checked the label. - segments, err := record.ParseContext(cph.context) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrEncoding, err) - } - under := *p - under.Context, under.ContextField = segments, "" - p = &under - } f := p.Field(field) if f == nil { return nil, fmt.Errorf("%w: the plan has no field %q", ErrEncoding, field) @@ -172,6 +157,34 @@ func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, }) } +// queryPlan is the plan a query on field derives under: the declaration's +// plan with this cipher's extension and, for a plan that takes its context +// from a field, the context named with [Cipher.Context] in that field's +// place. A cipher that names none is refused as [ErrEncoding]: a term +// derives under a context, and such a plan has none until a record names it. +func (cph *Cipher) queryPlan(plan *record.Plan, field string) (*record.Plan, error) { + p, err := cph.plan(plan) + if err != nil { + return nil, err + } + if p.ContextField == "" { + return p, nil + } + if cph.context == "" { + return nil, fmt.Errorf("%w: field %q: the type takes its context from its field %q, so a query names it with Cipher.Context", ErrEncoding, field, p.ContextField) + } + // The label a query compares against is /, as the + // record stored it: the plan under the named context, with no context + // field of its own. Validate has checked the label. + segments, err := record.ParseContext(cph.context) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + under := *p + under.Context, under.ContextField = segments, "" + return &under, nil +} + // Query derives the EQL query value of a field that names an EQL type, for // one value: the operand that matches stored values of the field, as the // JSON bytes the field's eql_v3.query_* domain takes. The engine runs the diff --git a/languages/golang/encrypt/targets_test.go b/languages/golang/encrypt/targets_test.go index 69a007acf..24b0ce9dd 100644 --- a/languages/golang/encrypt/targets_test.go +++ b/languages/golang/encrypt/targets_test.go @@ -19,20 +19,33 @@ func TestParseTargetsReadsEachEntry(t *testing.T) { vcvalue.Field{Key: "plaintext", Value: "string"}, vcvalue.Field{Key: "sql_domain", Value: "eql_v3_text_eq"}, vcvalue.Field{Key: "indexes", Value: []any{"eq"}}, vcvalue.Field{Key: "query", Value: "TextEqQuery"}, vcvalue.Field{Key: "query_sql_domain", Value: "eql_v3.query_text_eq"}, vcvalue.Field{Key: "producible", Value: true}, vcvalue.Field{Key: "reason", Value: nil}, + vcvalue.Field{Key: "operators", Value: []any{operator("eq", "exact", "one", "eq")}}, ) notYet := entry( vcvalue.Field{Key: "name", Value: "JSON"}, vcvalue.Field{Key: "family", Value: "JSON"}, vcvalue.Field{Key: "suffix", Value: ""}, vcvalue.Field{Key: "plaintext", Value: nil}, vcvalue.Field{Key: "sql_domain", Value: "eql_v3_json"}, vcvalue.Field{Key: "indexes", Value: []any{"json"}}, vcvalue.Field{Key: "query", Value: nil}, vcvalue.Field{Key: "query_sql_domain", Value: nil}, vcvalue.Field{Key: "producible", Value: false}, vcvalue.Field{Key: "reason", Value: "the JSON index is a new operation in the engine"}, + vcvalue.Field{Key: "operators", Value: []any{}}, ) - got, err := parseTargets(vcvalue.Object{{Key: "targets", Value: []any{textEq, notYet}}}) + ordered := entry( + vcvalue.Field{Key: "name", Value: "IntegerOrdOre"}, vcvalue.Field{Key: "plaintext", Value: "int64"}, + vcvalue.Field{Key: "indexes", Value: []any{"ore"}}, vcvalue.Field{Key: "producible", Value: true}, + vcvalue.Field{Key: "operators", Value: []any{operator("lt", "exact", "one", "ore"), operator("between", "exact", "pair", "ore"), operator("matches", "probabilistic", "one", "match")}}, + ) + got, err := parseTargets(vcvalue.Object{{Key: "targets", Value: []any{textEq, notYet, ordered}}}) if err != nil { t.Fatal(err) } want := []record.Target{ - {Name: "TextEq", Family: "Text", Suffix: "Eq", Plaintext: record.String, SQLDomain: "eql_v3_text_eq", Indexes: []record.Output{record.Equality}, Query: "TextEqQuery", QuerySQLDomain: "eql_v3.query_text_eq", Producible: true}, - {Name: "JSON", Family: "JSON", Plaintext: record.Untyped, SQLDomain: "eql_v3_json", Indexes: []record.Output{"json"}, Reason: "the JSON index is a new operation in the engine"}, + {Name: "TextEq", Family: "Text", Suffix: "Eq", Plaintext: record.String, SQLDomain: "eql_v3_text_eq", Indexes: []record.Output{record.Equality}, Query: "TextEqQuery", QuerySQLDomain: "eql_v3.query_text_eq", Producible: true, + Operators: []record.TargetOperator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}}}, + {Name: "JSON", Family: "JSON", Plaintext: record.Untyped, SQLDomain: "eql_v3_json", Indexes: []record.Output{"json"}, Reason: "the JSON index is a new operation in the engine", Operators: []record.TargetOperator{}}, + {Name: "IntegerOrdOre", Plaintext: record.Int64, Indexes: []record.Output{record.Ore}, Producible: true, Operators: []record.TargetOperator{ + {Name: "lt", Fidelity: record.Exact, Index: record.Ore}, + {Name: "between", Fidelity: record.Exact, Pair: true, Index: record.Ore}, + {Name: "matches", Fidelity: record.Probabilistic, Index: record.Match}, + }}, } if !reflect.DeepEqual(got, want) { t.Fatalf("parseTargets =\n%+v\nwant\n%+v", got, want) @@ -42,22 +55,45 @@ func TestParseTargetsReadsEachEntry(t *testing.T) { t.Fatalf("empty list: %v %v", empty, err) } for name, bad := range map[string]any{ - "not an object": "targets", - "another key": vcvalue.Object{{Key: "types", Value: []any{}}}, - "list not a list": vcvalue.Object{{Key: "targets", Value: "x"}}, - "entry not object": vcvalue.Object{{Key: "targets", Value: []any{"TextEq"}}}, - "numeric plaintext": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: int64(1)}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "misspelled kind": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: "strng"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "misspelled index": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{"equality"}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "unknown key": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "kind", Value: "string"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "no name": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "no producible": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}})}}}, - "index not string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{int64(1)}}, vcvalue.Field{Key: "producible", Value: true})}}}, - "producible string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: "yes"})}}}, - "key twice": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "A"}, vcvalue.Field{Key: "name", Value: "B"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "not an object": "targets", + "another key": vcvalue.Object{{Key: "types", Value: []any{}}}, + "list not a list": vcvalue.Object{{Key: "targets", Value: "x"}}, + "entry not object": vcvalue.Object{{Key: "targets", Value: []any{"TextEq"}}}, + "numeric plaintext": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: int64(1)}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "misspelled kind": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "plaintext", Value: "strng"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "misspelled index": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{"equality"}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "unknown key": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "kind", Value: "string"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "no name": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "no producible": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}})}}}, + "index not string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{int64(1)}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "producible string": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: "yes"})}}}, + "key twice": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "A"}, vcvalue.Field{Key: "name", Value: "B"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "no operators": vcvalue.Object{{Key: "targets", Value: []any{entry(vcvalue.Field{Key: "name", Value: "TextEq"}, vcvalue.Field{Key: "indexes", Value: []any{}}, vcvalue.Field{Key: "producible", Value: true})}}}, + "operators not list": withOperators("eq"), + "operator not object": withOperators([]any{"eq"}), + "unknown fidelity": withOperators([]any{operator("eq", "approximate", "one", "eq")}), + "unknown arity": withOperators([]any{operator("eq", "exact", "three", "eq")}), + "unknown op index": withOperators([]any{operator("eq", "exact", "one", "json")}), + "no op name": withOperators([]any{operator("", "exact", "one", "eq")}), + "op key missing": withOperators([]any{vcvalue.Object{{Key: "operator", Value: "eq"}}}), + "op unknown key": withOperators([]any{append(operator("eq", "exact", "one", "eq"), vcvalue.Field{Key: "case", Value: "insensitive"})}), + "op key twice": withOperators([]any{append(operator("eq", "exact", "one", "eq"), vcvalue.Field{Key: "index", Value: "eq"})}), + "op value not text": withOperators([]any{vcvalue.Object{{Key: "operator", Value: int64(1)}, {Key: "fidelity", Value: "exact"}, {Key: "arity", Value: "one"}, {Key: "index", Value: "eq"}}}), } { if _, err := parseTargets(bad); !errors.Is(err, ErrInternal) { t.Errorf("%s: %v, want ErrInternal", name, err) } } } + +// operator is one se_targets operators entry. +func operator(name, fidelity, arity, index string) vcvalue.Object { + return vcvalue.Object{{Key: "operator", Value: name}, {Key: "fidelity", Value: fidelity}, {Key: "arity", Value: arity}, {Key: "index", Value: index}} +} + +// withOperators is a well-formed TextEq entry whose operators are value. +func withOperators(value any) vcvalue.Object { + return vcvalue.Object{{Key: "targets", Value: []any{vcvalue.Object{ + {Key: "name", Value: "TextEq"}, {Key: "indexes", Value: []any{}}, {Key: "producible", Value: true}, {Key: "operators", Value: value}, + }}}} +} diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 0123bb3dc..70ad1582c 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -417,7 +417,9 @@ type Sealed = map[string]Outputs // format, written by eql-bindings' serialiser and read by [ParseTargets]: // name, family, suffix, plaintext (a ValueKind name, or null), sql_domain, // indexes (eq | match | ore | ope | json), query (or null), query_sql_domain -// (or null), producible and reason (or null). +// (or null), producible, reason (or null), and operators, which the engine +// adds: the operators it answers for the type, each an object of operator, +// fidelity (exact | probabilistic), arity (one | pair) and index. type Target struct { // Name is the Go type name, and the value of encrypt_into: TextEq. Name string @@ -439,4 +441,36 @@ type Target struct { // Reason says why not when it does not. Producible bool Reason string + // Operators are the operators the engine answers for a field that names + // the type, in the engine's order; none for a type it does not produce. + Operators []TargetOperator +} + +// Fidelity says how far the server's answer to an operator can be trusted: +// [Exact], or [Probabilistic], whose rows a caller verifies after decrypt. +type Fidelity string + +// The fidelities an operator has. +const ( + // Exact is an answer the server gives whole: nothing to verify. + Exact Fidelity = "exact" + // Probabilistic is an answer that can hold false positives and no false + // negatives: each decrypted row is verified, and the fetch is sized for + // the false positives. + Probabilistic Fidelity = "probabilistic" +) + +// TargetOperator is one operator the engine answers for an EQL type, as +// se_targets lists it. +type TargetOperator struct { + // Name is the operator's name, as a query names it: eq, lt, lte, gt, + // gte, between or matches. + Name string + // Fidelity is how far the server's answer can be trusted. + Fidelity Fidelity + // Pair says the operator takes two values, low then high, as between + // does, where every other takes one. + Pair bool + // Index is the type's index that answers it. + Index Output } From 6fd63bfce23833601da7cb7935f2f42923ec2b6a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 03:13:58 +1100 Subject: [PATCH 34/38] docs(stack-encrypt): amend ADR-0007 for queries named by operator, proposed until ADR-0009 is accepted The data grammar gains the operator and its input, the term comes back with its query domain, verify and target_limit become engine exports, and the catalog reports each EQL type's operators and their fidelity. The amendment rests on ADR-0009, which is proposed, and takes effect when it is accepted. Part of #1188 --- ...-through-a-plan-never-a-second-executor.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md b/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md index 36ed1f64d..4fa765af6 100644 --- a/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md +++ b/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md @@ -2,6 +2,7 @@ status: accepted date: 2026-10-04 extends: ADR-0003, ADR-0004 +amended: 2026-10-16, proposed with ADR-0009 and in effect when ADR-0009 is accepted --- # Bindings enter the engine through a plan lowered from data, never through a second executor @@ -29,6 +30,14 @@ extends: ADR-0003, ADR-0004 > Go SDK, as the `context_field` tag word. The typed parts that stay > Rust-only are the typed verb `encrypt_into`, the picker and the one-value > start. +> +> **Amended 2026-10-16**, by #1188, **proposed**. This amendment depends on +> ADR-0009, which is proposed, and takes effect when ADR-0009 is accepted; +> until then it is proposed with it, and changes with it. The decision is +> unchanged. Added, under its last bullet: the operator and its input in the +> data grammar, the query term and its query domain back, `verify` and +> `target_limit` as engine exports, and the operators each EQL type answers, +> with their fidelity, in the catalog. Stack Encrypt has one execution engine: the `Encryption` and `Decryption` descriptions in `target/` and the batched `Pending` they produce. ADR-0003 @@ -188,3 +197,99 @@ to Go (G1: one engine). The reasons recorded against this on 2026-10-04 were guest size and an EQL-free guest for programs that never store into EQL. Two guest builds answer the second, and the first is measured before the SDK ships one build or two (G7). + +## Amended 2026-10-16 (#1188), proposed with ADR-0009 + +**Status: proposed.** ADR-0009 is proposed, and this amendment rests on it: +it takes effect when ADR-0009 is accepted, and is revised or withdrawn with +it. The code it describes lands first, as ADR-0009's Rust side has +(#1182 to #1186), because Stack Encrypt has no stored data and no released +binding a change could break. + +ADR-0009 makes the operator, not the index, what a query names, and gives a +probabilistic term `verify` and `target_limit`. The data grammar a binding +lowered named a field and, for a query, took the term the field's EQL type +chose. It had no operator and no operator input, and no export could verify +a decrypted value or size a fetch. The last bullet of the decision says what +to do with a capability the grammar cannot express: add it to the grammar, +once, so every author has it. This amendment does that for the query side. +ADR-0009's appendix B is the sketch it was amended from. + +**A query names a field, an operator and the operator's input.** The +operator crosses as its name: `eq`, `lt`, `lte`, `gt`, `gte`, `between` or +`matches`. The input is one value, or for `between` a list of two, low then +high. The engine reads the input as the field's declared type first +(`dynamic::read`), so a host's float `34.0` queries a `uint64` field as the +`u64` the field stored. It then finds the index that answers the operator by +the table the typed path's `DefaultIndex` impls state, written over a type +and the indexes a field declares (`dynamic::OpSpec::answered_by`): `eq` +through an equality index, `matches` through a match index over text, the +orderings through an ORE index over an integer, or OPE when the field +declares no ORE. A field that does not answer the operator is refused with +the same shape of error as a mistyped field: an input error naming the field, +with the reason `operator_not_answered` (or `unknown_operator`, +`not_a_range`). The refusals the typed path makes before a term exists, a +range the wrong way round and match text that cuts no token, are refused the +same way here, before the cipher is consulted. + +**The term comes back with its query domain.** The guest's +`se_query_term(input, plan, field, operator, options)` returns one object: +the operator, its fidelity, the index that answered, the PostgreSQL domain +the term is cast to, and the term. A field with indexes derives that index's +term under the field's context, by the derivation its stored terms came +from, as bytes, and `between` derives one term per bound under the one +context, as a list of two. A field that names an EQL type runs the type's +query twin, and the term is the twin's JSON and the domain its +`query_sql_domain`. One query domain per operator is EQL v4's (ADR-0009, +decision 8, #1179); until then a type's twin carries every term of the type. + +**`verify` and `target_limit` are engine exports.** `se_verify(input, values, +plan, field, operator)` answers one boolean per decrypted value, a page in +one call. `se_target_limit(term, stats, plan, field, operator, want)` answers +the rows to fetch, or null for no finite fetch. Neither uses a key, so each +runs in a guest that holds no cipher, as `se_plan_check` does. `verify` names +the query as it was derived, by the field, the operator and the input, and +the engine cuts the input's tokens again with the field's own tokenizer and +options. The tokens are plaintext, so no term carries them across the +binding. Sending the input and a page of decrypted values back in is no new +exposure: the input came from the host, and the values came out of the +guest as a decrypt. The alternative, a tokenizer in each language, is the +"lives twice" cost this ADR accepts only for an encoder, and a tokenizer that +drifts from the one that built the stored filter verifies the wrong thing. +`target_limit` takes the term's bytes, because the number of bits the query +sets is a property of the term the engine built, and the fill formula +belongs beside the index configuration. Both refuse an exact operator, which +has nothing to verify and no fetch to size, as the typed path does not +compile the call. + +**The catalog reports the operators each EQL type answers, and their +fidelity.** Each `se_targets` entry gains `operators`: one object per +operator the engine answers for a field that names the type, with its name, +its fidelity (`exact` or `probabilistic`), its arity (`one` or `pair`) and +the type's index that answers it. The engine computes the list from the +type's row by the same table every plan field is held to, so a generator +asks the engine it embeds rather than holding a copy of the rules, as the +2026-10-06 amendment asks of every rule. ADR-0009 rejects a fidelity value +on a term, because Rust says it with a trait. A generator asks before the +program runs, so it needs the answer as data, and the code it generates +carries `Verify` only where the answer is `probabilistic`, which rebuilds +decision 4 of ADR-0009 in a language without traits. Today `TextEq` answers +`eq` and no other type is produced. A type answers no probabilistic operator +and no `between` yet: the catalog names a type's match index but not its +options, which `verify` must cut tokens by, and a twin is one value, not a +range. Both wait for the EQL v4 types (#1179). + +What does not change: a binding still reaches the engine through a plan, and +nothing here derives a term outside the engine's own derivation, so a query +from Go and the typed query chain derive the same bytes. A cross-language +fixture holds them together (`tests/fixtures/query_operators.json`): the Rust +side checks each recorded term against the typed chain, and the Go SDK's +tests derive, verify and size the same queries through the guest. +`se_query`, which derives a target field's twin with no operator, stays until +the generated Go code names operators (#1189), and is removed then. +`se_term`, a term by index under an explicit context, stays for the generated +code of fields declared with indexes until that code names operators too. + +A data plan's match index declares no `expected_tokens` prior, so +`target_limit` on a field declared as data is unbounded without a measured +histogram. A data form for the prior is later work. From 4e5474805548fd0f05d232fc6bd50de27c559510 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 04:16:36 +1100 Subject: [PATCH 35/38] feat(stack-encrypt): let a binding ask which operators a plan field answers, so a generator holds no copy of the table dynamic::field_operators lists the operators a field of a data plan answers, each with the index that answers it: for a field declared with indexes, what OpSpec::answered_by finds under its kind; for a field that names an EQL type, what the type's catalog entry lists. query_term reads the same table, so what a generator is told a field answers is what a query on it finds. operator_to_value writes one entry in the catalog's wire form, which TargetDescriptor::to_value now shares. Part of #1189 --- packages/stack-encrypt/CHANGELOG.md | 8 + packages/stack-encrypt/src/dynamic/mod.rs | 8 +- packages/stack-encrypt/src/dynamic/query.rs | 240 +++++++++++++++++-- packages/stack-encrypt/src/dynamic/target.rs | 16 +- 4 files changed, 238 insertions(+), 34 deletions(-) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index f58686bce..d6ab0e9cd 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -28,6 +28,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (`TargetDescriptor::to_value`) lists the operators the engine answers for it, each with its fidelity (`dynamic::Fidelity`), arity and index (`TargetDescriptor::operators`). +- **A binding asks which operators a field answers.** + `dynamic::field_operators(&plan, field, &resolver)` lists the operators a + field of a data plan answers, each with the index that answers it: what + `query_term` derives a term for on that field and nothing else, for a + field declared with indexes as for one that names an EQL type. + `dynamic::operator_to_value` writes one entry in the catalog's wire form. + A generator asks this before the program runs, so it writes one query + method per operator and holds no copy of the table. - **A query names an operator, not an index.** `cipher.query(op::Eq, "bob@example.com").on(&plan)` and `cipher.query(op::Matches, "ali").on(&plan)`: the operator comes first, as diff --git a/packages/stack-encrypt/src/dynamic/mod.rs b/packages/stack-encrypt/src/dynamic/mod.rs index 2f7752b47..b72b30f36 100644 --- a/packages/stack-encrypt/src/dynamic/mod.rs +++ b/packages/stack-encrypt/src/dynamic/mod.rs @@ -40,7 +40,9 @@ //! * [`query_term`], [`verify`] and [`target_limit`] — a query named by //! operator ([`OpSpec`]) on a plan field, and a probabilistic term's two //! questions, over a page of decrypted values and over [`Stats`](crate::op::Stats) -//! (ADR-0009; ADR-0007, amended 2026-10-16). +//! (ADR-0009; ADR-0007, amended 2026-10-16), and [`field_operators`], the +//! operators a plan field answers, for a generator to ask before the +//! program runs. //! * [`Scope`] — which cipher an opening operation decrypts through. //! //! # What is not here @@ -100,8 +102,8 @@ pub use ffi_value::{ }; pub use kind::{admits, read}; pub use query::{ - check_query, limit_to_value, query_term, target_limit, verify, Fidelity, OpSpec, Operand, - QueryTerm, + check_query, field_operators, limit_to_value, operator_to_value, query_term, target_limit, + verify, Fidelity, OpSpec, Operand, QueryTerm, }; pub use record::{FieldPlan, Output, Plan}; pub use target::{NoTargets, TargetDescriptor, TargetError, TargetResolver}; diff --git a/packages/stack-encrypt/src/dynamic/query.rs b/packages/stack-encrypt/src/dynamic/query.rs index 8386ee73d..c6a121b93 100644 --- a/packages/stack-encrypt/src/dynamic/query.rs +++ b/packages/stack-encrypt/src/dynamic/query.rs @@ -16,6 +16,10 @@ //! that names an EQL type, that type's query value. It refuses an operator //! the field does not answer as it refuses a value of the wrong type, with //! [`Error::Source`] naming the field. +//! - [`field_operators`] lists the operators a plan field answers, each with +//! the index that answers it ([`operator_to_value`] writes one): what a +//! generator asks before the program runs, so it writes one query method +//! per operator the engine answers and holds no copy of the table. //! - [`verify`] and [`target_limit`] are a probabilistic term's two //! questions, [`Probabilistic::verify`] and //! [`Probabilistic::target_limit`], over a page of decrypted values and @@ -370,29 +374,16 @@ fn answer<'p>( operator: OpSpec, resolver: &(impl TargetResolver + ?Sized), ) -> Result, Error> { - let found = plan - .fields() - .iter() - .find(|candidate| candidate.name() == field) - .ok_or_else(|| Error::Plan { - field: Some(field.to_owned()), - reason: Reason::NoSuchField, - })?; + let found = find(plan, field)?; let not_answered = || Error::Source { field: Some(field.to_owned()), reason: Reason::OperatorNotAnswered, }; let kind = found.field_type().ok_or_else(not_answered)?; - let (index, target) = match found.target() { - Some(name) => { - let target = resolver.resolve(name).map_err(Error::Target)?; - let index = target_operators(&target) - .into_iter() - .find_map(|(op, index)| (op == operator).then_some(index)); - (index, Some(target)) - } - None => (operator.answered_by(kind, &found.indexes()).cloned(), None), - }; + let (operators, target) = answers(found, resolver)?; + let index = operators + .into_iter() + .find_map(|(op, index)| (op == operator).then_some(index)); Ok(Answer { field: found, kind, @@ -401,6 +392,116 @@ fn answer<'p>( }) } +/// The field named `field`, or [`Error::Plan`] with +/// [`Reason::NoSuchField`]. +fn find<'p>(plan: &'p Plan, field: &str) -> Result<&'p FieldPlan, Error> { + plan.fields() + .iter() + .find(|candidate| candidate.name() == field) + .ok_or_else(|| Error::Plan { + field: Some(field.to_owned()), + reason: Reason::NoSuchField, + }) +} + +/// Operators a field answers, each with the index that answers it. +type Operators = Vec<(OpSpec, IndexSpec)>; + +/// Every operator `field` answers with the index that answers it, and the +/// EQL type it names, resolved: the one table [`answer`] and +/// [`field_operators`] both read, so what a generator is told a field +/// answers is what a query on it finds. +fn answers( + field: &FieldPlan, + resolver: &(impl TargetResolver + ?Sized), +) -> Result<(Operators, Option), Error> { + let Some(kind) = field.field_type() else { + return Ok((Vec::new(), None)); + }; + if let Some(name) = field.target() { + let target = resolver.resolve(name).map_err(Error::Target)?; + return Ok((target_operators(&target), Some(target))); + } + let indexes = field.indexes(); + let operators = OpSpec::ALL + .into_iter() + .filter_map(|op| { + op.answered_by(kind, &indexes) + .map(|index| (op, index.clone())) + }) + .collect(); + Ok((operators, None)) +} + +/// The operators a field of a plan answers, each with the index that +/// answers it, in [`OpSpec::ALL`]'s order: exactly the operators +/// [`query_term`] derives a term for on that field, so a generator asks the +/// engine which query methods to write rather than holding a copy of the +/// table. +/// +/// A field with indexes answers each operator [`OpSpec::answered_by`] finds +/// an index for under its declared kind; a field that names an EQL type +/// answers what the type's [`TargetDescriptor::operators`] lists; a field +/// with no declared kind answers none. The context does not matter, so a +/// plan whose context is a field of each record is taken. Write each entry +/// with [`operator_to_value`]. +/// +/// ``` +/// use stack_encrypt::dynamic::{field_operators, record, FfiValue, NoTargets, OpSpec}; +/// use stack_encrypt::target::IndexSpec; +/// +/// let text = |s: &str| FfiValue::String(s.into()); +/// let plan = record::plan(FfiValue::Object(vec![( +/// "age".into(), +/// FfiValue::Object(vec![ +/// ("context".into(), FfiValue::Array(vec![text("users"), text("age")])), +/// ("outputs".into(), FfiValue::Array(vec![text("c"), text("eq"), text("ore")])), +/// ("type".into(), text("uint32")), +/// ]), +/// )]))?; +/// +/// let operators = field_operators(&plan, "age", &NoTargets)?; +/// assert_eq!(operators[0], (OpSpec::Eq, IndexSpec::Equality)); +/// assert_eq!(operators[1], (OpSpec::Lt, IndexSpec::Ore)); +/// assert_eq!(operators.len(), 6, "eq, the four comparisons and between"); +/// # Ok::<(), stack_encrypt::dynamic::Error>(()) +/// ``` +/// +/// # Errors +/// +/// [`Error::Plan`] for a field the plan does not have +/// ([`Reason::NoSuchField`]), and [`Error::Target`] for a target the +/// resolver refuses. +pub fn field_operators( + plan: &Plan, + field: &str, + resolver: &(impl TargetResolver + ?Sized), +) -> Result, Error> { + answers(find(plan, field)?, resolver).map(|(operators, _)| operators) +} + +/// One operator a field answers in its wire form, as a catalog lists it: an +/// object with every key present. +/// +/// | key | value | +/// |---|---| +/// | `operator` | string: the operator's name, `eq` ([`OpSpec::name`]) | +/// | `fidelity` | string: `exact` or `probabilistic` ([`Fidelity::name`]) | +/// | `arity` | string: `one`, or `pair` for an operator whose input is two values, low then high | +/// | `index` | string: the [key](IndexSpec::key) of the index that answers it, `eq` | +pub fn operator_to_value(operator: OpSpec, index: &IndexSpec) -> FfiValue { + let text = |s: &str| FfiValue::String(s.into()); + FfiValue::Object(vec![ + ("operator".to_owned(), text(operator.name())), + ("fidelity".to_owned(), text(operator.fidelity().name())), + ( + "arity".to_owned(), + text(if operator.takes_pair() { "pair" } else { "one" }), + ), + ("index".to_owned(), text(index.key())), + ]) +} + /// Refuse a plan whose context is a field of each record: a term derives /// under a context, and such a plan has none until a record names it, so a /// binding sends the plan under the context its query names. @@ -1747,6 +1848,104 @@ mod tests { ); } + /// What a field is said to answer is what a query on it finds: every + /// listed operator derives a term through the listed index, and every + /// other is refused as not answered. + #[tokio::test] + async fn a_fields_operators_are_exactly_the_queries_it_answers() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let input = |kind: &str, op: OpSpec| { + let one = if kind == "string" { + text("alice smith") + } else { + FfiValue::UInt32(7) + }; + if op.takes_pair() { + FfiValue::Array(vec![one.clone(), one]) + } else { + one + } + }; + for (name, kind, want) in [ + ("email", "string", vec![(OpSpec::Eq, IndexSpec::Equality)]), + ( + "name", + "string", + vec![(OpSpec::Matches, IndexSpec::Match(MatchOptions::default()))], + ), + ( + "age", + "uint32", + vec![ + (OpSpec::Lt, IndexSpec::Ore), + (OpSpec::Lte, IndexSpec::Ore), + (OpSpec::Gt, IndexSpec::Ore), + (OpSpec::Gte, IndexSpec::Ore), + (OpSpec::Between, IndexSpec::Ore), + ], + ), + ("nickname", "string", vec![]), + ] { + let listed = field_operators(&users(), name, &NoTargets).unwrap(); + assert_eq!(listed, want, "{name}"); + for op in OpSpec::ALL { + let found = query(&keyset, name, op, input(kind, op)).await; + match (listed.iter().find(|(listed, _)| *listed == op), found) { + (Some((_, index)), Ok(term)) => { + assert_eq!(term.index(), index, "{name} {op}"); + } + (Some(_), Err(error)) => panic!("{name} {op} is listed and refused: {error}"), + (None, Ok(_)) => panic!("{name} {op} is not listed and answered"), + (None, Err(error)) => assert_eq!( + reason_of(&error), + (Some(name), Some(Reason::OperatorNotAnswered)), + "{name} {op}" + ), + } + } + } + } + + #[test] + fn a_field_the_plan_lacks_has_no_operators() { + let error = field_operators(&users(), "phone", &NoTargets).unwrap_err(); + assert_eq!( + reason_of(&error), + (Some("phone"), Some(Reason::NoSuchField)) + ); + } + + #[test] + fn an_operator_is_written_with_every_key_in_order() { + let FfiValue::Object(entries) = operator_to_value(OpSpec::Between, &IndexSpec::Ope) else { + panic!("an object"); + }; + let entries: Vec<(&str, &[u8])> = entries + .iter() + .map(|(k, v)| match v { + FfiValue::String(s) => (k.as_str(), s.risky_ref()), + _ => panic!("{k} is a string"), + }) + .collect(); + assert_eq!( + entries, + [ + ("operator", b"between".as_slice()), + ("fidelity", b"exact".as_slice()), + ("arity", b"pair".as_slice()), + ("index", b"ope".as_slice()) + ] + ); + let FfiValue::Object(entries) = + operator_to_value(OpSpec::Matches, &IndexSpec::Match(MatchOptions::default())) + else { + panic!("an object"); + }; + assert!(matches!(&entries[1].1, FfiValue::String(s) if s.risky_ref() == b"probabilistic")); + assert!(matches!(&entries[2].1, FfiValue::String(s) if s.risky_ref() == b"one")); + } + /// A resolver holding one type that answers `eq`, whose query is a /// fixed JSON value: what a target field's query term carries. struct OneType; @@ -1809,6 +2008,11 @@ mod tests { assert_eq!(term.operand(), &Operand::One(br#"{"hm":"00"}"#.to_vec())); assert_eq!(term.domain(), Some("eql_v3.query_some")); assert_eq!(term.index(), &IndexSpec::Equality); + assert_eq!( + field_operators(&plan, "email", &OneType).unwrap(), + [(OpSpec::Eq, IndexSpec::Equality)], + "a target field answers what its type lists" + ); for op in [OpSpec::Lt, OpSpec::Matches, OpSpec::Between] { let error = query_term(&keyset, &plan, "email", op, text("a"), &OneType) .map(drop) diff --git a/packages/stack-encrypt/src/dynamic/target.rs b/packages/stack-encrypt/src/dynamic/target.rs index 757c536b1..e7c857f21 100644 --- a/packages/stack-encrypt/src/dynamic/target.rs +++ b/packages/stack-encrypt/src/dynamic/target.rs @@ -27,7 +27,7 @@ use std::fmt; -use super::query::target_operators; +use super::query::{operator_to_value, target_operators}; use super::{FfiValue, OpSpec, ValueKind}; use crate::{KeysetCipher, Label, Pending, StackCipher}; @@ -153,18 +153,8 @@ impl TargetDescriptor { "operators".to_string(), FfiValue::Array( target_operators(self) - .into_iter() - .map(|(op, index)| { - FfiValue::Object(vec![ - ("operator".to_string(), text(op.name())), - ("fidelity".to_string(), text(op.fidelity().name())), - ( - "arity".to_string(), - text(if op.takes_pair() { "pair" } else { "one" }), - ), - ("index".to_string(), text(index.key())), - ]) - }) + .iter() + .map(|(op, index)| operator_to_value(*op, index)) .collect(), ), ), From f5e6f9a67e288a7bc84b54bfd2f2128c67b86123 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 04:16:55 +1100 Subject: [PATCH 36/38] feat(golang)!: generate one query method per operator the engine answers, with Verify and TargetLimit on a matches query stashgen asks the engine which operators each sealed field answers, through a new guest export, se_operators, and writes one method for each: Eq, Lt, Lte, Gt, Gte, Between and Matches, each taking the field's own Go type. A field has no method for an operator it does not answer, so such a query does not compile. The generator holds no copy of which index answers which operator; the static test engine is held to the guest by a test that generates the same declarations through both and compares the files. Eq returns an EqualityTerm (or, on an encrypt_into field, the EQL type's query value), an ordering an OreTerm or OpeTerm, and Between an encrypt.Range of the two bounds' terms. Matches returns an encrypt.MatchQuery, whose answer can hold false positives, so it carries Verify, one guest call for a page of decrypted values, and TargetLimit. Every query type implements driver.Valuer. MatchQuery keeps the text it was derived from, because Verify cuts its tokens again in the engine; it never prints it. se_query, Cipher.Query and gensupport's per-index term methods are gone: eq on a target field is its twin, and a query names an operator. se_term stays, now behind the unexported Cipher.derive, because a record call's term failure does not say which field it came from and the host asks se_term again to name it. The order tests read the terms Encrypt stores, since a query method reaches only the index that answers its operator. Part of #1189 --- languages/golang/encrypt/checker.go | 45 +++- .../golang/encrypt/context_field_test.go | 8 +- languages/golang/encrypt/eql_test.go | 4 +- languages/golang/encrypt/errors_test.go | 2 +- languages/golang/encrypt/example/main.go | 20 +- .../golang/encrypt/example/user_stash.go | 41 ++- languages/golang/encrypt/fixture_test.go | 12 +- .../golang/encrypt/generated_paths_test.go | 8 +- languages/golang/encrypt/gensupport/field.go | 80 +++--- languages/golang/encrypt/guest.go | 14 +- languages/golang/encrypt/guest/src/abi.rs | 67 +++-- languages/golang/encrypt/guest/src/lib.rs | 11 +- languages/golang/encrypt/guest/src/ops.rs | 111 ++++----- .../golang/encrypt/guest/tests/native_ops.rs | 88 +++++++ languages/golang/encrypt/guest_test.go | 47 ++-- .../internal/testpolicy/member_stash.go | 41 ++- .../internal/testusers/account_stash.go | 5 +- .../internal/testusers/contact_stash.go | 6 +- .../encrypt/internal/testusers/kinds_stash.go | 235 ++++++++++++++---- .../internal/testusers/lookup_stash.go | 30 ++- .../internal/testusers/memberstash_stash.go | 41 ++- .../encrypt/internal/testusers/note_stash.go | 5 +- .../encrypt/internal/testusers/probe_stash.go | 91 ++++--- .../internal/testusers/profile_stash.go | 213 ++++++++++++++++ .../encrypt/internal/testusers/user_stash.go | 41 ++- .../encrypt/internal/testusers/users.go | 13 + languages/golang/encrypt/kinds_test.go | 14 +- languages/golang/encrypt/live_test.go | 4 +- languages/golang/encrypt/locate_test.go | 6 +- languages/golang/encrypt/order_test.go | 61 ++--- languages/golang/encrypt/query.go | 114 +++++++++ languages/golang/encrypt/query_test.go | 186 +++++++++++++- languages/golang/encrypt/records.go | 60 +---- languages/golang/encrypt/roundtrip_test.go | 16 +- languages/golang/encrypt/targets_test.go | 6 +- languages/golang/internal/record/record.go | 11 +- languages/golang/stashgen/emit.go | 109 +++++++- languages/golang/stashgen/engine.go | 64 ++++- .../golang/stashgen/enginetest/enginetest.go | 77 +++++- languages/golang/stashgen/generate.go | 18 ++ languages/golang/stashgen/policy.go | 3 + languages/golang/stashgen/read.go | 18 +- languages/golang/stashgen/refusal_test.go | 14 +- .../cases/accounts/account_stash.go.golden | 6 +- .../testdata/cases/contacts/contacts.go | 2 +- .../contacts/contactstash_stash.go.golden | 16 +- .../cases/embedded/patient_stash.go.golden | 11 +- .../foreign/individualstash_stash.go.golden | 17 +- .../cases/orders/order_stash.go.golden | 40 ++- .../cases/orders/refund_stash.go.golden | 6 +- .../cases/tenants/note_stash.go.golden | 11 +- .../testdata/cases/users/user_stash.go.golden | 12 +- .../policy_individual_stash.go.golden | 17 +- .../testdata/stubsdk/encrypt/encrypt.go | 9 + .../stubsdk/encrypt/gensupport/gensupport.go | 17 +- 55 files changed, 1723 insertions(+), 501 deletions(-) create mode 100644 languages/golang/encrypt/internal/testusers/profile_stash.go diff --git a/languages/golang/encrypt/checker.go b/languages/golang/encrypt/checker.go index 807967325..026734cc0 100644 --- a/languages/golang/encrypt/checker.go +++ b/languages/golang/encrypt/checker.go @@ -79,6 +79,37 @@ func (k *Checker) Targets(ctx context.Context) ([]record.Target, error) { return parseTargets(decoded) } +// Operators lists the operators one field of a plan answers, in the +// engine's order, each with its fidelity, its arity and the index that +// answers it: exactly the operators [Cipher.QueryTerm] derives a term for +// on that field, so stashgen writes one query method for each and holds no +// copy of which index answers which operator. A field the plan does not +// have is [ErrEncoding]. +func (k *Checker) Operators(ctx context.Context, plan *record.Plan, field string) ([]record.Operator, error) { + if err := plan.Validate(); err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + encoded, err := vcffi.Marshal(plan.Wire()) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrEncoding, err) + } + out, err := k.c.call(ctx, func(inst *instance) ([]byte, error) { + return inst.call(ctx, inst.operators, buf(encoded), buf([]byte(field))) + }) + if err != nil { + return nil, err + } + decoded, err := vcffi.Unmarshal(out) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrInternal, err) + } + operators, err := parseOperators(decoded) + if err != nil { + return nil, fmt.Errorf("%w: se_operators: %v", ErrInternal, err) + } + return operators, nil +} + // parseTargets reads se_targets's decoded value: {"targets": [entry, ...]}, // each entry an object with exactly the keys record.Target documents. The // shape is a contract between eql-bindings' serialiser and this reader, so @@ -166,7 +197,7 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { case "reason": t.Reason, err = targetOptionalString(f) case "operators": - t.Operators, err = targetOperators(f.Value) + t.Operators, err = parseOperators(f.Value) default: return t, fmt.Errorf("unknown key %q", f.Key) } @@ -185,21 +216,21 @@ func parseTarget(entry vcvalue.Object) (record.Target, error) { return t, nil } -// targetOperators reads an entry's operators: a list of objects, each with -// exactly the keys operator, fidelity, arity and index, every value one the -// engine writes. -func targetOperators(value any) ([]record.TargetOperator, error) { +// parseOperators reads a list of operators, an se_targets entry's or +// se_operators's: a list of objects, each with exactly the keys operator, +// fidelity, arity and index, every value one the engine writes. +func parseOperators(value any) ([]record.Operator, error) { list, ok := value.([]any) if !ok { return nil, fmt.Errorf("operators is %T, not a list", value) } - operators := make([]record.TargetOperator, 0, len(list)) + operators := make([]record.Operator, 0, len(list)) for i, item := range list { entry, ok := item.(vcvalue.Object) if !ok { return nil, fmt.Errorf("operator %d is %T, not an object", i, item) } - var op record.TargetOperator + var op record.Operator seen := map[string]bool{} for _, f := range entry { if seen[f.Key] { diff --git a/languages/golang/encrypt/context_field_test.go b/languages/golang/encrypt/context_field_test.go index 7952df4e7..303d71ae4 100644 --- a/languages/golang/encrypt/context_field_test.go +++ b/languages/golang/encrypt/context_field_test.go @@ -109,19 +109,19 @@ func TestContextFieldQueriesNameTheContext(t *testing.T) { t.Fatal(err) } // A query through a cipher with no context has nothing to derive under. - if _, err := testusers.NoteFields.Text.Equality(ctx, cipher, "hello"); !errors.Is(err, encrypt.ErrEncoding) || !strings.Contains(err.Error(), "Cipher.Context") { + if _, err := testusers.NoteFields.Text.Eq(ctx, cipher, "hello"); !errors.Is(err, encrypt.ErrEncoding) || !strings.Contains(err.Error(), "Cipher.Context") { t.Fatalf("a query with no context: %v, want ErrEncoding naming Cipher.Context", err) } // Under the tenant it singles out that tenant's row. acme := cipher.Context("tenants/acme") - probe, err := testusers.NoteFields.Text.Equality(ctx, acme, "hello") + probe, err := testusers.NoteFields.Text.Eq(ctx, acme, "hello") if err != nil { t.Fatal(err) } if !probe.Equal(encrypted[0].Text.Equality) || probe.Equal(encrypted[1].Text.Equality) || probe.Equal(encrypted[2].Text.Equality) { t.Error("the probe under tenants/acme does not single out acme's hello") } - globex, err := testusers.NoteFields.Text.Equality(ctx, cipher.Context("tenants/globex"), "hello") + globex, err := testusers.NoteFields.Text.Eq(ctx, cipher.Context("tenants/globex"), "hello") if err != nil || !globex.Equal(encrypted[1].Text.Equality) { t.Fatalf("the probe under tenants/globex: %v", err) } @@ -223,7 +223,7 @@ func TestGeneratedCodeOpensTheRustContextFieldFixture(t *testing.T) { t.Fatalf("through the globex cipher: %v, want ErrContextMismatch", err) } // The term Go derives under the tenant is the bytes Rust stored. - eq, err := testusers.NoteFields.Text.Equality(ctx, cipher.Context(want.Tenant), want.Text) + eq, err := testusers.NoteFields.Text.Eq(ctx, cipher.Context(want.Tenant), want.Text) if err != nil || !eq.Equal(stored.Text.Equality) { t.Errorf("text equality: %v, equal=%v", err, eq.Equal(stored.Text.Equality)) } diff --git a/languages/golang/encrypt/eql_test.go b/languages/golang/encrypt/eql_test.go index 833724c85..0d8f6df04 100644 --- a/languages/golang/encrypt/eql_test.go +++ b/languages/golang/encrypt/eql_test.go @@ -162,7 +162,7 @@ func TestTextEqQueryMatchesTheStoredValueAndTheRustFixture(t *testing.T) { t.Fatalf("the fixture's column changed: %+v", fixture) } - query, err := testusers.ContactFields.Email.Query(ctx, cipher, fixture.Plaintext) + query, err := testusers.ContactFields.Email.Eq(ctx, cipher, fixture.Plaintext) if err != nil { t.Fatal(err) } @@ -184,7 +184,7 @@ func TestTextEqQueryMatchesTheStoredValueAndTheRustFixture(t *testing.T) { if env := decodeEnvelope(t, stored); env.HM != probe.HM { t.Fatalf("the query's hm %s does not match the stored %s", probe.HM, env.HM) } - if other, _ := testusers.ContactFields.Email.Query(ctx, cipher, "Bob@example.com"); string(other) == fixture.Query { + if other, _ := testusers.ContactFields.Email.Eq(ctx, cipher, "Bob@example.com"); string(other) == fixture.Query { t.Fatal("no case folding: a different plaintext has a different term") } } diff --git a/languages/golang/encrypt/errors_test.go b/languages/golang/encrypt/errors_test.go index e7253d348..2ba3f1566 100644 --- a/languages/golang/encrypt/errors_test.go +++ b/languages/golang/encrypt/errors_test.go @@ -46,7 +46,7 @@ func TestEachKindCarriesItsDiagnostic(t *testing.T) { return err }, encrypt.ErrAuthentication, "stack_encrypt::aead"}, {"match text with no token", func() error { - _, err := testusers.Fields.Email.Match(ctx, cipher, "") + _, err := testusers.Fields.Email.Matches(ctx, cipher, "") return err }, encrypt.ErrTerm, "stack_encrypt::empty_term_text"}, {"a ciphertext that is not a sealed value", func() error { diff --git a/languages/golang/encrypt/example/main.go b/languages/golang/encrypt/example/main.go index 2157699d7..ef451565f 100644 --- a/languages/golang/encrypt/example/main.go +++ b/languages/golang/encrypt/example/main.go @@ -47,7 +47,7 @@ func run(ctx context.Context) error { } // A query term for one field compares against the stored terms. - term, err := Fields.Email.Equality(ctx, cipher, "bob@example.com") + term, err := Fields.Email.Eq(ctx, cipher, "bob@example.com") if err != nil { return fmt.Errorf("query term: %w", err) } @@ -66,5 +66,23 @@ func run(ctx context.Context) error { for _, u := range back { fmt.Printf("decrypted id %d\n", u.ID) } + + // A matches query's answer can hold rows that do not match, so the + // rows it finds are decrypted and verified: one call for the page. + q, err := Fields.Email.Matches(ctx, cipher, "bob") + if err != nil { + return fmt.Errorf("matches: %w", err) + } + emails := make([]string, len(back)) + for i, u := range back { + emails[i] = u.Email + } + verified, err := q.Verify(ctx, emails) + if err != nil { + return fmt.Errorf("verify: %w", err) + } + for i, u := range back { + fmt.Printf("id %d holds bob: %v\n", u.ID, verified[i]) + } return nil } diff --git a/languages/golang/encrypt/example/user_stash.go b/languages/golang/encrypt/example/user_stash.go index 13cf2b6ef..4dc7ec03a 100644 --- a/languages/golang/encrypt/example/user_stash.go +++ b/languages/golang/encrypt/example/user_stash.go @@ -142,12 +142,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedUserEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type AgeField struct { @@ -162,10 +165,32 @@ func (f AgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (Enc return EncryptedUserAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f AgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f AgeField) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f AgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f AgeField) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } diff --git a/languages/golang/encrypt/fixture_test.go b/languages/golang/encrypt/fixture_test.go index e992a9679..5f92793a7 100644 --- a/languages/golang/encrypt/fixture_test.go +++ b/languages/golang/encrypt/fixture_test.go @@ -122,21 +122,21 @@ func TestGeneratedCodeOpensTheRustRecordFixture(t *testing.T) { } } // The terms Go derives are the bytes Rust stored. - eq, err := testusers.Fields.Age.Equality(ctx, cipher, f.Plaintext.Age) + eq, err := testusers.Fields.Age.Eq(ctx, cipher, f.Plaintext.Age) if err != nil || !eq.Equal(stored.Age.Equality) { t.Errorf("age equality: %v, equal=%v", err, eq.Equal(stored.Age.Equality)) } - ore, err := testusers.Fields.Age.Ore(ctx, cipher, f.Plaintext.Age) + ore, err := testusers.Fields.Age.Lt(ctx, cipher, f.Plaintext.Age) if err != nil || !bytes.Equal(ore, stored.Age.Ore) { t.Errorf("age ore: %v, equal=%v", err, bytes.Equal(ore, stored.Age.Ore)) } - emailEq, err := testusers.Fields.Email.Equality(ctx, cipher, f.Plaintext.Email) + emailEq, err := testusers.Fields.Email.Eq(ctx, cipher, f.Plaintext.Email) if err != nil || !emailEq.Equal(stored.Email.Equality) { t.Errorf("email equality: %v, equal=%v", err, emailEq.Equal(stored.Email.Equality)) } - match, err := testusers.Fields.Email.Match(ctx, cipher, f.Plaintext.Email) - if err != nil || !bytes.Equal(match, stored.Email.Match) { - t.Errorf("email match: %v, equal=%v", err, bytes.Equal(match, stored.Email.Match)) + match, err := testusers.Fields.Email.Matches(ctx, cipher, f.Plaintext.Email) + if err != nil || !bytes.Equal(match.Term(), stored.Email.Match) { + t.Errorf("email match: %v, equal=%v", err, bytes.Equal(match.Term(), stored.Email.Match)) } // A fresh Go record of the same plaintext derives the same terms. again, err := testusers.Encrypt(ctx, cipher, []testusers.User{{ID: int64(f.Plaintext.ID), Age: f.Plaintext.Age, Email: f.Plaintext.Email, Notes: f.Plaintext.Notes}}) diff --git a/languages/golang/encrypt/generated_paths_test.go b/languages/golang/encrypt/generated_paths_test.go index efd787fbc..03011a841 100644 --- a/languages/golang/encrypt/generated_paths_test.go +++ b/languages/golang/encrypt/generated_paths_test.go @@ -117,19 +117,19 @@ func TestGeneratedCodeFromForAndFromAPolicyRunsAgainstTheGuest(t *testing.T) { } // A term derived through each path's Fields is the tag path's. - probe, err := testusers.Fields.Email.Equality(ctx, cipher, "bob@example.com") + probe, err := testusers.Fields.Email.Eq(ctx, cipher, "bob@example.com") if err != nil { t.Fatal(err) } - forProbe, err := testusers.MemberFields.Email.Equality(ctx, cipher, "bob@example.com") + forProbe, err := testusers.MemberFields.Email.Eq(ctx, cipher, "bob@example.com") if err != nil || !forProbe.Equal(probe) { t.Fatalf("the -for path's probe: %v, equal=%v", err, forProbe.Equal(probe)) } - policyProbe, err := testpolicy.Fields.Email.Equality(ctx, cipher, "bob@example.com") + policyProbe, err := testpolicy.Fields.Email.Eq(ctx, cipher, "bob@example.com") if err != nil || !policyProbe.Equal(probe) { t.Fatalf("the policy path's probe: %v, equal=%v", err, policyProbe.Equal(probe)) } - ore, err := testpolicy.Fields.Age.Ore(ctx, cipher, 29) + ore, err := testpolicy.Fields.Age.Lt(ctx, cipher, 29) if err != nil || !bytes.Equal(ore, tagged[1].Age.Ore) { t.Fatalf("the policy path's ORE probe: %v", err) } diff --git a/languages/golang/encrypt/gensupport/field.go b/languages/golang/encrypt/gensupport/field.go index 8006b0a01..3881daf7a 100644 --- a/languages/golang/encrypt/gensupport/field.go +++ b/languages/golang/encrypt/gensupport/field.go @@ -10,7 +10,7 @@ import ( // Field is one sealed field's entry in a generated Fields value: it encrypts // one value of the field, for an update of one column, and derives the -// terms the field declares, for a query. T is the field's Go type. +// query term of each operator the field answers. T is the field's Go type. type Field[T any] struct { name string plan *record.Plan @@ -65,58 +65,58 @@ func (f Field[T]) Encrypt(ctx context.Context, c *encrypt.Cipher, v T) (Output, return outputOf(sealed[0][f.name]), nil } -// Query derives the EQL query value for one value of an encrypt_into field: -// the operand that matches stored values of the field, in Output.EQL. The -// engine runs the EQL type's own query plan, with no data key. -func (f Field[T]) Query(ctx context.Context, c *encrypt.Cipher, v T) (Output, error) { - if f.err != nil { - return Output{}, f.err - } - if c == nil { - return Output{}, fmt.Errorf("gensupport: %s: Query needs a cipher", f.name) - } - out, err := c.Query(ctx, f.plan, f.name, v) +// Term derives the term of an exact operator the field answers, for one +// value: an index term for a field declared with indexes, the query value's +// JSON for a field that names an EQL type. The generated method for the +// operator converts it to its query type. The engine picks the index that +// answers the operator ([encrypt.Cipher.QueryTerm]). +func (f Field[T]) Term(ctx context.Context, c *encrypt.Cipher, operator string, v T) ([]byte, error) { + terms, err := f.query(ctx, c, operator, v, 1) if err != nil { - return Output{}, err + return nil, err } - return Output{EQL: out}, nil -} - -// Equality derives the field's equality term for one value. -func (f Field[T]) Equality(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.EqualityTerm, error) { - return f.term(ctx, c, record.Equality, v) -} - -// Match derives the field's match term for one value. -func (f Field[T]) Match(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.MatchTerm, error) { - return f.term(ctx, c, record.Match, v) + return terms[0], nil } -// Ore derives the field's ORE term for one value. -func (f Field[T]) Ore(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.OreTerm, error) { - return f.term(ctx, c, record.Ore, v) -} - -// Ope derives the field's OPE term for one value. -func (f Field[T]) Ope(ctx context.Context, c *encrypt.Cipher, v T) (encrypt.OpeTerm, error) { - return f.term(ctx, c, record.Ope, v) +// Range derives the two terms of a range operator the field answers, such +// as between: the low bound's term, then the high bound's. +func (f Field[T]) Range(ctx context.Context, c *encrypt.Cipher, operator string, lo, hi T) ([]byte, []byte, error) { + terms, err := f.query(ctx, c, operator, []any{lo, hi}, 2) + if err != nil { + return nil, nil, err + } + return terms[0], terms[1], nil } -// JSON derives the field's json index term. The engine does not derive it -// yet, so it fails. -func (f Field[T]) JSON(context.Context, *encrypt.Cipher, T) (encrypt.JSONTerm, error) { +// query derives an exact operator's terms, and refuses an answer of another +// shape than the generated method's: a probabilistic term has a query type +// of its own (Matches). +func (f Field[T]) query(ctx context.Context, c *encrypt.Cipher, operator string, input any, parts int) ([][]byte, error) { if f.err != nil { return nil, f.err } - return nil, fmt.Errorf("gensupport: %s: the engine does not derive the json index yet", f.name) + if c == nil { + return nil, fmt.Errorf("gensupport: %s: a query needs a cipher", f.name) + } + term, err := c.QueryTerm(ctx, f.plan, f.name, operator, input) + if err != nil { + return nil, err + } + if term.Fidelity != record.Exact || len(term.Terms) != parts { + return nil, fmt.Errorf("gensupport: %s: %s came back as a %s term of %d parts, not an exact one of %d", f.name, operator, term.Fidelity, len(term.Terms), parts) + } + return term.Terms, nil } -func (f Field[T]) term(ctx context.Context, c *encrypt.Cipher, output record.Output, v T) ([]byte, error) { +// Matches derives the matches query for one value of a field declared with +// a match index: the term, and what Verify and TargetLimit need. A function, +// not a method, because only a string field answers matches. +func Matches[T ~string](ctx context.Context, f Field[T], c *encrypt.Cipher, v T) (encrypt.MatchQuery[T], error) { if f.err != nil { - return nil, f.err + return encrypt.MatchQuery[T]{}, f.err } if c == nil { - return nil, fmt.Errorf("gensupport: %s: a term needs a cipher", f.name) + return encrypt.MatchQuery[T]{}, fmt.Errorf("gensupport: %s: a query needs a cipher", f.name) } - return c.Derive(ctx, f.plan, f.name, output, v) + return encrypt.NewMatchQuery(ctx, c, f.plan, f.name, v) } diff --git a/languages/golang/encrypt/guest.go b/languages/golang/encrypt/guest.go index 78ef91c4d..def93995e 100644 --- a/languages/golang/encrypt/guest.go +++ b/languages/golang/encrypt/guest.go @@ -82,12 +82,12 @@ type instance struct { // on it. mem *guest.Allocator - exports guest.Exports - cipherInit, shutdown, keyset api.Function - term, query api.Function - queryTerm, verify, limit api.Function - encryptRecord, decryptRecord api.Function - planCheck, targets api.Function + exports guest.Exports + cipherInit, shutdown, keyset api.Function + term api.Function + queryTerm, verify, limit api.Function + encryptRecord, decryptRecord api.Function + planCheck, targets, operators api.Function } // guestModuleConfig is the module configuration every guest instance runs @@ -172,7 +172,6 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo "se_shutdown": &inst.shutdown, "se_keyset": &inst.keyset, "se_term": &inst.term, - "se_query": &inst.query, "se_query_term": &inst.queryTerm, "se_verify": &inst.verify, "se_target_limit": &inst.limit, @@ -180,6 +179,7 @@ func newInstance(ctx context.Context, wasm []byte, t *transport, policy guest.Lo "se_decrypt_record": &inst.decryptRecord, "se_plan_check": &inst.planCheck, "se_targets": &inst.targets, + "se_operators": &inst.operators, } for name, slot := range exports { if *slot = module.ExportedFunction(name); *slot == nil { diff --git a/languages/golang/encrypt/guest/src/abi.rs b/languages/golang/encrypt/guest/src/abi.rs index e608bb90b..679ce8371 100644 --- a/languages/golang/encrypt/guest/src/abi.rs +++ b/languages/golang/encrypt/guest/src/abi.rs @@ -6,7 +6,7 @@ //! //! - **Every export handed plaintext wipes that buffer in place before it //! returns**, rather than leaving it for `se_dealloc`: [`se_cipher_init`] -//! (the config carries the client key), and [`se_term`], [`se_query`], +//! (the config carries the client key), and [`se_term`], //! [`se_query_term`], [`se_verify`] and [`se_encrypt_record`] (their //! value, source and input buffers, and [`se_verify`]'s page of decrypted //! values). The host's plaintext therefore lives no longer @@ -54,8 +54,8 @@ //! fields, so their contexts must be non-empty (`STATUS_ENCODING` //! otherwise): each is a [`stack_encrypt::NonEmpty`] from the moment it is //! parsed, and the sealing and opening sides bind that one value. -//! [`se_plan_check`] and [`se_targets`] answer the Go generator's questions -//! and need no cipher; nor do [`se_verify`] and [`se_target_limit`], a +//! [`se_plan_check`], [`se_targets`] and [`se_operators`] answer the Go +//! generator's questions and need no cipher; nor do [`se_verify`] and [`se_target_limit`], a //! probabilistic query's two questions, which use no key. //! //! Every export decodes and validates *all* of its inputs — the operation @@ -467,42 +467,6 @@ pub unsafe extern "C" fn se_plan_check(plan_ptr: *const u8, plan_len: u32) -> u6 }) } -/// Derive the EQL query value of one target field under the keyset `opts` -/// selects: a codec-encoded plaintext, the codec-encoded plan and the -/// field's name (UTF-8 bytes) in, the query value's JSON bytes out -/// ([`ops::query`]). The plan is parsed against this build's resolver, so -/// a build without EQL types refuses it as `STATUS_ENCODING` before the -/// cipher is consulted, as it refuses the same plan at [`se_plan_check`]. -/// -/// # Safety -/// -/// As for [`se_term`]. -#[no_mangle] -pub unsafe extern "C" fn se_query( - val_ptr: *mut u8, - val_len: u32, - plan_ptr: *const u8, - plan_len: u32, - field_ptr: *const u8, - field_len: u32, - opt_ptr: *const u8, - opt_len: u32, -) -> u64 { - export(|| { - let value = unsafe { take_plaintext(val_ptr, val_len)? }; - let value = value.as_slice(); - // SAFETY: host-owned ranges the export was handed; the borrows end - // before it returns and before any wipe of an overlapping range. - let plan = unsafe { input(plan_ptr, plan_len)? }; - let field = unsafe { input(field_ptr, field_len)? }; - let opts = unsafe { input(opt_ptr, opt_len)? }; - ops::validate::query(value, plan, field)?; - with_keyset(opts, |keyset| { - block_on(ops::query(keyset, value, plan, field)) - }) - }) -} - /// Derive the query term for one plan field, an operator and its input /// under the keyset `opts` selects ([`ops::query_term`]): a codec-encoded /// input (one value, or for `between` a list of two), the codec-encoded @@ -625,3 +589,28 @@ pub unsafe extern "C" fn se_target_limit( pub extern "C" fn se_targets() -> u64 { export(ops::targets) } + +/// The operators one plan field answers ([`ops::operators`]): the +/// codec-encoded plan and the field's name (UTF-8 bytes) in, a codec-encoded +/// list of operator objects out. A field the plan does not have is +/// `STATUS_ENCODING`. Needs no cipher: it works in every instance state, as +/// [`se_plan_check`] does. +/// +/// # Safety +/// +/// As for [`se_term`]. +#[no_mangle] +pub unsafe extern "C" fn se_operators( + plan_ptr: *const u8, + plan_len: u32, + field_ptr: *const u8, + field_len: u32, +) -> u64 { + export(|| { + // SAFETY: host-owned ranges the export was handed; the borrows end + // before they return. + let plan = unsafe { input(plan_ptr, plan_len)? }; + let field = unsafe { input(field_ptr, field_len)? }; + ops::operators(plan, field) + }) +} diff --git a/languages/golang/encrypt/guest/src/lib.rs b/languages/golang/encrypt/guest/src/lib.rs index c22d3a2dc..fd25df371 100644 --- a/languages/golang/encrypt/guest/src/lib.rs +++ b/languages/golang/encrypt/guest/src/lib.rs @@ -45,11 +45,12 @@ //! `se_cipher_init`; derived data keys and index keys never leave. //! //! The exports are the record path (`se_encrypt_record`, `se_decrypt_record`), -//! per-field term derivation (`se_term`), a target field's EQL query value -//! (`se_query`), a query named by operator and a probabilistic query's two -//! questions (`se_query_term`, `se_verify`, `se_target_limit`; ADR-0007, -//! amended 2026-10-16), keyset resolution (`se_keyset`), the generator's two -//! questions (`se_plan_check`, `se_targets`) and the lifetime pair +//! the term of one index under a context (`se_term`, which the Go SDK asks +//! to name the field behind a record's term failure), a query named by +//! operator and a probabilistic query's two questions (`se_query_term`, +//! `se_verify`, `se_target_limit`; ADR-0007, amended 2026-10-16), keyset +//! resolution (`se_keyset`), the generator's three questions +//! (`se_plan_check`, `se_targets`, `se_operators`) and the lifetime pair //! (`se_cipher_init`, `se_shutdown`). There is no whole-value export: the //! Go SDK seals every value under a declaration (ADR-0007). //! diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index be09d95d9..6317c298d 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -213,31 +213,6 @@ where encode_value(value) } -/// Derive the EQL query value of one target field: a codec-encoded -/// plaintext, the codec-encoded plan and the field's name (UTF-8) in, the -/// query value's JSON bytes out — the operand `eql_v3.query_` takes. -/// Runs the EQL type's own query plan through this build's resolver -/// ([`crate::targets`]); a build without EQL types refuses the plan before -/// this is reached. No data key is minted: under the local backend the -/// equality term is one PRF derivation. -pub async fn query( - cipher: &KeysetCipher<'_, K>, - value: &[u8], - plan: &[u8], - field: &[u8], -) -> Result, u32> -where - K: DataKeySource + Sync + 'static, -{ - let plan = parse_plan(plan)?; - let field = std::str::from_utf8(field) - .map_err(|_| last_error::malformed("the field name is not UTF-8"))?; - dynamic::record::query(cipher, &plan, field, decode_value(value)?, &resolver()) - .map_err(|e| fail_dynamic(&e))? - .await - .map_err(|e| fail_error(&e)) -} - // ============================================================================= // Queries by operator // ============================================================================= @@ -430,6 +405,36 @@ pub fn targets() -> Result, u32> { )])) } +/// The operators one field of a codec-encoded plan answers, as a +/// codec-encoded list: one object per operator, in the order the engine +/// lists them, each written by +/// [`dynamic::operator_to_value`](stack_encrypt::dynamic::operator_to_value): +/// +/// ```text +/// [ { "operator": "eq", "fidelity": "exact", "arity": "one", "index": "eq" }, ... ] +/// ``` +/// +/// Exactly the operators [`query_term`] derives a term for on that field +/// ([`dynamic::field_operators`]): the operators a field declared with +/// indexes answers, and those its EQL type answers for a field that names +/// one. The Go generator (`stashgen`) asks this for every sealed field it +/// writes and generates one query method per entry, so it holds no copy of +/// which index answers which operator. A field the plan does not have is +/// [`STATUS_ENCODING`]. Needs no cipher. +pub fn operators(plan: &[u8], field: &[u8]) -> Result, u32> { + let plan = parse_plan(plan)?; + let field = std::str::from_utf8(field) + .map_err(|_| last_error::malformed("the field name is not UTF-8"))?; + let operators = + dynamic::field_operators(&plan, field, &resolver()).map_err(|e| fail_dynamic(&e))?; + encode_value(FfiValue::Array( + operators + .iter() + .map(|(operator, index)| dynamic::operator_to_value(*operator, index)) + .collect(), + )) +} + // ============================================================================= // Boundary validation // ============================================================================= @@ -463,37 +468,6 @@ pub mod validate { dynamic::record::check_source(decode_value(source)?, &plan).map_err(|e| fail_dynamic(&e)) } - /// A target query's inputs, as [`query`] takes them: the plan parses - /// against this build's resolver, the field is one of its target fields, - /// and the value is of the field's declared kind. - pub fn query(value: &[u8], plan: &[u8], field: &[u8]) -> Result<(), u32> { - let plan = parse_plan(plan)?; - let field = std::str::from_utf8(field) - .map_err(|_| last_error::malformed("the field name is not UTF-8"))?; - let refuse = |reason| { - fail_dynamic(&dynamic::Error::Plan { - field: Some(field.to_owned()), - reason, - }) - }; - let field = plan - .fields() - .iter() - .find(|candidate| candidate.name() == field) - .ok_or_else(|| refuse(dynamic::Reason::NoSuchField))?; - if field.target().is_none() { - return Err(refuse(dynamic::Reason::NotATarget)); - } - let value = decode_value(value)?; - if field.field_type().is_some_and(|kind| !kind.holds(&value)) { - return Err(fail_dynamic(&dynamic::Error::Source { - field: Some(field.name().to_owned()), - reason: dynamic::Reason::FieldType, - })); - } - Ok(()) - } - /// A query's inputs, as [`query_term`](super::query_term) takes them: /// the plan parses, the field is one of its fields, the operator is one /// the engine knows and the field answers, and the input reads as the @@ -925,11 +899,22 @@ mod tests { ); assert!(matches!(&values[1].1, FfiValue::String(s) if s.risky_ref() == b"likes cats")); - // The query value matches the stored term. + // The eq query's term is the type's query value, and it matches the + // stored term. let probe = encoded(text("alice@example.com")); - validate::query(&probe, &plan, b"email").expect("a target field"); - let query_bytes = block_on(query(&keyset, &probe, &plan, b"email")).expect("derives"); - let probe_json: serde_json::Value = serde_json::from_slice(&query_bytes).expect("JSON"); + validate::query_term(&probe, &plan, b"email", b"eq").expect("a target field answers eq"); + let term = block_on(query_term(&keyset, &probe, &plan, b"email", b"eq")).expect("derives"); + let FfiValue::Object(term) = decode_value(&term).expect("a value") else { + panic!("a query term is an object") + }; + assert!( + matches!(&term[3].1, FfiValue::String(s) if s.risky_ref() == b"eql_v3.query_text_eq") + ); + let FfiValue::Bytes(query_bytes) = &term[4].1 else { + panic!("the term is the query value's bytes") + }; + let probe_json: serde_json::Value = + serde_json::from_slice(query_bytes.risky_ref()).expect("JSON"); assert_eq!( probe_json["hm"], eql["hm"], "the query matches the stored value" @@ -941,15 +926,15 @@ mod tests { // A query on a sealed field, an unknown field, or a wrong kind is // refused at validation. assert_eq!( - validate::query(&probe, &plan, b"notes"), + validate::query_term(&probe, &plan, b"notes", b"eq"), Err(STATUS_ENCODING) ); assert_eq!( - validate::query(&probe, &plan, b"nope"), + validate::query_term(&probe, &plan, b"nope", b"eq"), Err(STATUS_ENCODING) ); assert_eq!( - validate::query(&encoded(FfiValue::UInt32(1)), &plan, b"email"), + validate::query_term(&encoded(FfiValue::UInt32(1)), &plan, b"email", b"eq"), Err(STATUS_ENCODING) ); } diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 4cb971ac3..28b4dfb7d 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -1745,6 +1745,94 @@ fn the_catalog_lists_each_types_operators() { } } +/// The operators of one field, each as `(operator, fidelity, arity, index)`. +fn operators_of(plan: &[u8], field: &str) -> Vec<[String; 4]> { + let FfiValue::Array(items) = + decode(&ops::operators(plan, field.as_bytes()).expect("operators")) + else { + panic!("operators are a list"); + }; + items + .iter() + .map(|item| { + let FfiValue::Object(entries) = item else { + panic!("an operator is an object"); + }; + let keys: Vec<&str> = entries.iter().map(|(k, _)| k.as_str()).collect(); + assert_eq!(keys, ["operator", "fidelity", "arity", "index"]); + [0, 1, 2, 3].map(|i| text(&entries[i].1).to_owned()) + }) + .collect() +} + +/// A field's operators are the queries the engine answers on it, with +/// their fidelity, arity and index: what the generator writes one method +/// for. Every operator listed derives a term, and every other is refused. +#[test] +fn a_fields_operators_are_the_queries_it_answers() { + let row = |op: &str, fidelity: &str, arity: &str, index: &str| { + [op, fidelity, arity, index].map(str::to_owned) + }; + assert_eq!( + operators_of(&plan(), "age"), + [ + row("eq", "exact", "one", "eq"), + row("lt", "exact", "one", "ore"), + row("lte", "exact", "one", "ore"), + row("gt", "exact", "one", "ore"), + row("gte", "exact", "one", "ore"), + row("between", "exact", "pair", "ore"), + ] + ); + assert_eq!( + operators_of(&plan(), "name"), + [row("matches", "probabilistic", "one", "match")] + ); + for (field, input) in [("age", FfiValue::UInt32(7)), ("name", s("alice smith"))] { + let listed: Vec = operators_of(&plan(), field) + .into_iter() + .map(|[op, ..]| op) + .collect(); + for op in ["eq", "lt", "lte", "gt", "gte", "between", "matches"] { + let input = if op == "between" { + FfiValue::Array(vec![input.clone(), input.clone()]) + } else { + input.clone() + }; + let checked = + ops::validate::query_term(&encode(input), &plan(), field.as_bytes(), op.as_bytes()); + if listed.iter().any(|l| l == op) { + assert_eq!(checked, Ok(()), "{field} {op} is listed and answered"); + } else { + assert_eq!(checked, Err(STATUS_ENCODING), "{field} {op} is not listed"); + } + } + } + assert_eq!( + ops::operators(&plan(), b"phone"), + Err(STATUS_ENCODING), + "a field the plan lacks" + ); + assert_eq!(ops::operators(b"\xff", b"age"), Err(STATUS_ENCODING)); +} + +/// In the `eql` build a field that names an EQL type answers what the +/// catalog lists for the type: `TextEq` answers `eq`. +#[test] +fn a_target_fields_operators_are_its_types() { + if !stack_encrypt_guest::targets::HOLDS_EQL { + return; + } + let plan = encode(obj(vec![( + "email", + obj(vec![("context", label("email")), ("target", s("TextEq"))]), + )])); + assert_eq!( + operators_of(&plan, "email"), + [["eq", "exact", "one", "eq"].map(str::to_owned)] + ); +} + // ============================================================================= // A plan whose context is a field of the record // ============================================================================= diff --git a/languages/golang/encrypt/guest_test.go b/languages/golang/encrypt/guest_test.go index 166d6d57d..a98266f3b 100644 --- a/languages/golang/encrypt/guest_test.go +++ b/languages/golang/encrypt/guest_test.go @@ -145,7 +145,7 @@ func TestImportSurfaceIsWASIPlusTransport(t *testing.T) { if !reflect.DeepEqual(transportImports, want) { t.Fatalf("transport imports = %v, want %v", transportImports, want) } - for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_term": true, "se_query": true, "se_query_term": true, "se_verify": true, "se_target_limit": true, "se_encrypt_record": true, "se_decrypt_record": true, "se_plan_check": true, "se_targets": true} { + for name := range map[string]bool{"se_alloc": true, "se_dealloc": true, "se_cipher_init": true, "se_shutdown": true, "se_keyset": true, "se_term": true, "se_query_term": true, "se_verify": true, "se_target_limit": true, "se_encrypt_record": true, "se_decrypt_record": true, "se_plan_check": true, "se_targets": true, "se_operators": true} { if _, ok := compiled.ExportedFunctions()[name]; !ok { t.Errorf("guest does not export %s", name) } @@ -649,15 +649,20 @@ func TestTargetWithoutEQLIsRefusedBeforeTheGuest(t *testing.T) { } } -// Query refuses a field the plan does not have, and a field that names no -// EQL type, before the guest is asked. -func TestQueryRejectsMissingAndNonTargetFieldsBeforeTheGuest(t *testing.T) { +// QueryTerm refuses a field the plan does not have, an operator the engine +// does not know and one the field does not answer before it looks for a +// cipher: ErrEncoding naming the field, on an uninitialised instance. +func TestQueryTermRefusesWhatTheFieldDoesNotAnswerBeforeState(t *testing.T) { ctx := context.Background() c := rawInstance(t) - for _, field := range []string{"email", "missing"} { - _, err := c.DefaultKeyset().Query(ctx, usersPlan(), field, "a@b.c") - if !errors.Is(err, ErrEncoding) || errors.Is(err, ErrState) || !strings.Contains(err.Error(), field) { - t.Errorf("Query(%q) = %v, want ErrEncoding before the guest, naming the field", field, err) + for _, q := range []struct{ field, operator string }{ + {"missing", "eq"}, + {"age", "like"}, + {"age", "matches"}, + } { + _, err := c.DefaultKeyset().QueryTerm(ctx, usersPlan(), q.field, q.operator, uint32(1)) + if !errors.Is(err, ErrEncoding) || errors.Is(err, ErrState) || !strings.Contains(err.Error(), q.field) { + t.Errorf("QueryTerm(%q, %q) = %v, want ErrEncoding before the guest, naming the field", q.field, q.operator, err) } } } @@ -705,13 +710,19 @@ func TestGuestAcceptsEveryEncodingThisPackageBuilds(t *testing.T) { "Open bound": func() error { _, err := byID.Open(ctx, usersPlan(), records); return err }, "Open extended": func() error { _, err := tenant.Open(ctx, usersPlan(), records); return err }, "Open any": func() error { _, err := c.Open(ctx, usersPlan(), records); return err }, - "Derive equality": func() error { _, err := def.Derive(ctx, usersPlan(), "age", record.Equality, uint32(34)); return err }, - "Derive match": func() error { _, err := named.Derive(ctx, usersPlan(), "email", record.Match, "alice"); return err }, - "Derive ore ext": func() error { _, err := tenant.Derive(ctx, usersPlan(), "age", record.Ore, uint32(1)); return err }, - "Derive ope": func() error { + "derive equality": func() error { _, err := def.derive(ctx, usersPlan(), "age", record.Equality, uint32(34)); return err }, + "derive match": func() error { _, err := named.derive(ctx, usersPlan(), "email", record.Match, "alice"); return err }, + "derive ore ext": func() error { _, err := tenant.derive(ctx, usersPlan(), "age", record.Ore, uint32(1)); return err }, + "QueryTerm eq": func() error { _, err := def.QueryTerm(ctx, usersPlan(), "age", "eq", uint32(34)); return err }, + "QueryTerm between": func() error { + _, err := tenant.QueryTerm(ctx, usersPlan(), "age", "between", []any{uint32(18), uint32(65)}) + return err + }, + "QueryTerm matches": func() error { _, err := named.QueryTerm(ctx, usersPlan(), "email", "matches", "alice"); return err }, + "derive ope": func() error { p := usersPlan() p.Fields[0].Outputs = append(p.Fields[0].Outputs, record.Ope) - _, err := byID.Derive(ctx, p, "age", record.Ope, uint32(1)) + _, err := byID.derive(ctx, p, "age", record.Ope, uint32(1)) return err }, "Seal untyped": func() error { @@ -737,9 +748,9 @@ func TestGuestRefusesMalformedInputsBeforeState(t *testing.T) { boolFlag := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "flag", Kind: record.Bool, Outputs: []record.Output{record.Ciphertext, record.Equality}}}} matchInt := &record.Plan{Context: []string{"users"}, Fields: []record.Field{{Name: "age", Kind: record.Uint32, Outputs: []record.Output{record.Ciphertext, record.Match}}}} calls := map[string]func() error{ - "bool under equality": func() error { _, err := def.Derive(ctx, boolFlag, "flag", record.Equality, true); return err }, - "integer under match": func() error { _, err := def.Derive(ctx, matchInt, "age", record.Match, uint32(1)); return err }, - "container as term value": func() error { _, err := def.Derive(ctx, plan, "age", record.Equality, []any{1}); return err }, + "bool under equality": func() error { _, err := def.derive(ctx, boolFlag, "flag", record.Equality, true); return err }, + "integer under match": func() error { _, err := def.derive(ctx, matchInt, "age", record.Match, uint32(1)); return err }, + "container as term value": func() error { _, err := def.derive(ctx, plan, "age", record.Equality, []any{1}); return err }, "a kind the type refuses": func() error { _, err := def.Seal(ctx, matchInt, []record.Source{usersRow(1, "x")}); return err }, "a value of another kind": func() error { _, err := def.Seal(ctx, plan, []record.Source{{"age": "thirty", "email": "x"}}) @@ -749,8 +760,8 @@ func TestGuestRefusesMalformedInputsBeforeState(t *testing.T) { _, err := def.Open(ctx, plan, []record.Sealed{{"age": {Ciphertext: []byte{0xff}}, "email": {Ciphertext: fixtureLeaf}}}) return err }, - "a term the field lacks": func() error { _, err := def.Derive(ctx, plan, "age", record.Match, uint32(1)); return err }, - "a field the plan lacks": func() error { _, err := def.Derive(ctx, plan, "name", record.Equality, "x"); return err }, + "a term the field lacks": func() error { _, err := def.derive(ctx, plan, "age", record.Match, uint32(1)); return err }, + "a field the plan lacks": func() error { _, err := def.derive(ctx, plan, "name", record.Equality, "x"); return err }, "an empty context segment": func() error { p := usersPlan(); p.Context = []string{""}; _, err := def.Seal(ctx, p, nil); return err }, "an extension the codec cannot carry": func() error { _, err := def.Extend(1.5).Seal(ctx, plan, nil); return err }, "a nil keyset selector": func() error { _, err := c.Keyset(nil).Seal(ctx, plan, nil); return err }, diff --git a/languages/golang/encrypt/internal/testpolicy/member_stash.go b/languages/golang/encrypt/internal/testpolicy/member_stash.go index de5f9ff2e..1b52480cf 100644 --- a/languages/golang/encrypt/internal/testpolicy/member_stash.go +++ b/languages/golang/encrypt/internal/testpolicy/member_stash.go @@ -157,12 +157,34 @@ func (f AgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (Enc return EncryptedMemberAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f AgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f AgeField) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f AgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f AgeField) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type EmailField struct { @@ -177,12 +199,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedMemberEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type NotesField struct { diff --git a/languages/golang/encrypt/internal/testusers/account_stash.go b/languages/golang/encrypt/internal/testusers/account_stash.go index c07695012..27cf2ba7b 100644 --- a/languages/golang/encrypt/internal/testusers/account_stash.go +++ b/languages/golang/encrypt/internal/testusers/account_stash.go @@ -143,6 +143,7 @@ func (f AccountEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v str return EncryptedAccountEmail{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f AccountEmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f AccountEmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } diff --git a/languages/golang/encrypt/internal/testusers/contact_stash.go b/languages/golang/encrypt/internal/testusers/contact_stash.go index 05869aea8..a30de7b20 100644 --- a/languages/golang/encrypt/internal/testusers/contact_stash.go +++ b/languages/golang/encrypt/internal/testusers/contact_stash.go @@ -124,9 +124,9 @@ func (f ContactEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v str return eql.TextEq(out.EQL), err } -func (f ContactEmailField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f ContactEmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } type ContactNotesField struct { diff --git a/languages/golang/encrypt/internal/testusers/kinds_stash.go b/languages/golang/encrypt/internal/testusers/kinds_stash.go index 3eb5866b2..1ed6d507d 100644 --- a/languages/golang/encrypt/internal/testusers/kinds_stash.go +++ b/languages/golang/encrypt/internal/testusers/kinds_stash.go @@ -555,12 +555,15 @@ func (f KindsSField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) ( return EncryptedKindsS{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f KindsSField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsSField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f KindsSField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f KindsSField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type KindsEField struct { @@ -575,16 +578,15 @@ func (f KindsEField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Email) (E return EncryptedKindsE{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match, Ore: out.Ore}, nil } -func (f KindsEField) Equality(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsEField) Eq(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f KindsEField) Match(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) -} - -func (f KindsEField) Ore(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f KindsEField) Matches(ctx context.Context, c *encrypt.Cipher, v Email) (encrypt.MatchQuery[Email], error) { + return gensupport.Matches(ctx, f.field, c, v) } type KindsBoField struct { @@ -608,12 +610,34 @@ func (f KindsI8Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int8) (E return EncryptedKindsI8{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f KindsI8Field) Equality(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsI8Field) Eq(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err +} + +func (f KindsI8Field) Lt(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsI8Field) Lte(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsI8Field) Gt(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err } -func (f KindsI8Field) Ore(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f KindsI8Field) Gte(ctx context.Context, c *encrypt.Cipher, v int8) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsI8Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int8) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type KindsI16Field struct { @@ -628,8 +652,29 @@ func (f KindsI16Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int16) return EncryptedKindsI16{Ciphertext: out.Ciphertext, Ope: out.Ope}, nil } -func (f KindsI16Field) Ope(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +func (f KindsI16Field) Lt(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsI16Field) Lte(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsI16Field) Gt(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsI16Field) Gte(ctx context.Context, c *encrypt.Cipher, v int16) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsI16Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int16) (encrypt.Range[encrypt.OpeTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OpeTerm]{Low: low, High: high}, err } type KindsI32Field struct { @@ -644,8 +689,9 @@ func (f KindsI32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int32) return EncryptedKindsI32{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f KindsI32Field) Equality(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsI32Field) Eq(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } type KindsIField struct { @@ -660,8 +706,29 @@ func (f KindsIField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int) (Enc return EncryptedKindsI{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil } -func (f KindsIField) Ore(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f KindsIField) Lt(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsIField) Lte(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsIField) Gt(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsIField) Gte(ctx context.Context, c *encrypt.Cipher, v int) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsIField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type KindsI64Field struct { @@ -685,8 +752,9 @@ func (f KindsU8Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint8) ( return EncryptedKindsU8{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f KindsU8Field) Equality(ctx context.Context, c *encrypt.Cipher, v uint8) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsU8Field) Eq(ctx context.Context, c *encrypt.Cipher, v uint8) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } type KindsU16Field struct { @@ -710,12 +778,34 @@ func (f KindsU32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) return EncryptedKindsU32{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f KindsU32Field) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsU32Field) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err +} + +func (f KindsU32Field) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsU32Field) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err } -func (f KindsU32Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f KindsU32Field) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsU32Field) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsU32Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type KindsUField struct { @@ -739,8 +829,29 @@ func (f KindsU64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint64) return EncryptedKindsU64{Ciphertext: out.Ciphertext, Ope: out.Ope}, nil } -func (f KindsU64Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +func (f KindsU64Field) Lt(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsU64Field) Lte(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsU64Field) Gt(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsU64Field) Gte(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OpeTerm(term), err +} + +func (f KindsU64Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint64) (encrypt.Range[encrypt.OpeTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OpeTerm]{Low: low, High: high}, err } type KindsF32Field struct { @@ -764,10 +875,6 @@ func (f KindsF64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v float64 return EncryptedKindsF64{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil } -func (f KindsF64Field) Ore(ctx context.Context, c *encrypt.Cipher, v float64) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) -} - type KindsByField struct { field gensupport.Field[[]byte] } @@ -780,8 +887,9 @@ func (f KindsByField) Encrypt(ctx context.Context, c *encrypt.Cipher, v []byte) return EncryptedKindsBy{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f KindsByField) Equality(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsByField) Eq(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } type KindsBlField struct { @@ -805,8 +913,29 @@ func (f KindsScField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Score) ( return EncryptedKindsSc{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil } -func (f KindsScField) Ore(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f KindsScField) Lt(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsScField) Lte(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsScField) Gt(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsScField) Gte(ctx context.Context, c *encrypt.Cipher, v Score) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsScField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi Score) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type KindsStField struct { @@ -821,8 +950,9 @@ func (f KindsStField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Status) return EncryptedKindsSt{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f KindsStField) Equality(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f KindsStField) Eq(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } type KindsDField struct { @@ -837,6 +967,27 @@ func (f KindsDField) Encrypt(ctx context.Context, c *encrypt.Cipher, v time.Dura return EncryptedKindsD{Ciphertext: out.Ciphertext, Ore: out.Ore}, nil } -func (f KindsDField) Ore(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f KindsDField) Lt(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsDField) Lte(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsDField) Gt(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f KindsDField) Gte(ctx context.Context, c *encrypt.Cipher, v time.Duration) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f KindsDField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi time.Duration) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } diff --git a/languages/golang/encrypt/internal/testusers/lookup_stash.go b/languages/golang/encrypt/internal/testusers/lookup_stash.go index 0ae987a04..6b47dbb05 100644 --- a/languages/golang/encrypt/internal/testusers/lookup_stash.go +++ b/languages/golang/encrypt/internal/testusers/lookup_stash.go @@ -151,8 +151,29 @@ func (f LookupScoreField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int3 return EncryptedLookupScore{Ore: out.Ore}, err } -func (f LookupScoreField) Ore(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f LookupScoreField) Lt(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f LookupScoreField) Lte(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f LookupScoreField) Gt(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f LookupScoreField) Gte(ctx context.Context, c *encrypt.Cipher, v int32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f LookupScoreField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type LookupCodeField struct { @@ -164,6 +185,7 @@ func (f LookupCodeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v Statu return EncryptedLookupCode{Equality: out.Equality}, err } -func (f LookupCodeField) Equality(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f LookupCodeField) Eq(ctx context.Context, c *encrypt.Cipher, v Status) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } diff --git a/languages/golang/encrypt/internal/testusers/memberstash_stash.go b/languages/golang/encrypt/internal/testusers/memberstash_stash.go index 286985abc..9efb5414e 100644 --- a/languages/golang/encrypt/internal/testusers/memberstash_stash.go +++ b/languages/golang/encrypt/internal/testusers/memberstash_stash.go @@ -157,12 +157,34 @@ func (f MemberAgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32 return EncryptedMemberAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f MemberAgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f MemberAgeField) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f MemberAgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f MemberAgeField) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f MemberAgeField) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f MemberAgeField) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f MemberAgeField) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f MemberAgeField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type MemberEmailField struct { @@ -177,12 +199,15 @@ func (f MemberEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v stri return EncryptedMemberEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f MemberEmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f MemberEmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f MemberEmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f MemberEmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type MemberNotesField struct { diff --git a/languages/golang/encrypt/internal/testusers/note_stash.go b/languages/golang/encrypt/internal/testusers/note_stash.go index a0a81c56e..e09124ffe 100644 --- a/languages/golang/encrypt/internal/testusers/note_stash.go +++ b/languages/golang/encrypt/internal/testusers/note_stash.go @@ -128,6 +128,7 @@ func (f NoteTextField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) return EncryptedNoteText{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f NoteTextField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f NoteTextField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } diff --git a/languages/golang/encrypt/internal/testusers/probe_stash.go b/languages/golang/encrypt/internal/testusers/probe_stash.go index 807745930..f6c10e99e 100644 --- a/languages/golang/encrypt/internal/testusers/probe_stash.go +++ b/languages/golang/encrypt/internal/testusers/probe_stash.go @@ -193,12 +193,29 @@ func (f ProbeU32Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) return EncryptedProbeU32{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil } -func (f ProbeU32Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f ProbeU32Field) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err } -func (f ProbeU32Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +func (f ProbeU32Field) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU32Field) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU32Field) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU32Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type ProbeU64Field struct { @@ -213,12 +230,29 @@ func (f ProbeU64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint64) return EncryptedProbeU64{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil } -func (f ProbeU64Field) Ore(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f ProbeU64Field) Lt(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err } -func (f ProbeU64Field) Ope(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +func (f ProbeU64Field) Lte(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU64Field) Gt(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU64Field) Gte(ctx context.Context, c *encrypt.Cipher, v uint64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeU64Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint64) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type ProbeI64Field struct { @@ -233,12 +267,29 @@ func (f ProbeI64Field) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) return EncryptedProbeI64{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil } -func (f ProbeI64Field) Ore(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f ProbeI64Field) Lt(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeI64Field) Lte(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeI64Field) Gt(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err } -func (f ProbeI64Field) Ope(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +func (f ProbeI64Field) Gte(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f ProbeI64Field) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int64) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type ProbeSField struct { @@ -253,14 +304,6 @@ func (f ProbeSField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) ( return EncryptedProbeS{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil } -func (f ProbeSField) Ore(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) -} - -func (f ProbeSField) Ope(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) -} - type ProbeBField struct { field gensupport.Field[[]byte] } @@ -272,11 +315,3 @@ func (f ProbeBField) Encrypt(ctx context.Context, c *encrypt.Cipher, v []byte) ( } return EncryptedProbeB{Ciphertext: out.Ciphertext, Ore: out.Ore, Ope: out.Ope}, nil } - -func (f ProbeBField) Ore(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) -} - -func (f ProbeBField) Ope(ctx context.Context, c *encrypt.Cipher, v []byte) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) -} diff --git a/languages/golang/encrypt/internal/testusers/profile_stash.go b/languages/golang/encrypt/internal/testusers/profile_stash.go new file mode 100644 index 000000000..f4e820947 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/profile_stash.go @@ -0,0 +1,213 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Profile prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedProfile struct { + Email EncryptedProfileEmail + Age EncryptedProfileAge + Name EncryptedProfileName +} + +type EncryptedProfileEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} + +type EncryptedProfileAge struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Ore encrypt.OreTerm +} + +type EncryptedProfileName struct { + Ciphertext encrypt.Ciphertext + Match encrypt.MatchTerm +} + +func (e EncryptedProfile) String() string { + return gensupport.Redacted("EncryptedProfile", nil, "Email", "Age", "Name") +} + +func (e EncryptedProfile) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "Email", "Age", "Name") +} + +// Stops compiling when Profile gains, loses, reorders or retypes a field. +var _ = profileShape(Profile{}) + +type profileShape struct { + _ struct{} + Email string + Age uint32 + Name string +} + +var profileDeclaration = gensupport.Declare("users"). + EncryptIndex("email", gensupport.String, encrypt.Equality). + EncryptIndex("age", gensupport.Uint32, encrypt.Equality, encrypt.Ore). + EncryptIndex("name", gensupport.String, encrypt.Match()) + +var profileCodec = gensupport.New(gensupport.Generated[Profile, EncryptedProfile]{ + TypeName: "Profile", + Declaration: profileDeclaration, + PrintsPlaintext: true, + Source: func(v Profile) gensupport.Values { + return gensupport.Values{ + "email": v.Email, + "age": v.Age, + "name": v.Name, + } + }, + Seal: func(rec gensupport.Record) (EncryptedProfile, error) { + var e EncryptedProfile + e.Email = EncryptedProfileEmail{ + Ciphertext: rec["email"].Ciphertext, + Equality: rec["email"].Equality, + } + e.Age = EncryptedProfileAge{ + Ciphertext: rec["age"].Ciphertext, + Equality: rec["age"].Equality, + Ore: rec["age"].Ore, + } + e.Name = EncryptedProfileName{ + Ciphertext: rec["name"].Ciphertext, + Match: rec["name"].Match, + } + return e, nil + }, + Open: func(e EncryptedProfile) gensupport.Record { + return gensupport.Record{ + "email": {Ciphertext: e.Email.Ciphertext, Equality: e.Email.Equality}, + "age": {Ciphertext: e.Age.Ciphertext, Equality: e.Age.Equality, Ore: e.Age.Ore}, + "name": {Ciphertext: e.Name.Ciphertext, Match: e.Name.Match}, + } + }, + Value: func(e EncryptedProfile, vals gensupport.Values) (Profile, error) { + var v Profile + var err error + if v.Email, err = gensupport.Get[string](vals, "email"); err != nil { + return Profile{}, err + } + if v.Age, err = gensupport.Get[uint32](vals, "age"); err != nil { + return Profile{}, err + } + if v.Name, err = gensupport.Get[string](vals, "name"); err != nil { + return Profile{}, err + } + return v, nil + }, +}) + +// EncryptProfile seals each Profile, with one ZeroKMS request for each 500 +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. +func EncryptProfile(ctx context.Context, cipher *encrypt.Cipher, values []Profile) ([]EncryptedProfile, error) { + return profileCodec.Encrypt(ctx, cipher, values) +} + +// DecryptProfile opens each EncryptedProfile, with one ZeroKMS request for each +// 500 sealed values, plus one the first time a keyset is used. +func DecryptProfile(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedProfile) ([]Profile, error) { + return profileCodec.Decrypt(ctx, d, encrypted) +} + +var ProfileFields = struct { + Email ProfileEmailField + Age ProfileAgeField + Name ProfileNameField +}{ + Email: ProfileEmailField{gensupport.NewField[string](profileDeclaration, "email")}, + Age: ProfileAgeField{gensupport.NewField[uint32](profileDeclaration, "age")}, + Name: ProfileNameField{gensupport.NewField[string](profileDeclaration, "name")}, +} + +type ProfileEmailField struct { + field gensupport.Field[string] +} + +func (f ProfileEmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedProfileEmail, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProfileEmail{}, err + } + return EncryptedProfileEmail{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil +} + +func (f ProfileEmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err +} + +type ProfileAgeField struct { + field gensupport.Field[uint32] +} + +func (f ProfileAgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (EncryptedProfileAge, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProfileAge{}, err + } + return EncryptedProfileAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil +} + +func (f ProfileAgeField) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err +} + +func (f ProfileAgeField) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f ProfileAgeField) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f ProfileAgeField) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f ProfileAgeField) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f ProfileAgeField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err +} + +type ProfileNameField struct { + field gensupport.Field[string] +} + +func (f ProfileNameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedProfileName, error) { + out, err := f.field.Encrypt(ctx, c, v) + if err != nil { + return EncryptedProfileName{}, err + } + return EncryptedProfileName{Ciphertext: out.Ciphertext, Match: out.Match}, nil +} + +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f ProfileNameField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) +} diff --git a/languages/golang/encrypt/internal/testusers/user_stash.go b/languages/golang/encrypt/internal/testusers/user_stash.go index 97f1cae6e..9b304cd6b 100644 --- a/languages/golang/encrypt/internal/testusers/user_stash.go +++ b/languages/golang/encrypt/internal/testusers/user_stash.go @@ -159,12 +159,34 @@ func (f AgeField) Encrypt(ctx context.Context, c *encrypt.Cipher, v uint32) (Enc return EncryptedUserAge{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f AgeField) Equality(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f AgeField) Eq(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f AgeField) Ore(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f AgeField) Lt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Lte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gt(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Gte(ctx context.Context, c *encrypt.Cipher, v uint32) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f AgeField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi uint32) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type EmailField struct { @@ -179,12 +201,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedUserEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type NotesField struct { diff --git a/languages/golang/encrypt/internal/testusers/users.go b/languages/golang/encrypt/internal/testusers/users.go index b834f0be8..c04346e3e 100644 --- a/languages/golang/encrypt/internal/testusers/users.go +++ b/languages/golang/encrypt/internal/testusers/users.go @@ -40,6 +40,19 @@ type UserRow struct { Notes encrypt.Ciphertext `stash:"notes"` } +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Profile -name Profile + +// Profile is the declaration of the Rust query fixture +// (packages/stack-encrypt/tests/fixtures/query_operators.json): the same +// context, field names, kinds and indexes, so each query its generated +// methods derive compares with the term the fixture records. +type Profile struct { + _ struct{} `stash:"context=users"` + Email string `stash:"email,encrypt,index=equality"` + Age uint32 `stash:"age,encrypt,index=equality;ore"` + Name string `stash:"name,encrypt,index=match"` +} + //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Document -name Document // Document is sealed as one value. diff --git a/languages/golang/encrypt/kinds_test.go b/languages/golang/encrypt/kinds_test.go index 56e5e77e7..065f878af 100644 --- a/languages/golang/encrypt/kinds_test.go +++ b/languages/golang/encrypt/kinds_test.go @@ -88,7 +88,7 @@ func TestEverySealedKindRoundTrips(t *testing.T) { t.Fatalf("DecryptKinds =\n%+v\nwant\n%+v", back, in) } // A defined type's field entry takes and derives at the defined type. - term, err := testusers.KindsFields.E.Equality(ctx, cipher, testusers.Email("alice@example.com")) + term, err := testusers.KindsFields.E.Eq(ctx, cipher, testusers.Email("alice@example.com")) if err != nil || !term.Equal(encrypted[0].E.Equality) { t.Fatalf("defined-type probe: %v", err) } @@ -96,7 +96,7 @@ func TestEverySealedKindRoundTrips(t *testing.T) { if err != nil || len(one.Ciphertext) == 0 { t.Fatalf("Score field: %v", err) } - if probe, err := testusers.KindsFields.St.Equality(ctx, cipher, "active"); err != nil || !probe.Equal(encrypted[0].St.Equality) { + if probe, err := testusers.KindsFields.St.Eq(ctx, cipher, "active"); err != nil || !probe.Equal(encrypted[0].St.Equality) { t.Fatalf("Status probe: %v", err) } } @@ -110,16 +110,16 @@ func TestModelRowsRoundTrip(t *testing.T) { if err != nil { t.Fatal(err) } - probe, err := testusers.Fields.Email.Equality(t.Context(), cipher, people[1].Email) + probe, err := testusers.Fields.Email.Eq(t.Context(), cipher, people[1].Email) if err != nil || !probe.Equal(rows[1].EmailEq) || probe.Equal(rows[0].EmailEq) { t.Fatalf("EmailEq does not hold the email's equality term: %v", err) } - ageProbe, err := testusers.Fields.Age.Ore(t.Context(), cipher, people[1].Age) + ageProbe, err := testusers.Fields.Age.Lt(t.Context(), cipher, people[1].Age) if err != nil || ageProbe.Compare(rows[1].AgeOre) != 0 || !rows[1].AgeOre.Less(rows[0].AgeOre) { t.Fatalf("AgeOre does not hold the age's ORE term: %v", err) } - match, err := testusers.Fields.Email.Match(t.Context(), cipher, people[1].Email) - if err != nil || !bytes.Equal(match, rows[1].EmailMatch) || len(rows[1].Notes) == 0 || rows[1].ID != people[1].ID { + match, err := testusers.Fields.Email.Matches(t.Context(), cipher, people[1].Email) + if err != nil || !bytes.Equal(match.Term(), rows[1].EmailMatch) || len(rows[1].Notes) == 0 || rows[1].ID != people[1].ID { t.Fatalf("the other columns: %v %+v", err, rows[1]) } want := append([]testusers.User(nil), people...) @@ -179,7 +179,7 @@ func TestIndexOnlyFieldsDecryptToTheirZeroValue(t *testing.T) { if !encrypted[1].Score.Ore.Less(encrypted[0].Score.Ore) || len(encrypted[0].Code.Equality) == 0 { t.Fatalf("index-only terms: %+v", encrypted) } - probe, err := testusers.LookupFields.Code.Equality(ctx, cipher, "gold") + probe, err := testusers.LookupFields.Code.Eq(ctx, cipher, "gold") if err != nil || !probe.Equal(encrypted[0].Code.Equality) { t.Fatalf("Code probe: %v", err) } diff --git a/languages/golang/encrypt/live_test.go b/languages/golang/encrypt/live_test.go index 8c0b3a406..8fc4308b0 100644 --- a/languages/golang/encrypt/live_test.go +++ b/languages/golang/encrypt/live_test.go @@ -36,7 +36,7 @@ func TestLiveRecordsAndTerms(t *testing.T) { t.Fatalf("encrypted = %+v", encrypted) } - probe, err := testusers.Fields.Age.Equality(ctx, cipher, 34) + probe, err := testusers.Fields.Age.Eq(ctx, cipher, 34) if err != nil { t.Fatal(err) } @@ -82,7 +82,7 @@ func TestLiveRecordsAndTerms(t *testing.T) { if err != nil { t.Fatal(err) } - scoped, err := testusers.Fields.Email.Equality(ctx, tenant7, "bob@example.com") + scoped, err := testusers.Fields.Email.Eq(ctx, tenant7, "bob@example.com") if err != nil { t.Fatal(err) } diff --git a/languages/golang/encrypt/locate_test.go b/languages/golang/encrypt/locate_test.go index 04413a1f9..403ef7248 100644 --- a/languages/golang/encrypt/locate_test.go +++ b/languages/golang/encrypt/locate_test.go @@ -10,7 +10,7 @@ import ( ) // A record stores match text that cuts no token as the empty filter, so the -// term-failure locator must not blame a row for it. Derive asks the query +// term-failure locator must not blame a row for it. derive asks the query // path, which refuses that text; the locator skips that refusal and goes on // to the rows after it. func TestLocateTermFailureSkipsShortMatchText(t *testing.T) { @@ -28,9 +28,9 @@ func TestLocateTermFailureSkipsShortMatchText(t *testing.T) { // The query path refuses the short email, so without the skip the // locator would name row 0 for it. - _, derr := cph.Derive(ctx, plan, "email", record.Match, "ab") + _, derr := cph.derive(ctx, plan, "email", record.Match, "ab") if !queryOnlyRefusal(record.Match, derr) { - t.Fatalf("Derive of a short email: %v, want the empty-term-text refusal", derr) + t.Fatalf("derive of a short email: %v, want the empty-term-text refusal", derr) } sentinel := fmt.Errorf("%w: from the record call", ErrTerm) diff --git a/languages/golang/encrypt/order_test.go b/languages/golang/encrypt/order_test.go index d1b9b8131..19ad9df9a 100644 --- a/languages/golang/encrypt/order_test.go +++ b/languages/golang/encrypt/order_test.go @@ -13,8 +13,11 @@ import ( "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" ) -// Property tests of term ordering: random plaintexts, terms derived by the -// guest through the generated Probe fields, comparison in Go. ORE is +// Property tests of term ordering: random plaintexts, the terms the guest +// stores for them through the generated Probe fields' Encrypt, comparison +// in Go. A query method derives the same bytes, but only for the index +// that answers its operator (ORE, where a field declares both), and only +// over integers, so the stored terms reach every index and kind. ORE is // probabilistic — a wrong comparator or a wrong derivation still agrees // with plaintext order on many pairs — so a fixed vector set says little; // hundreds of random pairs per type say more. The index key is the @@ -23,12 +26,19 @@ import ( // termOf derives one ORE or OPE term for v. type termOf[T any] func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) -func oreOf[T any](f func(context.Context, *encrypt.Cipher, T) (encrypt.OreTerm, error)) termOf[T] { - return func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) { return f(ctx, c, v) } -} - -func opeOf[T any](f func(context.Context, *encrypt.Cipher, T) (encrypt.OpeTerm, error)) termOf[T] { - return func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) { return f(ctx, c, v) } +// storedTerm is the ORE or OPE term a field's Encrypt stores for v, read +// from its output E. +func storedTerm[T, E any](seal func(context.Context, *encrypt.Cipher, T) (E, error), ope bool, ore, opeTerm func(E) []byte) termOf[T] { + return func(ctx context.Context, c *encrypt.Cipher, v T) ([]byte, error) { + out, err := seal(ctx, c, v) + if err != nil { + return nil, err + } + if ope { + return opeTerm(out), nil + } + return ore(out), nil + } } func compareTerms(ope bool, a, b []byte) int { @@ -142,34 +152,30 @@ func TestTermOrderIsPlaintextOrder(t *testing.T) { } t.Run(name, func(t *testing.T) { t.Run("uint32", func(t *testing.T) { - d := oreOf(f.U32.Ore) - if ope { - d = opeOf(f.U32.Ope) - } + d := storedTerm(f.U32.Encrypt, ope, + func(e testusers.EncryptedProbeU32) []byte { return e.Ore }, + func(e testusers.EncryptedProbeU32) []byte { return e.Ope }) orderProperty(t, cipher, ope, d, cmp.Compare[uint32]) adjacentProperty(t, cipher, ope, d, []uint32{0, 1, 2, 255, 256, 257, 65535, 65536, 1<<31 - 1, 1 << 31, 1<<32 - 2, 1<<32 - 1}, cmp.Compare[uint32]) }) t.Run("uint64", func(t *testing.T) { - d := oreOf(f.U64.Ore) - if ope { - d = opeOf(f.U64.Ope) - } + d := storedTerm(f.U64.Encrypt, ope, + func(e testusers.EncryptedProbeU64) []byte { return e.Ore }, + func(e testusers.EncryptedProbeU64) []byte { return e.Ope }) orderProperty(t, cipher, ope, d, cmp.Compare[uint64]) adjacentProperty(t, cipher, ope, d, []uint64{0, 1, 1<<32 - 1, 1 << 32, 1<<63 - 1, 1 << 63, 1<<64 - 1}, cmp.Compare[uint64]) }) t.Run("int64", func(t *testing.T) { - d := oreOf(f.I64.Ore) - if ope { - d = opeOf(f.I64.Ope) - } + d := storedTerm(f.I64.Encrypt, ope, + func(e testusers.EncryptedProbeI64) []byte { return e.Ore }, + func(e testusers.EncryptedProbeI64) []byte { return e.Ope }) orderProperty(t, cipher, ope, d, cmp.Compare[int64]) adjacentProperty(t, cipher, ope, d, []int64{-1 << 63, -1<<63 + 1, -2, -1, 0, 1, 2, 1<<63 - 1}, cmp.Compare[int64]) }) t.Run("string", func(t *testing.T) { - d := oreOf(f.S.Ore) - if ope { - d = opeOf(f.S.Ope) - } + d := storedTerm(f.S.Encrypt, ope, + func(e testusers.EncryptedProbeS) []byte { return e.Ore }, + func(e testusers.EncryptedProbeS) []byte { return e.Ope }) // Strings order by the UTF-8 bytes of their collated form; a // prefix orders before its extensions. See collatedAlphabet. orderProperty(t, cipher, ope, d, byteOrder, collatedString) @@ -182,10 +188,9 @@ func TestTermOrderIsPlaintextOrder(t *testing.T) { adjacentProperty(t, cipher, ope, d, []string{"e", "é"}, same) }) t.Run("bytes", func(t *testing.T) { - d := oreOf(f.B.Ore) - if ope { - d = opeOf(f.B.Ope) - } + d := storedTerm(f.B.Encrypt, ope, + func(e testusers.EncryptedProbeB) []byte { return e.Ore }, + func(e testusers.EncryptedProbeB) []byte { return e.Ope }) orderProperty(t, cipher, ope, d, bytes.Compare) adjacentProperty(t, cipher, ope, d, [][]byte{{}, {0}, {0, 0}, {0, 1}, {1}, {255}, {255, 0}}, bytes.Compare) }) diff --git a/languages/golang/encrypt/query.go b/languages/golang/encrypt/query.go index 21c96acfa..92fac949a 100644 --- a/languages/golang/encrypt/query.go +++ b/languages/golang/encrypt/query.go @@ -2,7 +2,10 @@ package encrypt import ( "context" + "database/sql/driver" "fmt" + "io" + "log/slog" "github.com/cipherstash/stack/languages/golang/internal/record" "github.com/cipherstash/vitaminc/bindings/go/vcffi" @@ -300,3 +303,114 @@ func (cph *Cipher) TargetLimit(ctx context.Context, plan *record.Plan, field, op return Limit{}, fmt.Errorf("%w: se_target_limit returned %T", ErrInternal, decoded) } } + +// Range is a between query on a field declared with an order index: the +// term of the low bound and the term of the high bound, both inclusive, +// derived under one context. Its answer is exact, so it has nothing to +// verify. A range binds as two parameters, one for each bound, and each +// bound implements driver.Valuer. +// +// An [OreTerm] does not order as its plaintext under plain byte order: a +// Range of them is compared in Go, with [OreTerm.Compare], not in SQL on a +// byte column. A Range of [OpeTerm] values orders under byte order, so a +// database compares it with no custom operator. +type Range[T OreTerm | OpeTerm] struct { + // Low is the low bound's term. + Low T + // High is the high bound's term. + High T +} + +// MatchQuery is a matches query on a field declared with a match index: the +// term to bind in SQL, and the two questions its probabilistic answer +// leaves, [MatchQuery.Verify] and [MatchQuery.TargetLimit]. The server +// answers a matches query with every row whose stored filter holds the +// query's bits, which can include rows that do not hold the query's text, +// so a caller decrypts the rows it fetched and verifies them. +// +// A MatchQuery keeps the text it was derived from, because Verify cuts that +// text's tokens again in the engine, with the field's own tokenizer: no +// term carries them, as they are plaintext. Go cannot wipe a string, so the +// text stays in memory until the garbage collector reclaims it, as the +// string the caller passed in does. A MatchQuery never prints it: fmt, +// slog and %#v show the field and the term's size only. +// +// T is the field's Go type. The zero MatchQuery holds no query, and its +// methods return [ErrEncoding]. +type MatchQuery[T ~string] struct { + term MatchTerm + cipher *Cipher + plan *record.Plan + field string + text T +} + +// NewMatchQuery derives the matches query for text on one field of the +// plan under this cipher's keyset, through [Cipher.QueryTerm]. For +// generated code: a program calls the field's Matches method. +func NewMatchQuery[T ~string](ctx context.Context, cph *Cipher, plan *record.Plan, field string, text T) (MatchQuery[T], error) { + if cph == nil { + return MatchQuery[T]{}, fmt.Errorf("%w: field %q: a matches query needs a cipher", ErrEncoding, field) + } + term, err := cph.QueryTerm(ctx, plan, field, "matches", string(text)) + if err != nil { + return MatchQuery[T]{}, err + } + if term.Fidelity != record.Probabilistic || term.Index != record.Match || len(term.Terms) != 1 { + return MatchQuery[T]{}, fmt.Errorf("%w: field %q: matches came back as a %s %s term of %d parts", ErrInternal, field, term.Fidelity, term.Index, len(term.Terms)) + } + return MatchQuery[T]{term: term.Terms[0], cipher: cph, plan: plan, field: field, text: text}, nil +} + +// Term is the match term to bind in SQL: the bits the query sets in the +// field's filter, which a stored filter must hold for the row to match. +func (q MatchQuery[T]) Term() MatchTerm { return q.term } + +// Value implements driver.Valuer: the term's bytes. +func (q MatchQuery[T]) Value() (driver.Value, error) { return []byte(q.term), nil } + +// Verify reports, for each of a page of decrypted values of the field, +// whether it really holds the query's text, in order: one call into the +// engine for the page, with no key. Keep the rows it reports true; the +// others are the false positives of the server's answer. +func (q MatchQuery[T]) Verify(ctx context.Context, values []T) ([]bool, error) { + if q.cipher == nil { + return nil, errNoMatchQuery + } + page := make([]any, len(values)) + for i, v := range values { + page[i] = string(v) + } + return q.cipher.Verify(ctx, q.plan, q.field, "matches", string(q.text), page) +} + +// TargetLimit is how many rows to fetch so that, at the confidence stats +// names, want of them verify: the LIMIT to send with the query. A field +// declared with tags states no expected token count, so with no measured +// [Stats.Histogram] the limit is unbounded, and a caller pages with +// [Limit.FetchOr]. Uses no key. +func (q MatchQuery[T]) TargetLimit(ctx context.Context, want uint64, stats Stats) (Limit, error) { + if q.cipher == nil { + return Limit{}, errNoMatchQuery + } + return q.cipher.TargetLimit(ctx, q.plan, q.field, "matches", q.term, want, stats) +} + +var errNoMatchQuery = fmt.Errorf("%w: the zero MatchQuery holds no query; derive one with the field's Matches", ErrEncoding) + +// String names the field and the term's size, never the text. +func (q MatchQuery[T]) String() string { + return fmt.Sprintf("encrypt.MatchQuery{field: %q, term: %d bytes}", q.field, len(q.term)) +} + +// GoString is what %#v prints: [MatchQuery.String]. +func (q MatchQuery[T]) GoString() string { return q.String() } + +// Format prints [MatchQuery.String] for every verb, so no verb reaches the +// text. +func (q MatchQuery[T]) Format(f fmt.State, _ rune) { _, _ = io.WriteString(f, q.String()) } + +// LogValue is what slog records: the field and the term's size. +func (q MatchQuery[T]) LogValue() slog.Value { + return slog.GroupValue(slog.String("field", q.field), slog.Int("term_bytes", len(q.term))) +} diff --git a/languages/golang/encrypt/query_test.go b/languages/golang/encrypt/query_test.go index 2deb0cd8f..c03d7f589 100644 --- a/languages/golang/encrypt/query_test.go +++ b/languages/golang/encrypt/query_test.go @@ -1,15 +1,21 @@ package encrypt_test import ( + "bytes" "context" + "database/sql/driver" "encoding/hex" "encoding/json" "errors" + "fmt" + "log/slog" "os" "path/filepath" + "strings" "testing" "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" "github.com/cipherstash/stack/languages/golang/internal/record" ) @@ -47,6 +53,7 @@ type queryFixture struct { TargetLimit []struct { Field string `json:"field"` Operator string `json:"operator"` + Input string `json:"input"` Term string `json:"term"` Want uint64 `json:"want"` Stats struct { @@ -351,7 +358,7 @@ func TestTheCatalogListsEachTypesOperators(t *testing.T) { for _, target := range targets { switch { case target.Name == "TextEq": - want := []record.TargetOperator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}} + want := []record.Operator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}} if len(target.Operators) != 1 || target.Operators[0] != want[0] { t.Fatalf("TextEq answers %+v, want %+v", target.Operators, want) } @@ -360,3 +367,180 @@ func TestTheCatalogListsEachTypesOperators(t *testing.T) { } } } + +// The engine lists the operators each field of a plan answers, with the +// fidelity, arity and index of each: what stashgen writes one method for. +func TestTheCheckerListsAFieldsOperators(t *testing.T) { + wasm, err := encrypt.EmbeddedGuest() + if errors.Is(err, encrypt.ErrGuestNotBuilt) { + t.Skip(err) + } + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + checker, err := encrypt.NewCheckerOver(ctx, wasm) + if err != nil { + t.Fatal(err) + } + defer func() { _ = checker.Close() }() + exact := func(name string, index record.Output) record.Operator { + return record.Operator{Name: name, Fidelity: record.Exact, Index: index} + } + for field, want := range map[string][]record.Operator{ + "email": {exact("eq", record.Equality)}, + "age": { + exact("eq", record.Equality), exact("lt", record.Ore), exact("lte", record.Ore), + exact("gt", record.Ore), exact("gte", record.Ore), + {Name: "between", Fidelity: record.Exact, Pair: true, Index: record.Ore}, + }, + "name": {{Name: "matches", Fidelity: record.Probabilistic, Index: record.Match}}, + } { + got, err := checker.Operators(ctx, queryPlan, field) + if err != nil { + t.Fatalf("%s: %v", field, err) + } + if fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("%s answers %+v, want %+v", field, got, want) + } + } + if _, err := checker.Operators(ctx, queryPlan, "phone"); !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("a field the plan lacks: %v, want ErrEncoding", err) + } +} + +// The generated query methods of Profile, whose declaration is the query +// fixture's, derive the terms the fixture records: one method per operator +// the engine answers, each returning the operator's query type. +func TestGeneratedQueriesAreTheRustFixtures(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + ctx := context.Background() + cipher := c.DefaultKeyset() + fields := testusers.ProfileFields + want := func(field, operator string) [][]byte { + t.Helper() + for _, q := range f.Queries { + if q.Field == field && q.Operator == operator { + return fixtureTerms(t, q.Term.Term) + } + } + t.Fatalf("the fixture has no %s on %s", operator, field) + return nil + } + check := func(name string, got []byte, err error, want []byte) { + t.Helper() + if err != nil || !bytes.Equal(got, want) { + t.Errorf("%s = %x, %v; want %x", name, got, err, want) + } + } + email, err := fields.Email.Eq(ctx, cipher, "bob@example.com") + check("Email.Eq", email, err, want("email", "eq")[0]) + age, err := fields.Age.Eq(ctx, cipher, 34) + check("Age.Eq", age, err, want("age", "eq")[0]) + lt, err := fields.Age.Lt(ctx, cipher, 40) + check("Age.Lt", lt, err, want("age", "lt")[0]) + lte, err := fields.Age.Lte(ctx, cipher, 40) + check("Age.Lte", lte, err, want("age", "lte")[0]) + gt, err := fields.Age.Gt(ctx, cipher, 18) + check("Age.Gt", gt, err, want("age", "gt")[0]) + gte, err := fields.Age.Gte(ctx, cipher, 18) + check("Age.Gte", gte, err, want("age", "gte")[0]) + between, err := fields.Age.Between(ctx, cipher, 18, 65) + bounds := want("age", "between") + check("Age.Between low", between.Low, err, bounds[0]) + check("Age.Between high", between.High, err, bounds[1]) + name, err := fields.Name.Matches(ctx, cipher, "alice") + check("Name.Matches", name.Term(), err, want("name", "matches")[0]) + + // Each query binds as a parameter: driver.Valuer gives the term's bytes. + for label, v := range map[string]driver.Valuer{"Eq": email, "Lt": lt, "Between.Low": between.Low, "Matches": name} { + got, err := v.Value() + if b, ok := got.([]byte); err != nil || !ok || len(b) == 0 { + t.Errorf("%s.Value() = %T %v", label, got, err) + } + } + // An inverted range is the typed path's refusal, from the engine. + if _, err := fields.Age.Between(ctx, cipher, 65, 18); !errors.Is(err, encrypt.ErrTerm) { + t.Errorf("an inverted range: %v, want ErrTerm", err) + } +} + +// A matches query verifies a page of decrypted values in one call, as the +// fixture's page verified, and sizes its fetch as the fixture's limits. +func TestAMatchesQueryVerifiesAndSizesAsTheRustFixtures(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + ctx := context.Background() + cipher := c.DefaultKeyset() + q, err := testusers.ProfileFields.Name.Matches(ctx, cipher, f.Verify.Input) + if err != nil { + t.Fatal(err) + } + // A string field has no null: the fixture's null value is left out. + var page []string + var verified []bool + for i, v := range f.Verify.Values { + if v != nil { + page = append(page, *v) + verified = append(verified, f.Verify.Verified[i]) + } + } + got, err := q.Verify(ctx, page) + if err != nil || fmt.Sprint(got) != fmt.Sprint(verified) { + t.Fatalf("Verify(%q) = %v, %v; want %v", page, got, err, verified) + } + if got, err := q.Verify(ctx, nil); err != nil || len(got) != 0 { + t.Fatalf("an empty page: %v, %v", got, err) + } + for i, l := range f.TargetLimit { + q, err := testusers.ProfileFields.Name.Matches(ctx, cipher, l.Input) + if err != nil { + t.Fatal(err) + } + if hex.EncodeToString(q.Term()) != l.Term { + t.Fatalf("case %d: Matches(%q) = %x, the fixture's term is %s", i, l.Input, q.Term(), l.Term) + } + stats := encrypt.Stats{Rows: l.Stats.Rows, Confidence: l.Stats.Confidence} + for _, b := range l.Stats.Histogram { + stats.Histogram = append(stats.Histogram, encrypt.Bucket{Bits: uint32(b[0]), Rows: b[1]}) + } + limit, err := q.TargetLimit(ctx, l.Want, stats) + if err != nil { + t.Fatalf("case %d: %v", i, err) + } + rows, bounded := limit.Rows() + if (l.Limit == nil) == bounded || (l.Limit != nil && rows != *l.Limit) { + t.Fatalf("case %d: %d (bounded %v), want %v", i, rows, bounded, l.Limit) + } + } + var zero encrypt.MatchQuery[string] + if _, err := zero.Verify(ctx, page); !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("the zero MatchQuery verifies: %v", err) + } + if _, err := zero.TargetLimit(ctx, 10, encrypt.Stats{Rows: 1}); !errors.Is(err, encrypt.ErrEncoding) { + t.Fatalf("the zero MatchQuery sizes a fetch: %v", err) + } +} + +// A matches query holds its text for Verify and never prints it: not +// through fmt with any verb, not through slog. +func TestAMatchesQueryNeverPrintsItsText(t *testing.T) { + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + const text = "zebrafish" + q, err := testusers.ProfileFields.Name.Matches(context.Background(), c.DefaultKeyset(), text) + if err != nil { + t.Fatal(err) + } + var out bytes.Buffer + fmt.Fprintf(&out, "%v %+v %#v %s %q %x %d", q, q, q, q, q, q, q) + slog.New(slog.NewTextHandler(&out, nil)).Info("query", "q", q) + slog.New(slog.NewJSONHandler(&out, nil)).Info("query", "q", q) + if strings.Contains(out.String(), text) || strings.Contains(out.String(), hex.EncodeToString([]byte(text))) { + t.Fatalf("the text was printed: %s", out.String()) + } + if !strings.Contains(out.String(), `field: "name"`) { + t.Fatalf("the field is not named: %s", out.String()) + } +} diff --git a/languages/golang/encrypt/records.go b/languages/golang/encrypt/records.go index a12b1cf4c..7a4748bbc 100644 --- a/languages/golang/encrypt/records.go +++ b/languages/golang/encrypt/records.go @@ -69,7 +69,7 @@ func (cph *Cipher) Seal(ctx context.Context, plan *record.Plan, rows []record.So return inst.call(ctx, inst.encryptRecord, buf(encodedSource), buf(encodedPlan), buf(opts)) }) if errors.Is(err, ErrTerm) { - // The plan as given: Derive applies the extension itself. + // The plan as given: derive applies the extension itself. return nil, cph.locateTermFailure(ctx, plan, rows, err) } if err != nil { @@ -111,13 +111,15 @@ func (cph *Cipher) Open(ctx context.Context, plan *record.Plan, records []record return cph.client.open(ctx, cph.keyset, cph.context, p, records) } -// Derive derives one index term for one value of a field, under the plan's -// context for that field and this cipher's extension: the term a query -// compares against the stored one. A plan that takes its context from a -// field has none of its own, so the term derives under the context named -// with [Cipher.Context], and a cipher that names none is refused as -// [ErrEncoding]. For generated code. -func (cph *Cipher) Derive(ctx context.Context, plan *record.Plan, field string, output record.Output, value any) ([]byte, error) { +// derive derives one index term for one value of a field, under the plan's +// context for that field and this cipher's extension: the term the record +// call stores for it. It is how locateTermFailure names the field +// behind a record's term failure; a query asks [Cipher.QueryTerm], which +// names an operator. A plan that takes its context from a field has none of +// its own, so the term derives under the context named with +// [Cipher.Context], and a cipher that names none is refused as +// [ErrEncoding]. +func (cph *Cipher) derive(ctx context.Context, plan *record.Plan, field string, output record.Output, value any) ([]byte, error) { p, err := cph.queryPlan(plan, field) if err != nil { return nil, err @@ -185,42 +187,6 @@ func (cph *Cipher) queryPlan(plan *record.Plan, field string) (*record.Plan, err return &under, nil } -// Query derives the EQL query value of a field that names an EQL type, for -// one value: the operand that matches stored values of the field, as the -// JSON bytes the field's eql_v3.query_* domain takes. The engine runs the -// EQL type's own query plan; no data key is minted. For generated code. -func (cph *Cipher) Query(ctx context.Context, plan *record.Plan, field string, value any) ([]byte, error) { - p, err := cph.plan(plan) - if err != nil { - return nil, err - } - f := p.Field(field) - if f == nil { - return nil, fmt.Errorf("%w: the plan has no field %q", ErrEncoding, field) - } - if !f.IsTarget() { - return nil, fmt.Errorf("%w: field %q names no EQL type; its terms are derived by index", ErrEncoding, field) - } - encodedValue, err := vcffi.Marshal(value) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrEncoding, err) - } - // The probe value is plaintext: its transport copy is wiped once it is - // in the guest. - defer wipe(encodedValue) - encodedPlan, err := vcffi.Marshal(p.Wire()) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrEncoding, err) - } - opts, err := vcffi.Marshal(options(cph.keyset, "")) - if err != nil { - return nil, err - } - return cph.client.call(ctx, func(inst *instance) ([]byte, error) { - return inst.call(ctx, inst.query, buf(encodedValue), buf(encodedPlan), buf([]byte(field)), buf(opts)) - }) -} - // Open decrypts records sealed under the plan by any keyset of this client: // each record is opened under the keyset that sealed it, with one ZeroKMS // request for each 500 sealed values from each keyset. For generated code. @@ -486,10 +452,10 @@ func (c *Client) open(ctx context.Context, sel KeysetSelector, context string, p // term at a time, which costs no key request: a term derives locally. A // stored value is never refused for its text (text with no token is stored // as the empty filter), so a term failure on a record is the scheme refusing -// the value's kind under that index. Derive asks the query path, which does +// the value's kind under that index. derive asks the query path, which does // refuse text with no token, so that one refusal is skipped here: it cannot // be why the record failed. p is the plan without this cipher's extension, -// which Derive adds. +// which derive adds. func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows []record.Source, err error) error { for i, row := range rows { for _, f := range p.Fields { @@ -497,7 +463,7 @@ func (cph *Cipher) locateTermFailure(ctx context.Context, p *record.Plan, rows [ if !o.IsTerm() { continue } - if _, derr := cph.Derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) && !queryOnlyRefusal(o, derr) { + if _, derr := cph.derive(ctx, p, f.Name, o, row[f.Name]); errors.Is(derr, ErrTerm) && !queryOnlyRefusal(o, derr) { return fmt.Errorf("%w: value %d, field %q: the engine derives no %s term for this value", err, i, f.Name, indexWord(o)) } } diff --git a/languages/golang/encrypt/roundtrip_test.go b/languages/golang/encrypt/roundtrip_test.go index c1658108b..2fcfb7914 100644 --- a/languages/golang/encrypt/roundtrip_test.go +++ b/languages/golang/encrypt/roundtrip_test.go @@ -93,15 +93,15 @@ func TestEncryptDecryptRoundTrip(t *testing.T) { if !one.Equality.Equal(encrypted[1].Email.Equality) || !bytes.Equal(one.Match, encrypted[1].Email.Match) { t.Error("a field's own Encrypt derives other terms than the record's") } - probe, err := testusers.Fields.Email.Equality(ctx, cipher, "bob@example.com") + probe, err := testusers.Fields.Email.Eq(ctx, cipher, "bob@example.com") if err != nil { t.Fatal(err) } if !probe.Equal(encrypted[1].Email.Equality) || probe.Equal(encrypted[0].Email.Equality) { t.Error("the probe does not single out its row") } - match, err := testusers.Fields.Email.Match(ctx, cipher, "bob@example.com") - if err != nil || !bytes.Equal(match, encrypted[1].Email.Match) { + match, err := testusers.Fields.Email.Matches(ctx, cipher, "bob@example.com") + if err != nil || !bytes.Equal(match.Term(), encrypted[1].Email.Match) { t.Errorf("match probe: %v", err) } // A column update: one field's ciphertext opens in a row. @@ -156,7 +156,7 @@ func TestExtendIsPartOfTheIdentity(t *testing.T) { t.Fatalf("with its extension: %v %+v", err, back) } // The probe takes the same cipher. - probe, err := testusers.Fields.Email.Equality(ctx, tenant7, "bob@example.com") + probe, err := testusers.Fields.Email.Eq(ctx, tenant7, "bob@example.com") if err != nil || !probe.Equal(ext[1].Email.Equality) || probe.Equal(plain[1].Email.Equality) { t.Fatalf("tenant probe: %v", err) } @@ -293,11 +293,11 @@ func TestAMatchIndexStoresShortText(t *testing.T) { } // No query matches the empty filter: a query has at least one // position, and the empty filter holds none of them. - q, err := testusers.Fields.Email.Match(ctx, cipher, "example") + q, err := testusers.Fields.Email.Matches(ctx, cipher, "example") if err != nil { t.Fatal(err) } - if positions, _ := q.Positions(); len(positions) == 0 { + if positions, _ := q.Term().Positions(); len(positions) == 0 { t.Fatalf("%s: a query with a token sets no bit", name) } } @@ -307,8 +307,8 @@ func TestAMatchIndexStoresShortText(t *testing.T) { } // A match query with no token is the one value the engine refuses. for _, text := range []string{"", " \t", "ab"} { - if _, err := testusers.Fields.Email.Match(ctx, cipher, text); !errors.Is(err, encrypt.ErrTerm) { - t.Fatalf("Fields.Email.Match(%q): %v, want ErrTerm", text, err) + if _, err := testusers.Fields.Email.Matches(ctx, cipher, text); !errors.Is(err, encrypt.ErrTerm) { + t.Fatalf("Fields.Email.Matches(%q): %v, want ErrTerm", text, err) } } // Zero values the engine does seal. diff --git a/languages/golang/encrypt/targets_test.go b/languages/golang/encrypt/targets_test.go index 24b0ce9dd..aa03b3656 100644 --- a/languages/golang/encrypt/targets_test.go +++ b/languages/golang/encrypt/targets_test.go @@ -39,9 +39,9 @@ func TestParseTargetsReadsEachEntry(t *testing.T) { } want := []record.Target{ {Name: "TextEq", Family: "Text", Suffix: "Eq", Plaintext: record.String, SQLDomain: "eql_v3_text_eq", Indexes: []record.Output{record.Equality}, Query: "TextEqQuery", QuerySQLDomain: "eql_v3.query_text_eq", Producible: true, - Operators: []record.TargetOperator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}}}, - {Name: "JSON", Family: "JSON", Plaintext: record.Untyped, SQLDomain: "eql_v3_json", Indexes: []record.Output{"json"}, Reason: "the JSON index is a new operation in the engine", Operators: []record.TargetOperator{}}, - {Name: "IntegerOrdOre", Plaintext: record.Int64, Indexes: []record.Output{record.Ore}, Producible: true, Operators: []record.TargetOperator{ + Operators: []record.Operator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}}}, + {Name: "JSON", Family: "JSON", Plaintext: record.Untyped, SQLDomain: "eql_v3_json", Indexes: []record.Output{"json"}, Reason: "the JSON index is a new operation in the engine", Operators: []record.Operator{}}, + {Name: "IntegerOrdOre", Plaintext: record.Int64, Indexes: []record.Output{record.Ore}, Producible: true, Operators: []record.Operator{ {Name: "lt", Fidelity: record.Exact, Index: record.Ore}, {Name: "between", Fidelity: record.Exact, Pair: true, Index: record.Ore}, {Name: "matches", Fidelity: record.Probabilistic, Index: record.Match}, diff --git a/languages/golang/internal/record/record.go b/languages/golang/internal/record/record.go index 70ad1582c..79ce40b8d 100644 --- a/languages/golang/internal/record/record.go +++ b/languages/golang/internal/record/record.go @@ -443,7 +443,7 @@ type Target struct { Reason string // Operators are the operators the engine answers for a field that names // the type, in the engine's order; none for a type it does not produce. - Operators []TargetOperator + Operators []Operator } // Fidelity says how far the server's answer to an operator can be trusted: @@ -460,9 +460,10 @@ const ( Probabilistic Fidelity = "probabilistic" ) -// TargetOperator is one operator the engine answers for an EQL type, as -// se_targets lists it. -type TargetOperator struct { +// Operator is one operator the engine answers for a field: for an EQL type, +// as se_targets lists it, and for any field of a plan, as se_operators +// lists it. +type Operator struct { // Name is the operator's name, as a query names it: eq, lt, lte, gt, // gte, between or matches. Name string @@ -471,6 +472,6 @@ type TargetOperator struct { // Pair says the operator takes two values, low then high, as between // does, where every other takes one. Pair bool - // Index is the type's index that answers it. + // Index is the index that answers it: the field's, or its EQL type's. Index Output } diff --git a/languages/golang/stashgen/emit.go b/languages/golang/stashgen/emit.go index f72ea02b3..dd3071062 100644 --- a/languages/golang/stashgen/emit.go +++ b/languages/golang/stashgen/emit.go @@ -20,6 +20,9 @@ func emit(f *genFile) ([]byte, error) { } } } + if err := planQueries(f); err != nil { + return nil, err + } w := &writer{} w.header(f) w.encryptedType(f) @@ -554,23 +557,105 @@ func (w *writer) fieldEntries(f *genFile) { w.p("\treturn %s{%s}, nil", g.outputType, strings.Join(parts, ", ")) } w.p("}") - if g.queryType != "" { + for _, q := range g.queries { w.nl() - w.p("func (f %s) Query(ctx context.Context, c *encrypt.Cipher, v %s) (%s, error) {", g.fieldType, g.typeExpr, g.queryType) - w.p("\tout, err := f.field.Query(ctx, c, v)") - w.p("\treturn %s(out.EQL), err", g.queryType) - w.p("}") + w.query(g, q) } - for _, o := range g.outputs { - if o.index == "" { - continue + } +} + +// queryMethod is one generated method for an operator the engine answers on +// a field. +type queryMethod struct { + method string // "Eq", "Between", "Matches" + operator Operator + resultType string // "encrypt.EqualityTerm", "encrypt.Range[encrypt.OreTerm]", "encrypt.MatchQuery[string]" + termType string // for a range, the bound's type: "encrypt.OreTerm" +} + +// planQueries picks each sealed field's query methods from the operators the +// engine answers on it: one method for each, named for the operator, whose +// result is the Go type of what the operator's term is. An operator the SDK +// has no Go type for stops the generator, naming the field, rather than +// leaving the field without a query the engine answers. +func planQueries(f *genFile) error { + for i := range f.fields { + g := &f.fields[i] + g.queries = nil + for _, op := range g.operators { + q, err := queryFor(g, op) + if err != nil { + return fieldErr(f.typeName, g.GoName, "%v", err) } - w.nl() - w.p("func (f %s) %s(ctx context.Context, c *encrypt.Cipher, v %s) (%s, error) {", g.fieldType, o.name, g.typeExpr, o.typeExpr) - w.p("\treturn f.field.%s(ctx, c, v)", o.name) - w.p("}") + g.queries = append(g.queries, q) + } + } + return nil +} + +func queryFor(g *genField, op Operator) (queryMethod, error) { + q := queryMethod{method: operatorMethod(op.Name), operator: op} + if !isIdent(q.method) || q.method == "Encrypt" { + return q, fmt.Errorf("the engine answers the operator %q, which has no Go method name", op.Name) + } + if g.Verb == VerbEncryptInto { + // The term of every operator on an EQL type is the type's query + // value. + switch { + case g.queryType == "": + return q, fmt.Errorf("the engine answers %s on %s, which has no query type", op.Name, g.EQLType) + case op.Probabilistic: + return q, fmt.Errorf("the engine answers %s on %s with a probabilistic term, and the SDK has no EQL query type that verifies yet", op.Name, g.EQLType) + case op.Pair: + return q, fmt.Errorf("the engine answers %s on %s with two values, and an EQL query value is one", op.Name, g.EQLType) } + q.resultType = g.queryType + return q, nil } + termType := map[IndexName]string{IndexEquality: "encrypt.EqualityTerm", IndexOre: "encrypt.OreTerm", IndexOpe: "encrypt.OpeTerm"}[op.Index] + switch { + case op.Probabilistic && op.Index == IndexMatch && !op.Pair && g.GoType.Kind == KindString: + q.resultType = "encrypt.MatchQuery[" + g.typeExpr + "]" + case op.Probabilistic || termType == "": + return q, fmt.Errorf("the engine answers %s through the %s index, and the SDK has no Go type for that query", op.Name, op.Index) + case op.Pair && op.Index != IndexEquality: + q.resultType = "encrypt.Range[" + termType + "]" + q.termType = termType + case op.Pair: + return q, fmt.Errorf("the engine answers %s with two values through the %s index, and the SDK has no Go type for that range", op.Name, op.Index) + default: + q.resultType = termType + } + return q, nil +} + +// operatorMethod is the Go method for an operator: its name with the first +// letter upper case, eq -> Eq, between -> Between. +func operatorMethod(name string) string { + if name == "" { + return "" + } + return strings.ToUpper(name[:1]) + name[1:] +} + +// query writes one operator's method on a field's entry. +func (w *writer) query(g genField, q queryMethod) { + recv := fmt.Sprintf("func (f %s) %s(ctx context.Context, c *encrypt.Cipher, ", g.fieldType, q.method) + switch { + case q.operator.Probabilistic: + w.b.WriteString(wrapComment(fmt.Sprintf("%s derives the %s query for v. Its answer can hold rows that do not match: decrypt the rows it fetches and keep those Verify reports true.", q.method, q.operator.Name), 80)) + w.p("%sv %s) (%s, error) {", recv, g.typeExpr, q.resultType) + w.p(" return gensupport.Matches(ctx, f.field, c, v)") + case q.operator.Pair: + w.p("%slo, hi %s) (%s, error) {", recv, g.typeExpr, q.resultType) + w.p(" low, high, err := f.field.Range(ctx, c, %q, lo, hi)", q.operator.Name) + w.p(" return %s{Low: low, High: high}, err", q.resultType) + default: + w.p("%sv %s) (%s, error) {", recv, g.typeExpr, q.resultType) + w.p(" term, err := f.field.Term(ctx, c, %q, v)", q.operator.Name) + w.p(" return %s(term), err", q.resultType) + } + w.p("}") } func (w *writer) model(f *genFile, m *genModel) { diff --git a/languages/golang/stashgen/engine.go b/languages/golang/stashgen/engine.go index baa20710f..040af40fc 100644 --- a/languages/golang/stashgen/engine.go +++ b/languages/golang/stashgen/engine.go @@ -11,10 +11,12 @@ import ( // Engine answers the generator's questions about what the Rust engine can do. // The generator holds no copy of the engine's rules: every refusal about an -// index, an EQL type or a field type comes from here. +// index, an EQL type or a field type comes from here, and so does every +// operator a field's query methods name. // // [GuestEngine] is the SDK's implementation: the WASI guest that package -// encrypt embeds, asked through se_plan_check and se_targets. +// encrypt embeds, asked through se_plan_check, se_targets and +// se_operators. type Engine interface { // EQLTypes lists the EQL types the engine can produce: each name, its // plaintext kind, its indexes and its query type. @@ -24,6 +26,29 @@ type Engine interface { // seal, an EQL type it cannot produce, or an index option it cannot // carry. The error names the field. Check(ctx context.Context, d Declaration) error + // Operators lists the operators the engine answers on each sealed field + // of a declaration it accepts, by the field's declared name, in the + // engine's order: the generator writes one query method for each. + Operators(ctx context.Context, d Declaration) (map[string][]Operator, error) +} + +// Operator is one operator the engine answers on a field, such as eq or +// matches: the generated method's name, its input and its result follow +// from it. +type Operator struct { + // Name is the operator's name, as a query names it: eq, lt, lte, gt, + // gte, between or matches. + Name string + // Probabilistic says the server's answer can hold false positives, so + // the query's type carries Verify and TargetLimit; an exact answer's + // carries neither. + Probabilistic bool + // Pair says the operator takes two values, low then high, as between + // does. + Pair bool + // Index is the index that answers it: the field's own, or its EQL + // type's. + Index IndexName } // EQLType is one EQL type the engine produces, such as TextEq. @@ -113,6 +138,41 @@ func (e *guestEngine) Check(ctx context.Context, d Declaration) error { return nil } +// Operators asks the guest, one sealed field at a time, which operators the +// engine answers on it (se_operators). +func (e *guestEngine) Operators(ctx context.Context, d Declaration) (map[string][]Operator, error) { + if d.Opaque { + // An opaque struct is one ciphertext with no index: no query. + return map[string][]Operator{}, nil + } + eqlTypes, err := e.EQLTypes(ctx) + if err != nil { + return nil, err + } + plan, err := lowerDeclaration(d, eqlTypes) + if err != nil { + return nil, err + } + out := make(map[string][]Operator, len(plan.Fields)) + for _, f := range plan.Fields { + one := &record.Plan{Context: plan.Context, ContextField: plan.ContextField, Fields: []record.Field{f}} + answered, err := e.checker.Operators(ctx, one, f.Name) + if err != nil { + return nil, &FieldError{Type: d.Type, Field: d.field(f.Name).GoName, Reason: fmt.Sprintf("the engine's operators: %v", err)} + } + ops := make([]Operator, 0, len(answered)) + for _, o := range answered { + index, ok := indexOfOutput[o.Index] + if !ok { + return nil, &FieldError{Type: d.Type, Field: d.field(f.Name).GoName, Reason: fmt.Sprintf("the engine answers %s through the index %q, which the generator does not know", o.Name, o.Index)} + } + ops = append(ops, Operator{Name: o.Name, Probabilistic: o.Fidelity == record.Probabilistic, Pair: o.Pair, Index: index}) + } + out[f.Name] = ops + } + return out, nil +} + // lowerDeclaration is the generator's lowering of a declaration to the plan // the engine reads: the same shape gensupport lowers the generated // declaration to. Passthrough and omitted fields stay on the host. diff --git a/languages/golang/stashgen/enginetest/enginetest.go b/languages/golang/stashgen/enginetest/enginetest.go index efb759584..1d8df0030 100644 --- a/languages/golang/stashgen/enginetest/enginetest.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -1,6 +1,7 @@ // Package enginetest is a static stand-in for the Rust engine in the // generator's tests. It knows what the engine produces today: the EQL type -// TextEq, and the four indexes with the Go kinds each applies to. +// TextEq, the four indexes with the Go kinds each applies to, and the +// operators each answers. // // It is a test double, not a copy of the rules the SDK ships: the SDK's // generator asks the embedded guest. When the engine learns a type or an @@ -64,6 +65,80 @@ func (e Static) Check(ctx context.Context, d stashgen.Declaration) error { return nil } +// operatorNames are the operators, in the engine's order (OpSpec::ALL). +var operatorNames = []string{"eq", "lt", "lte", "gt", "gte", "between", "matches"} + +// Operators applies the engine's table as it stands today: OpSpec::answered_by +// for a field with indexes, and for an EQL type the exact operators of one +// value its indexes answer. +func (e Static) Operators(ctx context.Context, d stashgen.Declaration) (map[string][]stashgen.Operator, error) { + out := map[string][]stashgen.Operator{} + if d.Opaque { + return out, nil + } + eqlTypes, _ := e.EQLTypes(ctx) + for _, f := range d.Fields { + if !f.Sealed() { + continue + } + indexes := make([]stashgen.IndexName, 0, len(f.Indexes)) + for _, idx := range f.Indexes { + indexes = append(indexes, idx.Name) + } + target := false + if f.Verb == stashgen.VerbEncryptInto { + target = true + indexes = nil + for _, t := range eqlTypes { + if t.Name == f.EQLType { + indexes = t.Indexes + } + } + } + ops := []stashgen.Operator{} + for _, name := range operatorNames { + op := stashgen.Operator{Name: name, Probabilistic: name == "matches", Pair: name == "between"} + if target && (op.Probabilistic || op.Pair) { + continue + } + if index, ok := answeredBy(name, f.GoType, indexes); ok { + op.Index = index + ops = append(ops, op) + } + } + out[f.Name] = ops + } + return out, nil +} + +// answeredBy mirrors OpSpec::answered_by in stack-encrypt: eq through an +// equality index, matches through a match index, and the orderings through +// ORE, or OPE when there is no ORE, over an integer alone; each only where +// the index applies to the type. +func answeredBy(operator string, t stashgen.GoType, indexes []stashgen.IndexName) (stashgen.IndexName, bool) { + declared := func(want stashgen.IndexName) bool { + for _, idx := range indexes { + if idx == want { + return indexApplies(want, t) == nil + } + } + return false + } + switch operator { + case "eq": + return stashgen.IndexEquality, declared(stashgen.IndexEquality) + case "matches": + return stashgen.IndexMatch, declared(stashgen.IndexMatch) + } + if t.Kind != stashgen.KindInt && t.Kind != stashgen.KindUint { + return "", false + } + if declared(stashgen.IndexOre) { + return stashgen.IndexOre, true + } + return stashgen.IndexOpe, declared(stashgen.IndexOpe) +} + // indexApplies mirrors dynamic::admits in stack-encrypt: equality over // integers, floats, text and bytes (the kinds vitaminc gives an equality // domain; a boolean has none), match over text alone, ORE and OPE over diff --git a/languages/golang/stashgen/generate.go b/languages/golang/stashgen/generate.go index ae584b43d..f4ee2726c 100644 --- a/languages/golang/stashgen/generate.go +++ b/languages/golang/stashgen/generate.go @@ -98,6 +98,9 @@ func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { if err := engine.Check(ctx, gf.decl); err != nil { return nil, err } + if err := gf.askOperators(ctx, engine); err != nil { + return nil, err + } src, err := emit(gf) if err != nil { return nil, err @@ -108,6 +111,21 @@ func FromTags(ctx context.Context, engine Engine, req Request) (*File, error) { return &File{Path: outPath, Content: src, Notices: gf.stderrNotices()}, nil } +// askOperators asks the engine which operators each sealed field answers, +// for the field's query methods. +func (f *genFile) askOperators(ctx context.Context, engine Engine) error { + operators, err := engine.Operators(ctx, f.decl) + if err != nil { + return err + } + for i := range f.fields { + if f.fields[i].Sealed() { + f.fields[i].operators = operators[f.fields[i].Name] + } + } + return nil +} + // checkNamesFree refuses a file that declares a package-level name the // package already declares: written, it would not compile, and the compiler // would point at the generated file rather than at the clash. The previous diff --git a/languages/golang/stashgen/policy.go b/languages/golang/stashgen/policy.go index e75452766..2cf724e23 100644 --- a/languages/golang/stashgen/policy.go +++ b/languages/golang/stashgen/policy.go @@ -176,6 +176,9 @@ func generateFor(ctx context.Context, cfg generateConfig, source policy.Source, if err := cfg.engine.Check(ctx, gf.decl); err != nil { return err } + if err := gf.askOperators(ctx, cfg.engine); err != nil { + return err + } src, err := emit(gf) if err != nil { return err diff --git a/languages/golang/stashgen/read.go b/languages/golang/stashgen/read.go index 6839e4a96..b3cf39a57 100644 --- a/languages/golang/stashgen/read.go +++ b/languages/golang/stashgen/read.go @@ -88,14 +88,16 @@ type encMember struct { // genField is one stored field with what the emitter writes for it. type genField struct { Field - typeExpr string // the field's Go type - via string // the embedded passthrough struct it is promoted from, or "" - tags string // the other libraries' tags, without stash - outputType string // the encrypted type's field type: "eql.TextEq" or "EncryptedContactEmail" - outputs []output // the outputs of a sealed field in separate columns - queryType string // "eql.TextEqQuery", or "" when the field has no query - fieldType string // "EmailField" - pathType string // the type with its package path, for model checks + typeExpr string // the field's Go type + via string // the embedded passthrough struct it is promoted from, or "" + tags string // the other libraries' tags, without stash + outputType string // the encrypted type's field type: "eql.TextEq" or "EncryptedContactEmail" + outputs []output // the outputs of a sealed field in separate columns + queryType string // "eql.TextEqQuery", or "" when the field has no query + operators []Operator // the operators the engine answers on the field + queries []queryMethod // the field's query methods, one for each operator + fieldType string // "EmailField" + pathType string // the type with its package path, for model checks } // definedScalar reports whether the field's type is a named type over a diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index 3b5bcba0d..5228ac5eb 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -283,18 +283,26 @@ func TestRefusalsHoldAgainstTheEmbeddedEngine(t *testing.T) { } }) } - // What the fake refuses, the engine refuses: the same cases against both. + // What the fake refuses, the engine refuses, and what both accept they + // write the same file for: the same query methods, from the same + // operators. The same cases against both. for _, c := range []struct{ name, line string }{ {"equality on a bool", "\tDone bool `stash:\"done,encrypt,index=equality\"`"}, {"order on a bool", "\tDone bool `stash:\"done,encrypt,index=ore\"`"}, {"equality on a float", "\tScore float64 `stash:\"score,encrypt,index=equality\"`"}, {"a declaration the engine runs", "\tAge uint32 `stash:\"age,encrypt,index=equality;ore\"`\n\tName string `stash:\"name,encrypt,index=match\"`"}, + {"every index on an integer", "\tScore int64 `stash:\"score,encrypt,index=equality;ore;ope\"`\n\tRank int32 `stash:\"rank,index=ope\"`"}, + {"orders on text, bytes and a float", "\tName string `stash:\"name,encrypt,index=ore;ope;match\"`\n\tBlob []byte `stash:\"blob,encrypt,index=equality;ore\"`\n\tScore float64 `stash:\"score,encrypt,index=ore\"`"}, } { dir := writeModule(t, map[string]string{"model.go": user(ctx + c.line), "crm/contact.go": crmPackage}) - _, real := stashgen.FromTags(context.Background(), engine, stashgen.Request{Dir: dir, Type: "User"}) - _, fake := stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: dir, Type: "User"}) + realFile, real := stashgen.FromTags(context.Background(), engine, stashgen.Request{Dir: dir, Type: "User"}) + fakeFile, fake := stashgen.FromTags(context.Background(), enginetest.Static{}, stashgen.Request{Dir: dir, Type: "User"}) if (real == nil) != (fake == nil) { t.Errorf("%s: the engine says %v, the fake says %v", c.name, real, fake) + continue + } + if real == nil && string(realFile.Content) != string(fakeFile.Content) { + t.Errorf("%s: the engine and the fake write different files:\n%s", c.name, diff(string(realFile.Content), string(fakeFile.Content))) } } } diff --git a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden index b63fb9cad..502a2e3e9 100644 --- a/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/accounts/account_stash.go.golden @@ -165,7 +165,7 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (e return eql.TextEq(out.EQL), err } -func (f EmailField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } diff --git a/languages/golang/stashgen/testdata/cases/contacts/contacts.go b/languages/golang/stashgen/testdata/cases/contacts/contacts.go index 0bf3717d3..2f43be0b4 100644 --- a/languages/golang/stashgen/testdata/cases/contacts/contacts.go +++ b/languages/golang/stashgen/testdata/cases/contacts/contacts.go @@ -66,7 +66,7 @@ func CreateWithGORM(ctx context.Context, db *gorm.DB, cipher *encrypt.Cipher, li } func IDByPhone(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, phone string) (int64, error) { - term, err := Fields.PhoneNumber.Equality(ctx, cipher, phone) + term, err := Fields.PhoneNumber.Eq(ctx, cipher, phone) if err != nil { return 0, err } diff --git a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden index 00d723533..366279cfe 100644 --- a/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/contacts/contactstash_stash.go.golden @@ -142,12 +142,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedContactEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type PhoneNumberField struct { @@ -162,8 +165,9 @@ func (f PhoneNumberField) Encrypt(ctx context.Context, c *encrypt.Cipher, v stri return EncryptedContactPhoneNumber{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f PhoneNumberField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f PhoneNumberField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } // The -model flag: ContactRow converts to and from its shape only while the diff --git a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden index 1adbd28a5..88f9c4999 100644 --- a/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/embedded/patient_stash.go.golden @@ -155,9 +155,9 @@ func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eq return eql.TextEq(out.EQL), err } -func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f NameField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } type EmailField struct { @@ -172,8 +172,9 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedPatientEmail{Ciphertext: out.Ciphertext, Equality: out.Equality}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } type ChartField struct { diff --git a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden index b6dd34a2f..b76f15a9d 100644 --- a/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/foreign/individualstash_stash.go.golden @@ -164,12 +164,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedIndividualEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type MedicareNoField struct { @@ -181,7 +184,7 @@ func (f MedicareNoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v strin return eql.TextEq(out.EQL), err } -func (f MedicareNoField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f MedicareNoField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } diff --git a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden index 519b731c2..3e4137ec7 100644 --- a/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden @@ -157,12 +157,34 @@ func (f OrderAmountField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int6 return EncryptedOrderAmount{Ciphertext: out.Ciphertext, Equality: out.Equality, Ore: out.Ore}, nil } -func (f OrderAmountField) Equality(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f OrderAmountField) Eq(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f OrderAmountField) Ore(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { - return f.field.Ore(ctx, c, v) +func (f OrderAmountField) Lt(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return encrypt.OreTerm(term), err +} + +func (f OrderAmountField) Lte(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return encrypt.OreTerm(term), err +} + +func (f OrderAmountField) Gt(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return encrypt.OreTerm(term), err +} + +func (f OrderAmountField) Gte(ctx context.Context, c *encrypt.Cipher, v int64) (encrypt.OreTerm, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return encrypt.OreTerm(term), err +} + +func (f OrderAmountField) Between(ctx context.Context, c *encrypt.Cipher, lo, hi int64) (encrypt.Range[encrypt.OreTerm], error) { + low, high, err := f.field.Range(ctx, c, "between", lo, hi) + return encrypt.Range[encrypt.OreTerm]{Low: low, High: high}, err } type OrderNoteField struct { @@ -186,10 +208,8 @@ func (f OrderCustomerField) Encrypt(ctx context.Context, c *encrypt.Cipher, v st return EncryptedOrderCustomer{Ciphertext: out.Ciphertext, Match: out.Match, Ope: out.Ope}, nil } -func (f OrderCustomerField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) -} - -func (f OrderCustomerField) Ope(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.OpeTerm, error) { - return f.field.Ope(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f OrderCustomerField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } diff --git a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden index a9d7a066b..6286d44d4 100644 --- a/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/orders/refund_stash.go.golden @@ -109,7 +109,7 @@ func (f ReasonField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) ( return eql.TextEq(out.EQL), err } -func (f ReasonField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f ReasonField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } diff --git a/languages/golang/stashgen/testdata/cases/tenants/note_stash.go.golden b/languages/golang/stashgen/testdata/cases/tenants/note_stash.go.golden index 067abfde0..ffa276f75 100644 --- a/languages/golang/stashgen/testdata/cases/tenants/note_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/tenants/note_stash.go.golden @@ -130,12 +130,15 @@ func (f TextField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (En return EncryptedNoteText{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f TextField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f TextField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f TextField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f TextField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } // The -model flag: NoteRow converts to and from its shape only while the diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden index 9a82606d3..938efd373 100644 --- a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -122,9 +122,9 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (e return eql.TextEq(out.EQL), err } -func (f EmailField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } type NameField struct { @@ -136,7 +136,7 @@ func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eq return eql.TextEq(out.EQL), err } -func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f NameField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } diff --git a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden index b6dd34a2f..b76f15a9d 100644 --- a/languages/golang/stashgen/testdata/policy_individual_stash.go.golden +++ b/languages/golang/stashgen/testdata/policy_individual_stash.go.golden @@ -164,12 +164,15 @@ func (f EmailField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (E return EncryptedIndividualEmail{Ciphertext: out.Ciphertext, Equality: out.Equality, Match: out.Match}, nil } -func (f EmailField) Equality(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { - return f.field.Equality(ctx, c, v) +func (f EmailField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.EqualityTerm, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return encrypt.EqualityTerm(term), err } -func (f EmailField) Match(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchTerm, error) { - return f.field.Match(ctx, c, v) +// Matches derives the matches query for v. Its answer can hold rows that do not +// match: decrypt the rows it fetches and keep those Verify reports true. +func (f EmailField) Matches(ctx context.Context, c *encrypt.Cipher, v string) (encrypt.MatchQuery[string], error) { + return gensupport.Matches(ctx, f.field, c, v) } type MedicareNoField struct { @@ -181,7 +184,7 @@ func (f MedicareNoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v strin return eql.TextEq(out.EQL), err } -func (f MedicareNoField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { - out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err +func (f MedicareNoField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextEqQuery(term), err } diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go index 216bbde6f..997e932e0 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/encrypt.go @@ -34,6 +34,15 @@ type ( JSONTerm []byte ) +// Range is a between query on an order index: two terms. +type Range[T OreTerm | OpeTerm] struct { + Low T + High T +} + +// MatchQuery is a matches query on a match index. +type MatchQuery[T ~string] struct{ _ T } + // Index is one index on a field, as generated code names it in a declaration. type Index interface { Output() string diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go index 8bf0a0df0..484c7f6d3 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/gensupport/gensupport.go @@ -119,17 +119,16 @@ type Field[T any] struct{ _ struct{} } // NewField makes the entry for one field of a declaration. func NewField[T any](Declaration, string) Field[T] { return Field[T]{} } -func (Field[T]) Encrypt(context.Context, *encrypt.Cipher, T) (Output, error) { return Output{}, nil } -func (Field[T]) Query(context.Context, *encrypt.Cipher, T) (Output, error) { return Output{}, nil } -func (Field[T]) Equality(context.Context, *encrypt.Cipher, T) (encrypt.EqualityTerm, error) { - return nil, nil +func (Field[T]) Encrypt(context.Context, *encrypt.Cipher, T) (Output, error) { return Output{}, nil } +func (Field[T]) Term(context.Context, *encrypt.Cipher, string, T) ([]byte, error) { return nil, nil } +func (Field[T]) Range(context.Context, *encrypt.Cipher, string, T, T) ([]byte, []byte, error) { + return nil, nil, nil } -func (Field[T]) Match(context.Context, *encrypt.Cipher, T) (encrypt.MatchTerm, error) { - return nil, nil + +// Matches derives a matches query. +func Matches[T ~string](context.Context, Field[T], *encrypt.Cipher, T) (encrypt.MatchQuery[T], error) { + return encrypt.MatchQuery[T]{}, nil } -func (Field[T]) Ore(context.Context, *encrypt.Cipher, T) (encrypt.OreTerm, error) { return nil, nil } -func (Field[T]) Ope(context.Context, *encrypt.Cipher, T) (encrypt.OpeTerm, error) { return nil, nil } -func (Field[T]) JSON(context.Context, *encrypt.Cipher, T) (encrypt.JSONTerm, error) { return nil, nil } // RecordsCodec encrypts into and decrypts from a model. type RecordsCodec[P, R any] struct{ _ struct{} } From 20bd1047f54014d020e62c95c2bd71c293a02bf8 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 04:17:04 +1100 Subject: [PATCH 37/38] test(golang): run a matches query and verify its rows against PostgreSQL, through database/sql and pgx The test stores generated rows in separate columns, finds one by its equality term, and runs a matches query whose filter containment PostgreSQL tests with @>, then decrypts the rows it returned and verifies them: the verified rows are the Rust query fixture's. It uses the deterministic guest's keys and skips unless DATABASE_URL is set; the Linux Go job now starts PostgreSQL 17 and sets it. pgx is a test-only import. Part of #1189 --- .github/workflows/tests-golang.yml | 19 +++ languages/golang/encrypt/postgres_test.go | 177 ++++++++++++++++++++++ languages/golang/go.mod | 6 + languages/golang/go.sum | 30 +++- 4 files changed, 228 insertions(+), 4 deletions(-) create mode 100644 languages/golang/encrypt/postgres_test.go diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index 9bfff334a..253fcc210 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -103,6 +103,24 @@ jobs: # caches) the whole test toolset, which `mise run` would otherwise # install uncached. MISE_ENV: test + # The Go module's PostgreSQL test (encrypt/postgres_test.go) stores + # generated rows and runs the query methods against it through + # database/sql; it skips where DATABASE_URL is unset, as on the macOS + # and Windows jobs. + services: + postgres: + image: postgres:17 + env: + POSTGRES_PASSWORD: postgres + # Container port only: Docker picks the host port, and the Go step + # reads it from `job.services`, so the number is written once. + ports: + - 5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: @@ -215,6 +233,7 @@ jobs: - name: Go binding env: STACK_ENCRYPT_TESTS_REQUIRE_LOCK: "1" + DATABASE_URL: "postgres://postgres:postgres@localhost:${{ job.services.postgres.ports['5432'] }}/postgres?sslmode=disable" run: | ulimit -l "$(ulimit -H -l)" echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB" diff --git a/languages/golang/encrypt/postgres_test.go b/languages/golang/encrypt/postgres_test.go new file mode 100644 index 000000000..3b1bc4cd5 --- /dev/null +++ b/languages/golang/encrypt/postgres_test.go @@ -0,0 +1,177 @@ +package encrypt_test + +import ( + "context" + "database/sql" + "os" + "slices" + "testing" + + _ "github.com/jackc/pgx/v5/stdlib" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// The query methods against a real PostgreSQL, through database/sql and the +// pgx driver: generated rows stored in separate columns, an eq query on the +// equality term, and a matches query whose rows are decrypted and verified. +// The keys are the deterministic test build's, so only the database is +// outside the test. Skipped unless DATABASE_URL names a database the test +// may create a temporary table in. + +// matchPositions reads a stored match filter, or a match query's term, as +// the int[] of its little-endian uint16 positions, so PostgreSQL can test +// that a row's filter holds every position of the query with @>. +const matchPositions = ` +CREATE FUNCTION pg_temp.match_positions(term bytea) RETURNS int[] +LANGUAGE sql IMMUTABLE STRICT AS $$ + SELECT coalesce(array_agg(get_byte(term, 2 * i) | (get_byte(term, 2 * i + 1) << 8)), '{}') + FROM generate_series(0, length(term) / 2 - 1) AS i +$$` + +func TestQueriesRunAgainstPostgres(t *testing.T) { + url := os.Getenv("DATABASE_URL") + if url == "" { + t.Skip("DATABASE_URL is not set") + } + f := readQueryFixture(t) + c := queryFixtureClient(t, f) + ctx := context.Background() + cipher := c.DefaultKeyset() + + db, err := sql.Open("pgx", url) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + // One connection: the table and the function are temporary, so they + // live in this session alone. + conn, err := db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = conn.Close() }) + for _, stmt := range []string{ + `CREATE TEMPORARY TABLE profiles ( + id bigint PRIMARY KEY, + email bytea NOT NULL, + email_eq bytea NOT NULL, + age bytea NOT NULL, + age_eq bytea NOT NULL, + age_ore bytea NOT NULL, + name bytea NOT NULL, + name_match bytea NOT NULL + )`, + matchPositions, + } { + if _, err := conn.ExecContext(ctx, stmt); err != nil { + t.Fatal(err) + } + } + + // The fixture's verify page, as names, and one email each. + var people []testusers.Profile + for i, v := range f.Verify.Values { + if v != nil { + people = append(people, testusers.Profile{Email: "p" + string(rune('a'+i)) + "@example.com", Age: uint32(20 + i), Name: *v}) + } + } + encrypted, err := testusers.EncryptProfile(ctx, cipher, people) + if err != nil { + t.Fatal(err) + } + for i, e := range encrypted { + if _, err := conn.ExecContext(ctx, + `INSERT INTO profiles VALUES ($1, $2, $3, $4, $5, $6, $7, $8)`, + i, e.Email.Ciphertext, e.Email.Equality, e.Age.Ciphertext, e.Age.Equality, e.Age.Ore, e.Name.Ciphertext, e.Name.Match, + ); err != nil { + t.Fatal(err) + } + } + if _, err := conn.ExecContext(ctx, `ANALYZE profiles`); err != nil { + t.Fatal(err) + } + + // eq: the equality term binds as a parameter and finds its one row. + eq, err := testusers.ProfileFields.Email.Eq(ctx, cipher, people[2].Email) + if err != nil { + t.Fatal(err) + } + var id int64 + if err := conn.QueryRowContext(ctx, `SELECT id FROM profiles WHERE email_eq = $1`, eq).Scan(&id); err != nil || id != 2 { + t.Fatalf("eq found row %d, %v; want 2", id, err) + } + + // matches: size the fetch, let the server answer, then decrypt the rows + // it returned and keep those Verify reports true. + q, err := testusers.ProfileFields.Name.Matches(ctx, cipher, f.Verify.Input) + if err != nil { + t.Fatal(err) + } + var rows uint64 + if err := conn.QueryRowContext(ctx, `SELECT greatest(reltuples, 0)::bigint FROM pg_class WHERE oid = 'profiles'::regclass`).Scan(&rows); err != nil { + t.Fatal(err) + } + limit, err := q.TargetLimit(ctx, 10, encrypt.Stats{Rows: rows}) + if err != nil { + t.Fatal(err) + } + if _, bounded := limit.Rows(); bounded { + t.Fatal("a field declared with tags has no expected token count, so with no histogram the limit is unbounded") + } + result, err := conn.QueryContext(ctx, ` + SELECT id, email, email_eq, age, age_eq, age_ore, name, name_match FROM profiles + WHERE pg_temp.match_positions(name_match) @> pg_temp.match_positions($1) + ORDER BY id LIMIT $2`, q, limit.FetchOr(50)) + if err != nil { + t.Fatal(err) + } + var ids []int64 + var fetched []testusers.EncryptedProfile + for result.Next() { + var e testusers.EncryptedProfile + if err := result.Scan(&id, &e.Email.Ciphertext, &e.Email.Equality, &e.Age.Ciphertext, &e.Age.Equality, &e.Age.Ore, &e.Name.Ciphertext, &e.Name.Match); err != nil { + t.Fatal(err) + } + ids = append(ids, id) + fetched = append(fetched, e) + } + if err := result.Err(); err != nil { + t.Fatal(err) + } + decrypted, err := testusers.DecryptProfile(ctx, cipher, fetched) + if err != nil { + t.Fatal(err) + } + names := make([]string, len(decrypted)) + for i, p := range decrypted { + names[i] = p.Name + } + verified, err := q.Verify(ctx, names) + if err != nil { + t.Fatal(err) + } + var hits []string + for i, ok := range verified { + if ok { + hits = append(hits, names[i]) + } + } + + var want []string + for i, v := range f.Verify.Values { + if v != nil && f.Verify.Verified[i] { + want = append(want, *v) + } + } + if !slices.Equal(hits, want) { + t.Fatalf("matches %q verified %q from the server's %q (rows %v), want %q", f.Verify.Input, hits, names, ids, want) + } + // The server's answer has no false negative: every true match came back. + for _, w := range want { + if !slices.Contains(names, w) { + t.Fatalf("the server did not return %q", w) + } + } +} diff --git a/languages/golang/go.mod b/languages/golang/go.mod index 52569fa25..b5e46eb5d 100644 --- a/languages/golang/go.mod +++ b/languages/golang/go.mod @@ -10,12 +10,18 @@ require ( ) require ( + github.com/jackc/pgx/v5 v5.7.6 golang.org/x/sys v0.48.0 golang.org/x/tools v0.51.0 google.golang.org/protobuf v1.36.12 ) require ( + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + golang.org/x/crypto v0.37.0 // indirect golang.org/x/mod v0.41.0 // indirect golang.org/x/sync v0.23.0 // indirect + golang.org/x/text v0.24.0 // indirect ) diff --git a/languages/golang/go.sum b/languages/golang/go.sum index e8211d9d1..4bb4ee0bd 100644 --- a/languages/golang/go.sum +++ b/languages/golang/go.sum @@ -1,15 +1,31 @@ -github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc h1:iGbTgv9Kx5SaTI2tPxaxwDvS5BXZUO565N00cxRVwC8= -github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20261015123909-2e895cb9caed h1:supKop+UADso55MvBLebaBjP2vyilWmbAxOtFZJOnQ8= github.com/cipherstash/vitaminc/bindings/go/vcffi v0.0.0-20261015123909-2e895cb9caed/go.mod h1:6jpaqAo6f7rjWuxoti0Ymoi9DQZyxpnieWVudhXv38Y= -github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc h1:vlrjoILAURGfpBWucVZEywK22k4lfky1xGDIKV6kqNg= -github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20260902024806-f2c7f7d17fbc/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20261015123909-2e895cb9caed h1:do5JX9ovxiHzLc9G4wcybbr4f3xaVAYeIKFnOJk9NKw= github.com/cipherstash/vitaminc/bindings/go/vcvalue v0.0.0-20261015123909-2e895cb9caed/go.mod h1:RJODA1DCSm4H+1pbmCzDdT2pAIkxhinwIu6ewjh5sPs= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.7.6 h1:rWQc5FwZSPX58r1OQmkuaNicxdmExaEz5A2DO2hUuTk= +github.com/jackc/pgx/v5 v5.7.6/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0= +golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE= +golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc= golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98= @@ -18,7 +34,13 @@ golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0= +golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU= golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 8bb65850a682087418a013bcf00ceccc9943c613 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 16 Oct 2026 04:17:04 +1100 Subject: [PATCH 38/38] docs(golang): show the query methods, the database/sql query and the decrypt-and-verify step, and say se_query is gone The README and the package documentation describe one method per operator, what each returns, why a matches query carries Verify and TargetLimit and an exact one neither, and that a MatchQuery holds its text because Go cannot wipe a string. The ADR-0007 amendment records se_operators, the removal of se_query, and the one reason se_term stays. Part of #1189 --- languages/golang/cmd/stashgen/README.md | 8 +- languages/golang/encrypt/README.md | 93 +++++++++++++++++-- languages/golang/encrypt/doc.go | 43 ++++++++- languages/golang/encrypt/eql/eql.go | 2 +- languages/golang/encrypt/example/README.md | 6 +- languages/golang/encrypt/guest/src/ops.rs | 2 +- ...-through-a-plan-never-a-second-executor.md | 18 +++- 7 files changed, 153 insertions(+), 19 deletions(-) diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index d3f38554e..2bef89f69 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -40,7 +40,7 @@ Your program calls those functions, and it never builds or names a plan. ```go encrypted, err := users.Encrypt(ctx, cipher, people) opened, err := users.Decrypt(ctx, cipher, encrypted) - query, err := users.Fields.Email.Query(ctx, cipher, "bob@example.com") + query, err := users.Fields.Email.Eq(ctx, cipher, "bob@example.com") ``` 6. Store the encrypted type. @@ -123,7 +123,11 @@ For `-type User`, the file `user_stash.go` holds: - `Encrypt` and `Decrypt`, which take a slice and return a slice, and send one ZeroKMS request for each 500 sealed values in it, plus one the first time a keyset is used. - `Fields`, with one entry for each sealed field. An opaque struct, or a struct with no sealed field outside an opaque one, gets no `Fields`: nothing in it is sealed on its own, so nothing can be queried. - An entry encrypts one value, and it has a query method only for what the field declares: `Query` for `encrypt_into`, and `Equality`, `Match`, `Ore` or `Ope` for `index=`. + An entry encrypts one value, and has one query method for each operator the engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` or `Matches`. + stashgen asks the embedded engine which operators each field answers, and which index answers each one, so it holds no copy of those rules: an equality index answers `Eq`, a match index on a string `Matches`, and an ORE or OPE index on an integer the four orderings and `Between`, through ORE when the field declares both. + `encrypt_into=TextEq` answers `Eq`, which returns the type's query value, `eql.TextEqQuery`. + The query of `Matches` carries `Verify` and `TargetLimit`, because its answer can hold rows that do not match; an exact operator's query carries neither. + See the encrypt package's README for each method's result and the query with `database/sql`. - `String` and `LogValue` on `EncryptedUser`, which print the passthrough fields and hide the sealed ones. - A copy of the fields of `User`, so a change to them stops the build. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 6f8f51904..d2b413606 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -56,11 +56,13 @@ The package reference is on [pkg.go.dev]; the generator's reference is encrypted, err := users.Encrypt(ctx, cipher, people) opened, err := users.Decrypt(ctx, cipher, encrypted) - term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") + term, err := users.Fields.Email.Eq(ctx, cipher, "bob@example.com") ``` `Encrypt` and `Decrypt` send one ZeroKMS request for each 500 sealed values in the batch, plus one request the first time a keyset is used. + A query derives its term locally and sends no request; see + [Queries](#queries). 6. Store the encrypted type. Each sealed field is one or more byte columns: `e.Email.Ciphertext`, `e.Email.Equality`, `e.Email.Match`. `Ciphertext` @@ -198,7 +200,7 @@ field and the index. So an optional or short value does not belong under `match`: give such a field `equality` alone, or make the value required. Terms are byte-equal to the ones the Rust crate derives, so a term from -`users.Fields.Email.Equality` compares against a stored term written from any +`users.Fields.Email.Eq` compares against a stored term written from any language. `EqualityTerm.Equal` compares in constant time; `OreTerm.Compare` and `OpeTerm.Compare` order as the plaintexts; `MatchTerm.Positions` decodes the token positions. @@ -211,8 +213,86 @@ A database can compare some terms by itself: - `Ore`: do not compare in SQL. Plain byte order is almost always right, and sometimes wrong, with no error. Compare in Go with `OreTerm.Compare`, or use an EQL type (`encrypt_into=`), which the database compares correctly. -- `Match`: do not compare in SQL. Decode the positions in Go with - `MatchTerm.Positions`, or use an EQL type. +- `Match`: a row matches when its stored filter holds every position of the + query's term. Decode both into positions in SQL and compare with `@>`, as + [Queries](#queries) shows, or in Go with `MatchTerm.Positions`. + +## Queries + +Each sealed field's entry in `Fields` has one method for each operator the +engine answers on the field: `Eq`, `Lt`, `Lte`, `Gt`, `Gte`, `Between` and +`Matches`. A field has no method for an operator it does not answer, so such +a query does not compile. The generator asks the engine which operators each +field answers, and which index answers each one; it holds no copy of those +rules. + +| The field declares | Methods | Each returns | +|---|---|---| +| `index=equality` | `Eq` | `encrypt.EqualityTerm` | +| `index=ore` or `index=ope`, on an integer | `Lt`, `Lte`, `Gt`, `Gte` | `encrypt.OreTerm`, or `encrypt.OpeTerm` when the field declares no `ore` | +| the same | `Between(ctx, cipher, lo, hi)` | `encrypt.Range[encrypt.OreTerm]` or `encrypt.Range[encrypt.OpeTerm]`, the two bounds' terms | +| `index=match`, on a string | `Matches` | `encrypt.MatchQuery[T]` | +| `encrypt_into=TextEq` | `Eq` | `eql.TextEqQuery` | + +A method takes the field's own Go type, so `Fields.Age.Lt` on a `uint32` +field takes a `uint32`. An order index on text, bytes or a float stores its +term but answers no ordering operator yet, so such a field has no `Lt`. + +Every query type implements `driver.Valuer`, so it binds as a parameter in +any library built on `database/sql` or pgx. A `Range` binds as two +parameters, `Low` and `High`. + +`Eq`, the orderings and `Between` are exact: the server's answer is the +answer. `Matches` is probabilistic: the server answers with every row whose +stored filter holds the query's bits, which can include rows that do not +hold the query's text. So `MatchQuery` carries two more methods, and an +exact query's type carries neither: + +- `TargetLimit(ctx, want, stats)` is how many rows to fetch for `want` of + them to verify. A field declared with tags states no expected number of + tokens, so with no measured histogram of the stored filters in + `encrypt.Stats` the limit is unbounded, and `Limit.FetchOr(page)` fetches + a page. +- `Verify(ctx, values)` says, for each decrypted value of a page, whether it + really holds the query's text: one call into the engine for the page, with + no key. + +```go +// Positions of a stored filter or a match term, for @>. +const positions = `CREATE FUNCTION match_positions(term bytea) RETURNS int[] +LANGUAGE sql IMMUTABLE STRICT AS $$ + SELECT coalesce(array_agg(get_byte(term, 2 * i) | (get_byte(term, 2 * i + 1) << 8)), '{}') + FROM generate_series(0, length(term) / 2 - 1) AS i +$$` + +q, err := users.Fields.Email.Matches(ctx, cipher, "alice") +limit, err := q.TargetLimit(ctx, 10, encrypt.Stats{Rows: rowCount}) + +rows, err := db.QueryContext(ctx, ` + SELECT id, email, email_eq, email_match, age, age_eq, age_ore FROM users + WHERE match_positions(email_match) @> match_positions($1) + ORDER BY id LIMIT $2`, q, limit.FetchOr(50)) +// scan each row into a users.Encrypted, then +people, err := users.Decrypt(ctx, cipher, encrypted) +emails := make([]string, len(people)) +for i, p := range people { + emails[i] = p.Email +} +verified, err := q.Verify(ctx, emails) +hits := people[:0] +for i, p := range people { + if verified[i] { + hits = append(hits, p) + } +} +``` + +`Verify` cuts the query's text into tokens again in the engine, with the +field's own tokenizer, so a `MatchQuery` keeps that text for as long as the +program holds the query. Go cannot wipe a string: the text stays in memory +until the garbage collector reclaims it, as the string you passed in does. +A `MatchQuery` never prints it; `fmt`, `slog` and `%#v` show the field and +the size of the term only. ## Errors @@ -280,8 +360,9 @@ PostgreSQL holds in a `public.eql_v3_text_eq` column, as an `eql.TextEq` from package `encrypt/eql`. The guest builds the value: the generated code imports `encrypt/eql`, which embeds the build of the engine that holds the EQL types and registers it on import, so the program runs that build and -stores what it returns. `Fields.Email.Query` returns the `eql.TextEqQuery` -the column's `eql_v3.query_text_eq` operand takes. `mise run +stores what it returns. `Fields.Email.Eq` returns the `eql.TextEqQuery` +the column's `eql_v3.query_text_eq` operand takes; `TextEq` answers `eq` +alone. `mise run wasm:guest:build:eql` builds that guest beside the other. A cipher extended with a tenant part refuses a struct with an `encrypt_into` field: an EQL value is stored under a table and a column, and the extended label has no diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 01179f7db..cc9904caa 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -35,10 +35,11 @@ // cipher := client.Keyset(encrypt.KeysetName("tenant-42")) // encrypted, err := users.Encrypt(ctx, cipher, people) // []users.EncryptedUser // opened, err := users.Decrypt(ctx, cipher, encrypted) // []users.User -// term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com") +// term, err := users.Fields.Email.Eq(ctx, cipher, "bob@example.com") // // Encrypt and Decrypt send one ZeroKMS request for each 500 sealed values in -// the batch, plus one the first time a keyset is used. +// the batch, plus one the first time a keyset is used. A query derives its +// term with no request; see Queries below. // // 6. Store the encrypted type. Each sealed field is one or more columns of // bytes ([Ciphertext] and the term types), each implementing driver.Valuer @@ -112,6 +113,44 @@ // refused with [ErrTerm], because an empty filter as a query would match every // row. // +// # Queries +// +// A generated Fields entry has one method for each operator the engine +// answers on the field, named for it: Eq, Lt, Lte, Gt, Gte, Between and +// Matches. A field has no method for an operator it does not answer, so +// such a query does not compile; the generator asks the engine which +// operators each field answers, and holds no copy of the rules. Each method +// takes the field's own Go type. +// +// Eq returns an [EqualityTerm] (or, for a field with encrypt_into, the EQL +// type's query value), an ordering an [OreTerm] or [OpeTerm], and Between a +// [Range] of two. Their answers are exact. Matches returns a [MatchQuery], +// whose answer can hold rows that do not match: size the fetch with +// [MatchQuery.TargetLimit], then decrypt the rows and keep those +// [MatchQuery.Verify] reports true, one call into the engine for the page. +// Every query type implements driver.Valuer, so it binds as a parameter +// through database/sql or pgx: +// +// q, err := users.Fields.Email.Matches(ctx, cipher, "alice") +// limit, err := q.TargetLimit(ctx, 10, encrypt.Stats{Rows: rowCount}) +// rows, err := db.QueryContext(ctx, `SELECT id, email, email_eq, email_match, age, age_eq, age_ore +// FROM users WHERE match_positions(email_match) @> match_positions($1) +// ORDER BY id LIMIT $2`, q, limit.FetchOr(50)) +// // scan into []users.EncryptedUser, then +// people, err := users.Decrypt(ctx, cipher, encrypted) +// emails := make([]string, len(people)) +// for i, p := range people { +// emails[i] = p.Email +// } +// verified, err := q.Verify(ctx, emails) // one bool per row +// +// match_positions is a SQL function of your own that reads a match term's +// little-endian uint16 positions into an int[]; the package README has one. +// A MatchQuery keeps the text it was derived from, because Verify cuts its +// tokens again in the engine. Go cannot wipe a string, so the text stays in +// memory until the garbage collector reclaims it; a MatchQuery never prints +// it. +// // # Transport and auth // // The guest imports exactly two host functions: an HTTP send, served by any diff --git a/languages/golang/encrypt/eql/eql.go b/languages/golang/encrypt/eql/eql.go index 24b3d7c17..95500bf1e 100644 --- a/languages/golang/encrypt/eql/eql.go +++ b/languages/golang/encrypt/eql/eql.go @@ -3,7 +3,7 @@ // the type's domain, and the query types that match them. // // A field with `stash:"email,encrypt_into=TextEq"` is an [TextEq] in the -// generated encrypted type, and its Fields entry's Query returns a +// generated encrypted type, and its Fields entry's Eq returns a // [TextEqQuery]. The guest builds every value: generated code stores what // the guest returns and never assembles an EQL value itself (ADR-0007, // amended 2026-10-06). Each type implements driver.Valuer and sql.Scanner diff --git a/languages/golang/encrypt/example/README.md b/languages/golang/encrypt/example/README.md index 0a995caca..1e1e3c2c1 100644 --- a/languages/golang/encrypt/example/README.md +++ b/languages/golang/encrypt/example/README.md @@ -2,7 +2,7 @@ A runnable tour of the Go SDK against real ZeroKMS: declare a struct with `stash` tags, generate its encrypted type, seal a batch in one request, derive -a query term, and open the batch again. +a query term, open the batch again, and verify a matches query over it. ## Running it @@ -28,5 +28,5 @@ until they are built. - `model.go`: the struct and its `go:generate` line. - `user_stash.go`: what `stashgen` wrote from the tags. Regenerate it with `go generate ./...`; CI fails when that changes a committed file. -- `main.go`: `Encrypt`, `Fields.Email.Equality` and `Decrypt`, the only three - calls a program makes. +- `main.go`: `Encrypt`, `Fields.Email.Eq`, `Decrypt`, and `Fields.Email.Matches` + with `Verify` on the decrypted rows. diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index 6317c298d..936b6c593 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -408,7 +408,7 @@ pub fn targets() -> Result, u32> { /// The operators one field of a codec-encoded plan answers, as a /// codec-encoded list: one object per operator, in the order the engine /// lists them, each written by -/// [`dynamic::operator_to_value`](stack_encrypt::dynamic::operator_to_value): +/// [`dynamic::operator_to_value`]: /// /// ```text /// [ { "operator": "eq", "fidelity": "exact", "arity": "one", "index": "eq" }, ... ] diff --git a/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md b/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md index 4fa765af6..40a5af01b 100644 --- a/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md +++ b/packages/stack-encrypt/docs/adr/0007-bindings-enter-through-a-plan-never-a-second-executor.md @@ -285,10 +285,20 @@ from Go and the typed query chain derive the same bytes. A cross-language fixture holds them together (`tests/fixtures/query_operators.json`): the Rust side checks each recorded term against the typed chain, and the Go SDK's tests derive, verify and size the same queries through the guest. -`se_query`, which derives a target field's twin with no operator, stays until -the generated Go code names operators (#1189), and is removed then. -`se_term`, a term by index under an explicit context, stays for the generated -code of fields declared with indexes until that code names operators too. +The catalog answers for an EQL type; a field declared with indexes has no +type to look up, so `se_operators` answers for any field of a plan: the +plan and the field's name in, the same operator objects out, computed by the +one table `se_query_term` reads (`dynamic::field_operators`). The Go +generator asks it for every sealed field, `encrypt_into` fields included, +and writes one method per operator (#1189). + +`se_query`, which derived a target field's twin with no operator, is +removed: the generated Go code names operators (#1189), and `eq` on a target +field is the twin. `se_term`, a term by index under an explicit context, no +longer serves a query. It stays for one host need: a record call's term +failure does not say which row and field it came from, and the Go SDK asks +`se_term` again for each term to name them. When the record exports name the +field themselves, it goes too. A data plan's match index declares no `expected_tokens` prior, so `target_limit` on a field declared as data is unbounded without a measured