Skip to content

feat(golang)!: generate one query method per operator, with Verify and TargetLimit on a matches query - #1216

Draft
coderdan wants to merge 4 commits into
feat/stack-encrypt-operator-exportsfrom
feat/golang-operator-queries
Draft

coderdan wants to merge 4 commits into
feat/stack-encrypt-operator-exportsfrom
feat/golang-operator-queries

Conversation

@coderdan

Copy link
Copy Markdown
Contributor

Summary

Stacked on #1215. This is the Go SDK half of ADR-0009 (appendix B): the generated code names an operator, not an index. PR 4 of 4, the top of the chain.

  • One query method per operator the engine answers. Each sealed field's Fields entry has Eq, Lt, Lte, Gt, Gte, Between or Matches, as the engine says the field answers, each taking the field's own Go type (Fields.Age.Lt(ctx, cipher, uint32(40)), Fields.Age.Between(ctx, cipher, lo, hi)). A field has no method for an operator it does not answer, so the query does not compile. Query and the per-index Equality, Match, Ore, Ope methods are gone.
  • The operators come from the engine. The catalog lists operators only for EQL types, and a field declared with indexes has no type to look up, so there is a new engine function and guest export: dynamic::field_operators(&plan, field, &resolver) and se_operators(plan, field), read in Go by Checker.Operators. They return the same {operator, fidelity, arity, index} objects as the catalog, from the same table that query_term reads, so the generator is told what a query on the field finds. stashgen asks for every sealed field, encrypt_into fields included, and holds no copy of the rules. enginetest.Static is still a stand-in. A test now generates the same declarations through it and through the guest and checks that the files match.
  • What each method returns. Eq returns an encrypt.EqualityTerm, or the EQL type's query value (eql.TextEqQuery) on an encrypt_into field. The orderings return an encrypt.OreTerm, or an encrypt.OpeTerm when the field declares no ORE. Between returns an encrypt.Range[T] with the two bounds' terms. Matches returns an encrypt.MatchQuery[T]. Every one implements driver.Valuer; a Range binds as two parameters, each bound a Valuer.
  • A probabilistic query carries Verify and TargetLimit, and an exact one carries neither. MatchQuery.Verify(ctx, values []T) checks a page of decrypted values in one guest call. MatchQuery.TargetLimit(ctx, want, stats) sizes the fetch. Both use no key. The generator only writes MatchQuery for an operator the engine reports as probabilistic. If the engine ever reports a combination the SDK has no Go type for (a probabilistic operator on an EQL type, for example), stashgen refuses the field and names it, rather than leaving a method out.
  • A MatchQuery holds its text. Verify cuts the query's tokens again in the engine, so the query keeps the text it was derived from. Go cannot wipe a string, so the text stays in memory until the garbage collector reclaims it. The README and the package docs say so. A MatchQuery never prints the text: Format, String, GoString and LogValue show only the field and the size of the term, and a test checks every fmt verb and both slog handlers.
  • se_query is removed. On a target field, eq gives the twin. Cipher.Query, the export, its guest ops and its tests are gone.
  • se_term stays. Generated code no longer uses it, and Cipher.Derive is now the unexported derive. One caller still needs it: when a record call fails on a term, the error does not say which row and field caused it, and locateTermFailure asks se_term again, one term at a time, to find them. se_query_term cannot do this job. It reaches only the index that answers an operator (ORE, when a field declares ORE and OPE), and it refuses text with no token, which storage accepts. The ADR-0007 amendment records this.
  • Generated code and goldens regenerated. This covers encrypt/internal/testusers (plus a new Profile type whose declaration matches the Rust query fixture), testpolicy, example, the stashgen goldens and the stub SDK. The order property tests now read the ORE and OPE terms that Encrypt stores, because a query method reaches only one index, and only on integers.

The width rules hold because each method takes the field's Go type: Lt on a uint32 field takes a uint32, and Matches on type Email string returns a MatchQuery[Email] whose Verify takes []Email.

Behaviour to note

  • An ORE or OPE index on text, bytes or a float answers no operator. This is the engine's rule (OpSpec::answered_by orders integers only, as the typed op::Ordered does). Such a field still stores its term, but no longer gets an Ore or Ope method.
  • When a field declares both ORE and OPE, its orderings use ORE, and ORE terms do not order in SQL. A field that should compare in SQL should declare OPE alone, or use an EQL type later.
  • A field declared with tags has no expected_tokens prior. Without a measured histogram, TargetLimit is unbounded and Limit.FetchOr(page) pages. The README says so.
  • A declared match column is bytes. The README shows a small match_positions(bytea) → int[] SQL function and @> for the containment test. This is what the PostgreSQL test runs.

What waits

Tests

  • Rust (dynamic::query): every operator a field is said to answer derives a term through that index, and every other operator is refused as operator_not_answered, for equality, match, ORE+OPE and text-with-ORE fields. Also covered: a missing field, the wire form of an operator, and a target field answering what its type lists.
  • Guest native tests: ops::operators for each field of the record plan, with every listed operator validating and every other refused; a missing field and a malformed plan; in the eql build, a TextEq field answers eq. The in-crate TextEq round trip now queries through query_term with eq.
  • Go:
    • Checker.Operators for each field of the fixture plan.
    • Profile's generated Eq, Lt, Lte, Gt, Gte, Between and Matches against the Rust query fixture's terms.
    • MatchQuery.Verify against the fixture's verified page, and MatchQuery.TargetLimit against all six fixture limits.
    • The zero MatchQuery refuses both methods.
    • The redaction test.
    • QueryTerm refusals before state.
    • The existing suites, moved onto the new methods.
  • stashgen: goldens and the policy golden are regenerated, and the stub SDK is updated (the agreement tests pass). The fake-versus-guest test now also compares the generated files for two new declarations: every index on an integer, and orderings on text, bytes and a float.
  • PostgreSQL (encrypt/postgres_test.go): the test writes Profile rows through database/sql and pgx, finds one row by Eq, then runs Matches with TargetLimit and @>, decrypts the rows, and checks that Verify keeps exactly the fixture's verified names and that the server returned every true match. It skips unless DATABASE_URL is set. The Linux Go job now starts postgres:17 and sets it.

Test plan

  • cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings (Rust 1.94.1).
  • cargo nextest run --workspace --all-features: 1159 passed.
  • mise run test:doc, mise run doc.
  • mise run wasm:guest:test (fmt, clippy on host and wasm32-wasip1 for all four feature sets, rustdoc, nextest with and without eql).
  • mise run wasm:guest:build, :eql, :deterministic, :eql:deterministic, wasm:auth-guest:build, then scripts/go-binding-test.sh languages/golang (gofmt, vet, tests; the 386 run is Linux only) and mise run go:lint.
  • go generate ./... in languages/golang leaves the tree clean.
  • The PostgreSQL test against a local PostgreSQL 17.5 (DATABASE_URL set).
  • cargo-mutants --in-diff over stack-auth and stack-encrypt: 10 mutants, 4 caught, 6 unviable, none missed.
  • Script tests (vitest --config scripts/vitest.config.mjs): 74 of 75 files pass. The other file, check-auth-npm-changeset, fails to import @changesets/parse from this worktree's linked node_modules, and is not touched here.
  • CI, including the new PostgreSQL service in the Linux Go job, and the macOS and Windows Go jobs.

Every checkbox of #1189 is done:

  • One method per operator, from the engine
  • No copy of the rules
  • Verify and TargetLimit on probabilistic queries only, each query keeping driver.Valuer
  • One guest call per page
  • The width rules
  • README and doc.go
  • The hermetic tests and the PostgreSQL Matches test, which runs locally and in the Linux Go job once CI runs

Closes #1189

…nswers, so a generator holds no copy of the table

dynamic::field_operators lists the operators a field of a data plan
answers, each with the index that answers it: for a field declared with
indexes, what OpSpec::answered_by finds under its kind; for a field that
names an EQL type, what the type's catalog entry lists. query_term reads
the same table, so what a generator is told a field answers is what a
query on it finds. operator_to_value writes one entry in the catalog's
wire form, which TargetDescriptor::to_value now shares.

Part of #1189
…ers, with Verify and TargetLimit on a matches query

stashgen asks the engine which operators each sealed field answers,
through a new guest export, se_operators, and writes one method for each:
Eq, Lt, Lte, Gt, Gte, Between and Matches, each taking the field's own Go
type. A field has no method for an operator it does not answer, so such a
query does not compile. The generator holds no copy of which index answers
which operator; the static test engine is held to the guest by a test that
generates the same declarations through both and compares the files.

Eq returns an EqualityTerm (or, on an encrypt_into field, the EQL type's
query value), an ordering an OreTerm or OpeTerm, and Between an
encrypt.Range of the two bounds' terms. Matches returns an
encrypt.MatchQuery, whose answer can hold false positives, so it carries
Verify, one guest call for a page of decrypted values, and TargetLimit.
Every query type implements driver.Valuer. MatchQuery keeps the text it was
derived from, because Verify cuts its tokens again in the engine; it never
prints it.

se_query, Cipher.Query and gensupport's per-index term methods are gone:
eq on a target field is its twin, and a query names an operator. se_term
stays, now behind the unexported Cipher.derive, because a record call's
term failure does not say which field it came from and the host asks
se_term again to name it. The order tests read the terms Encrypt stores,
since a query method reaches only the index that answers its operator.

Part of #1189
…SQL, through database/sql and pgx

The test stores generated rows in separate columns, finds one by its
equality term, and runs a matches query whose filter containment
PostgreSQL tests with @>, then decrypts the rows it returned and verifies
them: the verified rows are the Rust query fixture's. It uses the
deterministic guest's keys and skips unless DATABASE_URL is set; the Linux
Go job now starts PostgreSQL 17 and sets it. pgx is a test-only import.

Part of #1189
…decrypt-and-verify step, and say se_query is gone

The README and the package documentation describe one method per
operator, what each returns, why a matches query carries Verify and
TargetLimit and an exact one neither, and that a MatchQuery holds its text
because Go cannot wipe a string. The ADR-0007 amendment records
se_operators, the removal of se_query, and the one reason se_term stays.

Part of #1189
@coderabbitai

coderabbitai Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@changeset-bot

changeset-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d0869e7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
4 0 6 0

Every mutant in the changed lines was caught by a test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant