Skip to content

EQL JSON containment (@>, <@) matches across array elements, so Contains is not exact yet #1231

Open

Description

@coderdan

Background

EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database, as the eql_v3 schema today and eql_v4 next. It lets the database search encrypted values without decrypting them. For an encrypted JSON column, each stored document carries a SteVec index: a list with one entry per leaf of the document. Each entry has a selector, a keyed hash of the leaf's path, or of its path and value. A containment query (@>, "does this document contain that JSON?") encrypts the query document the same way, and the database compares selectors.

Stack Encrypt, the Rust encryption engine behind the Go SDK, declares JSON operators Contains (@>) and ContainedBy (<@) in #1209. It declares both as exact: the database's answer is the answer, and the library offers no verify step to check rows after decryption. We are keeping that declaration on purpose. This issue tracks the fact that the database does not yet deliver it.

Problem

What happens. EQL's containment is selector-set containment, not PostgreSQL's jsonb @>. A stored document "contains" a query document when every selector of the query is somewhere in the stored document (eql_v3.jsonb_document_contains, packages/eql/src/v3/json/functions.sql:298-334; packages/eql/docs/reference/json-support.md:214). Every element of an array shares one wildcard path ($.a[*]), so selectors cannot tell which array element a value came from (packages/eql/docs/plans/stevec-envelope-key-header-rfc.md:57).

A false positive (verified from the two facts above):

  • Stored: {"a":[{"x":1,"y":5},{"x":7,"y":2}]}
  • Query: {"a":[{"x":1,"y":2}]}
  • The query's value selectors are $.a[*].x = 1 and $.a[*].y = 2. Both are in the stored document, so the database returns the row.
  • Plain jsonb @> on the decrypted documents says false: no single element has both x = 1 and y = 2.

<@ has the same problem in reverse.

A possible false negative (plausible, not verified): in plain jsonb, '["a"]' @> '"a"' is true (a top-level array contains a top-level scalar). The query's selector sits at $, the stored element's at $[*]. Unless SteVec maps one onto the other, selector containment says false. A false negative is worse than a false positive, because no check after decryption can bring back a row the database never returned.

Who hits it. Anyone running a containment query on an encrypted JSON column with arrays of objects. This is today's behaviour in production through EQL v3, so it is a bug in what we ship now, not something #1209 introduces. #1209 only declares the operators; no Stack Encrypt index answers them yet.

Why nothing catches it. The EQL tests check containment against EQL's own selector rule, not against jsonb @> on the decrypted documents. The operator is declared exact, so callers are told the answer needs no checking.

Proposal

  1. Add a test that runs the same containment queries through EQL and through plain jsonb @> / <@ on the plaintext documents, and reports every disagreement. Start with the array-of-objects case above and the top-level-scalar case. That turns "plausible" into "verified" for the second case and pins the first.
  2. Make the database answer agree with jsonb containment for arrays. The selector would need to say which array element a value belongs to, without revealing more than today's index does. This is a SteVec and EQL change, and it has to work for EQL v4 too.
  3. Until step 2 ships, document the gap where callers will read it: the Contains and ContainedBy docs in stack-encrypt (src/op/mod.rs), ADR-0009, and the EQL JSON reference. Say plainly that arrays of objects can return extra rows.
  4. Keep the declaration exact. A step back to Probabilistic (false positives filtered by verify after decryption) would hide a database bug inside the library, and would still not cover the false-negative case.

Relationship to other work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    SDKbugSomething isn't workingneeds-triageMaintainer needs to evaluate the issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions