Skip to content

Document JSON path access in restriction and projection - #291

Open
dimitri-yatsenko wants to merge 1 commit into
mainfrom
docs/json-path-reference
Open

dimitri-yatsenko wants to merge 1 commit into
mainfrom
docs/json-path-reference

Conversation

@dimitri-yatsenko

@dimitri-yatsenko dimitri-yatsenko commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

attribute.path works in both restriction and projection, and DataJoint translates it per backend. None of it was documented.

  • reference/operators.md — zero mentions of JSON.
  • reference/specs/type-system.md — one line: "Supports JSON path queries where available." Eleven words, no syntax, and "where available" implies a portability caveat that does not exist.
  • The only pages showing the syntax were two written this week for _prov.

What this adds

A JSON path access section in the type-system spec:

  • The syntax on both surfaces — Equipment & {"specs.vendor": "Acme"} and Equipment.proj(vendor="specs.vendor") — and that both emit json_value() on MySQL and jsonb_extract_path_text() on PostgreSQL.
  • Nested and array paths: specs.probe.serial, specs.channels[0].
  • That extracted values are text, and the path:type annotation that fixes it.
  • Ordering comparisons, which need a typed projection first: proj(ch="specs.channels:unsigned") & "ch > 32".
  • What is not supported: whole-object comparison raises; hidden attributes cannot be reached by the mapping form.

Two defects stated rather than left to be discovered

A reader who follows the obvious path gets wrong answers, so the section says so:

  • Compare against a string. & {"specs.calibrated": True} returns no rows on MySQL and raises on PostgreSQL — the extraction yields text and the Python value is not coerced (#1564). The MySQL case is the dangerous one: an empty result reads as "none are calibrated."
  • :int is not portable — works on PostgreSQL, raises on MySQL, whose RETURNING clause does not accept it. unsigned and signed work on both (#1563).

Still to come

tutorials/advanced/json-type.ipynb teaches "Filtering on JSON Content — fetch then filter in Python" and lists "Filter in Python" as an inherent property of JSON in its Design Guidelines. That is wrong — it teaches a full table scan into Python for something the database does — and it is almost certainly a workaround for #1564 mistaken for a property of the type.

Rewriting it is deliberately held until #1564 merges, so the tutorial is not rewritten twice. Tracked in #292.

Verification

Every example run against MySQL 8.0 and postgres:15. mkdocs build --strict clean; check_links.py passes across 142 pages.

`attribute.path` works in both restriction and projection, and DataJoint
translates it per backend — json_value() on MySQL, jsonb_extract_path_text()
on PostgreSQL. None of that was documented. reference/operators.md does not
mention JSON at all, and the type-system spec said only "Supports JSON path
queries where available", whose "where available" implies a portability caveat
that does not exist.

Adds a JSON path access section to the type-system spec covering the syntax for
both surfaces, nested and array paths, the path:type annotation, and ordering
comparisons via a typed projection.

Also states the two defects a reader will otherwise discover by being given
wrong answers:

- a restriction value that is a Python int or bool is not coerced to match the
  extracted text, so {"specs.calibrated": True} returns no rows on MySQL and
  raises on PostgreSQL (#1564);
- :int works on PostgreSQL and raises on MySQL, where RETURNING does not accept
  it; unsigned and signed are portable (#1563).

Verified every example on MySQL 8.0 and postgres:15.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant