Public access to hidden attributes (_job_*, _prov): no reader exists

[dimitri-yatsenko]

Platform-managed columns — _job_start_time, _job_duration, _job_version, _prov, _singleton — are stored per row and have no public reader. to_arrays('_job_start_time') and proj('_prov') both raise DataJointError: Attribute '...' not found., because Heading.attributes (heading.py:258-267) excludes hidden names and every accessor derives from it.

reference/specs/job-metadata.md documented to_arrays('_job_start_time', ...) as the access path. That call has never worked; the docs are corrected in datajoint/datajoint-docs#288 to describe the SQL workaround until this lands. Supersedes datajoint-docs#1553.

Deferred out of 2.3.4 deliberately — the mechanism touches query machinery and should not be designed under release pressure.

What works today, so the gap is scoped correctly

Measured on MySQL 8.0, not assumed:

Restrict, string form — & "_prov IS NULL"	works
Restrict, mapping form — & {"_prov.system": ...}	silently returns every row — #1561
make_sql(["subject_id", "_prov"]) + decode_attribute	works, composes with restrictions, decodes JSON to a dict on both backends
to_arrays('_prov'), proj('_prov'), heading['_prov']	raise

So the missing piece is reading values through a supported surface. Restriction already works in its string form, and a correctly decoded read is reachable today in about six lines — which means the design can be judged on ergonomics and safety rather than on feasibility.

Proposed direction: QueryExpression.hidden

A property returning a copy whose heading also carries the hidden attributes:

Analysis.hidden.to_dicts()               # includes _job_* and a decoded _prov
Analysis.hidden.proj('_job_version')
Target.insert(Source.hidden)             # INSERT ... SELECT carries them

Returning a copy is what makes it safe: describe(), alter(), diagram and every un-opted query keep the heading they have now.

The invariant

.hidden changes what is selected and returned. It never changes what is matched on. Join and restriction keys must keep coming from the visible set unconditionally. Three findings make that non-negotiable:

• expression.py:398,430 build the USING list from heading.names. Hidden namesakes there reproduce exactly the NATURAL JOIN defect the USING clause was introduced to prevent (specs/job-metadata.md, "Excluding Hidden Attributes from Binary Operators").
• Hidden attributes never get a lineage row (lineage.py:366-369 skips any column starting with _), so assert_join_compatibility (condition.py:264-289) would raise "lineage missing on one side" for every join of two metadata-bearing tables.
• condition.py:441 selects restriction keys the same way, and that one fails silently.

Where the work is

• heading.py — an opt-in flag plus with_hidden(); an unconditionally-filtered name list for matching; select() (:641) must carry hidden through, join() (:698) must not. _attributes already holds everything with correct json/uuid/codec/dtype flags (:499-503), so nothing new is loaded.
• expression.py — the hidden property (copy idiom at :1507-1516); proj's validator at :572-576.
• table.py:877-886 — the INSERT ... SELECT branch currently appends _prov by name. Generalizing it is tempting but needs thought: copying _singleton, or _job_* from one table to another, is not meaningful. The _prov-specific carry may be correct because it is specific.

Two defects found on this path

• expression.py:578-583 is a dead validator. next(a for a in mentions if not self.heading.names) tests whether the heading is empty, not whether a is in it. It never fires, for any name.
• proj(x='_prov') silently drops the column — _prov matches rename_pattern, bypassing the positional check, and Heading.select's loop never reaches it. No error, no column. Same family as Mapping restriction naming a hidden attribute silently drops the predicate #1561.

The alternative worth taking seriously

Keep hidden attributes fully hidden and add a narrow documented reader instead — a function taking an expression and attribute names, doing the SELECT and decode_attribute itself. It serves the known consumer (provenance-export, which needs _job_version and _prov per row for audit packets) with decoded values and restriction composition, and touches no query machinery.

The case for it is not weak: attributes' filter currently does five distinct jobs — join-key selection, restriction-key selection, output shaping, insert validation, and DDL round-tripping through describe() ↔ prepare_declare ↔ alter. Splitting them is where silent breakage comes from. The case against is that it is a second way to query, outside the algebra.

Decide between them before implementing.

What must not change

Behavior	Pinned at
_job_* absent from heading.names and to_dicts()	test_hidden_job_metadata.py:184-201
Hidden attributes absent from a join result	test_hidden_job_metadata.py:208-224
_prov out of heading, fetch, joins	test_entry_provenance.py:140-152
An author cannot write _prov	test_entry_provenance.py:155-158
Config.heading.primary_key == [] for a singleton	test_declare.py:421
describe() omits _singleton	test_declare.py:477-490
Users cannot declare an underscore attribute	tests/unit/test_declare_hidden_attribute.py

.hidden would expose _singleton, so primary_key becomes ['_singleton'] in the opted-in view. Correct, but keys() and restriction run off the primary key — verify on a singleton table.

Acceptance

• Reading returns a decoded _prov dict on both MySQL (json) and PostgreSQL (jsonb); backend-consistent decoding is the likeliest thing to break.
• Default to_dicts() byte-identical to today.
• A * B with _job_* on both sides joins on the visible key only, no lineage error.
• describe() and alter() round-trip unchanged, including a singleton.
• test_entry_provenance.py's raw-SQL _raw_prov helper is replaced by the new API — that is the real acceptance test.
• Also closes the declare.py:965 error message, which tells users to "use proj()" — currently false.

[dimitri-yatsenko]

#1561 is in scope for this design, not a separate fix.

Same root cause: Heading.attributes (heading.py:258-267) masks hidden names, so a hidden attribute can be neither matched on (#1561 — a mapping restriction silently drops the predicate and returns every row) nor read back (this issue). Settling one without the other would ship an inconsistent surface: a user who can read _prov but cannot filter by it, or the reverse.

The constraint from #1561 that must survive whatever is designed here: a mapping restriction ignores attributes it cannot match by design — that is what lets Session & key work when key carries attributes from a more detailed downstream table, and make() depends on it. The fix is to distinguish absent from _attributes entirely (keep ignoring) from present but hidden (resolve), never to start raising on unmatched keys.

[dimitri-yatsenko]

#1567 proposes making every platform-managed column declarable through one mechanism in DataJoint notation, by separating the grammar (which should be able to spell _prov) from the policy (a user may not declare one). It works the same ground as this issue from the other side: #1567 makes hidden attributes uniformly declarable, this one makes them readable, and both turn on where the visible/hidden line is drawn.

Worth designing together — a decision here about how an explicitly-named hidden attribute resolves will likely settle how #1567's registry is read back, and vice versa.