Skip to content

Codec context argument, additive to key - #1556

Open
dimitri-yatsenko wants to merge 3 commits into
masterfrom
feat/codec-context
Open

dimitri-yatsenko wants to merge 3 commits into
masterfrom
feat/codec-context

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Closes #1550, scoped so that nothing breaks.

The problem

encode/decode overload one key dict with two unrelated things: primary key values, and connection context (_schema, _table, _field, _config). Two consequences, both in the issue:

  1. The underscore convention separating them is unenforced — it holds only because DataJoint's grammar disallows attribute names starting with _.
  2. _config is functionally required for correct store resolution in any multi-connection process, but arrives as an optional dict key an author must remember to read out and thread through. Forgetting doesn't fail loudly; it falls back to the global config and resolves a different store silently. That happened twice independently — dj-figpack-codecs#6 and dj-canvasxpress-codecs#3 — both following the SchemaCodec docstring.

Why this is not a breaking change

The issue proposes making key primary-key-only and config required. Both would break every existing codec, which does not fit a patch release on a line whose notes promise "no API breaks — every feature on the 2.3 line is additive" — and the 2.3.3 notes already told readers the opposite would happen: "the underscore key will keep working with a deprecation warning rather than changing under you."

Three things keep that promise:

  • context is passed only to codecs that declare it. table.py and decode_attribute introspect the signature first, reusing the mechanism already used for store_name (table.py:1432). A codec with the old signature is called exactly as before — not "works by accident", but never offered the argument at all.
  • The underscore keys stay in key, still populated. A codec reading key["_config"] directly keeps working, and can't be warned about anyway since that's a plain dict access.
  • _extract_context(key) still takes one argument. It warns only when it actually falls back to underscore keys — so a codec that passes context is silent, and one that never needed context is never nagged.

Verified against the real ecosystem

All four third-party codecs, checked by parsing their sources:

Package encode decode Passed context?
dj-figpack-codecs value, key, store_name stored, key no — called as before
dj-canvasxpress-codecs value, key, store_name stored, key no — called as before
dj-zarr-codecs value, key, store_name stored, key no — called as before
dj-photon-codecs value, key, store_name stored, key no — called as before

None needs changing.

What ships

  • context keyword on Codec.encode/decode and all five store-backed built-ins, carrying schema, table, field, config.
  • Codec._codec_config(key, context) replaces the hand-rolled (key or {}).get("_config") at eleven sites, preferring context and falling back to the legacy key.
  • _extract_context(key, context=None), with the deprecation warning on the fallback path.
  • SchemaCodec's docstring example — the one both third-party bugs were copied from — now shows the new signature and _codec_config.

Deliberately deferred to 2.4

key reverting to primary-key-only, and config becoming required on _build_path/_get_backend so that omitting it is a loud TypeError. That is the genuinely breaking half, and it wants the deprecation window this PR opens.

Verification

  • Full integration suite: 673 passed, 14 skipped, 0 failed across MySQL and PostgreSQL — every built-in codec exercised end to end.
  • 13 new unit tests pinning the non-breaking contract specifically: a legacy-signature codec resolving everything from key, a modern one from context, precedence between them, the warning firing on the fallback path and not firing when context is supplied or when only a plain primary key is passed, and that every built-in declares the argument.
  • One test in test_object.py called encode directly with a legacy key dict; it now uses context, so the suite exercises zero deprecated paths. That was the only such call site — every framework path already supplies context.
  • ruff, ruff-format, codespell, mypy pass via pre-commit.

@dimitri-yatsenko

Copy link
Copy Markdown
Member Author

On try/except versus introspection, raised in review — and a performance fix it surfaced (6f36ff2).

Catching TypeError would be simpler to write, but it is not safe here. An encode body serializes and uploads, so a TypeError raised inside it is indistinguishable at the call site from an unexpected-keyword error. Catching would mask the real failure, retry the call without context — resolving the global config and possibly a different store, which is the exact defect this PR exists to fix — and repeat the upload. Narrowing it by checking e.__traceback__.tb_next is None would distinguish the two, but that is subtler than reading the signature, not simpler.

The question did expose a real cost, though. inspect.signature measures 6.10 µs per call against 1.76 µs for the whole BlobCodec.encode it guards — 3.5× the work of the thing it protects — and it ran per attribute per row. The store_name check predates this PR and paid the same.

Now cached on the underlying function:

per call
inspect.signature 6.10 µs
cached lookup 0.06 µs

About 12 µs saved per encoded attribute across both checks, or roughly a second on a 100k-row insert carrying one blob.

Keyed on the unbound function (type(codec).encode), which is stable per class — a bound method is created fresh on every attribute access and would never hit the cache.

Re-verified: full integration suite 673 passed, 14 skipped, 0 failed; unit tests include a new one asserting the signature is not re-inspected after the first lookup.

`encode`/`decode` overloaded a single `key` dict with two unrelated things:
primary key values, and connection context (`_schema`, `_table`, `_field`,
`_config`). The underscore convention separating them is unenforced, and
`_config` -- functionally required for correct store resolution in any
multi-connection process -- arrived as an optional dict key a codec author had
to remember to read out and thread through by hand. Forgetting did not fail
loudly: it fell back to the global config and resolved a different store
silently. That happened twice independently, in dj-figpack-codecs#6 and
dj-canvasxpress-codecs#3, both following the SchemaCodec docstring.

Adds `context` as a separate keyword carrying schema, table, field and config,
leaving `key` to mean what it means everywhere else in DataJoint.

Nothing breaks:

- DataJoint passes `context` only to codecs whose signature declares it,
  reusing the introspection already used for `store_name`. A codec written
  before this keeps its old signature and is called exactly as before.
- The underscore keys stay in `key` and are still populated, so a codec
  reading `key["_config"]` directly keeps working.
- `_extract_context(key)` still accepts one argument. It warns only when it
  has to fall back to underscore keys, so a codec passing context is quiet and
  one that never needed context is never nagged.

Verified against all four third-party codecs in the ecosystem
(dj-figpack, dj-canvasxpress, dj-zarr, dj-photon): every one declares the old
signature, so none is passed `context` and none needs changing.

`Codec._codec_config(key, context)` replaces the hand-rolled
`(key or {}).get("_config")` at eleven sites across the built-ins, preferring
context and falling back to the legacy key.

The breaking half of #1550 -- `key` reverting to primary-key-only and `config`
becoming a required parameter on `_build_path`/`_get_backend` -- is deliberately
not done here. It belongs in 2.4, after the deprecation window this opens.
inspect.signature costs ~6.1 µs, against ~1.8 µs for the whole BlobCodec.encode
it guards, and it ran per attribute per row. Caching it on the underlying
function makes the lookup ~0.06 µs -- about 12 µs saved per encoded attribute
across the two checks, or roughly a second on a 100k-row insert carrying one
blob. The store_name check predates the context argument and paid this too.

Introspection rather than calling and catching TypeError: an encode body
serializes and uploads, so a TypeError raised inside it is indistinguishable at
the call site from an unexpected-keyword error. Catching would mask the real
failure, retry without context -- resolving the global config and possibly a
different store, the exact defect #1550 exists to fix -- and repeat the upload.

Keyed on the unbound function, which is stable per class, rather than a bound
method, which is created fresh on every attribute access.
Every place the library shows a codec now declares `context`, and the prose
says to declare it rather than describing it as something implementations
"may also accept". A codec written from today's documentation therefore
survives the 2.4 removal of the legacy key path without changing.

The compatibility guarantee is unchanged and stated where it belongs: the
underscore keys still work and are still populated, so codecs written before
2.3.4 keep running until 2.4.

Covers the module and class examples in codecs.py, the protocol sketches in
heading.py and spark.py, the builtin_codecs package example, and the
SchemaCodec guidance the two third-party config bugs were copied from.
@dimitri-yatsenko dimitri-yatsenko added the enhancement Indicates new improvements label Oct 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Indicates new improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codec encode/decode: separate connection context from primary key in the key argument

1 participant