Codec context argument, additive to key - #1556
dimitri-yatsenko wants to merge 3 commits into
Conversation
|
On try/except versus introspection, raised in review — and a performance fix it surfaced (6f36ff2). Catching The question did expose a real cost, though. Now cached on the underlying function:
About 12 µs saved per encoded attribute across both checks, or roughly a second on a 100k-row insert carrying one blob. Keyed on the unbound function ( Re-verified: full integration suite 673 passed, 14 skipped, 0 failed; unit tests include a new one asserting the signature is not re-inspected after the first lookup. |
`encode`/`decode` overloaded a single `key` dict with two unrelated things:
primary key values, and connection context (`_schema`, `_table`, `_field`,
`_config`). The underscore convention separating them is unenforced, and
`_config` -- functionally required for correct store resolution in any
multi-connection process -- arrived as an optional dict key a codec author had
to remember to read out and thread through by hand. Forgetting did not fail
loudly: it fell back to the global config and resolved a different store
silently. That happened twice independently, in dj-figpack-codecs#6 and
dj-canvasxpress-codecs#3, both following the SchemaCodec docstring.
Adds `context` as a separate keyword carrying schema, table, field and config,
leaving `key` to mean what it means everywhere else in DataJoint.
Nothing breaks:
- DataJoint passes `context` only to codecs whose signature declares it,
reusing the introspection already used for `store_name`. A codec written
before this keeps its old signature and is called exactly as before.
- The underscore keys stay in `key` and are still populated, so a codec
reading `key["_config"]` directly keeps working.
- `_extract_context(key)` still accepts one argument. It warns only when it
has to fall back to underscore keys, so a codec passing context is quiet and
one that never needed context is never nagged.
Verified against all four third-party codecs in the ecosystem
(dj-figpack, dj-canvasxpress, dj-zarr, dj-photon): every one declares the old
signature, so none is passed `context` and none needs changing.
`Codec._codec_config(key, context)` replaces the hand-rolled
`(key or {}).get("_config")` at eleven sites across the built-ins, preferring
context and falling back to the legacy key.
The breaking half of #1550 -- `key` reverting to primary-key-only and `config`
becoming a required parameter on `_build_path`/`_get_backend` -- is deliberately
not done here. It belongs in 2.4, after the deprecation window this opens.
inspect.signature costs ~6.1 µs, against ~1.8 µs for the whole BlobCodec.encode it guards, and it ran per attribute per row. Caching it on the underlying function makes the lookup ~0.06 µs -- about 12 µs saved per encoded attribute across the two checks, or roughly a second on a 100k-row insert carrying one blob. The store_name check predates the context argument and paid this too. Introspection rather than calling and catching TypeError: an encode body serializes and uploads, so a TypeError raised inside it is indistinguishable at the call site from an unexpected-keyword error. Catching would mask the real failure, retry without context -- resolving the global config and possibly a different store, the exact defect #1550 exists to fix -- and repeat the upload. Keyed on the unbound function, which is stable per class, rather than a bound method, which is created fresh on every attribute access.
Every place the library shows a codec now declares `context`, and the prose says to declare it rather than describing it as something implementations "may also accept". A codec written from today's documentation therefore survives the 2.4 removal of the legacy key path without changing. The compatibility guarantee is unchanged and stated where it belongs: the underscore keys still work and are still populated, so codecs written before 2.3.4 keep running until 2.4. Covers the module and class examples in codecs.py, the protocol sketches in heading.py and spark.py, the builtin_codecs package example, and the SchemaCodec guidance the two third-party config bugs were copied from.
c133198 to
65b5497
Compare
Closes #1550, scoped so that nothing breaks.
The problem
encode/decodeoverload onekeydict with two unrelated things: primary key values, and connection context (_schema,_table,_field,_config). Two consequences, both in the issue:_._configis functionally required for correct store resolution in any multi-connection process, but arrives as an optional dict key an author must remember to read out and thread through. Forgetting doesn't fail loudly; it falls back to the global config and resolves a different store silently. That happened twice independently —dj-figpack-codecs#6anddj-canvasxpress-codecs#3— both following theSchemaCodecdocstring.Why this is not a breaking change
The issue proposes making
keyprimary-key-only andconfigrequired. Both would break every existing codec, which does not fit a patch release on a line whose notes promise "no API breaks — every feature on the 2.3 line is additive" — and the 2.3.3 notes already told readers the opposite would happen: "the underscore key will keep working with a deprecation warning rather than changing under you."Three things keep that promise:
contextis passed only to codecs that declare it.table.pyanddecode_attributeintrospect the signature first, reusing the mechanism already used forstore_name(table.py:1432). A codec with the old signature is called exactly as before — not "works by accident", but never offered the argument at all.key, still populated. A codec readingkey["_config"]directly keeps working, and can't be warned about anyway since that's a plain dict access._extract_context(key)still takes one argument. It warns only when it actually falls back to underscore keys — so a codec that passescontextis silent, and one that never needed context is never nagged.Verified against the real ecosystem
All four third-party codecs, checked by parsing their sources:
encodedecodecontext?value, key, store_namestored, keyvalue, key, store_namestored, keyvalue, key, store_namestored, keyvalue, key, store_namestored, keyNone needs changing.
What ships
contextkeyword onCodec.encode/decodeand all five store-backed built-ins, carryingschema,table,field,config.Codec._codec_config(key, context)replaces the hand-rolled(key or {}).get("_config")at eleven sites, preferringcontextand falling back to the legacy key._extract_context(key, context=None), with the deprecation warning on the fallback path.SchemaCodec's docstring example — the one both third-party bugs were copied from — now shows the new signature and_codec_config.Deliberately deferred to 2.4
keyreverting to primary-key-only, andconfigbecoming required on_build_path/_get_backendso that omitting it is a loudTypeError. That is the genuinely breaking half, and it wants the deprecation window this PR opens.Verification
key, a modern one fromcontext, precedence between them, the warning firing on the fallback path and not firing when context is supplied or when only a plain primary key is passed, and that every built-in declares the argument.test_object.pycalledencodedirectly with a legacy key dict; it now usescontext, so the suite exercises zero deprecated paths. That was the only such call site — every framework path already supplies context.ruff,ruff-format,codespell,mypypass via pre-commit.