[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Removes the values-backend concept from user facing APIs. The flag allows explicitly picking a format for payload storage (Zarr vs Parquet) when building a dataset. However this is something that should only be decided at the Connector level, ensuring that a built dataset always has the same format. Without it releases of the same dataset and version can have different content & layout, breaking the inheritance assumptions of referencing datasets.

The PR also sets minor ground work for the rest of the stack.

Details

• The --values-backend flag is removed, and so does the values_backend argument of build_one(), ConnectorBuilder.build(), env_spec() and run_isolated().
• The connector's values_backend attribute decides the format. It is the only way to choose the backend.
• The parent planner moves from timenet_connectors/builder/plan.py to timenet/coordinator/plan.py. It imports nothing from timenet_connectors, and a test makes sure of that.

Breaking changes

• timenet-build build --values-backend is gone. The values_backend argument of build_one(), ConnectorBuilder.build(), env_spec() and run_isolated() is gone too.
• run_pipeline() and publish_pipeline() no longer take values_backend. BaseConnector.values_backend is typed ValuesBackend and defaults to ValuesBackend.PARQUET.
• The module timenet_connectors.builder.plan is gone, with plan_parents(), prepare_parents() and ParentStep.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• 👉 refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.

Merge message

refactor(build)!: let the connector choose the values backend

The connector alone chooses the storage backend of its values, because
a format override would change the files of a release and break the
child releases that pin them.

- Every isolated build includes the Zarr extra, because the connector
  cannot be imported before its dependencies are installed.
- Parent planning moves into timenet and no longer depends on
  timenet_connectors.

BREAKING CHANGE: timenet-build build loses --values-backend, and
build_one(), ConnectorBuilder.build(), env_spec(), run_isolated(),
run_pipeline() and publish_pipeline() lose values_backend.

BREAKING CHANGE: BaseConnector.values_backend and the values_backend
argument of store_dataset() use the ValuesBackend type and default to
ValuesBackend.PARQUET.

BREAKING CHANGE: plan_parents(), prepare_parents() and ParentStep move
from timenet_connectors.builder.plan to timenet.coordinator.plan.

[TimoStoff]

Thanks this LGTM! SOme minor comments. One thing I did notice is that with the override gone, values_backend: str = "parquet"is now the source of turth for the backend, so a typo like "zar" is only caught at build time. Maybe a good opportunity now to make this a named type. ValuesBackend is a StrEnum already, so typing this as ValuesBackend = ValuesBackend.PARQUET so you could maybe recycle that one.

[karhu]

and re top lvl comment:
Yes done: BaseConnector.values_backend is now a ValuesBackend with the default ValuesBackend.PARQUET.

[RealLast]

LMGTM; I would really like for us to take this opportunity to move some of the functions which are "outside" into classes that own their functionality. I see many functions like_build → why not make them part of the/a class that get's built

[claude]

I reviewed this PR and didn't find any bugs. It is a breaking change to the public build API and CLI, so a human look would still be worthwhile. One small thing to check: the generated api-nav block in zensical.toml now has a timenet.storage entry, but no timenet.storage module exists in the package. The nav may be out of sync with scripts/gen_api_docs.py.

What was reviewed:

• The values_backend parameter is removed from every layer: CLI, ConnectorBuilder.build, build_one, run_isolated, env_spec, run_pipeline and publish_pipeline. No references to the removed option or argument are left in cli.py.
• _check_values_backend still rejects unknown backends before any files change. env_spec now always adds the timenet[zarr] extra, at the same version pin as before.
• timenet/coordinator/plan.py moves unchanged and imports only from timenet and the standard library, as the new import test expects.

I ran nothing (no tests and no docs generator), so I didn't confirm the nav entry against a fresh generator run.

Extended reasoning...

The change touches the builder CLI, the isolated-env builder, the engine pipelines, the BaseConnector type and the docs nav, across 15 files. It is mostly code removal plus a planner module that moves unchanged. It touches no auth, crypto or data-exposure code. I deferred because it removes several public arguments and a CLI flag, and moves a module, all of which break callers. The zensical.toml nav also lists a timenet.storage page with no matching module. Two human approvals exist, but they came before the latest pushed commits.