[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Adds a coordinator for reads, with plan_read() and read_release(). The next PR moves the client onto it.

Details

A build plans and runs through one coordinator since #652, and it gets its parents from an ordered list of sources since #658 (TNET-157). A read still has its own code path in the client. This PR gives reads a coordinator of their own. The next PR moves the client onto it.

• timenet.coordinator.read has plan_read() and read_release(). plan_read() returns the plan of a read without running it. read_release() runs the plan and opens the release with its payload sources.
• A read opens a release where it is. The first source that provides the release gives a ReadStep, and the read copies nothing. Each payload source resolves on its own through the same list, at its exact version. So a release and its payload sources can come from different registries.
• If only InstalledConnectors() provides a release, the plan builds it into the store, as a build does. A copy or a build of a release replaces a read of it, and the read then opens the release from the store. This also holds when a build in the same plan needs a release that the read found in a registry: the release goes into the store, and the read opens it there.
• Reads and builds share one planner. A read is a fourth outcome of _Graph.visit(), next to no step, a copy and a build. visit() takes needed_in_local_registry. It is true for builds and downloads, which need a complete copy in the store, and false for reads, where a source that serves the release is enough.
• ReadStep(release, registry, depends_on) joins CopyStep and BuildStep in the Step union. run_plan() has an arm for it that runs nothing.
• BaseRegistry gets open_release(), which opens one exact release without its payload sources. It can take the manifest from the caller. So a remote read fetches each manifest once.
• open_version() goes away from every registry, and so does dependencies.open_cached_or_remote(). open_release() and download_release() handle one release without its payload sources, and copy_release() copies a release with them. A read of a release with its payload sources goes through the read coordinator.
• The read and the builds in its plan share one walk, with one set of answers and manifests.
• Size: This PR has about 1490 changed lines: about 740 in the source and 750 in tests. The cache of answers and manifests moves out of the build planner into one class that builds and reads share.

Design decisions

• Before any step runs, the plan checks each file size of each release that it reads from a local registry. A damaged copy in the store stops the read. The read never falls through to a later source, and it never builds over the damage. The check runs after the walk, so a copy that a build in the plan needs can first replace a damaged copy that the read found.
• A read can need one release twice. For example, a child reads P1 and P2, and the card of P1 declares P2. Each release is one node of the plan, with the pins of each reader. If two readers pin one file differently, no build can match both, and the plan raises TimeFFormatError.
• A cycle of payload sources through the requested release names that release first.
• The plan checks the pins of a payload source that the store already holds before any step runs.
• The plan builds a release with the InstalledBuilder that its lookup read, so it reads the card of each connector once.

Breaking changes

• BaseRegistry has a new abstract method, open_release(). A registry of another package must implement it.
• The abstract LazyReadRegistry._open_remote_version() takes an optional manifest.
• BaseRegistry.open_version() is gone from every registry. To read a release with its payload sources from one registry, use the client, for example TimeNet(registry).open(). From feat(client)!: take an ordered list of sources #661 on, the call is TimeNet(sources=[registry]).open().

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• 👉 feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.
• The next PR moves open(), load() and open_torch() onto read_release().

Merge message

feat(coordinator)!: plan and run reads through the list of sources

plan_read() and read_release() plan and run a read with the planner of
the builds. A read opens each release where the first source provides it
and copies nothing.

- Each payload source resolves on its own at its exact version, so a
  release and its payload sources can come from different registries.
- If only the installed connectors provide a release, the plan builds it
  into the local registry and the read opens it there.
- A damaged copy in the local registry stops the read, unless a copy in
  the same plan replaces it. The read never builds over the damage.
- BaseRegistry.open_release() opens one exact release without its
  payload sources.

BREAKING CHANGE: BaseRegistry gets the abstract open_release(), and
LazyReadRegistry._open_remote_version() takes an optional manifest.

BREAKING CHANGE: BaseRegistry.open_version() is gone. Read through the
client. open_cached_or_remote() becomes open_cached_or_remote_release()
and attaches no payload sources.

[TimoStoff]

alll greeen!

[karhu]

re top level comment:
Agreed, and done that way. _ReadPlanner is gone with its merge logic.
A read is now a fourth outcome, next to no step, a copy and a build.
Instead of need: Literal["read", "build"], visit() takes needed_in_local_registry: bool, because a download needs a local copy just like a build.

[claude]

One blocking issue that is not tied to a single line is posted as a comment on its whole file.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

Code review found no issues

No high-confidence issues detected in this change.