[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

download() copies a release into the store, through copy_release(), instead of into a separate storage folder.

Details

#661 gives the client a store, the first LocalRegistry of the list without read_only=True, but download() still copied into TIMENET_STORAGE as glue. This PR moves it into the store and makes it the way to repair a damaged copy.

• download() resolves the release like open(). If the store holds a complete copy of the release and of each payload source that it needs, it returns the release directory at once.
• Otherwise plan_download() in the coordinator plans the copies, and run_plan() runs them with copy_release(). The release comes from the first registry other than the store that provides the exact version. A payload source that this registry lacks comes from the first registry that provides it. So a release and its payload sources can come from different registries. The copy brings along each payload source that the store lacks, and it repairs a damaged copy in the store. A release needs only the payload sources that its manifest lists (the rule of feat(registry)!: copy a release between registries #649), so a payload source comes without its own payload sources.
• copy_release() and CopyStep take force=True, which copies the requested release again into a local registry. For each release that it copies into a local registry, it sends BytesMoved events, so a build that copies a parent sends them too.
• A read of a damaged copy in the store gets a hint that repairs it. If another registry provides the release, the hint names download('<release>'). If only the installed connectors provide it, the hint names the timenet-build command that builds it again.
• timenet download shows its progress with progress_renderer(), as timenet-build build does: a status line for each release, and a progress bar in a terminal. If there is nothing to copy, it prints Nothing to do: '<id>' <version> is already in <store>. It prints the release directory on stdout.
• BaseRegistry.download_version() goes away. TimeNet.download() and copy_release() replace it.
• Size: about 1210 changed lines. The download, its repair rules and their tests read best as one change.

Design decisions

• download() never builds. For a release that only InstalledConnectors() provides, it raises an error that names open() and load(). An installed connector can declare a newer version than any registry holds. Then download('org/name@latest') fails with the default list.
• A registry over the directory of the store counts as the store, so the store is never copied onto itself. The planner and download() share that check. The check compares the folders on disk. So a symlink to the store also counts as the store. On a volume that ignores case, so does a path in another case.
• A non-local registry refuses force=True, because a published version never changes. A different set of files or another card still raises TimeNetVersionConflictError.
• A damaged copy outside the store gets no repair hint, because download() writes only into the store.

Breaking changes

• download() copies into the store through copy_release(), and refuses a store copy with other files or another card.
• download() copies each release from the first registry that provides it, so a release and its payload sources can come from different registries.
• Without a version, download() takes the newest version of the first source that serves the dataset. With @latest, it asks every source in the list, so each registry must be reachable.
• download() takes progress= events instead of progress_cb=.
• The separate download storage folder is no longer used by download().
• BaseRegistry.download_version() is gone. Use TimeNet.download() or copy_release().

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• 👉 feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.
• default_sources(), _catalog() for get(), list() and search(), the --registry mapping, timenet_connectors.load() and TIMENET_STORAGE as the cache of remote registries are thin glue. feat(config)!: make the default list of sources from three variables #668 and feat(cli)!: take --local-registry, --remote-registry and --auto-build #669 replace default_sources(), the --registry mapping and the cache of remote registries. feat!: add timenet.build() and publish() #670 removes timenet_connectors.load(). feat(client)!: replace get() with get_manifest() and get_card() #672 and feat(client)!: list and search every source #673 replace _catalog().

Merge message

feat(client)!: download releases into the local registry

download() copies a release into the writable local registry through
copy_release(), instead of into a separate storage folder. It also
repairs a damaged copy there.

- The release comes from the first registry other than the local
  registry that provides it. Each missing payload source comes from
  the first registry that provides it.
- download() never builds. For a release that only the installed
  connectors provide, the error names open() and load().
- copy_release() takes force=True, which copies a release again into
  a local registry. A non-local registry refuses it.
- A read of a damaged local copy gets a hint that repairs it.

BREAKING CHANGE: download() writes into the local registry, takes
progress= instead of progress_cb=, and refuses a local copy with other
files or another card.

BREAKING CHANGE: BaseRegistry.download_version() is gone. Use
TimeNet.download() or copy_release().

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

No new blocking issues. 1 optional suggestion (a nit or a note on pre-existing code) was found and not posted. Nothing in this review needs a push before merging.