[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Adds timenet.registry.copy_release(), which copies a release between registries and never changes a published version.

Details

Builds, downloads and publishing all copy releases between registries. Other releases with inheritance pin the files of a published version, so a copy must not change it.

• timenet.registry.copy_release(reference, source, dest, *, with_payload_sources=True, progress=None) copies a release and the payload sources that its manifest lists into another registry. With with_payload_sources=False, it copies the release alone.
• BaseRegistry.download_release(reference, directory, ...) is public now. It takes the release as a DatasetRef. A failed download keeps the old copy.
• BaseRegistry.find_manifest() returns the manifest of a release. For a release that the registry does not hold, it returns None. LocalRegistry.damage() returns the error for the first missing or resized file of a stored release, or None.
• TimeNetVersionConflictError reports another build of the same version, with the release, the registries and the difference ("files" or "card") as attributes.
• The typed progress events StepStarted and StepFinished live in timenet.progress. copy_release() is their first sender.

Design decisions

• A published version cannot change. copy_release() compares the whole manifest of each release with timenet.manifest.same_release(), without the build environment. The file records (path, size and checksum) come first, then the card and the rest of the manifest. If the manifests are the same, nothing happens, or a local destination gets the release again from the source. If they differ, TimeNetVersionConflictError stops the copy before anything is written, and its message asks for a new version. An overwrite breaks every release that pins the old files.
• A release needs exactly the releases that its manifest lists as payload sources, never the payload sources of a payload source. A payload source is a release whose payload files the copied release reads, and the manifest lists every such release, at any depth. So a read in dest needs no more. Sources come first, and each release is copied once and alone.
• One walker, find_payload_sources() in timenet.registry.dependencies, applies this rule and the one access check. The planner and the download plan later in the stack use the same walker. A walk of one level cannot loop, so the copy needs no cycle check.
• A source can lack a payload source that the destination holds complete and with the pinned files. Then the copy uses the copy in the destination. A build can then copy a release from a registry that lacks a payload source that the store already holds or just built.
• Every check runs before the first copy. This includes access: BaseRegistry.requires_open_access is true by default and false on LocalRegistry. The copy refuses a release that is not open when dest, or the source that it fetches from, requires open access.
• Between two non-local registries, only the release that is missing is downloaded.
• The step events carry the release and the action, not a step object. Their shape stays the same while the step type of the coordinator grows in later PRs.

Breaking changes

• BaseRegistry._download_release(dataset_id, version, dest_dir, ...) is now download_release(reference, directory, ...). A registry of another package that overrides it must rename the override.
• TimeNetMissingParentError takes (missing, needed_by, searched, *, payload_source). Before, payload_source was the third positional argument.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• 👉 feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.
• From feat(coordinator)!: plan and run builds through build_release #652 on, the build side also sends the progress events.

Merge message

feat(registry)!: copy a release between registries

copy_release() copies a release between registries and keeps published
versions unchanged, because child releases pin their files.

- A release needs exactly the payload sources that its manifest lists.
  The copy takes each one once, before the release, and a complete copy
  in the destination can supply a missing one. find_payload_sources()
  applies this rule.
- The copy compares the whole manifest without the build environment,
  and it checks access before it copies anything. Another build of the
  same version raises TimeNetVersionConflictError.
- A complete copy is reused, a damaged local copy is fetched again, and
  a failed download keeps the old copy.
- Each copy sends StepStarted and StepFinished events.

BREAKING CHANGE: BaseRegistry._download_release() becomes
download_release(reference, directory, ...), with a DatasetRef in place
of dataset_id and version. Registry overrides use the new signature.

BREAKING CHANGE: TimeNetMissingParentError takes searched as its third
argument and payload_source as a keyword-only argument.

[github-actions]

✅ CI checks passed

Check	Status
Quick checks	✅ Passed
Tests (Python 3.13)	✅ Passed
Tests (Python 3.11)	✅ Passed

_{Updated for 3e30447 · run #8360}

[TimoStoff]

LGTM! Minor nits, but please fix them docstrings!

[RealLast]

Good direction, thank you! But please let's give it a bit more solid and declarative design; we are building a release system here and can have classes and well-defined interfaces to realize this

[RealLast]

As discussed, accepting to unblock : ) We can clean up later

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

⚠️ Code review skipped: your organization has reached its API usage limit, so Code Review can't run right now.

An organization admin can review the limit in your organization's usage and billing settings.

Once the limit resets or is raised, someone with write access to this repository can comment @claude review on this pull request to trigger a review.