[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

The store records where each release comes from, a copy or a build. The new groups timenet local downloads and timenet local builds use the records to show and clear the copies and the builds apart.

Details

The store did not know a copy from a build. A copy is easy to get again from its registry, but a build can take hours or need credentials. So a timenet local command deleted either every release in the store or none.

• Each release in the store gets a record in .origin.json: copy or build, the registry or the backend package with its version, and the time in UTC. A URL in the record loses its user info, query and fragment.
• run_plan() records each copy and each build. TimeNet.download() records each release that it copies. copy_release() returns the releases that it copied or repaired.
• BaseRegistry.location names each registry in a record. The new InstalledBuilder.package names the package that builds a dataset.
• The new groups timenet local downloads and timenet local builds split the store by the record. A download is a copy from a registry. A build is a release that a connector built, or a release without a record. Each group has info and clear, and each of these commands takes --local-registry.
• info lists the releases of the group with the source and the size of each. The source is the registry of a download, or the package of a build.
• clear deletes the releases of the group, but it keeps each one that a remaining release needs. It walks from each release that stays to the releases that it needs: the payload sources of a release, and the parents of a build. So the rule holds in both directions. local downloads clear keeps a download that a build needs, and local builds clear keeps a build that a download needs. A warning names each kept release and a release that needs it.
• timenet local info shows the count and the size of the downloads and of the builds, instead of the whole store. timenet local clear still deletes every release, and its prompt counts the builds.
• In timenet.cache, clear_releases(store, *, kinds) takes the kinds to delete, "download", "build" or both. Its result also names each kept release as a KeptRelease. releases_to_clear() gives the same answer without a delete. StoredRelease.kind gives the ReleaseKind of a release.
• Size: about 1100 changed lines. The record and the clear read best as one change. The safety rules of the clear depend on the record.

Design decisions

• A release without a record counts as a build, so local downloads clear never deletes a release that TimeNet did not copy.
• The record stays in the store. It is not in the manifest, and a copy never carries it, so two stores hold the same release files.
• A damaged record, a damaged manifest of a release that stays, or a symlinked dataset or release folder stops the clear before it deletes anything. The walk cannot see a release behind a link, so it cannot know which releases that release needs.
• timenet local clear needs no walk, because every release goes. So a damaged manifest or a symlinked folder does not stop it.
• The clear computes what to delete again after the prompt, and it skips a release whose record changed. So a build that lands during the prompt keeps the copies that it needs.

Breaking changes

• clear_releases() takes kinds=, and ClearedReleases has the new field kept.
• timenet local info shows the downloads and the builds instead of the local registry.
• timenet.registry.base.ProgressCallback is now ByteCountCallback.
• StoredRelease has the new field origin. TimeNet measures its size at the first read.
• BaseRegistry.location is abstract, so a registry of another package must implement it.
• find_card() raises TimeNetBuildError if no installed package declares the entry point of the backend.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• 👉 feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.

Merge message

feat(cache)!: record where each release in the local registry comes from

Each release in the local registry gets a record in .origin.json that
says whether it is a copy from a registry or a build. A copy is easy to
get again, but a build can take hours or need credentials.

- Each copy and each build of a plan, and each copy of download(),
  writes the record. A URL in the record loses its user info, query and
  fragment.
- timenet local downloads and timenet local builds show and clear the
  two kinds apart. A release without a record counts as a build.
- clear keeps each release that a remaining release needs, in both
  directions, and a warning names it.

BREAKING CHANGE: BaseRegistry subclasses must implement location, which
names the registry in a record.

BREAKING CHANGE: timenet.registry.base.ProgressCallback becomes
ByteCountCallback.

BREAKING CHANGE: find_card() raises TimeNetBuildError if no installed
package declares the entry point of the backend.

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

No new blocking issues. 1 optional suggestion (a nit or a note on pre-existing code) was found and not posted. Nothing in this review needs a push before merging.