[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Three environment variables make the default list of sources, each with one meaning. The new group timenet local replaces timenet cache. timenet local cache clear deletes only the folders that TimeNet marked as its own.

Details

TIMENET_REGISTRY had two meanings: a URL was the registry after the store, and a local path was the store itself. Downloads went to the storage folder and builds to the registry folder, both under ~/.cache/timenet. Cleanup tools delete files there without asking, and a build can take hours to make again.

• TIMENET_LOCAL_REGISTRY (default ~/.timenet/registry) places the store. TIMENET_REMOTE_REGISTRY (default timenet://) names the registry after it. TIMENET_AUTO_BUILD (default true) puts the installed connectors last.
• default_sources() reads the variables at call time, and its arguments override them. default_local_registry() returns the store alone. The first entry of the list is always the store.
• open_registry(), RemoteRegistry and S3Registry take local_registry= instead of cache_dir=. A remote registry reads a complete copy in that local registry only for the same release. For the same release, the two manifests are equal apart from the build environment. Without local_registry=, it reads only its own files. default_sources() passes its store.
• TIMENET_CACHE (default ~/.cache/timenet) holds only files that a build or a read can make again: the raw downloads in downloads/<dataset_id> and the control databases in control/.
• settings() raises TimeFValidationError for an empty or blank value and for another invalid value. The message names the variable.
• When TimeNet creates the downloads or the control folder of the cache, it writes a CACHEDIR.TAG marker file into it. If one of these two folders holds the marker, timenet local cache clear deletes it. It deletes no other folder of the cache. A folder without the marker stays, and a warning names it.
• timenet local info shows the path and the size of the cache and of the store. timenet local cache info shows the cache alone. timenet local clear deletes the marked folders of the cache and each release in the store, but no other file. Each clear asks first, and --yes skips the question.
• After a build, TimeNet deletes the raw downloads of the dataset. If the downloads folder has no marker, they stay.
• If no output is given, timenet-build and timenet_connectors.build() write to the store. An empty or blank --out is an error, not the current directory.
• The test fixture now sets the new variables instead of TIMENET_HOME.
• Size: about 2090 changed lines. The variables, the cache and its safety rules read best as one change, because each rule protects a folder that a variable places.

Design decisions

• The store lies outside ~/.cache, because it holds builds. The cache holds only files that TimeNet can make again, so a cleanup tool can delete all of it.
• settings() rejects a cache that is the home folder or a folder above it. With TIMENET_CACHE=~ on macOS, the downloads folder of the cache is ~/Downloads, because the volume ignores letter case.
• settings() rejects a store and a cache that are the same folder or lie in each other. If the store and a folder that clear_cache() deletes overlap, it deletes nothing and raises an error.
• These checks compare folders by device and inode number, not by the text of the path. So a symlink or a name in other letter case cannot hide an overlap.
• download() finds the store in the list with the same identity check as settings(). So the two checks cannot disagree on a symlink or a case variant.
• The marker file proves that TimeNet created the folder. CACHEDIR.TAG is a standard marker, and backup tools skip the folders that hold it. TimeNet does not mark a folder that exists already, because that folder can belong to the user.
• The group is timenet local, not timenet cache, because it also shows and clears the store, and the store is not a cache. timenet local cache clear clears the cache alone, and timenet local clear clears both. So no clear needs an extra flag.
• settings() checks each variable on each call, also a variable that an argument overrides. So a wrong value in the shell fails at once, not later in another command.
• A remote registry reads a copy in the store only for the same release. Another build of the same version can have other files or another card.
• A remote registry without local_registry= reads no local copy. So a list of sources that the caller gives does not read TIMENET_LOCAL_REGISTRY.
• The parser of a registry value lives in the private module timenet._registry_uri, which imports only timenet.errors. So timenet.config can import it at the top without a circular import. timenet.registry still exports local_registry_path and TIMENET_REGISTRY_URL.

Breaking changes

• TIMENET_HOME, TIMENET_STORAGE and TIMENET_REGISTRY are gone. TIMENET_LOCAL_REGISTRY, TIMENET_REMOTE_REGISTRY and TIMENET_AUTO_BUILD replace them.
• Raw downloads lie in <TIMENET_CACHE>/downloads.
• default_registry_path() is gone.
• The timenet cache group and the --all flag of timenet cache clear are gone. The new group timenet local has info, clear, cache info and cache clear. timenet local cache clear deletes only the downloads and control folders that TimeNet marked.
• TimeNetSettings loses home, storage, registry, home_dir, storage_dir and registry_path.
• In timenet.cache, cached_datasets(), raw_cache_size() and CachedDataset are gone. stored_releases(), cache_size() and StoredRelease replace them. clear_cache(*, include_registry) is now clear_cache(cache_dir, *, local_registry). It deletes only the folders of the cache and returns a ClearedCache. The new clear_releases(store) deletes the releases in the store and returns a ClearedReleases.
• open_registry(), RemoteRegistry and S3Registry take local_registry= instead of cache_dir=. Without local_registry=, a remote registry reads no local copy. Before, it read the copies in the storage folder.
• A remote registry reads a copy in the store only for the same release. Another build of the same version does not count.
• BaseRegistry.local_root is gone. Use LocalRegistry.root.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• 👉 feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.
• The CLI flags for the three variables come in the PR on CLI flags.
• feat(cache)!: record where each release in the local registry comes from #674 adds timenet local downloads and timenet local builds, which tell a downloaded release from a build.
• timenet_connectors.build() and the store fallback of timenet-build are glue. The PRs on the producer API and on timenet-build --publish-to replace them.

Merge message

feat(config)!: make the default list of sources from three variables

TIMENET_LOCAL_REGISTRY, TIMENET_REMOTE_REGISTRY and TIMENET_AUTO_BUILD
make the default list of sources, each with one meaning. The local
registry moves to ~/.timenet/registry, outside the cache, because a
build can take hours to make again.

- TIMENET_CACHE holds only files that TimeNet can make again: the raw
  downloads and the control databases.
- timenet local cache clear deletes only the cache folders that hold
  the CACHEDIR.TAG marker of TimeNet. timenet local clear also deletes
  the releases in the local registry.
- settings() refuses a cache that is the home folder or above it, and
  a local registry and a cache that overlap. The checks compare device
  and inode, not path text.
- A remote registry reads a local copy only for the same release.

BREAKING CHANGE: TIMENET_HOME, TIMENET_STORAGE and TIMENET_REGISTRY
are gone. Raw downloads lie in <TIMENET_CACHE>/downloads.

BREAKING CHANGE: The timenet cache group becomes timenet local, with
info, clear, cache info and cache clear.

BREAKING CHANGE: open_registry(), RemoteRegistry and S3Registry take
local_registry= instead of cache_dir=. Without it, a remote registry
reads no local copy. Registries lose local_root.

BREAKING CHANGE: default_registry_path() and the TimeNetSettings
fields home, storage and registry are gone. In timenet.cache,
stored_releases(), cache_size() and clear_releases() replace
cached_datasets() and raw_cache_size(). clear_cache() takes
(cache_dir, *, local_registry).

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

No new blocking issues. 1 optional suggestion (a nit or a note on pre-existing code) was found and not posted. Nothing in this review needs a push before merging.

[claude]

⚠️ Code review skipped: your organization has reached its API usage limit, so Code Review can't run right now.

An organization admin can review the limit in your organization's usage and billing settings.

Once the limit resets or is raised, someone with write access to this repository can comment @claude review on this pull request to trigger a review.