[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Adds timenet.sources, which resolves a release through an ordered list of sources. The build and the client do not use it yet.

Details

A build and a read must find each release in one ordered list of sources: registries and the installed connectors (TNET-157). Today each path asks its own places in its own order. This PR adds the parts that answer the question "which source provides this release?". The build and the client do not use them yet.

• The new module timenet.sources defines ReleaseSource, the union BaseRegistry | InstalledConnectors. Each source has one method: resolve(dataset_id, version), where version is a Version, or None for the latest version.
• A source gives one of four answers, a ReleaseLookup. FoundInRegistry names the version and the registry that serves the release. FoundInConnector keeps the InstalledBuilder that the lookup read, so a caller does not read the card again. Found is the union of the two. NotFound means that the source does not know the release. NotServed means that the source lists the release but does not serve its files, for example a credentialed release on a non-local registry. NotServed.access_error() turns it into a TimeNetAccessError that names the access page of each card that is not open.
• resolve_release() parses the version once and asks the sources in order.
• Every registry is now a source. BaseRegistry.resolve() reads the manifest. If the registry requires open access (requires_open_access), it answers NotServed for a release whose effective access is not open. LocalRegistry.resolve() serves every release that it holds, and looks only for the manifest file of that release.
• InstalledConnectors() answers with the version that the card of the connector declares. For another version, it answers NotFound. A caller gets the declared version from builders.declared(), which raises TimeNetVersionUnavailableError.
• A remote registry that cannot be reached raises TimeNetRegistryError, which names the registry or its storage, not an httpx error. This includes a "latest" lookup, which asks every source. It also includes a network drop during the parallel download of a release. An S3 registry that cannot reach its bucket raises TimeNetRegistryError, which names the bucket, not a botocore error.
• FoundInRegistry and NotServed keep the manifest that the source read, so a caller does not fetch it again.

Design decisions

• An exact version comes from the first source that serves it. Without a version, the release is the newest version of the first source that serves the dataset. "latest" is the highest version that any source serves, and the order of the sources breaks a tie. So with @latest, a store that holds an old version does not hide a newer one.
• NotFound and NotServed fall through to the next source. Any other error stops the resolution, so a broken source is never skipped in silence.
• None and an empty string mean no version, and "latest" asks every source. split_ref() returns "latest" for org/name@latest, so the two requests stay apart. resolve_release() parses the version once, so no source parses it again.
• ReleaseSource is a union, not a protocol. So code that holds an answer narrows it with isinstance, and the types rule out any other kind of source. A new kind of source extends the union.
• A local miss does not read every manifest in the registry, because a read of a dataset that the store lacks must stay cheap.
• Limit: a source that answers "latest" with NotServed for its newest version hides its older open versions, because no registry lists the versions that it serves.
• Both packages now have a module named sources: timenet.sources and timenet_connectors.sources. For such a pair, the API docs generator names each page after its full module path. The URL of the connectors page changes.

Breaking changes

• A registry that cannot be reached raises TimeNetRegistryError, not an httpx error.
• An S3 registry that cannot reach its bucket raises TimeNetRegistryError, not a botocore error.
• split_ref() returns "latest" for org/name@latest, not None.
• The API page of timenet_connectors.sources moves from api/sources.md to api/timenet_connectors.sources.md.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• 👉 feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.
• The next PR moves the build planner onto an ordered list of sources, and a later one moves client reads onto it.

Merge message

feat(sources)!: resolve a release through an ordered list of sources

timenet.sources resolves a release through an ordered list of
registries and installed connectors, so that builds and reads can find
each release in one way. Nothing uses it yet.

- Each source answers resolve() with FoundInRegistry,
  FoundInConnector, NotFound or NotServed. NotServed means that the
  source lists the release but does not serve its files.
- An exact version comes from the first source that serves it. No
  version means the newest version of the first source that serves
  the dataset. "latest" is the highest version of any source.
- NotFound and NotServed fall through to the next source. Any other
  error stops the lookup, so a broken source is never skipped.

BREAKING CHANGE: A registry or S3 bucket that cannot be reached raises
TimeNetRegistryError instead of an httpx or botocore error.

BREAKING CHANGE: split_ref() returns "latest" for org/name@latest
instead of None.

BREAKING CHANGE: The API page of timenet_connectors.sources moves from
api/sources.md to api/timenet_connectors.sources.md.

[github-actions]

✅ CI checks passed

Check	Status
Quick checks	✅ Passed
Tests (Python 3.13)	✅ Passed
Tests (Python 3.11)	✅ Passed

_{Updated for 3117782 · run #8357}

[TimoStoff]

some comments

[TimoStoff]

Actually…..lgtm

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I checked whether the HTTP registry has the same body-streaming gap. It does not: in _http.py, stream_to keeps _reaching around the whole iter_bytes loop, and RemoteRegistry.download_version wraps all of download_version_files.

Extended reasoning...

I compared the HTTP path with the S3 finding. In RegistryHttpClient.stream_to, _reaching is entered in the same with statement as self._client.stream(...), so a transport error while reading the body is still wrapped. RemoteRegistry.download_version also wraps the parallel downloader in _reaching.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.