[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

TimeNet.list() and TimeNet.search() ask every source of the list and merge the rows, so they also show the store and the installed connectors.

Details

list() and search() asked one registry of the list: the first registry that is not the store. So they missed the releases in the store and the datasets that only an installed connector provides, while open() reads all of them.

• Both methods ask each source of the list, one after another, in order. A registry answers from its catalog. InstalledConnectors() gets list_datasets() and search() like a registry, so the client treats every source the same. Its rows are the cards of the installed connectors, sorted by dataset ID. If two packages claim an ID, the first entry point wins, as it does for a build.
• The rows merge by dataset ID at the highest version. If two sources list the same version, the earlier source wins.
• search() asks each source without a limit. A DatasetFilter in timenet.types holds the filters of one search. Its matches_card() filters the cards, and BaseRegistry.search(filter) takes it too. limit applies after the merge, and a negative limit raises TimeFValidationError.
• A card never matches task or time_series_spec, because both filters read the schema of a manifest. So with one of these filters, search() does not read the cards.
• A registry without a catalog, such as an S3 registry, adds nothing and gets a CatalogWarning. If no source has a catalog, list() and search() raise TimeNetNoCatalogError. A card that cannot be read is left out with a ConnectorCardWarning. An error of a source, for example a registry that cannot be reached, stops the call.
• _catalog() is gone. timenet list and timenet search keep their flags. Their help, and the line of timenet list for no rows, name the sources.
• Size: about 590 changed lines.

Design decisions

• The call stops on an error of a source and does not skip it, because a partial list looks complete. A broken card is only a warning, because one bad connector must not hide the other connectors.
• search() asks registries without a limit, because the limit must apply to the merged rows and not to each source. With a task or time_series_spec filter, a registry search can therefore cost more.
• The earlier source wins on equal versions, so the order of the list stays the one rule for every command.
• The card filters live in one function that both searches use, so a registry and a connector card cannot match in different ways.

Breaking changes

• TimeNet.list() and TimeNet.search() merge every source of the list, connector cards included, and keep one row per dataset at the highest version.
• BaseRegistry.search() takes a DatasetFilter instead of filter keywords.
• If no source has a catalog, list() and search() raise TimeNetNoCatalogError, not NotImplementedError.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• 👉 feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.

Merge message

feat(client)!: list and search every source

TimeNet.list() and TimeNet.search() ask every source of the list and
merge the rows. So they also show the local registry and the datasets
that only an installed connector provides, as open() reads them.

- InstalledConnectors() answers list_datasets() and search() like a
  registry, with the cards of the installed connectors.
- The rows merge by dataset ID at the highest version. For the same
  version, the earlier source wins.
- A DatasetFilter holds the filters of one search, for the client and
  for each registry. The limit applies after the merge.
- A source without a catalog adds nothing and gets a CatalogWarning. Any
  other error of a source stops the call.

BREAKING CHANGE: TimeNet.list() and TimeNet.search() merge every source
of the list and keep one row for each dataset, at its highest version.
If no source has a catalog, they raise TimeNetNoCatalogError.

BREAKING CHANGE: BaseRegistry.search() takes a DatasetFilter instead of
filter keywords. A negative limit raises TimeFValidationError instead of
ValueError.

[TimoStoff]

OK so accepting to unblock, but I have a design suggestion here I would like you to consider: https://github.com/OpenTSLM/TimeNet/pull/673/changes#r4511197286

Will send in chat too

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.