[karhu]

The Stack

This stack covers refactoring the internal logic of how releases are resolved during read, download & build, in particular also in the context of parent releases / parent datasets. On top of that it refactors the public TimeNet APIs and CLI interfaces around these actions, to streamline the behavior across them.

This PR

Splits out the upload part from the current locally-store-then-upload flow of releasing an in-memory dataset to a registry. This allows reusing it for uploads from a version already on disk.

Details

Today the remote and S3 registries can only compile a dataset and upload the result inside store().

• WritableRegistry.put_release(from_directory) publishes a complete prebuilt local release on the local, remote and S3 registries.
• Before anything is published, it reads manifest.json from the directory and makes sure that every declared file has its recorded size. It publishes only the declared files.
• store() becomes put_dataset(), so the two write methods pair up: put_dataset() takes an in-memory dataset, put_release() a release that is already on disk. put_release() only reads from_directory.
• put_dataset() and put_release() both return the DatasetRef of the stored release.
• The remote and S3 put_dataset() now compile into staging and call put_release(). The local registry and the writer share one helper, publish_directory() in timenet/_fs.py. It swaps a staged copy into place. If the swap fails, it moves the old copy back.
• timenet.manifest gets read_manifest() and parse_manifest(). Every registry reads a stored manifest through them. A manifest that cannot be read or does not parse raises TimeFFormatError, and the message names the file or the release.

Design decisions

• put_release() does not compare with a copy that the destination already holds. The copy function in the next PR decides whether a destination can receive a version.
• A bad directory fails the same way on every backend, before any upload. A missing or unreadable manifest.json raises TimeFFormatError. A missing file or a wrong size raises TimeNetFileError.

Breaking changes

• WritableRegistry has a new abstract method, put_release(). A subclass outside this repository must implement it.
• WritableRegistry.store() is now put_dataset() on every registry, with the same parameters. Rename each call.
• A stored manifest that does not parse raises TimeFFormatError, not a pydantic ValidationError.

Stack

• docs: update the docs for the build coordinator #683
• feat(hints)!: give every error one hint for the SDK and one for the CLI #698
• feat(cache)!: record where each release in the local registry comes from #674
• feat(client)!: list and search every source #673
• feat(client)!: replace get() with get_manifest() and get_card() #672
• feat(build)!: build into the local registry, publish with --publish-to #671
• feat!: add timenet.build() and publish() #670
• feat(cli)!: take --local-registry, --remote-registry and --auto-build #669
• feat(config)!: make the default list of sources from three variables #668
• test: isolate every test from the real home and guard it #667
• feat(client)!: download releases into the local registry #662
• feat(client)!: take an ordered list of sources #661
• feat(client)!: read releases through the coordinator #660
• feat(coordinator)!: plan and run reads through the list of sources #659
• feat(coordinator)!: plan parents through an ordered list of sources #658
• feat(sources)!: resolve a release through an ordered list of sources #657
• feat(build)!: run every build through the coordinator #655
• feat(build)!: move the build CLI and build() onto the coordinator #654
• feat(build)!: build one prepared release in the built-in backend #653
• feat(coordinator)!: plan and run builds through build_release #652
• refactor(coordinator)!: name the plan step type Step #651
• feat(registry)!: copy a release between registries #649
• 👉 feat(registry)!: publish a local release with put_release #648
• refactor(build)!: let the connector choose the values backend #646

Changes later in the stack:

• docs: update the docs for the build coordinator #683 holds the doc changes of this PR.

Merge message

feat(registry)!: publish a local release with put_release

The local, remote and S3 registries publish releases already on disk
through put_release(), so uploads no longer need an in-memory dataset.

- put_release() reads manifest.json and checks declared file sizes
  before publishing. It leaves the source directory unchanged.
- It does not compare the release with an existing destination copy.
- Remote and S3 put_dataset() stage a build and call put_release().
- Local writes share a directory swap that tries to restore the old
  copy if publishing fails.

BREAKING CHANGE: Registry store() becomes put_dataset() and returns a
DatasetRef instead of a version string. WritableRegistry subclasses
must also implement put_release().

BREAKING CHANGE: Invalid stored manifests raise TimeFFormatError
instead of pydantic.ValidationError.

[github-actions]

✅ CI checks passed

Check	Status
Quick checks	✅ Passed
Tests (Python 3.13)	✅ Passed
Tests (Python 3.11)	✅ Passed

_{Updated for 3719fb9 · run #8249}

[TimoStoff]

LGTM! THere are some nits, so please do look at them before landing :)

[TimoStoff]

Mostly opportunities to delete more! SO you should be happy

[RealLast]

LMGTM, thanks! A few nits I'd like to see addressed please on ownership of functionality

[RealLast]

As discussed, accepting to unblock : ) We can clean up later

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

• 🟡 packages/timenet/src/timenet/registry/writable.py — A caller of put_release can publish a release whose payload sources the destination lacks; it then fails on every open. put_dataset never publishes such a release. _read_release only checks manifest.files.all_files() sizes and never looks at manifest.payload_sources. The local, remote and S3 put_release then publish without the per-source check TimeFWriter(registry=self) runs. Fix: before publishing, have _read_release (or each put_release) check every entry of manifest.payload_sources against self, the way the writer does.

  Why this was flagged

  Trigger: a release directory built in another registry whose manifest records payload_sources (reused parent files), passed to LocalRegistry.put_release (local.py:503), S3Registry.put_release (s3.py:790) or RemoteRegistry.put_release (remote/registry.py:605). WritableRegistry._read_release (writable.py:967-971) only runs check_file_sizes on manifest.files.all_files(), so a missing parent does not stop the publish. The release is copied or uploaded and committed. Every later open then fails: locally open_local_release raises TimeNetMissingParentError or TimeNetDatasetNotFoundError (dependencies.py:101-103). Before this change the only write path was store(), where TimeFWriter(..., registry=self) (writer.py:146-153) checks each payload source in the target registry before anything is published. The PR's design note only rules out comparing against an existing copy of the same release, not checking parents.

  Verification: Severity: nit. WritableRegistry._read_release (writable.py) only runs read_manifest(directory / MANIFEST_FILE) and check_file_sizes(version, manifest.files.all_files(), owner=...). Nothing in _read_release reads manifest.payload_sources. Later opens go through attach_payload_sources (dependencies.py:134-138), which raises TimeNetMissingParentError for a missing parent. The only caller of put_release in the repo is LazyReadRegistry.put_dataset (lazy_read.py:72).

• 🟡 packages/timenet/src/timenet/registry/local.py — A caller that replaces a release with put_release can keep getting the old manifest from the same LocalRegistry, so the new files are read through stale records. put_release copies manifest.json with shutil.copy2, which keeps the source file's mtime. The manifest cache at local.py:103-105 only checks st_mtime, so a new manifest with the same mtime counts as a cache hit. Fix: drop the _manifest_cache entry for the reference after publish_directory, or copy the manifest without its metadata (shutil.copy), so every publish shows the new manifest.

  Why this was flagged

  The trigger is a LocalRegistry instance that read a release (which cached (mtime, manifest) at local.py:113) and then gets put_release(from_directory) for the same dataset@ version. The new manifest's source has the same mtime as the old one. This is common when releases come from archives with a fixed mtime (reproducible tarballs with SOURCE_DATE_EPOCH, rsync -a, cp -p). The copy loop in put_release at local.py:211-214 uses shutil.copy2, which keeps the source mtime. It never invalidates self._manifest_cache. The next get_manifest/open_version compares only path.stat().st_mtime (local.py:103-105) and returns the old cached manifest for the new release directory. Its file records (sizes, checksums, paths) then no longer match the files on disk. On the base branch the only local write path was the writer, which writes a new manifest with a current mtime, so the mtime check always caught a replacement.

  Verification: get_manifest (local.py:103-106) returns the cached manifest when cached[0] == mtime. The new put_release copies with shutil.copy2 at local.py:211-214, which keeps the source mtime, and never drops _manifest_cache[(dataset_id, version)]. On the base, every local write went through TimeFWriter and produced a fresh mtime.