Repository navigation
feat(manifest)!: record TimeNet git provenance in each written dataset - #684
Merged
Merged
Conversation
karhu
force-pushed
the
feature/build-env-git-sha
branch
from
November 5, 2026 15:56
cf9bd6b to
d297a14
Compare
karhu
commented
Nov 6, 2026
karhu
marked this pull request as ready for review
November 6, 2026 15:25
RealLast
requested changes
Nov 8, 2026
RealLast
left a comment
RealLast
left a comment
Contributor
There was a problem hiding this comment.
Thanks! Would prefer something that is more declarative with types that own the interfaces and implemetations : )
karhu
force-pushed
the
feature/build-env-git-sha
branch
from
November 9, 2026 09:36
39272a7 to
c9999c2
Compare
karhu
commented
Nov 9, 2026
Contributor
Author
|
@RealLast Thanks for the review! Addressed your feedback. Please take another look! |
Contributor
Author
|
@RealLast Addressed the second round of feedback which we discussed on Slack. Please take another look at the PR stack! |
RealLast
approved these changes
Nov 10, 2026
RealLast
left a comment
RealLast
left a comment
Contributor
There was a problem hiding this comment.
Thank youu! Much cleaner, really appreciate it. Have some final nits on clarity and ambiguity, but feel free to address them as you go : )
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
A manifest now records which TimeNet code built a release. Before, a local checkout reported the same package version for every commit, so two builds from different code looked the same. `build_env.git` maps each installed TimeNet distribution to a `GitSource` with `sha` and `local_changes`. A local checkout asks git, and `local_changes` covers only the package directory, untracked files included. A wheel reads the commit from `_build_info.json`. `scripts/write_build_info.py` writes that file before `uv build`, and `make build` and the publish workflow run the script. A distribution with no known commit is absent from the map. `local_source_path` moves from the connector builder to `timenet.provenance`, so both use the same function. `test_a_built_version_starts_no_isolated_child` now stores its dataset before it blocks subprocesses, because a manifest write runs git. A reader without this change rejects a new manifest, because the manifest forbids unknown keys. This change lands before 1.0, so `timef_format_version` stays at 1. Part of TNET-165
Review findings on the first version: - A non-editable path install recorded the live checkout, not the installed copy. Only an editable install now asks git. Any other install reads the `_build_info.json` in the package. - A git install (`pip install git+https://...`) now records the commit in its `direct_url.json`. - `make build` left `_build_info.json` in the source tree, so a later plain `uv build` shipped an old commit. `scripts/build_dists.py` replaces `scripts/write_build_info.py`. It writes the file, runs `uv build`, and removes the file again, also on failure. - The publish workflow builds through the script. Its smoke step now makes sure that each artifact reports `$GITHUB_SHA` with no local changes. - git runs with `--no-optional-locks`, so a build never takes the index lock, and with a timeout. - A lookup that fails keeps its reason, so a later check can report it. A malformed `direct_url.json` or `_build_info.json`, or one with unknown keys, raises `TimeNetBuildError`. Tests: `packages/conftest.py` reports a clean commit to every test unless the test has the `real_git` marker. Test results no longer depend on the state of the checkout, and the core suite runs about 40% faster. The provenance tests go through `git_sources()` with fake installed distributions, and switch off the user's git configuration. Part of TNET-165
Drop the test that ran real git against this repository. Its one check, that build_env() carries the git block, moves into the editable-install test. Drop the invalid-sha case, which the manifest tests already cover, and keep the extra-key case, the only test of the forbid rule on GitSource. Remove a setup line that no assertion needed. Part of TNET-165
- Allow only 40-digit SHA-1 commit ids, because TimeNet uses SHA-1. - Shorten the docstrings of the provenance helpers and of `scripts/build_dists.py` to one line. Ruff now exempts a one-line docstring from the Returns and Raises sections (`ignore-one-line-docstrings`). - Rename `_validated` to `_validate_recorded_commit`. - `scripts/build_dists.py` registers each record for cleanup before it writes it, so a write that fails halfway leaves nothing behind. It compares the commit and `git status` of each package before and after `uv build`, and fails if a file changed during the build. - Simplify the build bullet in AGENTS.md. Part of TNET-165
The record sits at a known place, <import package>/_build_info.json, so look it up there instead of walking the distribution's RECORD. Part of TNET-165
Review feedback asked for types that own the provenance logic, instead of module functions and raw dicts. - `GitSource` is a strict model with `sha: GitSHA` and `contains_local_changes`, and it loads a wheel's build info itself. `GitSourceMap` names the `build_env.git` block. The manifest key `local_changes` is now `contains_local_changes`. - `TimeNetDistribution` owns one installed distribution: it lists the installed ones, finds the on-disk project directory, and reads the commit of the code it runs. The editable, git-install and wheel cases are its private methods. - `GitSnapshot` holds the commit of every installed distribution, read at one time. `GitSnapshot.read()` is the only call that runs git, and `recorded_sources()` gives the block a manifest records. - A private model validates PEP 610 `direct_url.json`, so the lookup reads named fields instead of dict keys. `builder/env.py`, the test stub, the publish workflow and `scripts/build_dists.py` use the new names. Part of TNET-165
PEP 610 lets a directory install name the repository in url and the project in subdirectory. The lookup ignored subdirectory, so it asked git about the whole repository and builder/env.py installed the repository root. Both now use the package directory. Part of TNET-165
It only renamed TimeNetDistribution.NAMES. The loop in _base_args reads the names directly, and its comment keeps the reason for the pin. Part of TNET-165
…nown Split the commit lookup into two layers. InstalledDistribution picks the InstallSource of one package: an editable checkout, a git install, or a built package. Each install source reads the commit in its own place. PackageSnapshot, GroupSnapshot and TimeNetSnapshot are typed views of the commits that a manifest records in build_env.git. TimeNetSnapshot is now the one list of the TimeNet distributions. The build environment and the test stub use it. An unknown commit now stops the build, also for a local registry. The error gives the reason: git is not on PATH, git failed, or a built package has no _build_info.json. UnknownGitSource is gone. BuildEnvironment is now a model with a capture constructor. The manifest requires it, because each manifest that the writer produces records it. Part of TNET-165
TimeNetSnapshot now derives directly from Snapshot. It owns read, from_source_map and to_source_map. The GroupSnapshot layer is gone. A validator makes each PackageSnapshot field hold the commit of its own distribution. The timenet field must hold timenet, and the timenet-connectors field must hold timenet-connectors. BuiltPackage now reads its _build_info.json file. GitSource only validates the JSON, so load_from_build_info is gone. The pytest marker real_git is now real_provenance, because it turns off the stub for every install type, wheels included. The autouse fixture is now _stub_installed_sources. Part of TNET-165
The snapshot and manifest layer now says package: PackageName, PackageSnapshot.package_name and TimeNetSnapshot.package_names(). "Distribution" also reads as a data distribution in TimeNet, and the classes are already called PackageSnapshot. InstalledDistribution and InstallSource keep the packaging term, because they wrap importlib.metadata.Distribution. Part of TNET-165
Part of TNET-165
The field holds a package name, not an importlib Distribution, so it now reads like every other name in the module. Only InstalledDistribution keeps the packaging term, because it wraps importlib.metadata.Distribution. Part of TNET-165
The docs for build_env.git and one test name still used "distribution" after the rename to "package". They now say "package". The TimeNetSnapshot docstring loses its second paragraph, because the one-line summary is enough.
…uild record An editable install read HEAD before git status. A commit between the two reads then looked like the old commit without local changes, so a build whose code changed could still pass the upload check. The lookup now reads the status first, so such a commit shows as a new commit. scripts/build_dists.py writes and removes the records of both packages. Two builds at the same time in one checkout could remove each other's records, and git status cannot see that, because git ignores the files. The script now makes sure that each record still holds what it wrote, and fails otherwise.
…heckout Review follow-ups for the provenance types: - EditableCheckout is now EditableInstall, and BuiltPackage is now WheelInstall, to match GitInstall. Their docstrings say why each reads its commit from a different place. - A private _GitCheckout class runs git. It owns the PATH lookup, the timeout and the error handling, and replaces _run_git. - The docstrings of InstallSource, PackageSnapshot and InstalledDistribution.from_name give examples and name the cases.
Main added tests that build a Manifest without build_env, call LocalRegistry.store(), or patch env.BASE_DISTRIBUTIONS. This branch makes build_env required and removes the other two names. The tests now pass a full build_env, call put_dataset(), and patch TimeNetSnapshot.package_names. Part of TNET-165
The writer read the TimeNet commit only when it wrote the manifest, after every data file. An unknown commit then failed after the whole write. A commit or checkout during the write also changed the record. The writer now reads the commit when it starts, and the manifest records that commit. Part of TNET-165
karhu
force-pushed
the
feature/build-env-git-sha
branch
from
November 10, 2026 18:58
f1bc8f6 to
db88bfc
Compare
build_dists.py removed the _build_info.json files only in a finally. A kill or a closed terminal ended the process before the finally ran. The files stayed, and git ignores them, so a later plain uv build packed a record of an old commit. SIGTERM and SIGHUP now raise SystemExit, so the finally removes the files. Only SIGKILL still leaves them. Part of TNET-165
karhu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A local checkout reported the same package version for every commit,
so two datasets written from different code looked the same.
and whether its directory had local changes.
direct_url.json, and a wheel reads the _build_info.json that
scripts/build_dists.py writes into it.
BREAKING CHANGE: A TimeNet install without git provenance fails every
dataset build and write with TimeNetBuildError.
BREAKING CHANGE: build_env with all its fields is required, so older
manifests do not load, and older clients reject new ones.
BREAKING CHANGE: BuildEnvironment is a model, not a TypedDict, and
timenet.provenance.build_env() is gone. Use BuildEnvironment.capture().