From 607593c2024bc50e671d329d56f13ccafd5d026d Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Sat, 7 Nov 2026 15:41:52 +0100 Subject: [PATCH 01/28] docs: update the docs for the build coordinator Update the user docs, the CLI pages and the agent skills for the changes of this stack. The pages describe the code as it is at the top of the stack: - A client reads through an ordered list of sources: the store, a remote registry and the installed connectors. A remote registry reads the complete copy of the same release in the store. - download() copies a release and its payload sources into the store. It never builds. - timenet-build builds into the store, and --publish-to publishes the release after the build. timenet.build(), plan_build() and publish() do the same in Python. - The timenet local commands show and clear the cache, the downloads and the builds in the store. - Each TimeNet error has a hint in the words of the SDK or the CLI. A registry flag that names no registry exits with code 2. - The examples take TimeNet, LocalRegistry and InstalledConnectors from the timenet package. This PR holds every doc change of the stack, so the docs get one review. Part of TNET-157. --- .agents/skills/add-dataset-connector/SKILL.md | 4 +- .agents/skills/timenet-datasets/SKILL.md | 114 ++++++-- README.md | 12 +- docs/architecture.md | 37 +-- docs/assets/architecture.svg | 2 +- docs/build.md | 179 +++++------- docs/cli/build.md | 82 ++++-- docs/cli/index.md | 18 +- docs/cli/timenet.md | 72 +++-- docs/client.md | 263 ++++++++++++------ docs/connectors.md | 24 +- docs/data-model/datasets.md | 8 +- docs/get-started.md | 17 +- docs/manifest.md | 6 +- docs/registry.md | 234 ++++++++++------ docs/timef-reader.md | 43 +-- docs/timef-writer.md | 3 +- docs/types.md | 13 +- docs/usage.md | 22 +- .../datasets/tsaqa/tsaqa_benchmark/README.md | 7 +- 20 files changed, 709 insertions(+), 451 deletions(-) diff --git a/.agents/skills/add-dataset-connector/SKILL.md b/.agents/skills/add-dataset-connector/SKILL.md index c100ef64f..bd66e3436 100644 --- a/.agents/skills/add-dataset-connector/SKILL.md +++ b/.agents/skills/add-dataset-connector/SKILL.md @@ -99,8 +99,8 @@ not by an id. A span takes Signal objects, for example in some docs are **not implemented**, so don't rely on them. - If you added or changed a `requirements.txt`, re-run `make sync` so the new library lands in your own environment. Otherwise `ty` reports your lazy import as unresolved and the connector test can't run. -- Round-trip end to end: `uv run timenet-build build --out `, then - `with TimeNet(registry="").open("") as reader: reader.read().describe()`. +- Round-trip end to end: `uv run timenet-build build --local-registry `, then + `with timenet.TimeNet(sources=[""]).open("") as reader: reader.read().describe()`. - Run `make check` and `make test`, and state which checks you ran (per AGENTS.md). ## Further reading diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index be2614649..0ba761233 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -1,13 +1,12 @@ --- name: timenet-datasets -description: Use when finding, searching, inspecting, downloading, or loading TimeNet datasets, via the `timenet` CLI or the Python `TimeNet` client (list, search, info, download, open, load, open_torch). Covers dataset ids, version pinning, and registry/storage configuration. +description: Use when finding, searching, inspecting, downloading, or loading TimeNet datasets, via the `timenet` CLI or the Python `TimeNet` client (list, search, info, download, open, load, open_torch). Covers dataset ids, version pinning, the list of sources, and the settings of the local and remote registries and the cache. --- # Using TimeNet datasets -TimeNet's consumer side is the `timenet` CLI and the `timenet.client.TimeNet` SDK. Both browse a -**registry** (the catalog of built datasets) and fetch dataset versions into local storage. The CLI -is a thin mirror of the SDK, so anything below works the same either way. +TimeNet's consumer side is the `timenet` CLI and the `timenet.TimeNet` SDK. Both find, read and +download TimeNet datasets: they read dataset versions from local and remote **registries** (catalogues of built datasets). A local registry can also serve to store locally built or downloaded datasets. The CLI is a thin mirror of the SDK, so anything below works the same either way. Producing datasets (writing connectors, running `timenet-build`) is a separate concern. For that, use the `add-dataset-connector` skill. @@ -26,12 +25,20 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | Command | What it does | | --- | --- | -| `timenet list` | Table of every dataset in the registry (id, name, license, domains). | -| `timenet search [filters]` | Filter datasets. See flags below. | -| `timenet info [version]` | Manifest summary: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). | -| `timenet download [version]` | Fetch a version's parquet to local storage; prints the target directory to stdout. | -| `timenet cache info` | Cached datasets with sizes, plus the raw download cache size. | -| `timenet cache clear` | Delete downloads and the raw cache. `--all` also removes the local registry; `-y`/`--yes` skips the prompt. | +| `timenet list` | Table of every dataset of the sources at its highest version, connector cards included (id, name, license, domains). | +| `timenet search [filters]` | Filter the datasets of the sources, connector cards included. See flags below. | +| `timenet info [@version]` | Manifest summary of the release that `open()` reads: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). A dataset that only a connector provides shows its card and the build command. | +| `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory to stdout. Never builds. Prints "Nothing to do" if the local registry holds everything. | +| `timenet publish [@version] --dest URL [--source URL]` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | +| `timenet local info` | Path and size of the cache, then one line each for the downloads and the builds in the local registry (count and size). | +| `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | +| `timenet local cache info` / `clear [--yes]` | Path and size of the cache (`$TIMENET_CACHE`), or delete its raw source files (`downloads/`) and control databases (`control/`) that carry TimeNet's `CACHEDIR.TAG`. | +| `timenet local downloads info` / `clear [--yes]` | List the downloads in the local registry (dataset, version, source, size, total), or delete them. | +| `timenet local builds info` / `clear [--yes]` | List the builds in the local registry (dataset, version, source, size, total), or delete them. | + +A download is a release whose provenance record says `copy`. A build is a release whose record says +`build`, or a release without a record. Releases can depend on each other (parent-child relationships). In that case `downloads clear` and `builds clear` might keep releases that are dependet on, +and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. Do not run a clear while a build or a download runs. `search` flags (repeat a flag to pass several values; all are ANDed across kinds): @@ -45,26 +52,67 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. - `--tag` - `--limit` (default 100) -Global: `-q`/`--quiet` suppresses status output. `-r`/`--registry` picks the registry (else -`$TIMENET_REGISTRY`, else the local default). `download` takes `--storage` (else `$TIMENET_STORAGE`). +Global: `-q`/`--quiet` suppresses status output; it goes before the command. + +Sources: `list`, `search`, `info` and `download` take three flags after the command. Each flag wins +over its environment variable, and the variable wins over its default. A registry flag that names no registry (a blank value, an unknown scheme, a URL for `--local-registry`) exits with code 2 and names the +flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. + +- `--local-registry PATH`: a local registry, the first source that a lookup checks, and the place + where downloads and builds land (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). +- `--remote-registry URL`: a remote registry, as a URL or a local path, the second source that a + lookup checks, if the local registry does not have the dataset (`$TIMENET_REMOTE_REGISTRY`, else + `timenet://`). +- `--auto-build`/`--no-auto-build`: whether installed connectors build a dataset that neither the + local nor the remote registry has (`$TIMENET_AUTO_BUILD`, else true). + +`list` and `search` merge every source, `info` resolves like `open()`, and `download` copies into +the local registry. Of these three flags, `publish` and the `local` commands other than `local cache` take +only `--local-registry`. The `local cache` commands take none of them. ## Python API ```python -from timenet.client import TimeNet +import timenet -tn = TimeNet() # registry: arg > $TIMENET_REGISTRY > local default +tn = timenet.TimeNet() # the default list of sources, see below tn.list() # list[DatasetMetadata] -tn.get("chengsenwang/tsqa") # Manifest (add a version or use id@version) +tn.get_manifest("chengsenwang/tsqa") # Manifest (add a version or use id@version) tn.search(domain=..., task=AnswerTask, tag="ecg", limit=50) # filters are scalar-or-list, ANDed -path = tn.download("chengsenwang/tsqa") # -> local /// dir +path = tn.download("chengsenwang/tsqa") # -> /// dir with tn.open("chengsenwang/tsqa") as reader: # TimeFReader; values stay in storage dataset = reader.read() dataset = tn.load("chengsenwang/tsqa") # every value read into memory ``` -- `TimeNet(registry=None, *, storage_path=None)`. `registry` accepts a `BaseRegistry`, a URL, a - `file://` URI, or a local path. +- `timenet.TimeNet(sources=None)`. `sources` is an ordered list. Each item is a registry object, a + URL, a `file://` URI, a local path, or `timenet.InstalledConnectors()`. Pass a list even for one + registry: `timenet.TimeNet(sources=[".timenet-registry"])`. An exact version comes from the + first source that has it; "latest" is the highest version of any source. +- One local registry in the list can declare `timenet.LocalRegistry(path, store=True)`. Builds land there. A + list with `InstalledConnectors()` needs a local registry with `store=True`, and only such a list builds on a read. A registry + URL in the list reads the complete copy of the same release in that local registry. Without one + in the list, it reads only its own files. +- `TimeNet()` uses `timenet.client.default_sources()`: the local registry `$TIMENET_LOCAL_REGISTRY`, then + `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. The + first entry is always the registry configured with `store=True`; `default_store()` returns it alone. +- `list()` and `search()` merge every source of the list, connector cards included: one row per + dataset at the highest version, sorted by ID. `limit` applies after the merge. A connector never + matches `task` or `time_series_spec`. A registry without a catalog (S3) adds nothing, and a card + that cannot be read is left out with a `ConnectorCardWarning`. To ask one registry alone, use + `open_registry("timenet://").search(...)` from `timenet.registry`. +- `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other + files. A registry that lists a release without serving it gives its listed manifest. A release + that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). + `get_card()` returns the card, also from a connector. `plan_read()` returns the plan of `open()` + (`timenet.Plan`, one step per release with its action and source) and runs nothing. +- `download()` needs a local registry configured with `store=True`. It resolves like `open()`. The release and each payload source that + the local registry lacks or holds damaged come from the first registry other than the local registry that provides + it, so they can come from different registries. If the local registry holds a complete copy of the + release, only the missing payload sources are copied. It never builds: a release that only a + connector provides raises `TimeNetNotBuiltError`, whose hint names `timenet.build()`. + `force=True` copies again, and `progress=` takes the events of `timenet.progress`. The hint of a + read of a damaged copy names the `download()` or the build that replaces it. - `open(...)` returns a `TimeFReader`; close it or use it in a `with` block. `reader.read()` returns a `TimeFDataset` whose values load only when asked: `record.signals[i].to_numpy()` / `.to_arrow()`. `reader.get(AnswerTask, task_id)` reads one object by its int ID. @@ -78,12 +126,12 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int ## End-to-end recipe ```python -from timenet.client import TimeNet +import timenet from timenet.types import AnswerTask -tn = TimeNet() +tn = timenet.TimeNet() hits = tn.search(task=AnswerTask, limit=10) # find candidates -tn.get(hits[0].dataset_id) # inspect the manifest +tn.get_manifest(hits[0].dataset_id) # inspect the manifest with tn.open(hits[0].dataset_id) as reader: # values stay in storage ds = reader.read() ds.describe() # identity, counts, per-spec columns, preview @@ -95,14 +143,22 @@ CLI equivalent: `uv run timenet search --task answer` then `timenet info ` t ## Configuration -All local state lives under `TIMENET_HOME` (default `~/.cache/timenet`). Env vars, prefix `TIMENET_`: - -- `TIMENET_HOME` relocates everything. -- `TIMENET_REGISTRY` selects the registry (URL or path). -- `TIMENET_STORAGE` where downloads are cached (default `/storage`). -- `TIMENET_CACHE` raw source / Hub download cache (default `/cache`). - -Precedence for any value: explicit argument or CLI flag, then env var, then default. +Env vars, prefix `TIMENET_`. Each sets one thing; none holds a list: + +- `TIMENET_LOCAL_REGISTRY` the first source that a lookup checks, and the place where + downloads and builds land (default `~/.timenet/registry`). +- `TIMENET_REMOTE_REGISTRY` a remote registry, as a URL or a local path: the second source that a + lookup checks, if the local registry does not have the dataset (default `timenet://`). +- `TIMENET_AUTO_BUILD` whether the default list ends with `InstalledConnectors()` (default `true`). +- `TIMENET_CACHE` raw downloads of builds in `downloads/` and control databases in `control/` + (default `~/.cache/timenet`). +- `TIMENET_TOKEN` bearer token of a remote registry; `TIMENET_ISOLATION` (default `on`). + +Precedence for any value: CLI flag, then env var, then default. An empty value is an error, and so +is a cache directory that overlaps the local registry or is the home folder or a folder above it. In the +CLI, a registry flag that names no registry exits with code 2, and a bad variable with code 1. A +`sources` list that you pass ignores `TIMENET_LOCAL_REGISTRY`, `TIMENET_REMOTE_REGISTRY` and +`TIMENET_AUTO_BUILD`. ## Further reading diff --git a/README.md b/README.md index 3146063ba..f38d5df93 100644 --- a/README.md +++ b/README.md @@ -22,22 +22,24 @@ TimeNet needs Python 3.11 or newer. ```bash uv add 'timenet[cli]' uv tool install timenet-connectors -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -Load the local build with Python: +The build lands in the local registry at `~/.timenet/registry/timenet/hello-world/1.0.0/`. Load it with Python: ```python -from timenet.client import TimeNet +import timenet +from timenet import TimeNet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +dataset = TimeNet().load("timenet/hello-world@1.0.0") dataset.describe() signal = dataset.records[0].signals[0] values = signal.to_numpy() ``` -This example uses a local registry, so it works without registry credentials or network access. +`TimeNet()` reads the local registry first, and the local registry holds this exact version. +So the example works without registry credentials or network access. See the [Get started guide](https://docs.timenet.ai/get-started.html) for installation options and a longer example. diff --git a/docs/architecture.md b/docs/architecture.md index dfdcc389f..ec4dc0410 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,8 +18,8 @@ for per-component detail. TimeNet splits into three parts. A **connector** builds a raw source into a TimeF version. The **client/SDK** reads its manifest from a **registry** and loads the data. The relational control plane is one immutable DuckDB file. The values plane can be Parquet or Zarr. Reading never runs -connector code. Against a local registry, `open` and `load` can first build a dataset that the -registry does not have from an installed connector. +connector code. If the sources of the client include the installed connectors, `open` and `load` +can first build a dataset that no registry has into the local registry. | | What it is | Ships | Used by | | --- | --- | --- | --- | @@ -43,13 +43,13 @@ PRODUCE dataset.yaml + connector registry │ ▼ -CONSUME SDK ─► open_version ─► TimeFReader ─► Arrow +CONSUME SDK ─► TimeNet.open ─► TimeFReader ─► Arrow ``` The compiled `manifest.json` is the entry point and commit marker. It contains the card metadata, -derived schema, counts, and file list. `open_version` returns a handle to the manifest and version -files. `TimeFReader` uses the handle to read the DuckDB hierarchy and lazy Signal values. The SDK -never imports connector code. +derived schema, counts, and file list. `TimeNet.open()` reads the manifest and opens a handle to +the files of the version. `TimeFReader` uses the handle to read the DuckDB hierarchy and lazy Signal +values. The SDK never imports connector code. --- @@ -61,7 +61,7 @@ Three producer-side pieces, each with one job: | --- | --- | --- | | **Connector** | one `BaseConnector` subclass per dataset ([connectors](connectors.md)) | the dataset-specific recipe: `download()` fetches raw files, `convert()` builds a `TimeFDataset`. Knows nothing about the engine or registry. | | **Engine** | `run_pipeline` ([build & publish](build.md)) | drives any connector through the fixed pipeline and owns caching, idempotency, and `force` / `keep_cache`. Knows no dataset specifics. | -| **Builder** | the `timenet-build` CLI ([build](build.md)) | the entry point: resolves the id to its connector and runs the engine into a registry. | +| **Builder** | the `timenet-build` CLI and `timenet.build()` ([build](build.md)) | the entry point: resolves the id to its connector and runs the engine into the local registry. | ``` timenet-build build org/name @@ -69,14 +69,14 @@ timenet-build build org/name ├─ builder discovery.resolve("org/name") -> Connector class │ datasets/// exposes CONNECTOR │ - └─ engine run_pipeline(connector, ) + └─ engine run_pipeline(connector, ) metadata -> download -> convert -> derive_schema -> store - -> /org/name// + -> /org/name// ``` `metadata()` reads the `dataset.yaml` card. The engine streams the converted dataset through -[`TimeFWriter`](timef-writer.md). A local output directory is also a valid registry, so the client -can read it directly. +[`TimeFWriter`](timef-writer.md) into the local registry, from where the client can +read it directly. --- @@ -115,9 +115,11 @@ can read it directly. 1. **Author** a connector under `timenet_connectors/datasets///`. Its `__init__.py` exposes `CONNECTOR`, and its `dataset.yaml` card sits beside it. -2. **Build**: `timenet-build build /` runs the engine, compiles the manifest, and writes a TimeF version. -3. **Verify** locally: point the SDK at the output directory (itself a valid local registry). -4. **Publish** the complete TimeF version to a registry. +2. **Build**: `timenet-build build /` or `timenet.build("/")` runs the engine, + compiles the manifest, and writes a TimeF version. +3. **Verify** locally: the SDK reads the local registry, so it finds the new version there. +4. **Publish** the complete TimeF version to a remote registry with `timenet publish` or + `timenet.publish()`. 5. **Consume**: use `TimeNet.open()` for lazy access, `TimeNet.load()` for a dataset with every value in memory, or `timenet download` from the CLI for a local copy. @@ -140,10 +142,9 @@ runs, the build prepares each parent release, as [Prepare parent releases](build.md#prepare-parent-releases) describes. `TimeNet.open()`, `TimeNet.load()`, `TimeNet.open_torch()`, and `TimeFReader.open()` resolve the -payload sources before the first read. A read never opens the catalog of a parent. Against a local -registry, the client reuses a complete copy of each release, copies it from `fetch_from`, or builds -it. `auto_build` is on by default. A remote client never runs connector code. See -[Open, load, or download](client.md#open-load-or-download). +payload sources before the first read. A read never opens the catalog of a parent. A client +reads each release where it is, through its ordered list of sources. If only +`InstalledConnectors()` in that list provides a release, the client builds it into the local registry. See [Open, load, or download](client.md#open-load-or-download). The child keeps an internal map from the ID of each parent object to the ID of its copy. Through this map, `get(..., parent=)` finds a copy. Through the same map, a later import reuses a copy. The diff --git a/docs/assets/architecture.svg b/docs/assets/architecture.svg index 6a1ef93e1..8a45a9381 100644 --- a/docs/assets/architecture.svg +++ b/docs/assets/architecture.svg @@ -42,7 +42,7 @@ timenet-build build timenet/hello-world - --out .timenet-registry + --local-registry ./registry diff --git a/docs/build.md b/docs/build.md index b413c0cb3..7073adfdc 100644 --- a/docs/build.md +++ b/docs/build.md @@ -14,17 +14,18 @@ version contains `manifest.json`, `control.duckdb`, and Parquet or Zarr Signal v Use `timenet-build` for normal builds: ```bash -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -The output can be a local directory or a writable registry URL. +The release lands in the local registry (default `~/.timenet/registry`), where +`TimeNet()` reads from. `--local-registry` selects another local registry. `--publish-to` publishes the release to another registry after the build: ```bash # Publish through the hosted registry API. -timenet-build build org/name --out timenet:// +timenet-build build org/name --publish-to timenet:// # Publish to S3. -timenet-build build org/name --out s3://bucket/prefix +timenet-build build org/name --publish-to s3://bucket/prefix ``` See the [`timenet-build` reference](cli/build.md) for all options. @@ -38,150 +39,98 @@ requirements do not affect each other or your current environment. Use `--no-isolation` while developing a connector in the current environment: ```bash -timenet-build build org/name --out .timenet-registry --no-isolation +timenet-build build org/name --local-registry .timenet-registry --no-isolation ``` -`TIMENET_ISOLATION=off` provides the same setting. The manifest records the Python and package -versions used for the build. +Most parameters can also be specified as environment variables. For example, `$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the Python SDK also honors them. + +For each build, the used Python and package and TimeNet versions are recorded to the build's manifest. ## Pipeline The engine runs these stages: -1. Read and validate the connector's `dataset.yaml` card. If the output registry is not local, the - card must declare `access: open`. -2. Open the parent releases that the card declares. The output registry must hold them, as +1. Read and validate the connector's `dataset.yaml` card. +2. Prepare the parent datasets that the card declares, in the local registry. Before the build + runs, the plan downloads or builds each missing parent there, as [Prepare parent releases](#prepare-parent-releases) describes. 3. Download raw artifacts into the build cache. 4. Compose the parents and the raw artifacts into a `TimeFDataset`. Record each parent in it. 5. Derive the dataset schema. -6. Write or publish the complete TimeF version. Then close the parent releases. Before a publish to - a registry that is not local, the build makes sure that each card in the lineage also declares - `access: open`. +6. Write the complete TimeF version into the local registry. 7. Remove the raw cache after success, unless `--keep-cache` is set. -`--force` rebuilds or republishes a version that already exists. Without this flag, the pipeline -returns the committed version and skips expensive work. A forced local build replaces the old -version only once the new one is complete, so a failed rebuild keeps it. - -The connector selects Parquet or Zarr by default. Override it when you need to inspect another -backend: - -```bash -timenet-build build timenet/hello-world \ - --out .timenet-registry \ - --values-backend zarr -``` +By default a build reuses an existing version already in the local registry and a status line says so. Use `--force` to skip this behavior. A forced build replaces the old version only after the new one is +complete, so a failed build keeps the old one. A published version never changes, so with +`--publish-to`, `--force` only affects the build to the local registry. The publish then only checks the rebuild against the already published version. ## Programmatic builds -Use `run_pipeline()` for a local directory: +Build and publish from Python with the producer functions of `timenet`: ```python -from pathlib import Path - -from timenet.engine import run_pipeline - -version_dir = run_pipeline( - connector, - Path(".timenet-registry"), - values_backend="parquet", -) -``` +import timenet -Use `publish_pipeline()` with a writable registry: +# Build into the local registry. +timenet.build("org/name") -```python -from timenet.engine import publish_pipeline -from timenet.registry import open_writable_registry +# Read the new release, like any other. +dataset = timenet.TimeNet().load("org/name") -registry = open_writable_registry("s3://bucket/prefix") -version = publish_pipeline(connector, registry) +# Copy the release and its payload sources to a registry. +timenet.publish("org/name", dest="s3://bucket/registry") ``` -Both functions accept `force`, `keep_cache`, `cache_dir`, `values_backend`, `pinned_by`, and a -writer progress callback. - -A registry that is not local holds only releases with access declared `open`. `publish_pipeline()` -checks the card before it downloads anything. A credentialed or restricted card raises -`TimeNetAccessError` for such a registry. After the connector composes the dataset, -`publish_pipeline()` checks the dataset again with its lineage, before it writes. If the card of a -child declares `access: open` but its parent is credentialed, the second check raises the error. - -The error names the release and its effective access level, which is the strictest level of its -card and of the cards in its lineage. It also names each `access_url` that a user needs, with the -release that sets it. For a card, this is the `access_url` of the card. - -## Prepare parent releases - -A connector card can declare exact parent releases under `parents`. The connector reads these -releases when it composes the child dataset. Before the build downloads anything, it prepares each -parent release in the output registry. It also prepares the payload sources of each parent. A -payload source is a release whose payload files the parent reads. +`timenet.build()` builds a release with the installed connector that claims its dataset ID, in an +[isolated environment](#isolated-build-environments). It returns the plan that ran: one step for +each release that it copied or built, in order. Each argument that you leave out comes from the +environment at the time of the call: -For each release, the build uses the first of these steps that works: +- `store` comes from `$TIMENET_LOCAL_REGISTRY`. +- `parents_from` comes from `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With + `TIMENET_AUTO_BUILD=false`, the connectors stay out. -1. Reuse a complete copy in the output registry. -2. Copy the release from the registry that `--fetch-from` names. -3. If `--build-parents` is set, build the release from its own connector. +So a default build lands where `TimeNet()` reads. `store` takes a local directory, a `file://` URI +or a `LocalRegistry`. `parents_from` lists, in order, where the parents and payload sources that the local registry lacks come from (see [Prepare parent releases](#prepare-parent-releases)). -If no step works, the build stops before any download. The error names the missing release. The -build prepares each parent before its children. If the output already holds the requested version -and `--force` is not set, the build prepares only the missing payload sources of that version. It -uses the same steps in the same order. The build does not run the connector of the requested -dataset. +```python +import timenet -```bash -timenet-build build org/name --out .timenet-registry \ - --fetch-from timenet:// --build-parents +plan = timenet.build( + "org/name@1.2.0", + store=".timenet-registry", + parents_from=["timenet://", timenet.InstalledConnectors()], +) +for step in plan.steps: + print(step.release, step.action) ``` -A parent builds like the requested dataset, in the environment of its own connector. A parent never -builds with `--force`. If the builder backend of another installed package claims a parent, that -backend builds it. - -The build never builds over a release that the output holds. If a file of the copy in the output -is missing or has another size, the build copies the release again from `--fetch-from`. If the build -cannot copy it, the build raises `TimeNetFileError`. If a copied file does not match the size and -checksum that its manifest records, the build raises `TimeNetFileError`. - -If `--build-parents` is set, a missing payload source builds as a pinned release. The build -publishes the payload source only if it writes the files that the reading release pins. See -[Rebuild a pinned release](#rebuild-a-pinned-release) for the check and the two fixes. +`timenet.publish()` copies a complete release and the payload sources that it needs from `source`, +by default the local registry, to `dest`. Without a version in the dataset ID, it +copies the latest version of `source`. It mirrors the [`timenet publish`](cli/timenet.md#commands) CLI command. -`--fetch-from` needs a local `--out`. An output that is not local holds only releases with access -declared `open`. So the build checks the access of each release before any download. A copy from a -registry that is not local also needs access declared `open`. +A published version cannot change. If `dest` already holds the same release, `publish()` copies nothing. -`timenet_connectors.build()` and `timenet_connectors.load()` take the same options as keywords. -`fetch_from` takes a registry or its URI. `build_parents=True` builds missing parents. -```python -import timenet_connectors - -dataset = timenet_connectors.load( - "org/name", - fetch_from="timenet://", - build_parents=True, -) -``` +A remote registry holds only releases with access declared `open`. Before +`timenet.publish()` copies anything to such a registry, it checks the card and the lineage of each +release. -## Rebuild a pinned release +## Prepare parent releases -A release can read payload files that another release stores. The reading release pins each of -these files by path, size, and checksum (see [Shared payloads](timef-format.md#shared-payloads)). -A rebuild of the stored release must write the same files. +A connector card can declare exact parent releases under `parents`. The connector reads these +releases when it composes the child dataset, preparing each parent release in the local registry that it builds into. -Pass the pins to `run_pipeline()` or `publish_pipeline()` as `pinned_by`, keyed by the reading -release. The writer builds the release in a staging directory. Before it publishes the release, it -compares each pinned file with the file that it wrote. If a pinned file differs, the writer deletes -the staging directory and raises `TimeFFormatError`. Nothing is published. +When resolving dependencies, the build uses the first of these steps that +works: -The error names two fixes: +1. Reuse a complete copy in the local registry. `--local-registry` selects it. +2. Copy the release from the remote registry. `--remote-registry` selects it, else + `$TIMENET_REMOTE_REGISTRY` (default `timenet://`). +3. Build the release from its own connector. `--no-auto-build`, or `TIMENET_AUTO_BUILD=false`, + removes this step. -- Fetch the exact release from a registry that holds it. -- Rebuild the reading release with force, so that it pins the new files. The releases that pin the - old files of the reading release then fail to open. +If no step works, the build stops before any download, and the error names the missing release. ## Authoring loop @@ -195,11 +144,9 @@ The error names two fixes: The local check keeps build and load paths explicit: ```python -from timenet.client import TimeNet +import timenet -with TimeNet(".timenet-registry").open("org/name") as reader: +client = timenet.TimeNet(sources=[".timenet-registry"]) +with client.open("org/name") as reader: reader.read().describe() ``` - -See [Connectors](connectors.md) for the `download()` and `convert()` contract. See the -[`timenet.engine` API](api/engine.md) for the programmatic interface. diff --git a/docs/cli/build.md b/docs/cli/build.md index 1c817410f..614b906da 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -8,7 +8,8 @@ tags: # `timenet-build` `timenet-build` is the producer CLI. It resolves an installed connector, runs the build pipeline, -and writes or publishes one TimeF version. It ships with `timenet-connectors`. +writes one TimeF version into the local registry, and can publish it. It ships with +`timenet-connectors`. ```bash uv tool install timenet-connectors @@ -20,26 +21,30 @@ author or change a connector. ## Build command ```bash -timenet-build [--quiet] build \ - [--out ] \ - [--values-backend parquet|zarr] \ - [--force] [--keep-cache] [--no-isolation] \ - [--fetch-from ] [--build-parents] +timenet-build [--quiet] build [@] \ + [--force] [--keep-cache] [--isolation | --no-isolation] \ + [--publish-to ] [--dry-run] \ + [--local-registry ] [--remote-registry ] \ + [--auto-build | --no-auto-build] ``` | Option | Default | Purpose | | --- | --- | --- | -| `` | required | Select the connector by its `org/name` ID. | -| `--out` | `$TIMENET_REGISTRY`, then the local registry | Select a local or remote output registry. | -| `--values-backend` | connector setting | Override the Parquet or Zarr values backend. | -| `--force`, `-f` | off | Rebuild or republish an existing version. | +| `[@]` | required | Select the connector by its `org/name` ID. A version must be the one that the card of the connector declares. | +| `--force`, `-f` | off | Build the release again in the local registry, even if the local registry holds it. | | `--keep-cache` | off | Keep downloaded source files after a successful build. | -| `--no-isolation` | isolation on | Use the current environment instead of a connector-specific one. | -| `--fetch-from` | off | Copy missing parent releases from the registry at this URI. `--out` must be local. | -| `--build-parents` | off | Build missing parent releases from their own connectors. | +| `--isolation`, `--no-isolation` | `$TIMENET_ISOLATION`, else on | Run this build in an environment built from the requirements of the connector, or in the current one. | +| `--publish-to` | off | After the build, publish the release and its payload sources to the registry at this URL or path. | +| `--dry-run` | off | Print the plan to stdout and stop. It copies, builds, and publishes nothing. | +| `--local-registry` | `$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the local registry where the build lands. | +| `--remote-registry` | `$TIMENET_REMOTE_REGISTRY`, else `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have a parent. | +| `--auto-build`, `--no-auto-build` | `$TIMENET_AUTO_BUILD`, else on | Whether installed connectors build a parent that neither the local registry nor the remote registry has. | | `--quiet`, `-q` | off | Hide status output. Place this option before `build`. | -The output selector accepts: +The build lands in the local registry, where `TimeNet()` reads. + +`--local-registry` takes a local path or a `file://` URI. `--publish-to` and `--remote-registry` +accept: - a local path or `file://` URI; - `s3://bucket/prefix`; @@ -49,46 +54,71 @@ The output selector accepts: Remote publishing needs the credentials required by that registry. For example, the hosted API can use `TIMENET_TOKEN`. +## Publish after the build + +`--publish-to` gets the release to another registry in two steps: + +1. The release builds into the local registry, as without `--publish-to`. Its parents land there too. +2. `timenet.publish()` copies the release, and each parent that the destination lacks, + from the local registry to the destination. + +If the destination holds the version and `--force` is not set, the command builds and copies +nothing. While `--force` builds the release again in the local registry. It cannot overwrite a published version. If the new build wrote the same files, the publish copies nothing. If it wrote other files, the publish +raises `TimeNetVersionConflictError`, and the change needs a new version. + ## Output -Status goes to stderr. A local build prints the committed version directory to stdout. A remote -build prints the published version string. +Status goes to stderr. Before the first step, the command shows its plan as a tree: the requested +release at the top, and below each release the releases that it needs. Each line names a release, +what the build does with it, and the source: + +```text +📋 Plan for 'physionet/ecg-qa-cot' 1.2.0: + physionet/ecg-qa-cot@1.2.0: build with installed connectors + └── physionet/ptb-xl@1.0.0: copy from registry https://registry.timenet.ai +``` + +Pass `--quiet` to hide the plan and the status lines. ```bash -VERSION_DIR=$( - timenet-build --quiet build timenet/hello-world \ - --out .timenet-registry -) +VERSION_DIR=$(timenet-build --quiet build timenet/hello-world) test -f "$VERSION_DIR/manifest.json" ``` This separation lets scripts capture the result without parsing progress messages. +`--dry-run` prints only the tree to stdout, and stops. It reads cards and manifests, but it copies, +builds, and publishes nothing. + ## Exit codes | Code | Meaning | | --- | --- | | `0` | The version was built, published, or reused. | | `1` | An expected TimeNet operation failed. | -| `2` | The command or one of its options is invalid. | +| `2` | The command or one of its options is invalid, for example an unknown dataset ID. | + +If an isolated build cannot make its environment, for example because `uv` is missing, the command +raises `TimeNetIsolationError`. Unexpected programming errors keep their traceback. ## Inspect a local build -Use the same explicit registry for the build and the consumer: +Build into a local registry of your own, and pass the same registry to the consumer: ```bash -timenet-build build timenet/hello-world --out .timenet-registry -timenet list --registry .timenet-registry +timenet-build build timenet/hello-world --local-registry .timenet-registry +timenet info timenet/hello-world --local-registry .timenet-registry ``` Or load it in Python: ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +client = timenet.TimeNet(sources=[".timenet-registry"]) +dataset = client.load("timenet/hello-world") ``` See [Build and publish](../build.md) for the pipeline and [Connectors](../connectors.md) for the diff --git a/docs/cli/index.md b/docs/cli/index.md index d09f32231..39882d26f 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -11,8 +11,8 @@ TimeNet ships two console scripts, one for each side of the workflow: | Command | Ships with | Role | Mirrors | | --- | --- | --- | --- | -| [`timenet`](timenet.md) | `timenet` | Consumer: browse a registry and download datasets. | The [client SDK](../client.md). | -| [`timenet-build`](build.md) | `timenet-connectors` | Producer: run a connector through the pipeline into a registry. | [Build & publish](../build.md). | +| [`timenet`](timenet.md) | `timenet` | Consumer: find, read, download and publish TimeNet datasets. | The [client SDK](../client.md). | +| [`timenet-build`](build.md) | `timenet-connectors` | Producer: run a connector through the pipeline into the local registry, and publish the result. | [Build & publish](../build.md). | Both tools use [Typer](https://typer.tiangolo.com). The `timenet` command needs the `cli` extra: @@ -23,12 +23,14 @@ uv add 'timenet[cli]' If you run it without the extra, it prints a one-line install hint instead of a traceback. Both tools split their output the same way: status lines go to stderr, the machine-readable result (a -path) goes to stdout. `--quiet`/`-q` silences the status. The flag belongs to the tool, not the -subcommand, so it comes first: `timenet --quiet list`, `timenet-build --quiet build `. - -Both tools honor `$TIMENET_REGISTRY`, but their final defaults differ. The consumer uses the hosted -registry when no registry is set. The build tool writes to the local registry. Pass `--registry` or -`--out` when a workflow must use a specific location. +path or a version) goes to stdout. `--quiet`/`-q` silences the status. The flag belongs to the tool, +not the subcommand, so it comes first: `timenet --quiet list`, `timenet-build --quiet build `. + +Both tools read the same [settings](../client.md#settings). The consumer copies into the local +registry, `$TIMENET_LOCAL_REGISTRY`, and the build tool writes to it. The commands that look up +datasets take `--local-registry`, `--remote-registry`, and `--auto-build`/`--no-auto-build`, which +override these settings for one command (see +[Selecting the sources](timenet.md#selecting-the-sources)). If you want to use datasets, start with [`timenet`](timenet.md). If you build a dataset, start with [`timenet-build`](build.md). diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 679e7356b..eb92eb64f 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -1,6 +1,6 @@ --- icon: lucide/terminal -description: "The timenet consumer CLI: browse a registry and download datasets from the shell." +description: "The timenet consumer CLI: find, read, download and publish TimeNet datasets from the shell." tags: - cli --- @@ -14,8 +14,7 @@ from the shell that you do in Python. Install it with the `cli` extra: uv add 'timenet[cli]' ``` -By default, the command uses the hosted TimeNet registry. You can also select a local or S3 -registry: +By default, the command uses the hosted TimeNet registry: ```bash timenet list @@ -24,32 +23,65 @@ timenet info chengsenwang/tsqa@1.0.0 # pin a version; omit @ for the latest timenet download chengsenwang/tsqa ``` +To use a local or S3 registry instead, see [Selecting the sources](#selecting-the-sources). + ## Commands | Command | What it does | | --- | --- | -| `timenet list` | Print every dataset in the registry (id, name, license, domains), from the metadata of its latest version. | -| `timenet search [flags]` | Filter datasets. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). | -| `timenet info [@version]` | Show a dataset's [manifest](../manifest.md): metadata, schema, and counts. | -| `timenet download [@version]` | Copy a version's files into local storage and print the directory. `--storage ` picks the target (else `$TIMENET_STORAGE`, then `/storage`). If a local version already exists, it skips the copy. | -| `timenet cache info` | List downloaded datasets on disk (location, id, version, size) and the total. | -| `timenet cache clear` | Remove downloads and the raw cache. It prompts first. `-y` skips the prompt. `--all` also clears built data. | - -Every command writes its status to stderr. It writes its machine-readable result (a path) to stdout. -Therefore, you can capture `timenet download ` in a script safely. - -`--quiet`/`-q` removes that status output. It belongs to `timenet` itself. Therefore, it goes before +| `timenet list` | Print every dataset of the sources (id, name, license, domains), from the metadata of the highest version that a source lists. The cards of the installed connectors count too. See [`list()`](../client.md#browse-datasets). | +| `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources. | +| `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | +| `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. It never builds. See [`download()`](../client.md#open-load-or-download). | +| `timenet publish [@version] --dest URL` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source URL`, and without a version it is the latest version there. A remote `--source` reads the complete copies of the same releases in the local registry. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | +| `timenet local info` | Print the path and size of the cache. Then print one line for the downloads and one line for the builds in the local registry, such as `Downloads in : 2 releases, 1.4 GB`. | +| `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | +| `timenet local cache info` | Print the path and size of the cache, `$TIMENET_CACHE`. | +| `timenet local cache clear [--yes]` | Delete the raw source files (`downloads/`) and the control databases (`control/`) of the cache that carry TimeNet's `CACHEDIR.TAG` file. | +| `timenet local downloads info` | List the downloads in the local registry: dataset, version, source, size, and the total. | +| `timenet local downloads clear [--yes]` | Delete the downloads in the local registry, except the ones that a release that stays needs. | +| `timenet local builds info` | List the builds in the local registry: dataset, version, source, size, and the total. | +| `timenet local builds clear [--yes]` | Delete the builds in the local registry, except the ones that a release that stays needs. | + +Every command writes its status to stderr and its machine-readable result (a path or a release) to +stdout, so you can capture `timenet download ` in a script safely. + +Most errors end with a hint that names a command or a flag that helps. + +`--quiet`/`-q` removes the status output. It belongs to `timenet` itself. Therefore, it goes before the subcommand: `timenet --quiet list`, not `timenet list --quiet`. Be careful with the collision: after `search`, `-q` is the short form of `--query`. -## Selecting a registry +## Downloads and builds in the local registry + +The local registry holds two kinds of release, downloads and local builds. Releases can depend on each other (parent-child relationships). In that case `timenet local downloads clear` and `timenet local builds clear` might keep releases that are dependet on, and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. Do not run a clear while a build or a download runs. + +## Selecting the sources + +`list`, `search`, `info`, and `download` create a client with the default list of sources, +`[local registry, remote registry, installed connectors]` (see +[Choose the sources](../client.md#choose-the-sources)). Three flags set this list, and each flag +overrides one [variable](../client.md#settings): + +| Flag | Variable | Default | What it sets | +| --- | --- | --- | --- | +| `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the place where downloads and builds land. | +| `--remote-registry URL` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | -The tool resolves the registry in this order: `--registry`, then `$TIMENET_REGISTRY`, then -`timenet://`. The last value selects the hosted registry. Local paths, `file://`, `s3://`, -`http(s)://`, and `timenet://` are valid selectors. +A flag wins over its variable, and the variable wins over its default. Flags go after the subcommand: -The build tool has a different final default: it writes to the local registry. Pass the same explicit -path to both tools when you build and inspect a dataset locally. +```bash +timenet list --remote-registry https://registry.example.com +timenet download chengsenwang/tsqa --local-registry ./data --no-auto-build +``` + +`publish` copies out of the local registry by default. To mirror a hosted release in a registry of +your own, name the hosted registry as the source: + +```bash +timenet publish org/name --source timenet:// --dest s3://my-bucket/timenet +``` ## Pinning versions diff --git a/docs/client.md b/docs/client.md index 0c70b48a6..22d8d0aa4 100644 --- a/docs/client.md +++ b/docs/client.md @@ -1,6 +1,6 @@ --- icon: lucide/code -description: "Browse a registry and load TimeF datasets from Python." +description: "Find, read and download TimeNet datasets from Python." tags: - guide - client @@ -8,14 +8,14 @@ tags: # Client -`TimeNet` is the main Python entry point for dataset consumers. It wraps one -[registry](registry.md) and one local storage directory. +`TimeNet` is the main Python entry point for dataset consumers. It reads releases through an +ordered list of sources: [registries](registry.md) and the installed connectors. ```python -from timenet.client import TimeNet +import timenet from timenet.types import Domain -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) for metadata in client.search(domain=Domain.CARDIOLOGY): print(metadata.dataset_id) @@ -24,58 +24,137 @@ dataset = client.load("timenet/hello-world") values = dataset.records[0].signals[0].to_numpy() ``` -## Choose a registry +## Choose the sources -Create a client with a registry object, path, or URL: +A client takes an ordered list of sources. A source is a local or remote registry, or +`InstalledConnectors()`: ```python -from timenet.client import TimeNet +import timenet + +client = timenet.TimeNet(sources=[ + timenet.LocalRegistry(".timenet-registry", store=True), + "https://registry.example.com", + "timenet://", + timenet.InstalledConnectors(), +]) +``` + +- `InstalledConnectors()` stands for every installed package that registers a builder backend, + such as `timenet-connectors`. If only this source provides a release, a read builds it. Without + it in the list, a client never runs connector code. +- `LocalRegistry(path, store=True)` marks the local registry where downloads and builds land. A + client accepts at most one such local registry, and a list with `InstalledConnectors()` needs + one. A `LocalRegistry` without the flag is only read, for example a mirror on a network drive. +- A registry URL in the list gets the directory of this local registry. If the local registry holds + a complete copy of the same release, with the same manifest apart from the build environment, the + registry reads that copy instead of its remote files. Without this local registry in the list, it + reads only its own files. + +An exact version comes from the first source that provides it, while "latest" is the highest +version that any source provides. Only "not found" and "not served" fall through to the next +source, and any other error stops the call. So if a registry in the list does not answer, a request +for "latest" fails. + +A client that only reads takes a list without `store=True`: + +```python +import timenet -hosted = TimeNet() -local = TimeNet(".timenet-registry") -s3 = TimeNet("s3://my-bucket/timenet") -private = TimeNet("https://registry.example.com") +hosted = timenet.TimeNet(sources=["timenet://"]) +local = timenet.TimeNet(sources=[".timenet-registry"]) +s3 = timenet.TimeNet(sources=["s3://my-bucket/timenet"]) ``` -`TimeNet()` selects a registry in this order: +`TimeNet()` uses the default list of `timenet.client.default_sources()`. If no variable is set, +the list is: -1. The `registry` argument. -2. The `TIMENET_REGISTRY` environment variable. -3. The hosted registry, `timenet://`. +```python +[ + LocalRegistry("~/.timenet/registry", store=True), + "timenet://", + InstalledConnectors(), +] +``` -The [getting-started example](get-started.md) uses an explicit local path so its build and load use -the same registry. +`TIMENET_LOCAL_REGISTRY` moves the local registry, `TIMENET_REMOTE_REGISTRY` replaces `timenet://`, +and `TIMENET_AUTO_BUILD=false` removes `InstalledConnectors()` (see [Settings](#settings)). The +arguments `local_registry`, `remote_registry` and `auto_build` of `default_sources()` override +them, and `timenet.client.default_store()` returns the local registry alone. -## Local state +## Settings -TimeNet stores local state under `~/.cache/timenet` by default. +These environment variables configure TimeNet. Each variable sets one thing. | Environment variable | Default | Purpose | | --- | --- | --- | -| `TIMENET_HOME` | `~/.cache/timenet` | Parent directory for local state. | -| `TIMENET_REGISTRY` | unset | Registry URL or local path. | -| `TIMENET_STORAGE` | `/storage` | Complete versions downloaded from a registry. | -| `TIMENET_CACHE` | `/cache` | Raw source files used during builds. | +| `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | The local registry, as a local path or a `file://` URI. It is the first source that a lookup checks, and the place where downloads and builds land. | +| `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. If `true`, the default list ends with `InstalledConnectors()`. | +| `TIMENET_CACHE` | `~/.cache/timenet` | Raw downloads of builds in `downloads/`, and control databases of remote reads in `control/`. | | `TIMENET_TOKEN` | unset | Bearer token for a remote registry. | | `TIMENET_ISOLATION` | `on` | Whether connector builds use isolated environments. | -An explicit argument takes precedence over an environment variable. An environment variable takes -precedence over its default. - -The model for these values is `timenet.config.TimeNetSettings`. +An explicit argument, for example a CLI flag, wins over its variable, and the variable wins over its +default. ## Browse datasets -The client provides these discovery methods: +`list()` and `search()` answer for the whole list of sources. They return one row per dataset, at +the highest version that a source lists, sorted by dataset ID. The card of each installed connector +counts as a row. So `list()` also shows a dataset that only a connector provides. | Method | Result | | --- | --- | -| `list()` | Metadata for the latest version of every dataset. | -| `get(dataset_id, version=None)` | One dataset manifest. | -| `search(...)` | Metadata that matches registry filters. | +| `list()` | Metadata for the highest version of every dataset across the sources. | +| `search(...)` | Metadata that matches registry filters, across the sources. | `search()` accepts filters for free text, domains, task classes, licenses, Signal specifications, -dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. +dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. A +connector never matches `task` or `time_series_spec`, because both filters read the schema of a +manifest. + +The calls ask the sources one after another, in order. If two sources list a dataset at the same +version, the row of the earlier source stays. To query one registry alone, open it and ask it: + +```python +from timenet.registry import open_registry + +hits = open_registry("timenet://").search(query="ecg") +``` + +## Inspect a release + +These methods describe the release that `open()` reads. They resolve the release through the +sources as `open()` does (see [Choose the sources](#choose-the-sources)), and copy and build +nothing. + +| Method | Result | +| --- | --- | +| `get_manifest(dataset_id, version=None)` | The [manifest](manifest.md) of the release. | +| `get_card(dataset_id, version=None)` | The card of the release. | +| `plan_read(dataset_id, version=None)` | The plan of the read, one step for each release. | + +`get_manifest()` reads only the manifest, and checks no other file. A registry can list a +release without serving its files, for example a credentialed release on the hosted registry. Then +`get_manifest()` returns the listed manifest, which shows the access level and the access page of +each card. A release gets its manifest when it is built. So for a release that only +`InstalledConnectors()` provides, `get_manifest()` raises `TimeNetNotBuiltError`. + +`get_card()` returns the card of the release. The card comes from the manifest, or from the +installed connector for a release that is not built yet: + +```python +import timenet +from timenet.errors import TimeNetNotBuiltError + +client = timenet.TimeNet() +try: + card = client.get_manifest("chengsenwang/tsqa").metadata +except TimeNetNotBuiltError: + card = client.get_card("chengsenwang/tsqa") +print(card.name, card.license) +``` ## Open, load, or download @@ -120,49 +199,31 @@ with client.open("timenet/hello-world") as reader: values = record.signals[0].to_numpy() ``` -For a local registry, the reader opens the version in place. A local read needs complete copies of -the version and of its payload sources. A payload source is a release whose payload files the -version reads. For each of these releases, `open()` and `load()` use the first of these steps that -works: - -1. Reuse a complete local copy. -2. Copy the release from the registry that `fetch_from` names. -3. If `auto_build` is set, build the release from its installed connector. `auto_build` is set by - default. - -If no step works, the call raises an error that names the release and the keyword that can get it. -If the call builds the version, the build prepares the declared parents of the version with the -same steps in the same order. +`open()`, `load()`, and `open_torch()` read each release where it is, and copy nothing. They ask +the sources by the rules of [Choose the sources](#choose-the-sources), so if an installed connector +declares a newer version than the local registry holds, the call builds the newer version. ```python -with TimeNet(".timenet-registry").open( - "org/name", - fetch_from="timenet://", -) as reader: +import timenet + +client = timenet.TimeNet(sources=[ + timenet.LocalRegistry(".timenet-registry", store=True), + "timenet://", +]) +with client.open("org/name") as reader: records = reader.read().records ``` -`fetch_from` takes a registry or its URI. `open_torch()` takes `auto_build` and `fetch_from` too. -Set `auto_build=False` to report a missing local release without starting a build. - -The call builds a payload source with its own connector. The build makes the exact release that -the version pins. The build publishes this release only if it writes the pinned files. See -[Rebuild a pinned release](build.md#rebuild-a-pinned-release). +This list has no `InstalledConnectors()`, so a missing release raises an error instead of starting +a build. -A local copy is damaged if its manifest is present but a file is missing or has another size. If -`fetch_from` is set, the call repairs a damaged copy from that registry. If `fetch_from` is not set, -the call raises `TimeNetFileError`. The error names the first missing or resized file. The call -never builds a release over a damaged copy. +If only `InstalledConnectors()` provides a release, the call builds it into the local registry +first and reads it there. The build copies or builds each missing parent from the same sources, as +[Prepare parent releases](build.md#prepare-parent-releases) describes. -Remote registries never run connector code. A remote client rejects `fetch_from`, because it reads -from its own registry. - -When `open()`, `load()`, or `open_torch()` runs, an HTTP or S3 registry resolves the manifests of -the payload sources. Reads then use complete copies in `TIMENET_STORAGE` where they exist. Otherwise -they read the remote files by range, as each read needs them. The first read of Records or tasks -downloads the catalog of the version and makes sure that it matches the manifest. A read never opens -the catalog of a parent release. A read also never fetches a parent release that is not a payload -source. See [Payload sources](registry.md#payload-sources). +When `open()`, `load()`, or `open_torch()` reads from an HTTP or S3 registry, it reads the +complete copy of the same release in the local registry, if it holds one. Otherwise it reads the +remote files by range. A child dataset gives the objects that it imports from a parent IDs of its own. To read such a copy by the ID that the parent gives it, pass the parent to `get()` with `parent=`: @@ -180,29 +241,71 @@ with client.open("physionet/ecg-qa-cot") as reader: parent as a `ParentAlias`, or its exact release as a `DatasetRef`. Without `parent=`, the lookup uses the ID that the child gives the object. -`download()` copies the complete release and its payload sources into local storage: +`download()` copies the complete release and its payload sources into the local registry, and +returns the release directory there: ```python path = client.download("chengsenwang/tsqa@1.0.0") ``` -Use `force=True` to replace an existing local copy. A later `open()` or `load()` reuses the complete -copies. A remote client still requests the manifests. To read without a network, open the storage -directory as a local registry with `TimeNet(storage_dir)`. Then pass `auto_build=False` to `open()` -or `load()`. A remote read keeps the byte ranges that it reads in memory only. The copies of -`download()` stay on disk. +The release resolves as in `open()`. The call copies the release and each parent that the +local registry lacks or holds damaged. Each one comes from the first other registry that provides +it, so they can come from different registries. If the local registry holds a complete copy of the +release, the call copies only the parents that it lacks. + +A download never builds. For a release that only `InstalledConnectors()` provides, it raises +`TimeNetNotBuiltError`. + +A published version cannot change. So if the local registry holds the release with other files or +another card than the registry that the call copies from, the call raises +`TimeNetVersionConflictError`. + +`progress=` takes a function that receives the progress events of `timenet.progress`. For each +release that the call copies, the function receives `StepStarted`, then `BytesMoved` events as the +files arrive, then `StepFinished`: + +```python +from timenet.progress import BytesMoved + +def show(event): + if isinstance(event, BytesMoved): + print(f"{event.url}: {event.done} of {event.total} bytes") + +client.download("chengsenwang/tsqa", progress=show) +``` + +The local registry comes first in the default list, so a later `open()` or `load()` of the same +version reads the copy in the local registry. A remote read keeps the byte ranges that it reads in +memory only, while the copies of `download()` stay on disk. + +## The local registry + +The local registry where downloads and builds land keeps a provenance record for each release that +a copy or a build puts into it. The record says where the release comes from: + +```json +{ + "origin": "copy", + "source": "https://registry.timenet.ai", + "at": "2026-11-03T14:02:11Z" +} +``` + +`origin` is `copy` for a release copied from a registry, and `build` for a release +that an installed connector built. `source` names the registry, or the connector package and its +version. To read it, call `timenet.registry.origin.read_origin(store, release)`. ## Pin a version Append `@` to a dataset ID: ```python -client.get("chengsenwang/tsqa@1.0.0") +client.get_manifest("chengsenwang/tsqa@1.0.0") client.load("chengsenwang/tsqa@latest") ``` -The `get`, `download`, `open`, `load`, and `open_torch` methods also accept `version=`. Do not -supply a version in both places. +The `get_manifest`, `get_card`, `plan_read`, `download`, `open`, `load`, and `open_torch` methods +also accept `version=`. Do not supply a version in both places. ## Open for PyTorch @@ -212,10 +315,10 @@ does not have to fit in memory. The view holds an open reader. Close the view wi use it in a `with` block. ```python -from timenet.client import TimeNet +import timenet from timenet.types import read_value, read_values -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) with client.open_torch("timenet/hello-world") as view: item = view[0] diff --git a/docs/connectors.md b/docs/connectors.md index 39490ba23..7446115be 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -12,8 +12,10 @@ A connector is the unit of dataset integration. Each dataset has one connector c fetches raw data and converts it into a [`TimeFDataset`](timef-dataset.md). A connector has no knowledge of the registry, the engine, or other connectors. A child connector reads its parent releases only through the readers that the engine passes to it. The consumer SDK never imports -connector code. On a local registry, `TimeNet.load` builds a missing dataset through the -`timenet.builders` entry point. Pass `auto_build=False` to turn this off. +connector code. If the [sources](client.md#choose-the-sources) of a client include +`InstalledConnectors()`, `TimeNet.load` builds a missing dataset through the `timenet.builders` +entry point. To turn this off, leave `InstalledConnectors()` out of the list, or set +`TIMENET_AUTO_BUILD=false` for the default list. `BaseConnector` is the connector contract in the `timenet` package. Concrete connectors ship in `timenet-connectors`, beside their [dataset cards](manifest.md). @@ -224,13 +226,15 @@ a connector that you did not check. Credentials come from the environment. For the HuggingFace Hub, a token is read from `HF_TOKEN` automatically (needed only for gated or private sources). Downloaded source files cache under -`` (see [client local state](client.md#local-state)). +`/downloads/`, by default in `~/.cache/timenet` (see +[Settings](client.md#settings)). A credentialed dataset (a PhysioNet DUA-gated one, for example) declares `access: credentialed` and an `access_url` on its card. TimeNet never hosts such data, so it is build-your-own: get access at the `access_url`, set the provider credential in the environment (the way `HF_TOKEN` already -works), and run `timenet-build build ` yourself. An `open`, `load`, or `download` of a -credentialed dataset from a hosted registry raises `TimeNetAccessError` instead of serving bytes. +works), and run `timenet-build build ` yourself. If no other source of the client provides a +credentialed dataset, an `open`, `load`, or `download` from a hosted registry raises +`TimeNetAccessError` instead of serving bytes. The error names each `access_url` that the dataset needs, with the release that sets it. A child dataset can need the access pages of its ancestors. @@ -306,10 +310,10 @@ timenet-build build chengsenwang/tsqa # live, from the Hub timenet-build build chengsenwang/tsqa --keep-cache # keep the raw sources ``` -A successful build removes the dataset's raw download cache, at `/`. Only -`convert` reads the sources, and they are often several times the size of the dataset they produce. -To keep them, pass `--keep-cache`, or `keep_cache=True` to `timenet_connectors.build()`. This helps -while you write a connector for a large source, because each rebuild downloads the source again. +A successful build removes the raw downloads of the dataset, at +`/downloads/`. To keep them, pass `--keep-cache`, or `keep_cache=True` to `timenet.build()`. This +helps while you write a connector for a large source, because each rebuild downloads the source +again. Keeping `download` and `convert` apart makes a connector testable offline. `convert` takes raw references and does not touch the network. A test can therefore pass small local inputs or synthetic @@ -334,7 +338,7 @@ The connector of a child overrides `compose(raw_refs, context)`. The default `co `convert(raw_refs)`. If the card declares parents and the connector does not override `compose`, the build raises before it opens a parent or downloads anything. -Before the connector runs, the build prepares each parent release in the output registry, as +Before the connector runs, the build prepares each parent release in the local registry, as [Prepare parent releases](build.md#prepare-parent-releases) describes. The engine then opens each parent as a `TimeFReader` and passes the readers in a `BuildContext`, by parent alias. The connector copies objects into the child with `dataset.import_from()`: diff --git a/docs/data-model/datasets.md b/docs/data-model/datasets.md index f94044d81..334df287e 100644 --- a/docs/data-model/datasets.md +++ b/docs/data-model/datasets.md @@ -28,13 +28,13 @@ The `dataset_version` changes to `1.0.1`, while `source_revision` stays `v2.1`. TimeNet commits a version atomically when its `manifest.json` lands. The version never changes after that. A pinned `@version` gives you exactly those bytes. With no suffix (or `@latest`), you get the -newest committed version. +highest version that a source of the client provides. ```python -from timenet.client import TimeNet +import timenet -client = TimeNet() -client.load("chengsenwang/tsqa") # latest committed version +client = timenet.TimeNet() +client.load("chengsenwang/tsqa") # the latest version client.load("chengsenwang/tsqa@1.0.0") # a pinned, immutable snapshot ``` diff --git a/docs/get-started.md b/docs/get-started.md index 37e9baf57..eb5d419e9 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -44,16 +44,16 @@ Use `pipx install timenet-connectors` if you use pipx for command-line tools. ## Build the example dataset -Build the offline `timenet/hello-world` dataset into a local registry: +Build the offline `timenet/hello-world` dataset: ```bash -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -The command needs no network access. It writes this version: +The command needs no network access. It writes this version into a local registry: ```text -.timenet-registry/timenet/hello-world/1.0.0/ +~/.timenet/registry/timenet/hello-world/1.0.0/ ├── manifest.json ├── control.duckdb └── time_series/ @@ -61,15 +61,16 @@ The command needs no network access. It writes this version: ## Load the dataset -Pass the same registry path to the client: +The default client reads the local registry first. An exact version that the local registry holds +needs no network access: ```python import pandas as pd -from timenet.client import TimeNet +import timenet -client = TimeNet(".timenet-registry") -dataset = client.load("timenet/hello-world") +client = timenet.TimeNet() +dataset = client.load("timenet/hello-world@1.0.0") dataset.describe() signal = dataset.records[0].signals[0] diff --git a/docs/manifest.md b/docs/manifest.md index 3501cd15a..f62ed6670 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -79,6 +79,10 @@ Direct construction and parsing with an unsupported `timef_format_version` raise and `counts` default to empty. Unknown fields are rejected. A malformed block raises Pydantic's `ValidationError` with its native field locations and error details. +Registries read a stored manifest with `read_manifest(path)` or `parse_manifest(text, where)` from +`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A file that cannot be +read raises a plain `TimeFFormatError`. + ### Serialization notes - Known units serialize to their pint names (`"hertz"`, `"millivolt"`, `"dimensionless"`). An @@ -126,7 +130,7 @@ packed text and time-offset files that this release stores itself. A build needs the parent releases. A read needs only the payload sources. `TimeNet.open()`, `TimeNet.load()`, `TimeNet.open_torch()`, and `TimeFReader.open()` resolve the payload sources before the first read. A read never opens the catalog of a parent. `download()` copies the release -and its payload sources. See [Payload sources](registry.md#payload-sources). +and its payload sources into the local registry. See [Payload sources](registry.md#payload-sources). --- diff --git a/docs/registry.md b/docs/registry.md index c2146ebce..f3a027fe4 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -35,13 +35,15 @@ registry = open_registry("./local_registry") | `http(s)://` | `RemoteRegistry` | | `timenet://` | `RemoteRegistry`, an alias for the hosted `https://registry.timenet.ai` | -An unrecognized scheme, for example `gs://` or `az://`, raises `ValueError` instead of becoming a -local path. `open_registry` expands `~` in a local path or a `file://` URI. +`open_registry()` returns a `WritableRegistry`, so you can also publish to the registry that it +opens (see [Writing to a registry](#writing-to-a-registry)). `store=` gives an HTTP or S3 registry +the directory of the local registry where downloads and builds land (see +[Remote reads](#remote-reads)). A local registry ignores it. !!! note "Catalog support" - Every backend serves reads and publishes with `store` (remote publishing needs a writer token; the - S3 backend reads AWS credentials from the environment). The S3 backend has no catalog, so `list` and - `search` are unsupported there. + Every backend serves reads and takes new releases. Remote publishing needs a writer token, and + the S3 backend reads AWS credentials from the environment. The S3 backend has no catalog, so + `list` and `search` do not work there. ### Remote registries @@ -68,27 +70,36 @@ hardcoded. Install the extra: uv add 'timenet[s3]' ``` -The layout under the prefix mirrors the local one, `org/name/version/…` with `manifest.json` beside the -data files. `store` uploads to a temporary `version.tmp-` prefix and moves each object into place -(`manifest.json` last), so a failed publish never leaves a partial version. `open` reads lazily and -`load` reads every value. Both reuse complete local copies where they exist. Otherwise they read -remote payload ranges through a pyarrow `S3FileSystem`. The backend has no catalog, so `list` and -`search` are unsupported: publish and fetch by an explicit `org/name@version`. +The layout under the prefix is `datasets/org/name/version/…`, with `manifest.json` beside the data +files. `put_dataset` uploads to a temporary `version.tmp-` prefix and moves each object into +place (`manifest.json` last), so that failed publishing does not leak partial versions. +`TimeNet.open()` reads lazily and `load()` reads every value. Both read the complete copy of the +same release in the local registry, if it holds one (see [Remote reads](#remote-reads)). Otherwise +they read remote payload ranges through a pyarrow `S3FileSystem`. The backend has no catalog, so +`list` and `search` are unsupported: publish and fetch by an explicit `org/name@version`. ## `BaseRegistry` -Every backend implements this contract: four data-access methods and two shared methods, -`download_version` and `search`. +Every backend implements this contract. | Method | Description | | --- | --- | | `list_datasets()` | Latest-version `DatasetMetadata` for every dataset, sorted by id. | -| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version, or `TimeFFormatError` if the stored manifest's own id disagrees with the directory it was loaded from. | +| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeFFormatError` if the stored manifest is not valid, or if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | -| `local_root` | The local registry or cache directory for complete releases. | -| `open_version(dataset_id, version=None)` | A `DatasetVersion`: the parsed manifest plus a filesystem-rooted, picklable handle to the version's files, with its [payload sources](#payload-sources) opened from this registry. This is the storage seam the [reader](timef-reader.md) reads through, so a read never re-opens the registry nor re-parses `manifest.json`. | -| `download_version(dataset_id, version, dest_dir, *, force=False, manifest=None, progress_cb=None)` | Copy a version's files into a local directory (shared implementation). It stages the files under `.tmp-*` with `manifest.json` last, then swaps the directory in with one rename. A subclass can override it: `RemoteRegistry` streams the files in parallel. | -| `search(...)` | Filter datasets (shared implementation). | +| `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Any other registry gives its name. | +| `local_root` | The directory whose complete releases the registry reads: the root of a local registry, or the local registry that `store=` gives a remote registry. `None` for a remote registry without `store=`. | +| `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | +| `find_manifest(release)` | The manifest of an exact release, or `None` if the registry does not hold it. Raises `TimeFFormatError` if the stored manifest names another release. | +| `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | +| `open_release(reference, *, manifest=None)` | A `DatasetVersion` of one exact release without its payload sources: the parsed manifest and a handle to the files. Pass `manifest` if you already read it from this registry. A local registry raises `TimeNetFileError` for a missing or resized file. | +| `resolve(dataset_id, version)` | Whether the registry serves a release, as a [source](#sources) answer: `Found`, `NotFound` or `NotServed`. | +| `search(...)` | Filter datasets. | + +To copy a release with its payload sources, use +[`TimeNet.download()`](client.md#open-load-or-download) for the local registry where downloads and +builds land, or [`copy_release()`](#copying-a-release) for another registry. To read a release +with its payload sources, use [`TimeNet.open()`](client.md#open-load-or-download). `LocalRegistry` serves a `///` tree. `RemoteRegistry` serves the same layout over the versioned REST contract (`GET /api/v1/datasets`, `/api/v1/datasets/{id}/{version}/manifest`, @@ -103,95 +114,132 @@ lowercase ids to avoid casing clashes on case-insensitive filesystems. A version can reuse payload files that other versions store: its payload sources. Its manifest names each one and pins the files it reads from it (see -[Shared payloads](timef-format.md#shared-payloads)). A registry serves a version together with its -payload sources: - -- `open_version` opens each payload source from the same registry and checks it against the pins, - reading only manifests and, for local files, file sizes. A remote registry prefers complete - cached copies and otherwise reads remotely. See [Opening a version](timef-reader.md#opening-a-version) - for the checks and their errors. -- `download_version` downloads complete copies of the version and of each payload source, and checks - each source against the pins. If a downloaded file has another size or checksum than its manifest - records, the download raises `TimeNetFileError` and keeps no partial copy. -- `store` refuses a version whose payload sources the registry does not hold, holds in another form, - or that read payloads of the version itself. The [writer](timef-writer.md) checks this at the - first reuse of each source, and raises the errors that opening raises for a missing or differing - source. - -A version lists every payload source it reads from, so a registry resolves only those, never the -sources' own sources. - -A client of a local registry prepares the version and its payload sources before a read. For each -of these releases, the client reuses a complete copy, or copies the release from `fetch_from`. If -`auto_build` is set, the client can also build the release. If the client cannot get a release, -the read raises an error. The client builds a payload source with its own connector, at the exact -version of the source. - -If `fetch_from` is set, the client repairs a damaged copy from that registry. Without `fetch_from`, -the read raises `TimeNetFileError` for a damaged copy. A remote client never runs connector code. -It rejects `fetch_from`. See [Open, load, or download](client.md#open-load-or-download). - -### Remote reads of payload sources - -The `open_version()` method of an HTTP or S3 registry resolves the manifests of the payload sources -before it returns. Reads then use complete cached copies where they exist. Otherwise they read byte -ranges of the remote files, as each read needs them. The first read of Records or tasks downloads -the catalog of the version and makes sure that it matches the manifest. A read never opens the -catalog of a parent release. +[Shared payloads](timef-format.md#shared-payloads)). The version needs its payload sources: + +- A read through `TimeNet.open()` resolves each payload source through the list of sources, and + checks it against the pins. It reads only manifests and, for local files, file sizes. See + [Opening a version](timef-reader.md#opening-a-version) for the checks and their errors. +- `TimeNet.download()` and `copy_release()` copy complete copies of the version and of each payload + source, and check each source against the pins. If a downloaded file has another size or checksum + than its manifest records, the copy raises `TimeNetFileError` and keeps no partial copy. +- `put_dataset` refuses a version if the registry does not hold one of its parents. + +### Remote reads HTTP reads use the range reader and the read-ahead cache of fsspec. The object server must support byte ranges. The client requests a new signed URL after one expires. The registry token goes only with API requests. -A read of byte ranges cannot make sure that a whole file matches its checksum. `download()` and -`download_version()` copy complete files, and make sure that each file has its recorded size and -checksum. `TimeFReader.verify()` reads and hashes every file that a release reads. +A read of byte ranges cannot make sure that a whole file matches its checksum. `TimeNet.download()` +copies complete files, and makes sure that each file has its recorded size and checksum. +`TimeFReader.verify()` reads and hashes every file that a release reads. + +## Sources + +A source is something that can provide a release. `timenet.sources` defines the protocol +`ReleaseSource`, with one method: `resolve(dataset_id, version)`. `version` is an exact version, +such as `"1.2.0"`, or `None` for the latest version. Two kinds of source exist: + +- A registry. Every `BaseRegistry` is a source. +- `InstalledConnectors()`: every installed package that registers a builder backend under the + `timenet.builders` entry point group. + +A source gives one of three answers: + +| Answer | Meaning | +| --- | --- | +| `Found(version, entry)` | The source can serve or build the release. `entry` is the source that does it: the registry or `InstalledConnectors()`. | +| `NotFound()` | The source does not know the release. | +| `NotServed(version, access, access_url, requirements)` | The source lists the release, but does not serve its files. `access` is the effective access of the release, `access_url` the page of the first card that is not open, and `requirements` each such card. `access_error(dataset_id)` returns the `TimeNetAccessError` that names each page. | + +`Found` and `NotServed` also carry the `manifest` that the source read, or `None`, so that you do +not need to fetch it again. + +A `LocalRegistry` serves every release that it holds, so it never answers `NotServed`. A registry +that is not local serves the files of a release only if its effective access is `open`, and +answers `NotServed` otherwise. + +`InstalledConnectors()` answers `Found` only for the version that the card of a connector declares, +because a connector builds only that version. For any other request, it answers `NotFound`. + +`resolve_release(sources, dataset_id, version)` asks an ordered list of sources and follows three +rules: + +1. An exact version comes from the first source that provides it. +2. "Latest" is the highest version across all sources, the connectors included. The order then + decides only which source provides that version. +3. Only `NotFound` and `NotServed` fall through to the next source. Any other error of a source + stops the resolution. + +```python +from timenet.registry import LocalRegistry, open_registry +from timenet.sources import InstalledConnectors, resolve_release + +sources = [ + LocalRegistry("~/.timenet/registry"), + open_registry("timenet://"), + InstalledConnectors(), +] +answer = resolve_release(sources, "timenet/hello-world", None) +``` + +A build uses such a list to find the parent releases that its local registry lacks. Each missing +parent comes from the first source that provides its exact version. + +`TimeNet` takes such a list as `sources`. `LocalRegistry(path, store=True)` marks the local +registry where downloads and builds land. A client accepts at most one such local registry, and a +list with `InstalledConnectors()` needs one. See [Choose the sources](client.md#choose-the-sources). ## Writing to a registry -A `WritableRegistry` adds one write method to the read contract. As a result, -[build](build.md) can publish into any backend, not only a local directory: +A `WritableRegistry` adds two write methods to the read contract. Every backend implements them, +so you can publish to a local directory, a remote registry, or an S3 bucket: | Method | Description | | --- | --- | -| `store(dataset, *, force=False, values_backend="parquet", progress_cb=None)` | Compile a dataset and publish it; returns the stored version. Derives the schema first if absent, skips an already-committed version unless `force`, and checks every payload source in this registry. `values_backend` picks the values plane format: `"parquet"` or `"zarr"`. | +| `put_dataset(dataset, *, force=False, values_backend="parquet", progress_cb=None)` | Compile a dataset and publish it. Returns the stored release as a `DatasetRef`. Derives the schema first if absent, skips an already-committed version unless `force`, and checks every payload source in this registry. `values_backend` picks the values plane format: `"parquet"` or `"zarr"`. On a remote or S3 registry, `force=True` raises `TimeFValidationError` and uploads nothing, because a published version never changes. | +| `put_release(from_directory)` | Publish a complete local release directory as it is; returns the published release as a `DatasetRef`. Only reads `from_directory` and never changes it. Makes sure that the directory holds a readable `manifest.json` and every file that the manifest declares with its recorded size, and raises `TimeFFormatError` or `TimeNetFileError` otherwise. Publishes only those files and `manifest.json`. | | `exists(dataset_id, version)` | Whether a committed version already exists (shared implementation). | -`LocalRegistry` implements `store` by streaming the dataset through a [`TimeFWriter`](timef-writer.md), -which stages under `.tmp-*` and publishes with a single atomic rename. With `force`, the old -version stays in place until the new one is complete. A replace is not isolated from readers that -already opened the old version, or from versions that reuse its payloads, which then fail to open. -`RemoteRegistry` runs the [publish flow](#remote-publishing) below. `S3Registry` uploads to a temporary -prefix and moves each object into place (`manifest.json` last). Each backend passes itself to the -writer, which checks every payload source in that registry while it stages the release locally. -`open_writable_registry(uri)` resolves a URI like `open_registry` but returns a `WritableRegistry`. +`put_dataset` writes the dataset with a [`TimeFWriter`](timef-writer.md) and checks each payload +source of the new release in the registry that receives it. In a local registry, `force=True` +replaces a committed version only once the new one is complete, so a failed write keeps the old +one. A replace can break readers that already opened the old version, and versions that reuse its +payloads then fail to open. +`RemoteRegistry` publishes through the [publish flow](#remote-publishing) below. + +`put_release` publishes a release directory that already exists, without a new compile. The +directory must hold `manifest.json` and every file that it lists, with the recorded size. Only those +files reach the registry. + +`open_registry(uri)` returns a `WritableRegistry`, so you can publish to every registry that it +opens: ```python -from timenet.registry import open_writable_registry +from timenet.registry import open_registry -registry = open_writable_registry("./local_registry") -version = registry.store(dataset) # schema derived if needed, atomic commit +registry = open_registry("./local_registry") +# schema derived if needed, atomic commit +release = registry.put_dataset(dataset) ``` Every backend is a `WritableRegistry`. As a result, this type alone does not show if a backend is a directory the engine can write to or a remote stub. `local_registry_path(uri)` gives this information. It returns the directory that a `file://` URI or a plain path names. It raises -`TimeNetRegistryError` for a remote scheme. `default_registry_path()` uses `local_registry_path` to find -the default local registry. It uses `$TIMENET_REGISTRY` when its value is a local path. Otherwise, -it uses `/registry`. This is how [`timenet-build build`](cli/build.md) resolves its output -when `--out` is absent. The `timenet_connectors.builder` and `load` helpers use it too. +`TimeNetRegistryError` for a remote scheme or an empty string. ### Remote publishing -`RemoteRegistry.store` compiles the dataset locally, then hands the artifacts to the service over the -REST contract. Dataset bytes never pass through the API: the service issues a presigned PUT per file and -the SDK uploads straight to the object store. The token must be a writer token (`$TIMENET_TOKEN`); an -anonymous or read-only caller is rejected by the service. +`RemoteRegistry.put_dataset` compiles the dataset locally and then sends the files to the service. +The data does not pass through the API. The service gives a presigned URL for each file, and the SDK +uploads the file to the object store directly. You need a writer token in `$TIMENET_TOKEN`. The +service refuses an anonymous or read-only caller. `RemoteRegistry.put_release` runs steps 2 to 4 +for a release directory that already exists. 1. Compile into a temporary staging directory with a [`TimeFWriter`](timef-writer.md), deriving the - schema first if the dataset has none. An already-committed version returns early unless `force`. - The writer checks every payload source against the service, so a version whose sources the - service does not hold is refused before anything is published. + schema first if the dataset has none. An already-committed version returns early. The writer + checks every payload source against the service, so a version whose sources the service does not + hold is refused before anything is published. 2. `POST /datasets/{id}/{version}/publish` with the compiled `manifest.json` as the body. The service registers the pending version and returns the list of files it expects. 3. For each file, `POST /datasets/{id}/{version}/publish/upload-url` with `{"path": relpath}` to get a @@ -201,12 +249,23 @@ anonymous or read-only caller is rejected by the service. whether or not the publish succeeds. ```python -from timenet.registry import open_writable_registry +from timenet.registry import open_registry -registry = open_writable_registry("timenet://") # needs $TIMENET_TOKEN -version = registry.store(dataset) # compile, upload, finalize +registry = open_registry("timenet://") # needs $TIMENET_TOKEN +release = registry.put_dataset(dataset) # compile, upload, finalize ``` +### Copying a release + +`timenet.registry.copy_release(reference, source, dest, *, force=False, progress=None)` copies one +release from `source` to the writable registry `dest`. + +A dataset can depend on payload files stored as part of another dataset (parent/child relationship). +`copy_release` copies each parent release too, so that +`dest` can open the copied release. It copies only the releases that `dest` does not hold yet. + +`copy_release` raises each of these errors before it copies anything, so `dest` stays unchanged. + ## `search` ```python @@ -217,9 +276,10 @@ registry.search( ``` Every filter takes a scalar value or a list. The registry ignores `None` filters and combines all -non-`None` filters with AND logic. The consumer CLI, `timenet search`, mirrors this behavior -exactly. `limit` caps the number of results, with a default of 100. `limit=0` returns no results. -A negative `limit` raises `ValueError`. +non-`None` filters with AND logic. `TimeNet.search()` and the consumer CLI, `timenet search`, apply +the same filters to each source of their list, and merge the hits (see +[Browse datasets](client.md#browse-datasets)). `limit` caps the number of results, with a default +of 100. `limit=0` returns no results. | Filter | Matches | | --- | --- | diff --git a/docs/timef-reader.md b/docs/timef-reader.md index 3cd120d0d..9350282c0 100644 --- a/docs/timef-reader.md +++ b/docs/timef-reader.md @@ -16,11 +16,11 @@ There are two everyday ways to get a reader. `TimeFReader.open()` opens a version directory: ```python -from timenet.client import TimeNet +import timenet from timenet.reader import TimeFReader # By dataset ID, through the client. -with TimeNet().open("timenet/hello-world") as reader: +with timenet.TimeNet().open("timenet/hello-world") as reader: dataset = reader.read() values = dataset.records[0].signals[0].to_arrow() @@ -159,11 +159,11 @@ source gave a recording, read the objects with `iter_records()` or `read_tasks() `metadata`. ```python -from timenet.client import TimeNet +import timenet from timenet.dataset import Signal from timenet.types import AnswerTask, read_value -with TimeNet().open("timenet/hello-world") as reader: +with timenet.TimeNet().open("timenet/hello-world") as reader: task = reader.get(AnswerTask, task_id) # Reads this one text and keeps nothing; None without a prompt. prompt = read_value(task.prompt) @@ -298,27 +298,32 @@ and decoded Zarr chunks until it closes. ## Opening a version -To read a version without the client, pass a [`DatasetVersion`](registry.md) to -`TimeFReader(version)`. `open_local_version()` and the `open_version()` method of a registry -return one. A `DatasetVersion` holds the parsed manifest and the local or remote file systems of -the version and its payload sources. A reader that you open this way skips three parts of the -client: the download cache, the access check, and `auto_build`. The two examples open a version -from a registry and from a directory: +To read a version directory without the client, pass a [`DatasetVersion`](registry.md) to +`TimeFReader(version)`. `open_local_version()` returns one. A `DatasetVersion` holds the parsed +manifest and the file systems of the version and its payload sources. A reader that you open this +way reads only from that directory and the registry root around it, without other sources or +builds: ```python from timenet.reader import TimeFReader -from timenet.registry import open_local_version, open_registry +from timenet.registry import open_local_version -registry = open_registry("~/timenet/registry") -with TimeFReader(registry.open_version("timenet/hello-world")) as reader: +with TimeFReader(open_local_version(version_dir)) as reader: dataset = reader.read() +``` -with TimeFReader(open_local_version(version_dir)) as reader: +To read from one registry, give the client a list with only that registry: + +```python +import timenet + +client = timenet.TimeNet(sources=["~/timenet/registry"]) +with client.open("timenet/hello-world") as reader: dataset = reader.read() ``` -`open_local_version()` and the `open_version()` method of a registry open a version at once, but -they read only metadata. Opening makes sure that: +`open_local_version()` opens a version at once, but it reads only metadata. Opening makes sure +that: - `manifest.json` is valid and has a supported format version. - Each file that the manifest lists, `control.duckdb` included, is present with its recorded size. @@ -340,7 +345,7 @@ releases that source reuses. | A payload source differs from its pins | `TimeFFormatError` | The source and the first differing file | | A file is missing or has another size | `TimeNetFileError` | The file, its expected size, and its actual size | -An invalid or unsupported manifest raises a validation error, and a missing one raises +An invalid or unsupported manifest raises `TimeNetManifestError`, and a missing one raises `FileNotFoundError`. ### Where payload sources are found @@ -348,8 +353,8 @@ An invalid or unsupported manifest raises a validation error, and a missing one Payload sources resolve relative to a registry root. By default, `open_local_version()` takes the root that holds the version as `///`, so a version and its sources keep working when they move to another root together. Pass `sources_root=` to find them under another -root. `TimeFReader.open()` takes `sources_root=` too. A registry's `open_version()` finds them in -that registry. +root. `TimeFReader.open()` takes `sources_root=` too. The client finds them through its list of +sources. A version without payload sources opens from any directory. A version with payload sources that is not stored as `///` needs `sources_root=`, and opening it without one diff --git a/docs/timef-writer.md b/docs/timef-writer.md index 2deaa9a21..d7547c912 100644 --- a/docs/timef-writer.md +++ b/docs/timef-writer.md @@ -79,8 +79,7 @@ therefore needs a different dataset ID or version than its sources. Its readers find the sources in the registry the release is published to, so the writer checks each source there at its first reuse, during `write()`. Pass that registry as `registry=`. It defaults to a -local registry over `root`, and `S3Registry.store()` and `RemoteRegistry.store()` pass themselves while -they stage the release locally. The writer fetches the source's manifest from the registry once and +local registry over `root`. The writer fetches the manifest of the source from the registry once and compares the record of each reused file. It refuses the write, before it writes the remaining values and with nothing published, when: diff --git a/docs/types.md b/docs/types.md index 2fb87b1bf..19f34e2c3 100644 --- a/docs/types.md +++ b/docs/types.md @@ -294,19 +294,27 @@ TimeNet raises its own exceptions from `timenet.errors`. | `TimeNetError` | Base class for TimeNet failures. | | `TimeNetRegistryError` | A registry cannot complete an operation. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | +| `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | +| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. For a publish, the hint asks for a new version. For a copy into the local registry, the hint names the release folder to delete. | | `TimeFValidationError` | Input violates a TimeF invariant. | | `TimeFNullValuesError` | Values contain nulls that NumPy cannot hold. A `TimeFValidationError`. | | `TimeFNotFoundError` | An ID, parent alias, or parent release names nothing in the dataset, or the stored object has another kind. A `TimeFValidationError` and a `LookupError`. | | `TimeFFormatError` | A stored TimeF artifact is corrupt or unsupported. | | `TimeNetFileError` | A file of a release is missing, or does not match its manifest record. A `TimeFFormatError`. | +| `TimeNetManifestError` | A stored manifest is not a valid manifest. `where` names the file or the place in a registry, and `detail` names the first problem. A `TimeFFormatError`. | | `TimeFReaderError` | A reader could not complete a read, e.g. its connection was closed while the read ran. | | `TimeNetInvalidCardError` | `dataset.yaml` cannot be read or is not a mapping. | | `TimeNetConnectorError` | A connector found that an assumption about its source data, a download, or a parent does not hold. `TimeNetDownloadError` is a `TimeNetConnectorError`. | +| `TimeNetBuildError` | A build failed: the connector build, or the environment that it needs. | +| `TimeNetIsolationError` | An isolated build cannot make its environment, for example because `uv` is missing. A `TimeNetBuildError`. | + +Most TimeNet errors come with a `hint`: what you can do about the error. Pydantic `ValidationError` reports field and schema errors in a dataset card. The error includes the native field location and details. -Manifest parsing uses the same error for invalid JSON and invalid fields. +`Manifest.model_validate_json()` uses the same error for invalid JSON and invalid fields. A registry +that reads a stored manifest raises `TimeNetManifestError` instead. `TimeFValidationError` also subclasses `ValueError`. Existing input-validation handlers can catch it without losing the TimeNet-specific error type. @@ -314,4 +322,7 @@ without losing the TimeNet-specific error type. `SpanOutsideWindowWarning` reports a span outside a Signal window when the caller has chosen warning behavior. It derives from `TimeNetWarning` and `UserWarning`. +`ConnectorCardWarning` reports a connector card that cannot be read. The list of installed +connectors leaves its dataset out, so the card fails only its own dataset. + See the [API reference for `timenet.types`](api/types.md) for all symbols. diff --git a/docs/usage.md b/docs/usage.md index bcc04449d..09736ea7e 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -14,9 +14,10 @@ Every TimeF dataset loads through the same client. Each Record contains a Source Source contains Signals. ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +client = timenet.TimeNet(sources=[".timenet-registry"]) +dataset = client.load("timenet/hello-world") record = dataset.records[0] signal = record.signals[0] ``` @@ -25,7 +26,7 @@ signal = record.signals[0] instead. The reader's reads leave each Signal's values in storage until an array method reads them: ```python -with TimeNet(".timenet-registry").open("timenet/hello-world") as reader: +with client.open("timenet/hello-world") as reader: dataset = reader.read() values = dataset.records[0].signals[0].to_numpy() ``` @@ -96,9 +97,9 @@ Install the adapter with `uv add 'timenet[torch]'`. Then open a map-style PyTorc view holds an open reader, so use it in a `with` block or call its `close()`: ```python -from timenet.client import TimeNet +import timenet -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) with client.open_torch("timenet/hello-world") as view: item = view[0] @@ -126,12 +127,11 @@ from pathlib import Path from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split -from timenet.client import TimeNet -import timenet_connectors +import timenet registry = Path(".timenet-registry") -timenet_connectors.build("timenet/test-mean", out=registry) -dataset = TimeNet(registry).load("timenet/test-mean") +timenet.build("timenet/test-mean", store=registry) +dataset = timenet.TimeNet(sources=[registry]).load("timenet/test-mean") x, y = dataset.to_features_and_targets(output="numpy") x_train, x_test, y_train, y_test = train_test_split( @@ -156,10 +156,10 @@ The runnable version is ## Work with downloaded files -`download()` returns a local TimeF version directory: +`download()` copies a release into the local registry and returns its TimeF version directory there: ```python -path = TimeNet().download("org/name@1.0.0") +path = timenet.TimeNet().download("org/name@1.0.0") ``` The directory contains `manifest.json`, `control.duckdb`, and a Parquet or Zarr values plane. Use the diff --git a/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md b/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md index f86e0f7e1..edfb7bed6 100644 --- a/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md +++ b/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md @@ -436,14 +436,15 @@ matches its row: The build downloads the three files, 387 MB, from the Hub at the pinned commit: ```bash -uv run timenet-build build tsaqa/tsaqa-benchmark --out /tmp/timenet +uv run timenet-build build tsaqa/tsaqa-benchmark --local-registry /tmp/timenet ``` On 2026-10-29, building in-process took 79 s and peaked at 4.1 GB of memory, most of it the 510,093 Records. The built dataset takes 123 MB. To load it: ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet("/tmp/timenet").load("tsaqa/tsaqa-benchmark") +client = timenet.TimeNet(sources=["/tmp/timenet"]) +dataset = client.load("tsaqa/tsaqa-benchmark") ``` From 838da7b443294433fbfaa2987b1b4cd22c2fb966 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 10:06:10 +0100 Subject: [PATCH 02/28] docs: match the code names of the docs to the final code The docs named arguments and types that the stack removed or renamed. This commit changes them to the names in the code: - timenet.build() takes local_registry= and sources=, and returns the release directory. store=, parents_from= and the returned plan are gone (build.md, usage.md). - default_store() is now default_local_registry() (client.md, the timenet-datasets skill). - LocalRegistry takes no store=True. The first LocalRegistry of a list that is not read_only=True gets the downloads and builds (client.md, registry.md, the skill). - Remote registries take no local folder. The docs drop store= of open_registry(), RemoteRegistry and S3Registry, the local_root row, and the reads of local copies through a remote registry (client.md, registry.md, cli/timenet.md, the skill). - timenet.Plan is not exported (the skill). - BytesMoved has subject, not url (client.md). - The answers of a source are FoundInRegistry, FoundInConnector, NotFound and NotServed, and NotServed has no access_url (registry.md). - copy_release() takes with_payload_sources= (registry.md). - The metavar of --remote-registry, --publish-to, --dest and --source is URL|PATH (cli/build.md, cli/timenet.md, the skill). The trailing-whitespace hook also removes a space at the end of a line in build.md and in registry.md. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 24 +++++++------- docs/build.md | 26 +++++++-------- docs/cli/build.md | 4 +-- docs/cli/timenet.md | 4 +-- docs/client.md | 32 +++++++++---------- docs/registry.md | 40 ++++++++++++------------ docs/usage.md | 2 +- 7 files changed, 65 insertions(+), 67 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 0ba761233..24cd3784b 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -29,7 +29,7 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | `timenet search [filters]` | Filter the datasets of the sources, connector cards included. See flags below. | | `timenet info [@version]` | Manifest summary of the release that `open()` reads: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). A dataset that only a connector provides shows its card and the build command. | | `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory to stdout. Never builds. Prints "Nothing to do" if the local registry holds everything. | -| `timenet publish [@version] --dest URL [--source URL]` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | +| `timenet publish [@version] --dest URL|PATH [--source URL|PATH]` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | | `timenet local info` | Path and size of the cache, then one line each for the downloads and the builds in the local registry (count and size). | | `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | | `timenet local cache info` / `clear [--yes]` | Path and size of the cache (`$TIMENET_CACHE`), or delete its raw source files (`downloads/`) and control databases (`control/`) that carry TimeNet's `CACHEDIR.TAG`. | @@ -60,9 +60,9 @@ flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. - `--local-registry PATH`: a local registry, the first source that a lookup checks, and the place where downloads and builds land (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). -- `--remote-registry URL`: a remote registry, as a URL or a local path, the second source that a - lookup checks, if the local registry does not have the dataset (`$TIMENET_REMOTE_REGISTRY`, else - `timenet://`). +- `--remote-registry URL|PATH`: a remote registry, as a URL or a local path, the second source that + a lookup checks, if the local registry does not have the dataset (`$TIMENET_REMOTE_REGISTRY`, + else `timenet://`). - `--auto-build`/`--no-auto-build`: whether installed connectors build a dataset that neither the local nor the remote registry has (`$TIMENET_AUTO_BUILD`, else true). @@ -79,7 +79,7 @@ tn = timenet.TimeNet() # the default list of sources, see below tn.list() # list[DatasetMetadata] tn.get_manifest("chengsenwang/tsqa") # Manifest (add a version or use id@version) tn.search(domain=..., task=AnswerTask, tag="ecg", limit=50) # filters are scalar-or-list, ANDed -path = tn.download("chengsenwang/tsqa") # -> /// dir +path = tn.download("chengsenwang/tsqa") # -> /// dir with tn.open("chengsenwang/tsqa") as reader: # TimeFReader; values stay in storage dataset = reader.read() dataset = tn.load("chengsenwang/tsqa") # every value read into memory @@ -89,13 +89,13 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int URL, a `file://` URI, a local path, or `timenet.InstalledConnectors()`. Pass a list even for one registry: `timenet.TimeNet(sources=[".timenet-registry"])`. An exact version comes from the first source that has it; "latest" is the highest version of any source. -- One local registry in the list can declare `timenet.LocalRegistry(path, store=True)`. Builds land there. A - list with `InstalledConnectors()` needs a local registry with `store=True`, and only such a list builds on a read. A registry - URL in the list reads the complete copy of the same release in that local registry. Without one - in the list, it reads only its own files. +- The first `timenet.LocalRegistry(path)` of the list that is not `read_only=True` gets the + downloads and builds. A path in the list opens such a registry too. A list with + `InstalledConnectors()` needs a writable local registry, and only such a list builds on a read. + A remote registry reads only its own files. - `TimeNet()` uses `timenet.client.default_sources()`: the local registry `$TIMENET_LOCAL_REGISTRY`, then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. The - first entry is always the registry configured with `store=True`; `default_store()` returns it alone. + first entry is always the local registry; `default_local_registry()` returns it alone. - `list()` and `search()` merge every source of the list, connector cards included: one row per dataset at the highest version, sorted by ID. `limit` applies after the merge. A connector never matches `task` or `time_series_spec`. A registry without a catalog (S3) adds nothing, and a card @@ -105,8 +105,8 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int files. A registry that lists a release without serving it gives its listed manifest. A release that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). `get_card()` returns the card, also from a connector. `plan_read()` returns the plan of `open()` - (`timenet.Plan`, one step per release with its action and source) and runs nothing. -- `download()` needs a local registry configured with `store=True`. It resolves like `open()`. The release and each payload source that + (one step per release with its action and source) and runs nothing. +- `download()` needs a writable local registry in the list. It resolves like `open()`. The release and each payload source that the local registry lacks or holds damaged come from the first registry other than the local registry that provides it, so they can come from different registries. If the local registry holds a complete copy of the release, only the missing payload sources are copied. It never builds: a release that only a diff --git a/docs/build.md b/docs/build.md index 7073adfdc..b2829d443 100644 --- a/docs/build.md +++ b/docs/build.md @@ -42,7 +42,7 @@ Use `--no-isolation` while developing a connector in the current environment: timenet-build build org/name --local-registry .timenet-registry --no-isolation ``` -Most parameters can also be specified as environment variables. For example, `$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the Python SDK also honors them. +Most parameters can also be specified as environment variables. For example, `$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the Python SDK also honors them. For each build, the used Python and package and TimeNet versions are recorded to the build's manifest. @@ -82,27 +82,27 @@ timenet.publish("org/name", dest="s3://bucket/registry") ``` `timenet.build()` builds a release with the installed connector that claims its dataset ID, in an -[isolated environment](#isolated-build-environments). It returns the plan that ran: one step for -each release that it copied or built, in order. Each argument that you leave out comes from the -environment at the time of the call: +[isolated environment](#isolated-build-environments). It returns the release directory in the +local registry. Each argument that you leave out comes from the environment at the time of the +call: -- `store` comes from `$TIMENET_LOCAL_REGISTRY`. -- `parents_from` comes from `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With +- `local_registry` comes from `$TIMENET_LOCAL_REGISTRY`. +- `sources` comes from `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With `TIMENET_AUTO_BUILD=false`, the connectors stay out. -So a default build lands where `TimeNet()` reads. `store` takes a local directory, a `file://` URI -or a `LocalRegistry`. `parents_from` lists, in order, where the parents and payload sources that the local registry lacks come from (see [Prepare parent releases](#prepare-parent-releases)). +So a default build lands where `TimeNet()` reads. `local_registry` takes a local directory, a +`file://` URI or a `LocalRegistry`. `sources` lists, in order, where the parents and payload sources +that the local registry lacks come from (see [Prepare parent releases](#prepare-parent-releases)). ```python import timenet -plan = timenet.build( +path = timenet.build( "org/name@1.2.0", - store=".timenet-registry", - parents_from=["timenet://", timenet.InstalledConnectors()], + local_registry=".timenet-registry", + sources=["timenet://", timenet.InstalledConnectors()], ) -for step in plan.steps: - print(step.release, step.action) +print(path) ``` `timenet.publish()` copies a complete release and the payload sources that it needs from `source`, diff --git a/docs/cli/build.md b/docs/cli/build.md index 614b906da..4a16b1469 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -23,8 +23,8 @@ author or change a connector. ```bash timenet-build [--quiet] build [@] \ [--force] [--keep-cache] [--isolation | --no-isolation] \ - [--publish-to ] [--dry-run] \ - [--local-registry ] [--remote-registry ] \ + [--publish-to ] [--dry-run] \ + [--local-registry ] [--remote-registry ] \ [--auto-build | --no-auto-build] ``` diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index eb92eb64f..f271296d5 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -33,7 +33,7 @@ To use a local or S3 registry instead, see [Selecting the sources](#selecting-th | `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources. | | `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | | `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. It never builds. See [`download()`](../client.md#open-load-or-download). | -| `timenet publish [@version] --dest URL` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source URL`, and without a version it is the latest version there. A remote `--source` reads the complete copies of the same releases in the local registry. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | +| `timenet publish [@version] --dest URL|PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source URL|PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | | `timenet local info` | Print the path and size of the cache. Then print one line for the downloads and one line for the builds in the local registry, such as `Downloads in : 2 releases, 1.4 GB`. | | `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | | `timenet local cache info` | Print the path and size of the cache, `$TIMENET_CACHE`. | @@ -66,7 +66,7 @@ overrides one [variable](../client.md#settings): | Flag | Variable | Default | What it sets | | --- | --- | --- | --- | | `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the place where downloads and builds land. | -| `--remote-registry URL` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `--remote-registry URL|PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | | `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | A flag wins over its variable, and the variable wins over its default. Flags go after the subcommand: diff --git a/docs/client.md b/docs/client.md index 22d8d0aa4..ba91a89d9 100644 --- a/docs/client.md +++ b/docs/client.md @@ -33,7 +33,7 @@ A client takes an ordered list of sources. A source is a local or remote registr import timenet client = timenet.TimeNet(sources=[ - timenet.LocalRegistry(".timenet-registry", store=True), + timenet.LocalRegistry(".timenet-registry"), "https://registry.example.com", "timenet://", timenet.InstalledConnectors(), @@ -43,26 +43,25 @@ client = timenet.TimeNet(sources=[ - `InstalledConnectors()` stands for every installed package that registers a builder backend, such as `timenet-connectors`. If only this source provides a release, a read builds it. Without it in the list, a client never runs connector code. -- `LocalRegistry(path, store=True)` marks the local registry where downloads and builds land. A - client accepts at most one such local registry, and a list with `InstalledConnectors()` needs - one. A `LocalRegistry` without the flag is only read, for example a mirror on a network drive. -- A registry URL in the list gets the directory of this local registry. If the local registry holds - a complete copy of the same release, with the same manifest apart from the build environment, the - registry reads that copy instead of its remote files. Without this local registry in the list, it - reads only its own files. +- The first `LocalRegistry` of the list that is not `read_only=True` is the local registry where + downloads and builds land. A path in the list opens such a registry too. The client only reads a + later local registry. `LocalRegistry(path, read_only=True)` is never written, for example a + mirror on a network drive. A list with `InstalledConnectors()` needs a writable local registry. +- A remote registry reads only its own files. An exact version comes from the first source that provides it, while "latest" is the highest version that any source provides. Only "not found" and "not served" fall through to the next source, and any other error stops the call. So if a registry in the list does not answer, a request for "latest" fails. -A client that only reads takes a list without `store=True`: +A client that must never write takes `read_only=True`. A list of remote registries alone has no +local registry, so it never writes either: ```python import timenet +local = timenet.TimeNet(sources=[timenet.LocalRegistry(".timenet-registry", read_only=True)]) hosted = timenet.TimeNet(sources=["timenet://"]) -local = timenet.TimeNet(sources=[".timenet-registry"]) s3 = timenet.TimeNet(sources=["s3://my-bucket/timenet"]) ``` @@ -71,7 +70,7 @@ the list is: ```python [ - LocalRegistry("~/.timenet/registry", store=True), + LocalRegistry("~/.timenet/registry"), "timenet://", InstalledConnectors(), ] @@ -80,7 +79,7 @@ the list is: `TIMENET_LOCAL_REGISTRY` moves the local registry, `TIMENET_REMOTE_REGISTRY` replaces `timenet://`, and `TIMENET_AUTO_BUILD=false` removes `InstalledConnectors()` (see [Settings](#settings)). The arguments `local_registry`, `remote_registry` and `auto_build` of `default_sources()` override -them, and `timenet.client.default_store()` returns the local registry alone. +them, and `timenet.client.default_local_registry()` returns the local registry alone. ## Settings @@ -207,7 +206,7 @@ declares a newer version than the local registry holds, the call builds the newe import timenet client = timenet.TimeNet(sources=[ - timenet.LocalRegistry(".timenet-registry", store=True), + timenet.LocalRegistry(".timenet-registry"), "timenet://", ]) with client.open("org/name") as reader: @@ -221,9 +220,8 @@ If only `InstalledConnectors()` provides a release, the call builds it into the first and reads it there. The build copies or builds each missing parent from the same sources, as [Prepare parent releases](build.md#prepare-parent-releases) describes. -When `open()`, `load()`, or `open_torch()` reads from an HTTP or S3 registry, it reads the -complete copy of the same release in the local registry, if it holds one. Otherwise it reads the -remote files by range. +When `open()`, `load()`, or `open_torch()` reads from an HTTP or S3 registry, it reads the remote +files by range. A child dataset gives the objects that it imports from a parent IDs of its own. To read such a copy by the ID that the parent gives it, pass the parent to `get()` with `parent=`: @@ -269,7 +267,7 @@ from timenet.progress import BytesMoved def show(event): if isinstance(event, BytesMoved): - print(f"{event.url}: {event.done} of {event.total} bytes") + print(f"{event.subject}: {event.done} of {event.total} bytes") client.download("chengsenwang/tsqa", progress=show) ``` diff --git a/docs/registry.md b/docs/registry.md index f3a027fe4..eac232464 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -36,9 +36,9 @@ registry = open_registry("./local_registry") | `timenet://` | `RemoteRegistry`, an alias for the hosted `https://registry.timenet.ai` | `open_registry()` returns a `WritableRegistry`, so you can also publish to the registry that it -opens (see [Writing to a registry](#writing-to-a-registry)). `store=` gives an HTTP or S3 registry -the directory of the local registry where downloads and builds land (see -[Remote reads](#remote-reads)). A local registry ignores it. +opens (see [Writing to a registry](#writing-to-a-registry)). A remote registry reads only its own +files. To read a local copy first, put its `LocalRegistry` before the remote registry in the +sources. !!! note "Catalog support" Every backend serves reads and takes new releases. Remote publishing needs a writer token, and @@ -73,9 +73,8 @@ uv add 'timenet[s3]' The layout under the prefix is `datasets/org/name/version/…`, with `manifest.json` beside the data files. `put_dataset` uploads to a temporary `version.tmp-` prefix and moves each object into place (`manifest.json` last), so that failed publishing does not leak partial versions. -`TimeNet.open()` reads lazily and `load()` reads every value. Both read the complete copy of the -same release in the local registry, if it holds one (see [Remote reads](#remote-reads)). Otherwise -they read remote payload ranges through a pyarrow `S3FileSystem`. The backend has no catalog, so +`TimeNet.open()` reads lazily and `load()` reads every value. Both read remote payload ranges +through a pyarrow `S3FileSystem` (see [Remote reads](#remote-reads)). The backend has no catalog, so `list` and `search` are unsupported: publish and fetch by an explicit `org/name@version`. ## `BaseRegistry` @@ -88,7 +87,6 @@ Every backend implements this contract. | `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeFFormatError` if the stored manifest is not valid, or if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | | `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Any other registry gives its name. | -| `local_root` | The directory whose complete releases the registry reads: the root of a local registry, or the local registry that `store=` gives a remote registry. `None` for a remote registry without `store=`. | | `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | | `find_manifest(release)` | The manifest of an exact release, or `None` if the registry does not hold it. Raises `TimeFFormatError` if the stored manifest names another release. | | `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | @@ -136,9 +134,9 @@ copies complete files, and makes sure that each file has its recorded size and c ## Sources -A source is something that can provide a release. `timenet.sources` defines the protocol -`ReleaseSource`, with one method: `resolve(dataset_id, version)`. `version` is an exact version, -such as `"1.2.0"`, or `None` for the latest version. Two kinds of source exist: +A source is something that can provide a release. `timenet.sources` defines the type +`ReleaseSource`. Each source has the method `resolve(dataset_id, version)`. `version` is an exact +version, such as `"1.2.0"`, or `None` for the latest version. Two kinds of source exist: - A registry. Every `BaseRegistry` is a source. - `InstalledConnectors()`: every installed package that registers a builder backend under the @@ -148,12 +146,14 @@ A source gives one of three answers: | Answer | Meaning | | --- | --- | -| `Found(version, entry)` | The source can serve or build the release. `entry` is the source that does it: the registry or `InstalledConnectors()`. | +| `FoundInRegistry(version, registry)` | The registry serves the release. | +| `FoundInConnector(builder)` | An installed connector builds the release. `version` is the version that its card declares. | | `NotFound()` | The source does not know the release. | -| `NotServed(version, access, access_url, requirements)` | The source lists the release, but does not serve its files. `access` is the effective access of the release, `access_url` the page of the first card that is not open, and `requirements` each such card. `access_error(dataset_id)` returns the `TimeNetAccessError` that names each page. | +| `NotServed(version, access, requirements)` | The source lists the release, but does not serve its files. `access` is the effective access of the release, and `requirements` names each card that is not open, with its access page. `access_error(dataset_id)` returns the `TimeNetAccessError` that names each page. | -`Found` and `NotServed` also carry the `manifest` that the source read, or `None`, so that you do -not need to fetch it again. +The type `Found` covers `FoundInRegistry` and `FoundInConnector`. `FoundInRegistry` and +`NotServed` also carry the `manifest` that the source read, or `None`, so that you do not need to +fetch it again. A `LocalRegistry` serves every release that it holds, so it never answers `NotServed`. A registry that is not local serves the files of a release only if its effective access is `open`, and @@ -186,9 +186,9 @@ answer = resolve_release(sources, "timenet/hello-world", None) A build uses such a list to find the parent releases that its local registry lacks. Each missing parent comes from the first source that provides its exact version. -`TimeNet` takes such a list as `sources`. `LocalRegistry(path, store=True)` marks the local -registry where downloads and builds land. A client accepts at most one such local registry, and a -list with `InstalledConnectors()` needs one. See [Choose the sources](client.md#choose-the-sources). +`TimeNet` takes such a list as `sources`. The first `LocalRegistry` of the list that is not +`read_only=True` gets the downloads and builds, and a list with `InstalledConnectors()` needs one. +See [Choose the sources](client.md#choose-the-sources). ## Writing to a registry @@ -257,10 +257,10 @@ release = registry.put_dataset(dataset) # compile, upload, finalize ### Copying a release -`timenet.registry.copy_release(reference, source, dest, *, force=False, progress=None)` copies one -release from `source` to the writable registry `dest`. +`timenet.registry.copy_release(reference, source, dest, *, with_payload_sources=True, force=False, +progress=None)` copies one release from `source` to the writable registry `dest`. -A dataset can depend on payload files stored as part of another dataset (parent/child relationship). +A dataset can depend on payload files stored as part of another dataset (parent/child relationship). `copy_release` copies each parent release too, so that `dest` can open the copied release. It copies only the releases that `dest` does not hold yet. diff --git a/docs/usage.md b/docs/usage.md index 09736ea7e..c5001face 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -130,7 +130,7 @@ from sklearn.model_selection import train_test_split import timenet registry = Path(".timenet-registry") -timenet.build("timenet/test-mean", store=registry) +timenet.build("timenet/test-mean", local_registry=registry) dataset = timenet.TimeNet(sources=[registry]).load("timenet/test-mean") x, y = dataset.to_features_and_targets(output="numpy") From 3c155416716cf911f0d58882d6e2bb1c02c8f71a Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 10:07:54 +0100 Subject: [PATCH 03/28] docs: name every API page by the full dotted path of its module The API pages took the last part of the module path as their name, and only a clash, such as timenet.sources and timenet_connectors.sources, gave a page its full path. So most names did not show the package, and a new module could rename the pages of an old one. scripts/gen_api_docs.py now names every page by its full dotted path, for example api/timenet.client.md. It needs no clash check, because two modules never share a dotted path. The commit regenerates the nav in zensical.toml and updates the links to the API pages in client.md, connectors.md, manifest.md, registry.md, timef-dataset.md, timef-reader.md and types.md. Part of TNET-157. --- docs/client.md | 2 +- docs/connectors.md | 2 +- docs/manifest.md | 2 +- docs/registry.md | 2 +- docs/timef-dataset.md | 2 +- docs/timef-reader.md | 2 +- docs/types.md | 2 +- scripts/gen_api_docs.py | 40 ++++++++++------------------------------ zensical.toml | 2 +- 9 files changed, 18 insertions(+), 38 deletions(-) diff --git a/docs/client.md b/docs/client.md index ba91a89d9..3637ced1f 100644 --- a/docs/client.md +++ b/docs/client.md @@ -340,4 +340,4 @@ workflows. --- -See the [API reference for `timenet.client`](api/client.md) for every parameter and return type. +See the [API reference for `timenet.client`](api/timenet.client.md) for every parameter and return type. diff --git a/docs/connectors.md b/docs/connectors.md index 7446115be..1d7596bbe 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -397,7 +397,7 @@ that split keeps them. A child with a split of its own removes them and adds its --- -See the [API reference for `timenet.connectors`](api/connectors.md) for the full symbol listing. +See the [API reference for `timenet.connectors`](api/timenet.connectors.md) for the full symbol listing. ## Reading source rows diff --git a/docs/manifest.md b/docs/manifest.md index f62ed6670..7137ea718 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -134,4 +134,4 @@ and its payload sources into the local registry. See [Payload sources](registry. --- -See the [API reference for `timenet.manifest`](api/manifest.md) for the full symbol listing. +See the [API reference for `timenet.manifest`](api/timenet.manifest.md) for the full symbol listing. diff --git a/docs/registry.md b/docs/registry.md index eac232464..6c7c02cbb 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -297,4 +297,4 @@ manifest. The manifest always carries the derived schema, so the registry does n --- -See the [API reference for `timenet.registry`](api/registry.md) for the full symbol listing. +See the [API reference for `timenet.registry`](api/timenet.registry.md) for the full symbol listing. diff --git a/docs/timef-dataset.md b/docs/timef-dataset.md index e06bfea83..a2ac0cdc9 100644 --- a/docs/timef-dataset.md +++ b/docs/timef-dataset.md @@ -265,7 +265,7 @@ The stored version contains `manifest.json`, an immutable `control.duckdb`, and values plane. See [TimeF format](timef-format.md) and [TimeFWriter](timef-writer.md) for the physical layout. -See the [API reference for `timenet.dataset`](api/dataset.md) for the full symbol listing. +See the [API reference for `timenet.dataset`](api/timenet.dataset.md) for the full symbol listing. ## Importing from a parent diff --git a/docs/timef-reader.md b/docs/timef-reader.md index 9350282c0..ea49c3c22 100644 --- a/docs/timef-reader.md +++ b/docs/timef-reader.md @@ -402,4 +402,4 @@ Relationships return as object references: task inputs are Records, spans hold S Sources, axes, and annotations have no id. The reader does not keep the Python object identity from the writer process. -See the [API reference for `timenet.reader`](api/reader.md) for all members. +See the [API reference for `timenet.reader`](api/timenet.reader.md) for all members. diff --git a/docs/types.md b/docs/types.md index 19f34e2c3..62f7d0971 100644 --- a/docs/types.md +++ b/docs/types.md @@ -325,4 +325,4 @@ behavior. It derives from `TimeNetWarning` and `UserWarning`. `ConnectorCardWarning` reports a connector card that cannot be read. The list of installed connectors leaves its dataset out, so the card fails only its own dataset. -See the [API reference for `timenet.types`](api/types.md) for all symbols. +See the [API reference for `timenet.types`](api/timenet.types.md) for all symbols. diff --git a/scripts/gen_api_docs.py b/scripts/gen_api_docs.py index ae2f66a01..8fa586505 100644 --- a/scripts/gen_api_docs.py +++ b/scripts/gen_api_docs.py @@ -19,8 +19,7 @@ from __future__ import annotations import ast -from collections import Counter -from dataclasses import dataclass, replace +from dataclasses import dataclass from pathlib import Path import re @@ -62,14 +61,12 @@ class ApiModule: """A module's API page settings. Attributes: - dotted: The import path, such as ``timenet.registry``. - slug: The page name under ``docs/api/``: the last part of the import path, or the full path on a clash. + dotted: The import path, such as ``timenet.registry``. It is also the page name under ``docs/api/``. summary: The cleaned first line of the module docstring, or ``""``. show_submodules: Whether to include submodules recursively. True for a package without ``__all__``. """ dotted: str - slug: str summary: str show_submodules: bool @@ -154,9 +151,9 @@ def _discover(package: str, root: Path) -> list[ApiModule]: # A package that builds a public API via __all__ is documented at the # package level; one that doesn't keeps its surface in submodules, so # render them recursively instead of an empty page. - modules.append(ApiModule(dotted, name, _module_summary(init_file), not _has_dunder_all(init_file))) + modules.append(ApiModule(dotted, _module_summary(init_file), not _has_dunder_all(init_file))) else: - modules.append(ApiModule(dotted, name, _module_summary(child), False)) + modules.append(ApiModule(dotted, _module_summary(child), False)) return modules @@ -170,7 +167,7 @@ def _write_stub(mod: ApiModule) -> None: lines = ["---", f'description: "{_yaml_escape(desc)}"', "---", "", f"::: {mod.dotted}"] if mod.show_submodules: lines += [" options:", " show_submodules: true"] - (OUTPUT_DIR / f"{mod.slug}.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + (OUTPUT_DIR / f"{mod.dotted}.md").write_text("\n".join(lines) + "\n", encoding="utf-8") def _yaml_escape(text: str) -> str: @@ -208,7 +205,7 @@ def _write_index(by_package: dict[str, list[ApiModule]]) -> None: for package, mods in by_package.items(): parts.append(f"## `{package}`\n") for mod in mods: - bullet = f"- [`{mod.dotted}`]({mod.slug}.md)" + bullet = f"- [`{mod.dotted}`]({mod.dotted}.md)" if mod.summary: bullet += f" — {mod.summary}" parts.append(bullet) @@ -227,7 +224,7 @@ def _nav_line(by_package: dict[str, list[ApiModule]]) -> str: """ items = ['{ "Overview" = "api/index.md" }'] for mods in by_package.values(): - items += [f'{{ "{mod.dotted}" = "api/{mod.slug}.md" }}' for mod in mods] + items += [f'{{ "{mod.dotted}" = "api/{mod.dotted}.md" }}' for mod in mods] return '{ "API Reference" = [' + ", ".join(items) + "] }," @@ -261,33 +258,16 @@ def main() -> None: """Generate the API reference pages and landing page, and rewrite the nav region of ``zensical.toml``. It also deletes each page in ``docs/api/`` that no current module maps to. - - Raises: - SystemExit: If page names clash or the nav markers are missing or duplicated. """ OUTPUT_DIR.mkdir(parents=True, exist_ok=True) by_package = {pkg: _discover(pkg, root) for pkg, root in PACKAGE_ROOTS.items()} - # Both packages can have a module with the same name, such as timenet.sources and - # timenet_connectors.sources. The page of each such module takes its full dotted path. - shared = { - slug for slug, count in Counter(mod.slug for mods in by_package.values() for mod in mods).items() if count > 1 - } - by_package = { - pkg: [replace(mod, slug=mod.dotted) if mod.slug in shared else mod for mod in mods] - for pkg, mods in by_package.items() - } all_mods = [mod for mods in by_package.values() for mod in mods] - - # "index" is reserved for the landing page written by _write_index below. - slugs: dict[str, str] = {"index": "(API index landing page)"} - for mod in all_mods: - if mod.slug in slugs: - raise SystemExit(f"slug collision: {mod.dotted} and {slugs[mod.slug]} both map to {mod.slug}.md") - slugs[mod.slug] = mod.dotted + # Each page takes the full dotted path of its module, so two pages never share a name. + pages = {mod.dotted for mod in all_mods} # Drop stale stubs left by a previous run with a different module set. for stale in OUTPUT_DIR.glob("*.md"): - if stale.name != "index.md" and stale.stem not in slugs: + if stale.name != "index.md" and stale.stem not in pages: stale.unlink() for mod in all_mods: diff --git a/zensical.toml b/zensical.toml index 6342476d2..cfbe032a8 100644 --- a/zensical.toml +++ b/zensical.toml @@ -22,7 +22,7 @@ nav = [ { "Datasets" = [{ "Use datasets" = [{ "The client (SDK)" = "client.md" }, { "Registries" = "registry.md" }, { "Load into your stack" = "usage.md" }] }, { "Build datasets" = [{ "Connectors" = "connectors.md" }, { "Build & publish" = "build.md" }] }, { "Catalog" = [{ "Datasets" = "catalog/datasets.md" }, { "Benchmarks" = "catalog/benchmarks.md" }, { "Licensing" = "catalog/licensing.md" }] }] }, { "CLI" = [{ "Overview" = "cli/index.md" }, { "timenet (consumer)" = "cli/timenet.md" }, { "timenet-build (producer)" = "cli/build.md" }] }, # >>> api-nav: generated by scripts/gen_api_docs.py — do not edit >>> - { "API Reference" = [{ "Overview" = "api/index.md" }, { "timenet.builders" = "api/builders.md" }, { "timenet.client" = "api/client.md" }, { "timenet.config" = "api/config.md" }, { "timenet.connectors" = "api/connectors.md" }, { "timenet.dataset" = "api/dataset.md" }, { "timenet.engine" = "api/engine.md" }, { "timenet.errors" = "api/errors.md" }, { "timenet.format" = "api/format.md" }, { "timenet.json" = "api/json.md" }, { "timenet.manifest" = "api/manifest.md" }, { "timenet.producer" = "api/producer.md" }, { "timenet.progress" = "api/progress.md" }, { "timenet.provenance" = "api/provenance.md" }, { "timenet.reader" = "api/reader.md" }, { "timenet.registry" = "api/registry.md" }, { "timenet.sources" = "api/timenet.sources.md" }, { "timenet.storage" = "api/storage.md" }, { "timenet.types" = "api/types.md" }, { "timenet.values_backends" = "api/values_backends.md" }, { "timenet.writer" = "api/writer.md" }, { "timenet_connectors.bases" = "api/bases.md" }, { "timenet_connectors.discovery" = "api/discovery.md" }, { "timenet_connectors.download" = "api/download.md" }, { "timenet_connectors.sources" = "api/timenet_connectors.sources.md" }, { "timenet_connectors.splitting" = "api/splitting.md" }, { "timenet_connectors.time_axes" = "api/time_axes.md" }] }, + { "API Reference" = [{ "Overview" = "api/index.md" }, { "timenet.builders" = "api/timenet.builders.md" }, { "timenet.client" = "api/timenet.client.md" }, { "timenet.config" = "api/timenet.config.md" }, { "timenet.connectors" = "api/timenet.connectors.md" }, { "timenet.dataset" = "api/timenet.dataset.md" }, { "timenet.engine" = "api/timenet.engine.md" }, { "timenet.errors" = "api/timenet.errors.md" }, { "timenet.format" = "api/timenet.format.md" }, { "timenet.json" = "api/timenet.json.md" }, { "timenet.manifest" = "api/timenet.manifest.md" }, { "timenet.producer" = "api/timenet.producer.md" }, { "timenet.progress" = "api/timenet.progress.md" }, { "timenet.provenance" = "api/timenet.provenance.md" }, { "timenet.reader" = "api/timenet.reader.md" }, { "timenet.registry" = "api/timenet.registry.md" }, { "timenet.sources" = "api/timenet.sources.md" }, { "timenet.storage" = "api/timenet.storage.md" }, { "timenet.types" = "api/timenet.types.md" }, { "timenet.values_backends" = "api/timenet.values_backends.md" }, { "timenet.writer" = "api/timenet.writer.md" }, { "timenet_connectors.bases" = "api/timenet_connectors.bases.md" }, { "timenet_connectors.discovery" = "api/timenet_connectors.discovery.md" }, { "timenet_connectors.download" = "api/timenet_connectors.download.md" }, { "timenet_connectors.sources" = "api/timenet_connectors.sources.md" }, { "timenet_connectors.splitting" = "api/timenet_connectors.splitting.md" }, { "timenet_connectors.time_axes" = "api/timenet_connectors.time_axes.md" }] }, # <<< api-nav <<< ] From 12aa58a5648b57d3275aa8f99c51d5ae3949d3e2 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 10:11:01 +0100 Subject: [PATCH 04/28] docs: describe the read order and the later rules of the stack The later rounds of the stack changed rules that the docs describe. This commit adds them to the docs: - The order of a sources list is the read order. A release that two sources hold is read from the first one, so the local registry goes first to read offline (client.md, build.md, registry.md, the timenet-datasets skill). - Without a version, the first source that serves the dataset gives its newest version. Only @latest reads the highest version of all sources. The docs said that no version means latest (client.md, registry.md, cli/timenet.md, data-model/datasets.md, the skill). - timenet.build() takes the default list with the local registry first. It refuses a remote or read-only local_registry, and it takes force= and open_access_only=. publish() takes a DatasetRef too (build.md). - Registries take read_only=True and then refuse each write with TimeNetRegistryError (registry.md). - copy_release() and download() copy the payload sources that the manifest lists, not the parents. with_payload_sources=False copies the release alone (registry.md, client.md, cli/build.md). - A registry flag that names no registry exits with code 2 (cli/build.md). - The cache checks run before a clear deletes and before a build removes its raw downloads, not when the settings load (the skill). - A download adds its copy to the local registry, not to the cache (timef-format.md). The build diagram names the local registry, not the store (architecture.md). Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 21 ++++++++++------- docs/architecture.md | 4 ++-- docs/build.md | 23 ++++++++++++++----- docs/cli/build.md | 4 ++-- docs/cli/timenet.md | 10 ++++---- docs/client.md | 25 ++++++++++++-------- docs/data-model/datasets.md | 8 ++++--- docs/registry.md | 29 ++++++++++++++++-------- docs/timef-format.md | 10 ++++---- 9 files changed, 86 insertions(+), 48 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 24cd3784b..255f1aaf3 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -16,7 +16,8 @@ the `add-dataset-connector` skill. - A dataset id is `org/name` (HuggingFace-style, exactly one slash), for example `chengsenwang/tsqa`. - Pin a version with `@` or `@latest`, e.g. `chengsenwang/tsqa@1.0.0`. You can also pass the version as a separate argument. Don't do both (it raises). -- Omitting the version means the latest. +- Omitting the version means the newest version of the first source that serves the dataset, so a + local copy reads offline. `@latest` means the highest version of all sources. ## CLI @@ -87,8 +88,11 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int - `timenet.TimeNet(sources=None)`. `sources` is an ordered list. Each item is a registry object, a URL, a `file://` URI, a local path, or `timenet.InstalledConnectors()`. Pass a list even for one - registry: `timenet.TimeNet(sources=[".timenet-registry"])`. An exact version comes from the - first source that has it; "latest" is the highest version of any source. + registry: `timenet.TimeNet(sources=[".timenet-registry"])`. The order of the list is the read + order: a release that two sources hold comes from the first one, so put the local registry first + to read offline. An exact version comes from the first source that has it, no version gives the + newest version of the first source that serves the dataset, and `@latest` is the highest version + of any source. - The first `timenet.LocalRegistry(path)` of the list that is not `read_only=True` gets the downloads and builds. A path in the list opens such a registry too. A list with `InstalledConnectors()` needs a writable local registry, and only such a list builds on a read. @@ -154,11 +158,12 @@ Env vars, prefix `TIMENET_`. Each sets one thing; none holds a list: (default `~/.cache/timenet`). - `TIMENET_TOKEN` bearer token of a remote registry; `TIMENET_ISOLATION` (default `on`). -Precedence for any value: CLI flag, then env var, then default. An empty value is an error, and so -is a cache directory that overlaps the local registry or is the home folder or a folder above it. In the -CLI, a registry flag that names no registry exits with code 2, and a bad variable with code 1. A -`sources` list that you pass ignores `TIMENET_LOCAL_REGISTRY`, `TIMENET_REMOTE_REGISTRY` and -`TIMENET_AUTO_BUILD`. +Precedence for any value: CLI flag, then env var, then default. An empty value is an error. Before +`timenet local cache clear` or `timenet local clear` deletes, and before a build removes its raw +downloads, TimeNet refuses a cache that is the home folder or a folder above it, or that overlaps +the local registry. In the CLI, a registry flag that names no registry exits with code 2, and a bad +variable with code 1. A `sources` list that you pass ignores `TIMENET_LOCAL_REGISTRY`, +`TIMENET_REMOTE_REGISTRY` and `TIMENET_AUTO_BUILD`. ## Further reading diff --git a/docs/architecture.md b/docs/architecture.md index ec4dc0410..ad12bb3b5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -69,9 +69,9 @@ timenet-build build org/name ├─ builder discovery.resolve("org/name") -> Connector class │ datasets/// exposes CONNECTOR │ - └─ engine run_pipeline(connector, ) + └─ engine run_pipeline(connector, ) metadata -> download -> convert -> derive_schema -> store - -> /org/name// + -> /org/name// ``` `metadata()` reads the `dataset.yaml` card. The engine streams the converted dataset through diff --git a/docs/build.md b/docs/build.md index b2829d443..c8e4db19e 100644 --- a/docs/build.md +++ b/docs/build.md @@ -87,12 +87,22 @@ local registry. Each argument that you leave out comes from the environment at t call: - `local_registry` comes from `$TIMENET_LOCAL_REGISTRY`. -- `sources` comes from `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With - `TIMENET_AUTO_BUILD=false`, the connectors stay out. +- `sources` is the default list of sources with the local registry first: the local registry, + then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With `TIMENET_AUTO_BUILD=false`, + the connectors stay out. So a default build lands where `TimeNet()` reads. `local_registry` takes a local directory, a -`file://` URI or a `LocalRegistry`. `sources` lists, in order, where the parents and payload sources -that the local registry lacks come from (see [Prepare parent releases](#prepare-parent-releases)). +`file://` URI or a `LocalRegistry`. A remote registry or a `LocalRegistry` with `read_only=True` +raises `TimeFValidationError`, because a build writes into its local registry. To get the release +to such a registry, build it locally and publish it with `timenet.publish()`. + +`sources` lists, in order, where the parents and payload sources that the local registry lacks +come from (see [Prepare parent releases](#prepare-parent-releases)). The order of the list is the +read order: a release that two sources hold comes from the first one. + +`force=True` builds the release again, even if the local registry holds it. With +`open_access_only=True`, the build refuses a release without open access with +`TimeNetAccessError` before any step runs. ```python import timenet @@ -106,8 +116,9 @@ print(path) ``` `timenet.publish()` copies a complete release and the payload sources that it needs from `source`, -by default the local registry, to `dest`. Without a version in the dataset ID, it -copies the latest version of `source`. It mirrors the [`timenet publish`](cli/timenet.md#commands) CLI command. +by default the local registry, to `dest`. Without a version in the dataset ID, it copies the latest +version of `source`. A `DatasetRef` names the exact release, so `publish()` copies it without a +lookup in `source`. It mirrors the [`timenet publish`](cli/timenet.md#commands) CLI command. A published version cannot change. If `dest` already holds the same release, `publish()` copies nothing. diff --git a/docs/cli/build.md b/docs/cli/build.md index 4a16b1469..be588b0fb 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -59,7 +59,7 @@ use `TIMENET_TOKEN`. `--publish-to` gets the release to another registry in two steps: 1. The release builds into the local registry, as without `--publish-to`. Its parents land there too. -2. `timenet.publish()` copies the release, and each parent that the destination lacks, +2. `timenet.publish()` copies the release, and each payload source that the destination lacks, from the local registry to the destination. If the destination holds the version and `--force` is not set, the command builds and copies @@ -96,7 +96,7 @@ builds, and publishes nothing. | --- | --- | | `0` | The version was built, published, or reused. | | `1` | An expected TimeNet operation failed. | -| `2` | The command or one of its options is invalid, for example an unknown dataset ID. | +| `2` | The command or one of its options is invalid, for example an unknown dataset ID, or a registry flag that names no registry. | If an isolated build cannot make its environment, for example because `uv` is missing, the command raises `TimeNetIsolationError`. diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index f271296d5..4a02c734f 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -19,7 +19,7 @@ By default, the command uses the hosted TimeNet registry: ```bash timenet list timenet search --query ecg --domain cardiology --limit 10 -timenet info chengsenwang/tsqa@1.0.0 # pin a version; omit @ for the latest +timenet info chengsenwang/tsqa@1.0.0 # pin a version timenet download chengsenwang/tsqa ``` @@ -85,12 +85,14 @@ timenet publish org/name --source timenet:// --dest s3://my-bucket/timenet ## Pinning versions -Add the suffix `@` to an id to pin it. You can also use `@latest` (the default when you omit -the suffix): +Add the suffix `@` to an id to pin it. Without a suffix, the command reads the newest +version of the first source that serves the dataset, so a copy in the local registry reads offline. +`@latest` reads the highest version of all sources: ```bash timenet info chengsenwang/tsqa@1.0.0 # pinned -timenet download chengsenwang/tsqa # latest +timenet download chengsenwang/tsqa # newest of the first source +timenet info chengsenwang/tsqa@latest # highest of all sources ``` If you pin a version that is not committed, the tool exits with a `TimeNetDatasetNotFoundError`. diff --git a/docs/client.md b/docs/client.md index 3637ced1f..c642c2633 100644 --- a/docs/client.md +++ b/docs/client.md @@ -49,10 +49,14 @@ client = timenet.TimeNet(sources=[ mirror on a network drive. A list with `InstalledConnectors()` needs a writable local registry. - A remote registry reads only its own files. -An exact version comes from the first source that provides it, while "latest" is the highest -version that any source provides. Only "not found" and "not served" fall through to the next -source, and any other error stops the call. So if a registry in the list does not answer, a request -for "latest" fails. +The order of the list is the read order. A release that two sources hold is read from the first +one. So to read offline, put the local registry first in the list. + +Without a version, the first source that serves the dataset gives its newest version. An exact +version comes from the first source that provides it, while `@latest` is the highest version that +any source provides. Only "not found" and "not served" fall through to the next source, and any +other error stops the call. So if a registry in the list does not answer, a request for `@latest` +fails. A client that must never write takes `read_only=True`. A list of remote registries alone has no local registry, so it never writes either: @@ -114,7 +118,8 @@ connector never matches `task` or `time_series_spec`, because both filters read manifest. The calls ask the sources one after another, in order. If two sources list a dataset at the same -version, the row of the earlier source stays. To query one registry alone, open it and ask it: +version, the row of the earlier source stays. A listed version can be newer than the version that +`open()` reads without a version. To query one registry alone, open it and ask it: ```python from timenet.registry import open_registry @@ -199,8 +204,10 @@ with client.open("timenet/hello-world") as reader: ``` `open()`, `load()`, and `open_torch()` read each release where it is, and copy nothing. They ask -the sources by the rules of [Choose the sources](#choose-the-sources), so if an installed connector -declares a newer version than the local registry holds, the call builds the newer version. +the sources by the rules of [Choose the sources](#choose-the-sources). So without a version, the +call reads the newest version of the first source that serves the dataset, for example the local +registry. With `@latest`, if an installed connector declares a newer version than the local +registry holds, the call builds the newer version. ```python import timenet @@ -246,10 +253,10 @@ returns the release directory there: path = client.download("chengsenwang/tsqa@1.0.0") ``` -The release resolves as in `open()`. The call copies the release and each parent that the +The release resolves as in `open()`. The call copies the release and each payload source that the local registry lacks or holds damaged. Each one comes from the first other registry that provides it, so they can come from different registries. If the local registry holds a complete copy of the -release, the call copies only the parents that it lacks. +release, the call copies only the payload sources that it lacks. A download never builds. For a release that only `InstalledConnectors()` provides, it raises `TimeNetNotBuiltError`. diff --git a/docs/data-model/datasets.md b/docs/data-model/datasets.md index 334df287e..6948c499c 100644 --- a/docs/data-model/datasets.md +++ b/docs/data-model/datasets.md @@ -27,14 +27,16 @@ The `dataset_version` changes to `1.0.1`, while `source_revision` stays `v2.1`. ## Versions are immutable TimeNet commits a version atomically when its `manifest.json` lands. The version never changes after -that. A pinned `@version` gives you exactly those bytes. With no suffix (or `@latest`), you get the -highest version that a source of the client provides. +that. A pinned `@version` gives you exactly those bytes. With no suffix, you get the newest version +of the first source of the client that serves the dataset. With `@latest`, you get the highest +version that a source of the client provides. ```python import timenet client = timenet.TimeNet() -client.load("chengsenwang/tsqa") # the latest version +client.load("chengsenwang/tsqa") # the newest version of the first source +client.load("chengsenwang/tsqa@latest") # the highest version of all sources client.load("chengsenwang/tsqa@1.0.0") # a pinned, immutable snapshot ``` diff --git a/docs/registry.md b/docs/registry.md index 6c7c02cbb..301fb83c0 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -136,13 +136,14 @@ copies complete files, and makes sure that each file has its recorded size and c A source is something that can provide a release. `timenet.sources` defines the type `ReleaseSource`. Each source has the method `resolve(dataset_id, version)`. `version` is an exact -version, such as `"1.2.0"`, or `None` for the latest version. Two kinds of source exist: +version, such as `"1.2.0"`, or `None` for the newest version of that source. Two kinds of source +exist: - A registry. Every `BaseRegistry` is a source. - `InstalledConnectors()`: every installed package that registers a builder backend under the `timenet.builders` entry point group. -A source gives one of three answers: +A source gives one of these answers: | Answer | Meaning | | --- | --- | @@ -162,15 +163,20 @@ answers `NotServed` otherwise. `InstalledConnectors()` answers `Found` only for the version that the card of a connector declares, because a connector builds only that version. For any other request, it answers `NotFound`. -`resolve_release(sources, dataset_id, version)` asks an ordered list of sources and follows three +`resolve_release(sources, dataset_id, version)` asks an ordered list of sources and follows four rules: -1. An exact version comes from the first source that provides it. -2. "Latest" is the highest version across all sources, the connectors included. The order then +1. Without a version, the first source that serves the dataset gives its newest version. A source + that lists the dataset without serving it falls through to the next source. +2. An exact version comes from the first source that provides it. +3. `"latest"` is the highest version across all sources, the connectors included. The order then decides only which source provides that version. -3. Only `NotFound` and `NotServed` fall through to the next source. Any other error of a source +4. Only `NotFound` and `NotServed` fall through to the next source. Any other error of a source stops the resolution. +So the order of the list is the read order: a release that two sources hold comes from the first +one. + ```python from timenet.registry import LocalRegistry, open_registry from timenet.sources import InstalledConnectors, resolve_release @@ -201,6 +207,9 @@ so you can publish to a local directory, a remote registry, or an S3 bucket: | `put_release(from_directory)` | Publish a complete local release directory as it is; returns the published release as a `DatasetRef`. Only reads `from_directory` and never changes it. Makes sure that the directory holds a readable `manifest.json` and every file that the manifest declares with its recorded size, and raises `TimeFFormatError` or `TimeNetFileError` otherwise. Publishes only those files and `manifest.json`. | | `exists(dataset_id, version)` | Whether a committed version already exists (shared implementation). | +`LocalRegistry`, `RemoteRegistry` and `S3Registry` take `read_only=True`. A read-only registry +refuses each write with `TimeNetRegistryError`. + `put_dataset` writes the dataset with a [`TimeFWriter`](timef-writer.md) and checks each payload source of the new release in the registry that receives it. In a local registry, `force=True` replaces a committed version only once the new one is complete, so a failed write keeps the old @@ -261,10 +270,12 @@ release = registry.put_dataset(dataset) # compile, upload, finalize progress=None)` copies one release from `source` to the writable registry `dest`. A dataset can depend on payload files stored as part of another dataset (parent/child relationship). -`copy_release` copies each parent release too, so that -`dest` can open the copied release. It copies only the releases that `dest` does not hold yet. +`copy_release` copies each payload source that the manifest of the release lists too, so that +`dest` can open the copied release. With `with_payload_sources=False`, it copies the release alone. +It copies only the releases that `dest` does not hold yet. -`copy_release` raises each of these errors before it copies anything, so `dest` stays unchanged. +`copy_release` checks every release before it copies anything, so a refused copy leaves `dest` +unchanged. ## `search` diff --git a/docs/timef-format.md b/docs/timef-format.md index 36ef02c65..c3b939d79 100644 --- a/docs/timef-format.md +++ b/docs/timef-format.md @@ -270,11 +270,11 @@ constraints. The writer then checkpoints and closes the database, records final SQL scans, for tests and tooling. `TimeFReader.verify()` hashes every file that the release reads. A download makes sure that each -file has its recorded size and checksum before it adds the complete copy to the cache. Opening a -version makes sure that its local files exist with their recorded sizes, and hashes nothing. Reads -then use the local files, or byte ranges of the remote files. Normal reads stay lazy. Before a -reader opens a cached remote `control.duckdb` read-only, it makes sure that the file matches the -manifest. +file has its recorded size and checksum before it adds the complete copy to the local registry. +Opening a version makes sure that its local files exist with their recorded sizes, and hashes +nothing. Reads then use the local files, or byte ranges of the remote files. Normal reads stay lazy. +Before a reader opens a cached remote `control.duckdb` read-only, it makes sure that the file +matches the manifest. ## Catalogs of child datasets From 48bfa600db49d52938dd0f9cda31530e04fe4416 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 10:12:36 +0100 Subject: [PATCH 05/28] docs: rewrap the long prose lines of the docs pass and fix a typo The docs pass added prose lines over 100 columns. AGENTS.md asks for prose near 100 columns. This commit rewraps each such line, through the end of its paragraph, with the words unchanged. A code span stays on one line. It also fixes the typo "dependet on" in docs/cli/timenet.md and in the same sentence of the timenet-datasets skill. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 42 ++++++++++++++---------- README.md | 3 +- docs/architecture.md | 3 +- docs/build.md | 24 +++++++++----- docs/cli/build.md | 8 +++-- docs/cli/timenet.md | 11 +++++-- docs/connectors.md | 6 ++-- docs/manifest.md | 4 +-- 8 files changed, 64 insertions(+), 37 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 255f1aaf3..4316b4921 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -6,7 +6,9 @@ description: Use when finding, searching, inspecting, downloading, or loading Ti # Using TimeNet datasets TimeNet's consumer side is the `timenet` CLI and the `timenet.TimeNet` SDK. Both find, read and -download TimeNet datasets: they read dataset versions from local and remote **registries** (catalogues of built datasets). A local registry can also serve to store locally built or downloaded datasets. The CLI is a thin mirror of the SDK, so anything below works the same either way. +download TimeNet datasets: they read dataset versions from local and remote **registries** +(catalogues of built datasets). A local registry can also serve to store locally built or downloaded +datasets. The CLI is a thin mirror of the SDK, so anything below works the same either way. Producing datasets (writing connectors, running `timenet-build`) is a separate concern. For that, use the `add-dataset-connector` skill. @@ -38,8 +40,12 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | `timenet local builds info` / `clear [--yes]` | List the builds in the local registry (dataset, version, source, size, total), or delete them. | A download is a release whose provenance record says `copy`. A build is a release whose record says -`build`, or a release without a record. Releases can depend on each other (parent-child relationships). In that case `downloads clear` and `builds clear` might keep releases that are dependet on, -and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. Do not run a clear while a build or a download runs. +`build`, or a release without a record. Releases can depend on each other (parent-child +relationships). In that case `downloads clear` and `builds clear` might keep releases that are +depended on, and print `Kept : needs it.` for each. Because `local clear` deletes +everything, no dependency relationships can persist, and everything is removed. A symlinked dataset +or release folder in the local registry stops `downloads clear` and `builds clear` before they +delete anything. Do not run a clear while a build or a download runs. `search` flags (repeat a flag to pass several values; all are ANDed across kinds): @@ -56,8 +62,9 @@ and print `Kept : needs it.` for each. Because `local clear` Global: `-q`/`--quiet` suppresses status output; it goes before the command. Sources: `list`, `search`, `info` and `download` take three flags after the command. Each flag wins -over its environment variable, and the variable wins over its default. A registry flag that names no registry (a blank value, an unknown scheme, a URL for `--local-registry`) exits with code 2 and names the -flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. +over its environment variable, and the variable wins over its default. A registry flag that names no +registry (a blank value, an unknown scheme, a URL for `--local-registry`) exits with code 2 and +names the flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. - `--local-registry PATH`: a local registry, the first source that a lookup checks, and the place where downloads and builds land (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). @@ -68,8 +75,8 @@ flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. local nor the remote registry has (`$TIMENET_AUTO_BUILD`, else true). `list` and `search` merge every source, `info` resolves like `open()`, and `download` copies into -the local registry. Of these three flags, `publish` and the `local` commands other than `local cache` take -only `--local-registry`. The `local cache` commands take none of them. +the local registry. Of these three flags, `publish` and the `local` commands other than +`local cache` take only `--local-registry`. The `local cache` commands take none of them. ## Python API @@ -97,9 +104,9 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int downloads and builds. A path in the list opens such a registry too. A list with `InstalledConnectors()` needs a writable local registry, and only such a list builds on a read. A remote registry reads only its own files. -- `TimeNet()` uses `timenet.client.default_sources()`: the local registry `$TIMENET_LOCAL_REGISTRY`, then - `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. The - first entry is always the local registry; `default_local_registry()` returns it alone. +- `TimeNet()` uses `timenet.client.default_sources()`: the local registry `$TIMENET_LOCAL_REGISTRY`, + then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. + The first entry is always the local registry; `default_local_registry()` returns it alone. - `list()` and `search()` merge every source of the list, connector cards included: one row per dataset at the highest version, sorted by ID. `limit` applies after the merge. A connector never matches `task` or `time_series_spec`. A registry without a catalog (S3) adds nothing, and a card @@ -110,13 +117,14 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). `get_card()` returns the card, also from a connector. `plan_read()` returns the plan of `open()` (one step per release with its action and source) and runs nothing. -- `download()` needs a writable local registry in the list. It resolves like `open()`. The release and each payload source that - the local registry lacks or holds damaged come from the first registry other than the local registry that provides - it, so they can come from different registries. If the local registry holds a complete copy of the - release, only the missing payload sources are copied. It never builds: a release that only a - connector provides raises `TimeNetNotBuiltError`, whose hint names `timenet.build()`. - `force=True` copies again, and `progress=` takes the events of `timenet.progress`. The hint of a - read of a damaged copy names the `download()` or the build that replaces it. +- `download()` needs a writable local registry in the list. It resolves like `open()`. The release + and each payload source that the local registry lacks or holds damaged come from the first + registry other than the local registry that provides it, so they can come from different + registries. If the local registry holds a complete copy of the release, only the missing payload + sources are copied. It never builds: a release that only a connector provides raises + `TimeNetNotBuiltError`, whose hint names `timenet.build()`. `force=True` copies again, and + `progress=` takes the events of `timenet.progress`. The hint of a read of a damaged copy names the + `download()` or the build that replaces it. - `open(...)` returns a `TimeFReader`; close it or use it in a `with` block. `reader.read()` returns a `TimeFDataset` whose values load only when asked: `record.signals[i].to_numpy()` / `.to_arrow()`. `reader.get(AnswerTask, task_id)` reads one object by its int ID. diff --git a/README.md b/README.md index f38d5df93..3676c4b55 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,8 @@ uv tool install timenet-connectors timenet-build build timenet/hello-world ``` -The build lands in the local registry at `~/.timenet/registry/timenet/hello-world/1.0.0/`. Load it with Python: +The build lands in the local registry at `~/.timenet/registry/timenet/hello-world/1.0.0/`. Load it +with Python: ```python import timenet diff --git a/docs/architecture.md b/docs/architecture.md index ad12bb3b5..d98f023c0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -144,7 +144,8 @@ runs, the build prepares each parent release, as `TimeNet.open()`, `TimeNet.load()`, `TimeNet.open_torch()`, and `TimeFReader.open()` resolve the payload sources before the first read. A read never opens the catalog of a parent. A client reads each release where it is, through its ordered list of sources. If only -`InstalledConnectors()` in that list provides a release, the client builds it into the local registry. See [Open, load, or download](client.md#open-load-or-download). +`InstalledConnectors()` in that list provides a release, the client builds it into the local +registry. See [Open, load, or download](client.md#open-load-or-download). The child keeps an internal map from the ID of each parent object to the ID of its copy. Through this map, `get(..., parent=)` finds a copy. Through the same map, a later import reuses a copy. The diff --git a/docs/build.md b/docs/build.md index c8e4db19e..e73c629a0 100644 --- a/docs/build.md +++ b/docs/build.md @@ -18,7 +18,8 @@ timenet-build build timenet/hello-world ``` The release lands in the local registry (default `~/.timenet/registry`), where -`TimeNet()` reads from. `--local-registry` selects another local registry. `--publish-to` publishes the release to another registry after the build: +`TimeNet()` reads from. `--local-registry` selects another local registry. `--publish-to` publishes +the release to another registry after the build: ```bash # Publish through the hosted registry API. @@ -42,9 +43,12 @@ Use `--no-isolation` while developing a connector in the current environment: timenet-build build org/name --local-registry .timenet-registry --no-isolation ``` -Most parameters can also be specified as environment variables. For example, `$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the Python SDK also honors them. +Most parameters can also be specified as environment variables. For example, +`$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the +Python SDK also honors them. -For each build, the used Python and package and TimeNet versions are recorded to the build's manifest. +For each build, the used Python and package and TimeNet versions are recorded to the build's +manifest. ## Pipeline @@ -60,9 +64,11 @@ The engine runs these stages: 6. Write the complete TimeF version into the local registry. 7. Remove the raw cache after success, unless `--keep-cache` is set. -By default a build reuses an existing version already in the local registry and a status line says so. Use `--force` to skip this behavior. A forced build replaces the old version only after the new one is -complete, so a failed build keeps the old one. A published version never changes, so with -`--publish-to`, `--force` only affects the build to the local registry. The publish then only checks the rebuild against the already published version. +By default a build reuses an existing version already in the local registry and a status line says +so. Use `--force` to skip this behavior. A forced build replaces the old version only after the new +one is complete, so a failed build keeps the old one. A published version never changes, so with +`--publish-to`, `--force` only affects the build to the local registry. The publish then only checks +the rebuild against the already published version. ## Programmatic builds @@ -120,7 +126,8 @@ by default the local registry, to `dest`. Without a version in the dataset ID, i version of `source`. A `DatasetRef` names the exact release, so `publish()` copies it without a lookup in `source`. It mirrors the [`timenet publish`](cli/timenet.md#commands) CLI command. -A published version cannot change. If `dest` already holds the same release, `publish()` copies nothing. +A published version cannot change. If `dest` already holds the same release, `publish()` copies +nothing. A remote registry holds only releases with access declared `open`. Before @@ -130,7 +137,8 @@ release. ## Prepare parent releases A connector card can declare exact parent releases under `parents`. The connector reads these -releases when it composes the child dataset, preparing each parent release in the local registry that it builds into. +releases when it composes the child dataset, preparing each parent release in the local registry +that it builds into. When resolving dependencies, the build uses the first of these steps that works: diff --git a/docs/cli/build.md b/docs/cli/build.md index be588b0fb..5cb04c77e 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -58,13 +58,15 @@ use `TIMENET_TOKEN`. `--publish-to` gets the release to another registry in two steps: -1. The release builds into the local registry, as without `--publish-to`. Its parents land there too. +1. The release builds into the local registry, as without `--publish-to`. Its parents land there + too. 2. `timenet.publish()` copies the release, and each payload source that the destination lacks, from the local registry to the destination. If the destination holds the version and `--force` is not set, the command builds and copies -nothing. While `--force` builds the release again in the local registry. It cannot overwrite a published version. If the new build wrote the same files, the publish copies nothing. If it wrote other files, the publish -raises `TimeNetVersionConflictError`, and the change needs a new version. +nothing. While `--force` builds the release again in the local registry. It cannot overwrite a +published version. If the new build wrote the same files, the publish copies nothing. If it wrote +other files, the publish raises `TimeNetVersionConflictError`, and the change needs a new version. ## Output diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 4a02c734f..8e5b7ad22 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -54,7 +54,13 @@ after `search`, `-q` is the short form of `--query`. ## Downloads and builds in the local registry -The local registry holds two kinds of release, downloads and local builds. Releases can depend on each other (parent-child relationships). In that case `timenet local downloads clear` and `timenet local builds clear` might keep releases that are dependet on, and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. Do not run a clear while a build or a download runs. +The local registry holds two kinds of release, downloads and local builds. Releases can depend on +each other (parent-child relationships). In that case `timenet local downloads clear` and +`timenet local builds clear` might keep releases that are depended on, and print +`Kept : needs it.` for each. Because `local clear` deletes everything, no +dependency relationships can persist, and everything is removed. A symlinked dataset or release +folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. +Do not run a clear while a build or a download runs. ## Selecting the sources @@ -69,7 +75,8 @@ overrides one [variable](../client.md#settings): | `--remote-registry URL|PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | | `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | -A flag wins over its variable, and the variable wins over its default. Flags go after the subcommand: +A flag wins over its variable, and the variable wins over its default. Flags go after the +subcommand: ```bash timenet list --remote-registry https://registry.example.com diff --git a/docs/connectors.md b/docs/connectors.md index 1d7596bbe..a6705ecbc 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -311,9 +311,9 @@ timenet-build build chengsenwang/tsqa --keep-cache # keep the raw sources ``` A successful build removes the raw downloads of the dataset, at -`/downloads/`. To keep them, pass `--keep-cache`, or `keep_cache=True` to `timenet.build()`. This -helps while you write a connector for a large source, because each rebuild downloads the source -again. +`/downloads/`. To keep them, pass `--keep-cache`, or `keep_cache=True` to +`timenet.build()`. This helps while you write a connector for a large source, because each rebuild +downloads the source again. Keeping `download` and `convert` apart makes a connector testable offline. `convert` takes raw references and does not touch the network. A test can therefore pass small local inputs or synthetic diff --git a/docs/manifest.md b/docs/manifest.md index 7137ea718..fc7abfd36 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -80,8 +80,8 @@ and `counts` default to empty. Unknown fields are rejected. A malformed block ra `ValidationError` with its native field locations and error details. Registries read a stored manifest with `read_manifest(path)` or `parse_manifest(text, where)` from -`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A file that cannot be -read raises a plain `TimeFFormatError`. +`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A file that cannot +be read raises a plain `TimeFFormatError`. ### Serialization notes From ea857bbc4309dd540cbfc380e87bf3e2babaca98 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 11:56:50 +0100 Subject: [PATCH 06/28] docs: describe the access row of timenet info before a build Before a build, a release has no lineage. timenet info now says so in the access row of a card that declares parents. The page on access levels names this case. Part of TNET-157. --- docs/timef-dataset.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/timef-dataset.md b/docs/timef-dataset.md index a2ac0cdc9..50493bc82 100644 --- a/docs/timef-dataset.md +++ b/docs/timef-dataset.md @@ -305,6 +305,9 @@ still say `open`. The effective access level of a dataset is the strictest level of the cards in its `lineage`. A user must have this level of access to use the data of the dataset. `timenet info` shows the effective access level and the page of each card that requires access. +Before a build, a release has no lineage yet. So for a release that only a connector provides, +`timenet info` shows the level of its own card. If the card declares parents, the access row says +so, for example `open (this card only: the access of its 2 parents is known after a build)`. `effective_access()` and `access_requirements()` in `timenet.types.access` return the same information. From ec4e2adb2a708cfb598071f2ff59700c550e52e3 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 11:57:06 +0100 Subject: [PATCH 07/28] docs: describe DatasetFilter and has_catalog in the search docs BaseRegistry.search() takes a DatasetFilter in place of the filter keywords. The search section of the registry page shows the filter, DatasetFilter.of() and the error for a negative limit. The examples that ask one registry pass a DatasetFilter. The registry page names has_catalog of the S3 backend. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 3 ++- docs/client.md | 3 ++- docs/registry.md | 22 ++++++++++++---------- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 4316b4921..f1373c8fb 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -111,7 +111,8 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int dataset at the highest version, sorted by ID. `limit` applies after the merge. A connector never matches `task` or `time_series_spec`. A registry without a catalog (S3) adds nothing, and a card that cannot be read is left out with a `ConnectorCardWarning`. To ask one registry alone, use - `open_registry("timenet://").search(...)` from `timenet.registry`. + `open_registry("timenet://").search(DatasetFilter.of(...))`, with `DatasetFilter` from + `timenet.types`. - `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other files. A registry that lists a release without serving it gives its listed manifest. A release that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). diff --git a/docs/client.md b/docs/client.md index c642c2633..64fb905d8 100644 --- a/docs/client.md +++ b/docs/client.md @@ -123,8 +123,9 @@ version, the row of the earlier source stays. A listed version can be newer than ```python from timenet.registry import open_registry +from timenet.types import DatasetFilter -hits = open_registry("timenet://").search(query="ecg") +hits = open_registry("timenet://").search(DatasetFilter(query=("ecg",))) ``` ## Inspect a release diff --git a/docs/registry.md b/docs/registry.md index 301fb83c0..2733ac560 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -92,7 +92,7 @@ Every backend implements this contract. | `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | | `open_release(reference, *, manifest=None)` | A `DatasetVersion` of one exact release without its payload sources: the parsed manifest and a handle to the files. Pass `manifest` if you already read it from this registry. A local registry raises `TimeNetFileError` for a missing or resized file. | | `resolve(dataset_id, version)` | Whether the registry serves a release, as a [source](#sources) answer: `Found`, `NotFound` or `NotServed`. | -| `search(...)` | Filter datasets. | +| `search(filter)` | Find the datasets that match a `DatasetFilter`. | To copy a release with its payload sources, use [`TimeNet.download()`](client.md#open-load-or-download) for the local registry where downloads and @@ -102,7 +102,8 @@ with its payload sources, use [`TimeNet.open()`](client.md#open-load-or-download `LocalRegistry` serves a `///` tree. `RemoteRegistry` serves the same layout over the versioned REST contract (`GET /api/v1/datasets`, `/api/v1/datasets/{id}/{version}/manifest`, `.../download/{relpath}`); `S3Registry` serves the same layout under an `s3://bucket/prefix` root. The -S3 backend has no catalog, so `list_datasets` and `search` raise `NotImplementedError`. +S3 backend has no catalog: its `has_catalog` is `False`, and `list_datasets` and `search` raise +`NotImplementedError`. A dataset id is an `org/name` pair, for example `chengsenwang/tsqa`. The id nests one level deep on disk, at `/chengsenwang/tsqa//`. `list_datasets` finds these ids at any depth. Use @@ -280,17 +281,18 @@ unchanged. ## `search` ```python -registry.search( - query=None, domain=None, task=None, license=None, - time_series_spec=None, dataset_id=None, tag=None, limit=100, -) +from timenet.types import DatasetFilter, Domain + +registry.search(DatasetFilter.of(domain=Domain.CARDIOLOGY, tag="ecg", limit=10)) ``` -Every filter takes a scalar value or a list. The registry ignores `None` filters and combines all -non-`None` filters with AND logic. `TimeNet.search()` and the consumer CLI, `timenet search`, apply -the same filters to each source of their list, and merge the hits (see +A `DatasetFilter` from `timenet.types` holds the filters and the limit. Each filter is a tuple, and +an empty tuple sets no restriction. `DatasetFilter.of()` takes one value, a list, or `None` for +each filter. The registry combines the filters with AND logic. `TimeNet.search()` and the consumer +CLI, `timenet search`, take the same filters as keywords. They build one `DatasetFilter`, apply it +to each source of their list, and merge the hits (see [Browse datasets](client.md#browse-datasets)). `limit` caps the number of results, with a default -of 100. `limit=0` returns no results. +of 100. `limit=0` returns no results, and a negative limit raises `TimeFValidationError`. | Filter | Matches | | --- | --- | From 1c8758565b9468d863f7d325e2cbb7a21eabbc33 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 11:57:07 +0100 Subject: [PATCH 08/28] docs: describe the provenance record with OriginKind The record now stores its kind as "kind": "download" or "build", and LocalRegistry.read_origin() reads it. The page of the local registry shows the new record and the method. Part of TNET-157. --- docs/client.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/client.md b/docs/client.md index 64fb905d8..fadb94308 100644 --- a/docs/client.md +++ b/docs/client.md @@ -291,15 +291,16 @@ a copy or a build puts into it. The record says where the release comes from: ```json { - "origin": "copy", + "kind": "download", "source": "https://registry.timenet.ai", "at": "2026-11-03T14:02:11Z" } ``` -`origin` is `copy` for a release copied from a registry, and `build` for a release +`kind` is `download` for a release copied from a registry, and `build` for a release that an installed connector built. `source` names the registry, or the connector package and its -version. To read it, call `timenet.registry.origin.read_origin(store, release)`. +version. To read it, call `store.read_origin(release)` on the `LocalRegistry`. It returns an +`Origin` from `timenet.registry.origin`, whose `kind` is an `OriginKind`. ## Pin a version From 4aee1d132835884bffa092e78f3dbe2278dd069c Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 11:57:07 +0100 Subject: [PATCH 09/28] docs: read the card of a release that is not built from the error TimeNetNotBuiltError holds the card of the connector. The example of the client page reads error.card in place of a second call of get_card(), and the dataset skill names the attribute. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 3 ++- docs/client.md | 7 ++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index f1373c8fb..7d38570ac 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -115,7 +115,8 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int `timenet.types`. - `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other files. A registry that lists a release without serving it gives its listed manifest. A release - that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). + that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). The + error holds the card of the connector as `error.card`. `get_card()` returns the card, also from a connector. `plan_read()` returns the plan of `open()` (one step per release with its action and source) and runs nothing. - `download()` needs a writable local registry in the list. It resolves like `open()`. The release diff --git a/docs/client.md b/docs/client.md index fadb94308..6d1899427 100644 --- a/docs/client.md +++ b/docs/client.md @@ -147,7 +147,8 @@ each card. A release gets its manifest when it is built. So for a release that o `InstalledConnectors()` provides, `get_manifest()` raises `TimeNetNotBuiltError`. `get_card()` returns the card of the release. The card comes from the manifest, or from the -installed connector for a release that is not built yet: +installed connector for a release that is not built yet. `TimeNetNotBuiltError` holds the release +as `release` and the card of the connector as `card`, so a caller needs no second call: ```python import timenet @@ -156,8 +157,8 @@ from timenet.errors import TimeNetNotBuiltError client = timenet.TimeNet() try: card = client.get_manifest("chengsenwang/tsqa").metadata -except TimeNetNotBuiltError: - card = client.get_card("chengsenwang/tsqa") +except TimeNetNotBuiltError as error: + card = error.card print(card.name, card.license) ``` From 1d54a1d654aee28447f551f434c02209f4194d28 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 12:28:58 +0100 Subject: [PATCH 10/28] docs: describe Manifest.required_releases() in the manifest guide The manifest guide says that a build needs the parents and that a read needs only the payload sources. It now names Manifest.required_releases(), which lists these releases. Part of TNET-157. --- docs/manifest.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/manifest.md b/docs/manifest.md index fc7abfd36..10eaf3ebc 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -132,6 +132,9 @@ A build needs the parent releases. A read needs only the payload sources. `TimeN before the first read. A read never opens the catalog of a parent. `download()` copies the release and its payload sources into the local registry. See [Payload sources](registry.md#payload-sources). +`required_releases(with_parents=False)` lists the releases of the payload sources, which a read +needs. `required_releases(with_parents=True)` also lists the parents, which a build needs. + --- See the [API reference for `timenet.manifest`](api/timenet.manifest.md) for the full symbol listing. From 3fcec52b5a5b29e4479dff23330a69e68e2bd57c Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 12:29:18 +0100 Subject: [PATCH 11/28] docs: give the record kind of a download as download in the skill The provenance record now says "kind": "download" for a copy from a registry. The skill of the datasets still said that the record of a download says copy. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 7d38570ac..dd756b43e 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -39,8 +39,8 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | `timenet local downloads info` / `clear [--yes]` | List the downloads in the local registry (dataset, version, source, size, total), or delete them. | | `timenet local builds info` / `clear [--yes]` | List the builds in the local registry (dataset, version, source, size, total), or delete them. | -A download is a release whose provenance record says `copy`. A build is a release whose record says -`build`, or a release without a record. Releases can depend on each other (parent-child +A download is a release whose provenance record says `download`. A build is a release whose +record says `build`, or a release without a record. Releases can depend on each other (parent-child relationships). In that case `downloads clear` and `builds clear` might keep releases that are depended on, and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset From b40177ce6fc924a4129a018a40303b8fb90f5bdb Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 12:44:15 +0100 Subject: [PATCH 12/28] docs: say that a registry search keeps the first matches by dataset ID BaseRegistry.search() now sorts the catalog by dataset ID before it filters, and TimeNet.search() gives each source the limit. The search section of the registry guide says that the search returns the first matches by dataset ID, and that a task or time_series_spec filter stops to read manifests at the limit. Part of TNET-157. --- docs/registry.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/registry.md b/docs/registry.md index 2733ac560..9108633f1 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -292,7 +292,10 @@ each filter. The registry combines the filters with AND logic. `TimeNet.search() CLI, `timenet search`, take the same filters as keywords. They build one `DatasetFilter`, apply it to each source of their list, and merge the hits (see [Browse datasets](client.md#browse-datasets)). `limit` caps the number of results, with a default -of 100. `limit=0` returns no results, and a negative limit raises `TimeFValidationError`. +of 100. `limit=0` returns no results, and a negative limit raises `TimeFValidationError`. A +registry sorts its catalog by dataset ID before it filters, so `search()` returns the first matches +by dataset ID. A `task` or `time_series_spec` filter reads the manifests only until it has `limit` +matches. | Filter | Matches | | --- | --- | From 78951d5b9301ace37385e7cc9cf19aee78d211ad Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:27:49 +0100 Subject: [PATCH 13/28] docs: add the --force option of timenet download timenet download takes --force. It copies the release again, as download(force=True) does. Part of TNET-157. --- docs/cli/timenet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 8e5b7ad22..076200bb0 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -32,7 +32,7 @@ To use a local or S3 registry instead, see [Selecting the sources](#selecting-th | `timenet list` | Print every dataset of the sources (id, name, license, domains), from the metadata of the highest version that a source lists. The cards of the installed connectors count too. See [`list()`](../client.md#browse-datasets). | | `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources. | | `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | -| `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. It never builds. See [`download()`](../client.md#open-load-or-download). | +| `timenet download [@version] [--force]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. With `--force`, the release is copied again, also when the local registry holds a complete copy. It never builds. See [`download()`](../client.md#open-load-or-download). | | `timenet publish [@version] --dest URL|PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source URL|PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | | `timenet local info` | Print the path and size of the cache. Then print one line for the downloads and one line for the builds in the local registry, such as `Downloads in : 2 releases, 1.4 GB`. | | `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | From 4dfbbf03afb3601754a3d66d0ab873c372f55a2d Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:27:50 +0100 Subject: [PATCH 14/28] docs: describe the error and the hint of a damaged provenance record A provenance record that is not valid raises TimeNetOriginError, which names its first problem. Its hint names the forced download and the forced build, which replace the release and its record. The client guide, the error table and the page of the timenet CLI say so. Part of TNET-157. --- docs/cli/timenet.md | 4 +++- docs/client.md | 4 ++++ docs/types.md | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 076200bb0..f29932e4c 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -60,7 +60,9 @@ each other (parent-child relationships). In that case `timenet local downloads c `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. -Do not run a clear while a build or a download runs. +A provenance record that is not valid also stops them. Its hint names +`timenet download @ --force` and `timenet-build build @ --force`, which +replace the release and its record. Do not run a clear while a build or a download runs. ## Selecting the sources diff --git a/docs/client.md b/docs/client.md index 6d1899427..7632b603f 100644 --- a/docs/client.md +++ b/docs/client.md @@ -303,6 +303,10 @@ that an installed connector built. `source` names the registry, or the connector version. To read it, call `store.read_origin(release)` on the `LocalRegistry`. It returns an `Origin` from `timenet.registry.origin`, whose `kind` is an `OriginKind`. +A record that is not valid raises `TimeNetOriginError`, which names its first problem. To replace +the release and its record, get the release again: call `download(release, force=True)` if it came +from a registry, or `timenet.build(release, force=True)` if it was built. + ## Pin a version Append `@` to a dataset ID: diff --git a/docs/types.md b/docs/types.md index 62f7d0971..cbed081a7 100644 --- a/docs/types.md +++ b/docs/types.md @@ -302,6 +302,7 @@ TimeNet raises its own exceptions from `timenet.errors`. | `TimeFFormatError` | A stored TimeF artifact is corrupt or unsupported. | | `TimeNetFileError` | A file of a release is missing, or does not match its manifest record. A `TimeFFormatError`. | | `TimeNetManifestError` | A stored manifest is not a valid manifest. `where` names the file or the place in a registry, and `detail` names the first problem. A `TimeFFormatError`. | +| `TimeNetOriginError` | The provenance record of a release in the local registry is not valid. `release` names the release, `path` the file, and `detail` the first problem. A `TimeFFormatError`. | | `TimeFReaderError` | A reader could not complete a read, e.g. its connection was closed while the read ran. | | `TimeNetInvalidCardError` | `dataset.yaml` cannot be read or is not a mapping. | | `TimeNetConnectorError` | A connector found that an assumption about its source data, a download, or a parent does not hold. `TimeNetDownloadError` is a `TimeNetConnectorError`. | From 61e6ca7ac6641b95f5e78882d070c985da94e482 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:27:51 +0100 Subject: [PATCH 15/28] docs: say that each registry backend defines location BaseRegistry.location is abstract now, so no registry gives its name in place of a location. Part of TNET-157. --- docs/registry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/registry.md b/docs/registry.md index 9108633f1..b1dcc96c5 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -86,7 +86,7 @@ Every backend implements this contract. | `list_datasets()` | Latest-version `DatasetMetadata` for every dataset, sorted by id. | | `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeFFormatError` if the stored manifest is not valid, or if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | -| `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Any other registry gives its name. | +| `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Each backend must define it. | | `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | | `find_manifest(release)` | The manifest of an exact release, or `None` if the registry does not hold it. Raises `TimeFFormatError` if the stored manifest names another release. | | `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | From acf1b50b44cdc052db7e7c37c948479245a8edf2 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:28:38 +0100 Subject: [PATCH 16/28] docs: describe TimeNetUnreachableError and the new facts of two errors The error table names TimeNetUnreachableError, the difference and the registry_location of TimeNetVersionConflictError, and the path of TimeNetManifestError. Part of TNET-157. --- docs/types.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/types.md b/docs/types.md index cbed081a7..136aef7f9 100644 --- a/docs/types.md +++ b/docs/types.md @@ -293,15 +293,16 @@ TimeNet raises its own exceptions from `timenet.errors`. | --- | --- | | `TimeNetError` | Base class for TimeNet failures. | | `TimeNetRegistryError` | A registry cannot complete an operation. | +| `TimeNetUnreachableError` | A request to a registry or its storage failed on the network. The original error is its cause. A `TimeNetRegistryError`. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | | `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | -| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. For a publish, the hint asks for a new version. For a copy into the local registry, the hint names the release folder to delete. | +| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For a copy into the local registry, the hint names the release folder to delete. | | `TimeFValidationError` | Input violates a TimeF invariant. | | `TimeFNullValuesError` | Values contain nulls that NumPy cannot hold. A `TimeFValidationError`. | | `TimeFNotFoundError` | An ID, parent alias, or parent release names nothing in the dataset, or the stored object has another kind. A `TimeFValidationError` and a `LookupError`. | | `TimeFFormatError` | A stored TimeF artifact is corrupt or unsupported. | | `TimeNetFileError` | A file of a release is missing, or does not match its manifest record. A `TimeFFormatError`. | -| `TimeNetManifestError` | A stored manifest is not a valid manifest. `where` names the file or the place in a registry, and `detail` names the first problem. A `TimeFFormatError`. | +| `TimeNetManifestError` | A stored manifest is not a valid manifest. `where` names the file or the place in a registry, and `detail` names the first problem. `path` is the file, or `None` for a manifest that a registry served. A `TimeFFormatError`. | | `TimeNetOriginError` | The provenance record of a release in the local registry is not valid. `release` names the release, `path` the file, and `detail` the first problem. A `TimeFFormatError`. | | `TimeFReaderError` | A reader could not complete a read, e.g. its connection was closed while the read ran. | | `TimeNetInvalidCardError` | `dataset.yaml` cannot be read or is not a mapping. | From 0b9e2c3139a40fc5641be97ccdcb52b5ec0abef9 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:41:37 +0100 Subject: [PATCH 17/28] docs: drop TimeNet.plan_read() from the client guide and the skill TimeNet.plan_read() left the public API, because Plan stays internal. The method table and the list of methods that take version= in docs/client.md, and the timenet-datasets skill, no longer name it. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 3 +-- docs/client.md | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index dd756b43e..288ca3ca4 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -117,8 +117,7 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int files. A registry that lists a release without serving it gives its listed manifest. A release that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). The error holds the card of the connector as `error.card`. - `get_card()` returns the card, also from a connector. `plan_read()` returns the plan of `open()` - (one step per release with its action and source) and runs nothing. + `get_card()` returns the card, also from a connector. - `download()` needs a writable local registry in the list. It resolves like `open()`. The release and each payload source that the local registry lacks or holds damaged come from the first registry other than the local registry that provides it, so they can come from different diff --git a/docs/client.md b/docs/client.md index 7632b603f..dc84dd489 100644 --- a/docs/client.md +++ b/docs/client.md @@ -138,7 +138,6 @@ nothing. | --- | --- | | `get_manifest(dataset_id, version=None)` | The [manifest](manifest.md) of the release. | | `get_card(dataset_id, version=None)` | The card of the release. | -| `plan_read(dataset_id, version=None)` | The plan of the read, one step for each release. | `get_manifest()` reads only the manifest, and checks no other file. A registry can list a release without serving its files, for example a credentialed release on the hosted registry. Then @@ -316,7 +315,7 @@ client.get_manifest("chengsenwang/tsqa@1.0.0") client.load("chengsenwang/tsqa@latest") ``` -The `get_manifest`, `get_card`, `plan_read`, `download`, `open`, `load`, and `open_torch` methods +The `get_manifest`, `get_card`, `download`, `open`, `load`, and `open_torch` methods also accept `version=`. Do not supply a version in both places. ## Open for PyTorch From 1d4ac9da3d261d1273d22e970c3dee0c7037354c Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:41:38 +0100 Subject: [PATCH 18/28] docs: name store_dataset() as the last stage of the pipeline The docs named the last stage store, after a BaseConnector.store() method that does not exist. The engine writes the dataset with timenet.engine.store_dataset(), so the pipeline diagrams, the tables and the connector skill name that function. Part of TNET-157. --- .agents/skills/add-dataset-connector/SKILL.md | 2 +- .../add-dataset-connector/references/connector-anatomy.md | 4 ++-- docs/architecture.md | 4 ++-- docs/concepts.md | 2 +- docs/connectors.md | 2 +- docs/data-model/index.md | 2 +- 6 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.agents/skills/add-dataset-connector/SKILL.md b/.agents/skills/add-dataset-connector/SKILL.md index bd66e3436..0b32a171e 100644 --- a/.agents/skills/add-dataset-connector/SKILL.md +++ b/.agents/skills/add-dataset-connector/SKILL.md @@ -7,7 +7,7 @@ description: Use when adding a new TimeNet dataset connector, i.e. converting an A connector fetches a dataset's raw source and converts it into a `TimeFDataset`. It implements the `BaseConnector` contract (in the `timenet` package) and lives in `timenet-connectors`. The engine drives -it `download -> convert -> derive_schema -> store`; `timenet-build build ` runs that pipeline and +it `download -> convert -> derive_schema -> store_dataset`; `timenet-build build ` runs that pipeline and writes the result into a registry. Reuse a base connector wherever the source allows: `BaseHuggingFaceConnector` for Hub datasets, diff --git a/.agents/skills/add-dataset-connector/references/connector-anatomy.md b/.agents/skills/add-dataset-connector/references/connector-anatomy.md index f43463321..45463b585 100644 --- a/.agents/skills/add-dataset-connector/references/connector-anatomy.md +++ b/.agents/skills/add-dataset-connector/references/connector-anatomy.md @@ -21,8 +21,8 @@ this branch. Anchor files: - `metadata(self) -> DatasetMetadata` (**concrete**, do not override): loads and validates the card via `DatasetMetadata.from_yaml`. By convention the card is `dataset.yaml` beside the connector module; set the `CARD` class var to point elsewhere. (Some docs call `metadata` abstract; it isn't.) -- `store(...)` (concrete): derives the schema if missing and streams the dataset through `TimeFWriter`. - Most connectors never override it. +- The connector has no write method. The engine function `timenet.engine.store_dataset()` derives the + schema if missing and streams the dataset through `TimeFWriter`. Connectors take **no constructor arguments** (configuration comes from the environment). End with a module-level `CONNECTOR = `. diff --git a/docs/architecture.md b/docs/architecture.md index d98f023c0..3078ab469 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -37,7 +37,7 @@ There can be several registries: one public, private internal ones, or a local d PRODUCE dataset.yaml + connector │ ▼ - engine download ─► convert ─► derive_schema ─► store + engine download ─► convert ─► derive_schema ─► store_dataset │ ▼ registry @@ -70,7 +70,7 @@ timenet-build build org/name │ datasets/// exposes CONNECTOR │ └─ engine run_pipeline(connector, ) - metadata -> download -> convert -> derive_schema -> store + metadata -> download -> convert -> derive_schema -> store_dataset -> /org/name// ``` diff --git a/docs/concepts.md b/docs/concepts.md index f98d2658e..918c7a5f5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -19,7 +19,7 @@ This page collects the vocabulary that appears across these docs. Follow a link | **[Signal](data-model/time-series.md)** | One named value stream with its own TimeAxis and TimeSeriesSpec. Values remain lazy until read. | | **[Manifest](manifest.md)** | The `manifest.json` entry point for a version. It contains metadata, schema, counts, checksums, and file paths. Its presence marks a committed version. | | **[Connector](connectors.md)** | One `BaseConnector` per dataset. `download()` fetches the raw source. `convert()` builds a `TimeFDataset`. It knows nothing about the engine or registry. | -| **[Engine](build.md#pipeline)** | `run_pipeline`: drives any connector through the fixed `download -> convert -> derive_schema -> store` pipeline, and owns caching and idempotency. | +| **[Engine](build.md#pipeline)** | `run_pipeline`: drives any connector through the fixed `download -> convert -> derive_schema -> store_dataset` pipeline, and owns caching and idempotency. | | **[Build](build.md)** | Running a connector through the engine to compile a dataset and publish it with the [`timenet-build`](cli/build.md) CLI. | | **[Registry](registry.md)** | A served location of compiled TimeF versions that the [SDK](client.md) reads. It serves a DuckDB control plane plus a Parquet or Zarr values plane. It never runs connector code. It can be a [local directory, S3, or a remote host](registry.md#choosing-a-registry). | | **[Client (SDK)](client.md)** | `TimeNet`: the consumer entry point to search, download, and load datasets. | diff --git a/docs/connectors.md b/docs/connectors.md index a6705ecbc..812a4a8d0 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -35,7 +35,7 @@ class MyConnector(BaseConnector[MyRawRef]): ``` The two stages stay separate. This lets the engine drive -`download -> convert -> derive_schema -> store`. +`download -> convert -> derive_schema -> store_dataset`. | Method | Nature | Contract | | --- | --- | --- | diff --git a/docs/data-model/index.md b/docs/data-model/index.md index 4f905f955..d059f9a69 100644 --- a/docs/data-model/index.md +++ b/docs/data-model/index.md @@ -34,7 +34,7 @@ flowchart LR D["download()
I/O · fetch raw refs"] C["convert()
CPU only"] S["derive_schema()
types from data"] - W["store()
TimeFWriter"] + W["store_dataset()
TimeFWriter"] R[("registry
TimeF version")] D --> C --> S --> W --> R ``` From b094f4ddf30c6e49d6b9eaa7bd5e2cdac9f3960d Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 14:41:39 +0100 Subject: [PATCH 19/28] docs: drop an unused import and keep URL|PATH out of table cells The README example imported timenet without using it. In the tables of docs/cli/timenet.md and the timenet-datasets skill, GitHub split a cell at the bar of URL|PATH, so these cells say URL or PATH. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 2 +- README.md | 1 - docs/cli/timenet.md | 4 ++-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 288ca3ca4..6740c0336 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -32,7 +32,7 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | `timenet search [filters]` | Filter the datasets of the sources, connector cards included. See flags below. | | `timenet info [@version]` | Manifest summary of the release that `open()` reads: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). A dataset that only a connector provides shows its card and the build command. | | `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory to stdout. Never builds. Prints "Nothing to do" if the local registry holds everything. | -| `timenet publish [@version] --dest URL|PATH [--source URL|PATH]` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | +| `timenet publish [@version] --dest` `URL` or `PATH`, optionally `--source` `URL` or `PATH` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | | `timenet local info` | Path and size of the cache, then one line each for the downloads and the builds in the local registry (count and size). | | `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | | `timenet local cache info` / `clear [--yes]` | Path and size of the cache (`$TIMENET_CACHE`), or delete its raw source files (`downloads/`) and control databases (`control/`) that carry TimeNet's `CACHEDIR.TAG`. | diff --git a/README.md b/README.md index 3676c4b55..1c6a723b1 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,6 @@ The build lands in the local registry at `~/.timenet/registry/timenet/hello-worl with Python: ```python -import timenet from timenet import TimeNet dataset = TimeNet().load("timenet/hello-world@1.0.0") diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index f29932e4c..c9411ec16 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -33,7 +33,7 @@ To use a local or S3 registry instead, see [Selecting the sources](#selecting-th | `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources. | | `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | | `timenet download [@version] [--force]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. With `--force`, the release is copied again, also when the local registry holds a complete copy. It never builds. See [`download()`](../client.md#open-load-or-download). | -| `timenet publish [@version] --dest URL|PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source URL|PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | +| `timenet publish [@version] --dest` `URL` or `PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source` `URL` or `PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | | `timenet local info` | Print the path and size of the cache. Then print one line for the downloads and one line for the builds in the local registry, such as `Downloads in : 2 releases, 1.4 GB`. | | `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | | `timenet local cache info` | Print the path and size of the cache, `$TIMENET_CACHE`. | @@ -74,7 +74,7 @@ overrides one [variable](../client.md#settings): | Flag | Variable | Default | What it sets | | --- | --- | --- | --- | | `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the place where downloads and builds land. | -| `--remote-registry URL|PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `--remote-registry` `URL` or `PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | | `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | A flag wins over its variable, and the variable wins over its default. Flags go after the From 50da3abb8aa6ed8c19e887b4dc447726c8acc215 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 15:56:36 +0100 Subject: [PATCH 20/28] docs: describe the search over the merged rows of all sources TimeNet.search() now filters the merged rows of list(). The search section of docs/registry.md describes DatasetFilter.select() and ListedRelease: select() checks the card first, reads the schema of a matching card from its registry, and stops at limit matches. A row without a registry never matches a task or time_series_spec filter. docs/client.md and the timenet-datasets skill say that search() checks the highest version of each dataset only, and that it reads a manifest only for a row whose card matches. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 11 ++++++----- docs/client.md | 8 +++++--- docs/registry.md | 22 ++++++++++++++-------- 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 6740c0336..cd1b6bc60 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -108,11 +108,12 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. The first entry is always the local registry; `default_local_registry()` returns it alone. - `list()` and `search()` merge every source of the list, connector cards included: one row per - dataset at the highest version, sorted by ID. `limit` applies after the merge. A connector never - matches `task` or `time_series_spec`. A registry without a catalog (S3) adds nothing, and a card - that cannot be read is left out with a `ConnectorCardWarning`. To ask one registry alone, use - `open_registry("timenet://").search(DatasetFilter.of(...))`, with `DatasetFilter` from - `timenet.types`. + dataset at the highest version, sorted by ID. `search()` filters these merged rows and stops at + `limit` matches. It reads a manifest for `task` or `time_series_spec` only for a row whose card + matches. A connector never matches these two filters. A registry without a catalog (S3) adds + nothing, and a card that cannot be read is left out with a `ConnectorCardWarning`. To ask one + registry alone, use `open_registry("timenet://").search(DatasetFilter.of(...))`, with + `DatasetFilter` from `timenet.types`. - `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other files. A registry that lists a release without serving it gives its listed manifest. A release that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). The diff --git a/docs/client.md b/docs/client.md index dc84dd489..f748857ca 100644 --- a/docs/client.md +++ b/docs/client.md @@ -113,9 +113,11 @@ counts as a row. So `list()` also shows a dataset that only a connector provides | `search(...)` | Metadata that matches registry filters, across the sources. | `search()` accepts filters for free text, domains, task classes, licenses, Signal specifications, -dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. A -connector never matches `task` or `time_series_spec`, because both filters read the schema of a -manifest. +dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. +`search()` filters the rows of `list()`, so it checks the highest version of each dataset only. A +`task` or `time_series_spec` filter reads the manifest of a row from the registry that lists it. +It reads only for a row whose card matches the other filters, and only until `limit` rows match. A +connector card has no manifest, so it never matches these two filters. The calls ask the sources one after another, in order. If two sources list a dataset at the same version, the row of the earlier source stays. A listed version can be newer than the version that diff --git a/docs/registry.md b/docs/registry.md index b1dcc96c5..4bd6ff19f 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -288,14 +288,20 @@ registry.search(DatasetFilter.of(domain=Domain.CARDIOLOGY, tag="ecg", limit=10)) A `DatasetFilter` from `timenet.types` holds the filters and the limit. Each filter is a tuple, and an empty tuple sets no restriction. `DatasetFilter.of()` takes one value, a list, or `None` for -each filter. The registry combines the filters with AND logic. `TimeNet.search()` and the consumer -CLI, `timenet search`, take the same filters as keywords. They build one `DatasetFilter`, apply it -to each source of their list, and merge the hits (see -[Browse datasets](client.md#browse-datasets)). `limit` caps the number of results, with a default -of 100. `limit=0` returns no results, and a negative limit raises `TimeFValidationError`. A -registry sorts its catalog by dataset ID before it filters, so `search()` returns the first matches -by dataset ID. A `task` or `time_series_spec` filter reads the manifests only until it has `limit` -matches. +each filter. The registry combines the filters with AND logic. `limit` caps the number of +results, with a default of 100. `limit=0` returns no results, and a negative limit raises +`TimeFValidationError`. + +`DatasetFilter.select()` does the filtering. It takes `ListedRelease` rows, each a card and the +registry that lists it, and keeps their order. It checks the card first. A `task` or +`time_series_spec` filter then reads the manifest of a matching card from its registry. A row +without a registry never matches these two filters. `select()` stops at `limit` matches, so it +reads no more manifests than it needs. + +A registry sorts its catalog by dataset ID and gives it to `select()`, so `search()` returns the +first matches by dataset ID. `TimeNet.search()` and the consumer CLI, `timenet search`, take the +same filters as keywords. They give `select()` the merged rows of all sources (see +[Browse datasets](client.md#browse-datasets)). | Filter | Matches | | --- | --- | From cff8ad373bdcb863a73e3a95d0ea28e62e1e491c Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 15:57:29 +0100 Subject: [PATCH 21/28] docs: describe CatalogWarning and TimeNetNoCatalogError TimeNet.list() and TimeNet.search() now leave a source without a catalog out with a CatalogWarning. If no source has a catalog, they raise TimeNetNoCatalogError, whose hint names the read of an exact release. docs/registry.md, docs/client.md and the timenet-datasets skill say this. docs/types.md adds the error to its table and the warning to its warnings, and says that both warnings point at the first line outside the timenet package. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 9 +++++---- docs/client.md | 9 ++++++--- docs/registry.md | 4 +++- docs/types.md | 5 +++++ 4 files changed, 19 insertions(+), 8 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index cd1b6bc60..655edbbc3 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -110,10 +110,11 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int - `list()` and `search()` merge every source of the list, connector cards included: one row per dataset at the highest version, sorted by ID. `search()` filters these merged rows and stops at `limit` matches. It reads a manifest for `task` or `time_series_spec` only for a row whose card - matches. A connector never matches these two filters. A registry without a catalog (S3) adds - nothing, and a card that cannot be read is left out with a `ConnectorCardWarning`. To ask one - registry alone, use `open_registry("timenet://").search(DatasetFilter.of(...))`, with - `DatasetFilter` from `timenet.types`. + matches. A connector never matches these two filters. A registry without a catalog (S3) is left + out with a `CatalogWarning`, and a list without any catalog raises `TimeNetNoCatalogError`. A card + that cannot be read is left out with a `ConnectorCardWarning`. To ask one registry alone, use + `open_registry("timenet://").search(DatasetFilter.of(...))`, with `DatasetFilter` from + `timenet.types`. - `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other files. A registry that lists a release without serving it gives its listed manifest. A release that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). The diff --git a/docs/client.md b/docs/client.md index f748857ca..69e899b47 100644 --- a/docs/client.md +++ b/docs/client.md @@ -119,9 +119,12 @@ dataset IDs, and tags. See [Registry search](registry.md#search) for their match It reads only for a row whose card matches the other filters, and only until `limit` rows match. A connector card has no manifest, so it never matches these two filters. -The calls ask the sources one after another, in order. If two sources list a dataset at the same -version, the row of the earlier source stays. A listed version can be newer than the version that -`open()` reads without a version. To query one registry alone, open it and ask it: +The calls ask the sources one after another, in order. A registry without a catalog, such as an +S3 registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls raise +`TimeNetNoCatalogError`, and its hint names `open('@')`, the read of an exact release. +If two sources list a dataset at the same version, the row of the earlier source stays. A listed +version can be newer than the version that `open()` reads without a version. To query one registry +alone, open it and ask it: ```python from timenet.registry import open_registry diff --git a/docs/registry.md b/docs/registry.md index 4bd6ff19f..ef3189eda 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -103,7 +103,9 @@ with its payload sources, use [`TimeNet.open()`](client.md#open-load-or-download over the versioned REST contract (`GET /api/v1/datasets`, `/api/v1/datasets/{id}/{version}/manifest`, `.../download/{relpath}`); `S3Registry` serves the same layout under an `s3://bucket/prefix` root. The S3 backend has no catalog: its `has_catalog` is `False`, and `list_datasets` and `search` raise -`NotImplementedError`. +`NotImplementedError`. `TimeNet.list()` and `TimeNet.search()` leave a registry without a catalog +out, with a `CatalogWarning` that names it. If no source of the list has a catalog, they raise +`TimeNetNoCatalogError`. A dataset id is an `org/name` pair, for example `chengsenwang/tsqa`. The id nests one level deep on disk, at `/chengsenwang/tsqa//`. `list_datasets` finds these ids at any depth. Use diff --git a/docs/types.md b/docs/types.md index 136aef7f9..af50766b5 100644 --- a/docs/types.md +++ b/docs/types.md @@ -294,6 +294,7 @@ TimeNet raises its own exceptions from `timenet.errors`. | `TimeNetError` | Base class for TimeNet failures. | | `TimeNetRegistryError` | A registry cannot complete an operation. | | `TimeNetUnreachableError` | A request to a registry or its storage failed on the network. The original error is its cause. A `TimeNetRegistryError`. | +| `TimeNetNoCatalogError` | No source of a list or a search has a catalog. The hint names the read of an exact release. A `TimeNetRegistryError`. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | | `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | | `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For a copy into the local registry, the hint names the release folder to delete. | @@ -327,4 +328,8 @@ behavior. It derives from `TimeNetWarning` and `UserWarning`. `ConnectorCardWarning` reports a connector card that cannot be read. The list of installed connectors leaves its dataset out, so the card fails only its own dataset. +`CatalogWarning` reports a source without a catalog, which `TimeNet.list()` and +`TimeNet.search()` leave out. Both warnings point at the first line of the call stack outside the +`timenet` package, which is usually your own call. + See the [API reference for `timenet.types`](api/timenet.types.md) for all symbols. From 9d75f8a83a2a59de70983fd9c034e32b6bd79976 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 15:57:43 +0100 Subject: [PATCH 22/28] docs: say that a symlinked folder stops every clear A clear of the local registry now checks for symlinked dataset and release folders also when every kind goes. So a symlinked folder stops timenet local clear too, before it deletes anything. docs/cli/timenet.md and the timenet-datasets skill say this. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 4 ++-- docs/cli/timenet.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 655edbbc3..501f69941 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -44,8 +44,8 @@ record says `build`, or a release without a record. Releases can depend on each relationships). In that case `downloads clear` and `builds clear` might keep releases that are depended on, and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset -or release folder in the local registry stops `downloads clear` and `builds clear` before they -delete anything. Do not run a clear while a build or a download runs. +or release folder in the local registry stops every clear, `local clear` included, before it +deletes anything. Do not run a clear while a build or a download runs. `search` flags (repeat a flag to pass several values; all are ANDed across kinds): diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index c9411ec16..c82fa4f03 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -59,7 +59,7 @@ each other (parent-child relationships). In that case `timenet local downloads c `timenet local builds clear` might keep releases that are depended on, and print `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release -folder in the local registry stops `downloads clear` and `builds clear` before they delete anything. +folder in the local registry stops every clear, `local clear` included, before it deletes anything. A provenance record that is not valid also stops them. Its hint names `timenet download @ --force` and `timenet-build build @ --force`, which replace the release and its record. Do not run a clear while a build or a download runs. From affbf4c000c10cefbf01ee562c3516c56ef9d21f Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 17:12:39 +0100 Subject: [PATCH 23/28] docs: say that a search checks only the highest version of a dataset TimeNet.search() and timenet search filter the merged rows of list(). So they check only the highest version of each dataset, and they do not find an older version that matches. The search section of the registry guide, the browse section of the client guide and the search row of the CLI guide now say so. Part of TNET-157. --- docs/cli/timenet.md | 2 +- docs/client.md | 3 ++- docs/registry.md | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index c82fa4f03..d451c8547 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -30,7 +30,7 @@ To use a local or S3 registry instead, see [Selecting the sources](#selecting-th | Command | What it does | | --- | --- | | `timenet list` | Print every dataset of the sources (id, name, license, domains), from the metadata of the highest version that a source lists. The cards of the installed connectors count too. See [`list()`](../client.md#browse-datasets). | -| `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources. | +| `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources, and the search checks only the highest version of each dataset. | | `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | | `timenet download [@version] [--force]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. With `--force`, the release is copied again, also when the local registry holds a complete copy. It never builds. See [`download()`](../client.md#open-load-or-download). | | `timenet publish [@version] --dest` `URL` or `PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source` `URL` or `PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | diff --git a/docs/client.md b/docs/client.md index 69e899b47..39a8fe50f 100644 --- a/docs/client.md +++ b/docs/client.md @@ -114,7 +114,8 @@ counts as a row. So `list()` also shows a dataset that only a connector provides `search()` accepts filters for free text, domains, task classes, licenses, Signal specifications, dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. -`search()` filters the rows of `list()`, so it checks the highest version of each dataset only. A +`search()` filters the rows of `list()`, so it checks the highest version of each dataset only. It +does not find an older version that matches. A `task` or `time_series_spec` filter reads the manifest of a row from the registry that lists it. It reads only for a row whose card matches the other filters, and only until `limit` rows match. A connector card has no manifest, so it never matches these two filters. diff --git a/docs/registry.md b/docs/registry.md index ef3189eda..834aded62 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -303,7 +303,8 @@ reads no more manifests than it needs. A registry sorts its catalog by dataset ID and gives it to `select()`, so `search()` returns the first matches by dataset ID. `TimeNet.search()` and the consumer CLI, `timenet search`, take the same filters as keywords. They give `select()` the merged rows of all sources (see -[Browse datasets](client.md#browse-datasets)). +[Browse datasets](client.md#browse-datasets)). So they check only the highest version of each +dataset, and they do not find an older version that matches. | Filter | Matches | | --- | --- | From 9d341551f4e5d3896c4e2581c474f1c24211cdb0 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 17:12:46 +0100 Subject: [PATCH 24/28] docs: describe the warning lines of timenet list and search timenet list and timenet search leave out a source without a catalog and a connector card that cannot be read. For each, they print a warning line to stderr, without a Python file and line number. The CLI guide now says so. Part of TNET-157. --- docs/cli/timenet.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index d451c8547..855ce037f 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -48,6 +48,9 @@ stdout, so you can capture `timenet download ` in a script safely. Most errors end with a hint that names a command or a flag that helps. +`list` and `search` leave out a source without a catalog and a connector card that cannot be read. +For each, they print a warning line to stderr, without a Python file and line number. + `--quiet`/`-q` removes the status output. It belongs to `timenet` itself. Therefore, it goes before the subcommand: `timenet --quiet list`, not `timenet list --quiet`. Be careful with the collision: after `search`, `-q` is the short form of `--query`. From 1d3933fe1dd8a57ddec687e968a1ebd87e10acc3 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 17:12:47 +0100 Subject: [PATCH 25/28] docs: give --local-registry the help text of the CLI Every command of both CLIs now has the same help for --local-registry: the local registry, where downloads and builds land and which a lookup checks first. The guides of timenet and timenet-build build and the timenet-datasets skill use this text. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 4 ++-- docs/cli/build.md | 2 +- docs/cli/timenet.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 501f69941..9200b19ee 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -66,8 +66,8 @@ over its environment variable, and the variable wins over its default. A registr registry (a blank value, an unknown scheme, a URL for `--local-registry`) exits with code 2 and names the flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. -- `--local-registry PATH`: a local registry, the first source that a lookup checks, and the place - where downloads and builds land (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). +- `--local-registry PATH`: the local registry, where downloads and builds land and which a lookup + checks first (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). - `--remote-registry URL|PATH`: a remote registry, as a URL or a local path, the second source that a lookup checks, if the local registry does not have the dataset (`$TIMENET_REMOTE_REGISTRY`, else `timenet://`). diff --git a/docs/cli/build.md b/docs/cli/build.md index 5cb04c77e..762f9f915 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -36,7 +36,7 @@ timenet-build [--quiet] build [@] \ | `--isolation`, `--no-isolation` | `$TIMENET_ISOLATION`, else on | Run this build in an environment built from the requirements of the connector, or in the current one. | | `--publish-to` | off | After the build, publish the release and its payload sources to the registry at this URL or path. | | `--dry-run` | off | Print the plan to stdout and stop. It copies, builds, and publishes nothing. | -| `--local-registry` | `$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the local registry where the build lands. | +| `--local-registry` | `$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry` | The local registry, where downloads and builds land and which a lookup checks first. | | `--remote-registry` | `$TIMENET_REMOTE_REGISTRY`, else `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have a parent. | | `--auto-build`, `--no-auto-build` | `$TIMENET_AUTO_BUILD`, else on | Whether installed connectors build a parent that neither the local registry nor the remote registry has. | | `--quiet`, `-q` | off | Hide status output. Place this option before `build`. | diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 855ce037f..7f5f3f134 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -76,7 +76,7 @@ overrides one [variable](../client.md#settings): | Flag | Variable | Default | What it sets | | --- | --- | --- | --- | -| `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | A local registry: the first source that a lookup checks, and the place where downloads and builds land. | +| `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | The local registry, where downloads and builds land and which a lookup checks first. | | `--remote-registry` `URL` or `PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | | `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | From 99f126799ca4854efbce0a84a0317fe189d33951 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 17:12:48 +0100 Subject: [PATCH 26/28] docs: describe the new hints of the info commands and of a build conflict The info commands of timenet local now give a provenance record that is not valid the same hint as the clear commands. Before the build, timenet-build build --publish-to names --force when the destination and the local registry hold two different releases under the version. ReleaseDifference comes from timenet.types.references. The CLI guides and the error table say so. Part of TNET-157. --- docs/cli/build.md | 8 +++++--- docs/cli/timenet.md | 2 +- docs/types.md | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/cli/build.md b/docs/cli/build.md index 762f9f915..f6768a3ef 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -64,9 +64,11 @@ use `TIMENET_TOKEN`. from the local registry to the destination. If the destination holds the version and `--force` is not set, the command builds and copies -nothing. While `--force` builds the release again in the local registry. It cannot overwrite a -published version. If the new build wrote the same files, the publish copies nothing. If it wrote -other files, the publish raises `TimeNetVersionConflictError`, and the change needs a new version. +nothing. If the local registry holds another release under that version, the command fails before +the build, and the hint names `--force`, which compares a fresh build with the published release. +While `--force` builds the release again in the local registry. It cannot overwrite a published +version. If the new build wrote the same files, the publish copies nothing. If it wrote other files, +the publish raises `TimeNetVersionConflictError`, and the change needs a new version. ## Output diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 7f5f3f134..4419ad754 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -63,7 +63,7 @@ each other (parent-child relationships). In that case `timenet local downloads c `Kept : needs it.` for each. Because `local clear` deletes everything, no dependency relationships can persist, and everything is removed. A symlinked dataset or release folder in the local registry stops every clear, `local clear` included, before it deletes anything. -A provenance record that is not valid also stops them. Its hint names +A provenance record that is not valid also stops them and the info commands. Its hint names `timenet download @ --force` and `timenet-build build @ --force`, which replace the release and its record. Do not run a clear while a build or a download runs. diff --git a/docs/types.md b/docs/types.md index af50766b5..b1a6f0173 100644 --- a/docs/types.md +++ b/docs/types.md @@ -297,7 +297,7 @@ TimeNet raises its own exceptions from `timenet.errors`. | `TimeNetNoCatalogError` | No source of a list or a search has a catalog. The hint names the read of an exact release. A `TimeNetRegistryError`. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | | `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | -| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For a copy into the local registry, the hint names the release folder to delete. | +| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, from `timenet.types.references`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For `timenet-build build --publish-to` before the build, the hint names `--force`. For a copy into the local registry, the hint names the release folder to delete. | | `TimeFValidationError` | Input violates a TimeF invariant. | | `TimeFNullValuesError` | Values contain nulls that NumPy cannot hold. A `TimeFValidationError`. | | `TimeFNotFoundError` | An ID, parent alias, or parent release names nothing in the dataset, or the stored object has another kind. A `TimeFValidationError` and a `LookupError`. | From 1cf6b4a20a6765efa30ec83e13fda5941ea2f345 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 17:47:22 +0100 Subject: [PATCH 27/28] docs: name timenet.errors as the module of ReleaseDifference ReleaseDifference moved back into timenet.errors, next to TimeNetVersionConflictError, whose difference attribute has this type. The row of the error in the types page now names that module. Part of TNET-157. --- docs/types.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/types.md b/docs/types.md index b1a6f0173..63ec36ebd 100644 --- a/docs/types.md +++ b/docs/types.md @@ -297,7 +297,7 @@ TimeNet raises its own exceptions from `timenet.errors`. | `TimeNetNoCatalogError` | No source of a list or a search has a catalog. The hint names the read of an exact release. A `TimeNetRegistryError`. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | | `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | -| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, from `timenet.types.references`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For `timenet-build build --publish-to` before the build, the hint names `--force`. For a copy into the local registry, the hint names the release folder to delete. | +| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, from `timenet.errors`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For `timenet-build build --publish-to` before the build, the hint names `--force`. For a copy into the local registry, the hint names the release folder to delete. | | `TimeFValidationError` | Input violates a TimeF invariant. | | `TimeFNullValuesError` | Values contain nulls that NumPy cannot hold. A `TimeFValidationError`. | | `TimeFNotFoundError` | An ID, parent alias, or parent release names nothing in the dataset, or the stored object has another kind. A `TimeFValidationError` and a `LookupError`. | From 04830ed17b6b683aad9f223cafa47df99b07d434 Mon Sep 17 00:00:00 2001 From: Adrian Blumer Date: Mon, 9 Nov 2026 21:28:30 +0100 Subject: [PATCH 28/28] docs: fix the facts that the review of the docs found - put_dataset refuses a version when the registry lacks one of its payload sources, not one of its parents. - A source's resolve() takes a Version object, or None for the newest version of that source. - get_manifest() raises TimeNetManifestError for a stored manifest that is not valid. - Only the registries, auto-build, isolation, the cache and the token have environment variables. The build guide links to the settings table. - The build steps describe the order of the default list of sources. A Python build with its own list uses that order instead. - read_only=True belongs to the local registry in the list, not to the client. - The timenet-datasets skill names the three source flags and links to the CLI docs for the details, instead of repeating that section. Part of TNET-157. --- .agents/skills/timenet-datasets/SKILL.md | 15 +++------------ docs/build.md | 9 ++++++--- docs/client.md | 4 ++-- docs/registry.md | 8 ++++---- 4 files changed, 15 insertions(+), 21 deletions(-) diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index 9200b19ee..f27cfbf37 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -61,18 +61,9 @@ deletes anything. Do not run a clear while a build or a download runs. Global: `-q`/`--quiet` suppresses status output; it goes before the command. -Sources: `list`, `search`, `info` and `download` take three flags after the command. Each flag wins -over its environment variable, and the variable wins over its default. A registry flag that names no -registry (a blank value, an unknown scheme, a URL for `--local-registry`) exits with code 2 and -names the flag. A bad `TIMENET_*` variable exits with code 1 and names the variable. - -- `--local-registry PATH`: the local registry, where downloads and builds land and which a lookup - checks first (`$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry`). -- `--remote-registry URL|PATH`: a remote registry, as a URL or a local path, the second source that - a lookup checks, if the local registry does not have the dataset (`$TIMENET_REMOTE_REGISTRY`, - else `timenet://`). -- `--auto-build`/`--no-auto-build`: whether installed connectors build a dataset that neither the - local nor the remote registry has (`$TIMENET_AUTO_BUILD`, else true). +Sources: `list`, `search`, `info` and `download` take `--local-registry`, `--remote-registry` and +`--auto-build`/`--no-auto-build` after the command. Each flag wins over its environment variable +(see [Selecting the sources](../../../docs/cli/timenet.md#selecting-the-sources) for details). `list` and `search` merge every source, `info` resolves like `open()`, and `download` copies into the local registry. Of these three flags, `publish` and the `local` commands other than diff --git a/docs/build.md b/docs/build.md index e73c629a0..4b33def97 100644 --- a/docs/build.md +++ b/docs/build.md @@ -43,7 +43,8 @@ Use `--no-isolation` while developing a connector in the current environment: timenet-build build org/name --local-registry .timenet-registry --no-isolation ``` -Most parameters can also be specified as environment variables. For example, +The local and remote registries, auto-build, isolation, the cache and the token have environment +variables (see the [settings table](client.md#settings)). For example, `$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the Python SDK also honors them. @@ -140,8 +141,7 @@ A connector card can declare exact parent releases under `parents`. The connecto releases when it composes the child dataset, preparing each parent release in the local registry that it builds into. -When resolving dependencies, the build uses the first of these steps that -works: +With the default list of sources, the build uses the first of these steps that works: 1. Reuse a complete copy in the local registry. `--local-registry` selects it. 2. Copy the release from the remote registry. `--remote-registry` selects it, else @@ -149,6 +149,9 @@ works: 3. Build the release from its own connector. `--no-auto-build`, or `TIMENET_AUTO_BUILD=false`, removes this step. +A Python build with its own list, `timenet.build(..., sources=[...])`, uses the order of that list +instead. + If no step works, the build stops before any download, and the error names the missing release. ## Authoring loop diff --git a/docs/client.md b/docs/client.md index 39a8fe50f..a0a4930c8 100644 --- a/docs/client.md +++ b/docs/client.md @@ -58,8 +58,8 @@ any source provides. Only "not found" and "not served" fall through to the next other error stops the call. So if a registry in the list does not answer, a request for `@latest` fails. -A client that must never write takes `read_only=True`. A list of remote registries alone has no -local registry, so it never writes either: +A client whose local registry is `LocalRegistry(path, read_only=True)` never writes. A list of +remote registries alone has no local registry, so it never writes either: ```python import timenet diff --git a/docs/registry.md b/docs/registry.md index 834aded62..05b74a9bc 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -84,7 +84,7 @@ Every backend implements this contract. | Method | Description | | --- | --- | | `list_datasets()` | Latest-version `DatasetMetadata` for every dataset, sorted by id. | -| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeFFormatError` if the stored manifest is not valid, or if its own id disagrees with the directory it was loaded from. | +| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeNetManifestError` if the stored manifest is not valid, or `TimeFFormatError` if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | | `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Each backend must define it. | | `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | @@ -123,7 +123,7 @@ names each one and pins the files it reads from it (see - `TimeNet.download()` and `copy_release()` copy complete copies of the version and of each payload source, and check each source against the pins. If a downloaded file has another size or checksum than its manifest records, the copy raises `TimeNetFileError` and keeps no partial copy. -- `put_dataset` refuses a version if the registry does not hold one of its parents. +- `put_dataset` refuses a version if the registry does not hold one of its payload sources. ### Remote reads @@ -138,8 +138,8 @@ copies complete files, and makes sure that each file has its recorded size and c ## Sources A source is something that can provide a release. `timenet.sources` defines the type -`ReleaseSource`. Each source has the method `resolve(dataset_id, version)`. `version` is an exact -version, such as `"1.2.0"`, or `None` for the newest version of that source. Two kinds of source +`ReleaseSource`. Each source has the method `resolve(dataset_id, version)`. `version` is a `Version` +object for an exact version, or `None` for the newest version of that source. Two kinds of source exist: - A registry. Every `BaseRegistry` is a source.