diff --git a/.agents/skills/add-dataset-connector/SKILL.md b/.agents/skills/add-dataset-connector/SKILL.md index c100ef64f..0b32a171e 100644 --- a/.agents/skills/add-dataset-connector/SKILL.md +++ b/.agents/skills/add-dataset-connector/SKILL.md @@ -7,7 +7,7 @@ description: Use when adding a new TimeNet dataset connector, i.e. converting an A connector fetches a dataset's raw source and converts it into a `TimeFDataset`. It implements the `BaseConnector` contract (in the `timenet` package) and lives in `timenet-connectors`. The engine drives -it `download -> convert -> derive_schema -> store`; `timenet-build build ` runs that pipeline and +it `download -> convert -> derive_schema -> store_dataset`; `timenet-build build ` runs that pipeline and writes the result into a registry. Reuse a base connector wherever the source allows: `BaseHuggingFaceConnector` for Hub datasets, @@ -99,8 +99,8 @@ not by an id. A span takes Signal objects, for example in some docs are **not implemented**, so don't rely on them. - If you added or changed a `requirements.txt`, re-run `make sync` so the new library lands in your own environment. Otherwise `ty` reports your lazy import as unresolved and the connector test can't run. -- Round-trip end to end: `uv run timenet-build build --out `, then - `with TimeNet(registry="").open("") as reader: reader.read().describe()`. +- Round-trip end to end: `uv run timenet-build build --local-registry `, then + `with timenet.TimeNet(sources=[""]).open("") as reader: reader.read().describe()`. - Run `make check` and `make test`, and state which checks you ran (per AGENTS.md). ## Further reading diff --git a/.agents/skills/add-dataset-connector/references/connector-anatomy.md b/.agents/skills/add-dataset-connector/references/connector-anatomy.md index f43463321..45463b585 100644 --- a/.agents/skills/add-dataset-connector/references/connector-anatomy.md +++ b/.agents/skills/add-dataset-connector/references/connector-anatomy.md @@ -21,8 +21,8 @@ this branch. Anchor files: - `metadata(self) -> DatasetMetadata` (**concrete**, do not override): loads and validates the card via `DatasetMetadata.from_yaml`. By convention the card is `dataset.yaml` beside the connector module; set the `CARD` class var to point elsewhere. (Some docs call `metadata` abstract; it isn't.) -- `store(...)` (concrete): derives the schema if missing and streams the dataset through `TimeFWriter`. - Most connectors never override it. +- The connector has no write method. The engine function `timenet.engine.store_dataset()` derives the + schema if missing and streams the dataset through `TimeFWriter`. Connectors take **no constructor arguments** (configuration comes from the environment). End with a module-level `CONNECTOR = `. diff --git a/.agents/skills/timenet-datasets/SKILL.md b/.agents/skills/timenet-datasets/SKILL.md index be2614649..f27cfbf37 100644 --- a/.agents/skills/timenet-datasets/SKILL.md +++ b/.agents/skills/timenet-datasets/SKILL.md @@ -1,13 +1,14 @@ --- name: timenet-datasets -description: Use when finding, searching, inspecting, downloading, or loading TimeNet datasets, via the `timenet` CLI or the Python `TimeNet` client (list, search, info, download, open, load, open_torch). Covers dataset ids, version pinning, and registry/storage configuration. +description: Use when finding, searching, inspecting, downloading, or loading TimeNet datasets, via the `timenet` CLI or the Python `TimeNet` client (list, search, info, download, open, load, open_torch). Covers dataset ids, version pinning, the list of sources, and the settings of the local and remote registries and the cache. --- # Using TimeNet datasets -TimeNet's consumer side is the `timenet` CLI and the `timenet.client.TimeNet` SDK. Both browse a -**registry** (the catalog of built datasets) and fetch dataset versions into local storage. The CLI -is a thin mirror of the SDK, so anything below works the same either way. +TimeNet's consumer side is the `timenet` CLI and the `timenet.TimeNet` SDK. Both find, read and +download TimeNet datasets: they read dataset versions from local and remote **registries** +(catalogues of built datasets). A local registry can also serve to store locally built or downloaded +datasets. The CLI is a thin mirror of the SDK, so anything below works the same either way. Producing datasets (writing connectors, running `timenet-build`) is a separate concern. For that, use the `add-dataset-connector` skill. @@ -17,7 +18,8 @@ the `add-dataset-connector` skill. - A dataset id is `org/name` (HuggingFace-style, exactly one slash), for example `chengsenwang/tsqa`. - Pin a version with `@` or `@latest`, e.g. `chengsenwang/tsqa@1.0.0`. You can also pass the version as a separate argument. Don't do both (it raises). -- Omitting the version means the latest. +- Omitting the version means the newest version of the first source that serves the dataset, so a + local copy reads offline. `@latest` means the highest version of all sources. ## CLI @@ -26,12 +28,24 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. | Command | What it does | | --- | --- | -| `timenet list` | Table of every dataset in the registry (id, name, license, domains). | -| `timenet search [filters]` | Filter datasets. See flags below. | -| `timenet info [version]` | Manifest summary: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). | -| `timenet download [version]` | Fetch a version's parquet to local storage; prints the target directory to stdout. | -| `timenet cache info` | Cached datasets with sizes, plus the raw download cache size. | -| `timenet cache clear` | Delete downloads and the raw cache. `--all` also removes the local registry; `-y`/`--yes` skips the prompt. | +| `timenet list` | Table of every dataset of the sources at its highest version, connector cards included (id, name, license, domains). | +| `timenet search [filters]` | Filter the datasets of the sources, connector cards included. See flags below. | +| `timenet info [@version]` | Manifest summary of the release that `open()` reads: version, name, license, domains, specs, task types, and counts (records / annotations / chunks). A dataset that only a connector provides shows its card and the build command. | +| `timenet download [@version]` | Copy a release and its data files into the local registry, and print the release directory to stdout. Never builds. Prints "Nothing to do" if the local registry holds everything. | +| `timenet publish [@version] --dest` `URL` or `PATH`, optionally `--source` `URL` or `PATH` | Copy a complete release and its payload sources from the local registry (or `--source`) to another registry, and print the release to stdout. Same release: nothing happens. Another release under the version: an error. | +| `timenet local info` | Path and size of the cache, then one line each for the downloads and the builds in the local registry (count and size). | +| `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | +| `timenet local cache info` / `clear [--yes]` | Path and size of the cache (`$TIMENET_CACHE`), or delete its raw source files (`downloads/`) and control databases (`control/`) that carry TimeNet's `CACHEDIR.TAG`. | +| `timenet local downloads info` / `clear [--yes]` | List the downloads in the local registry (dataset, version, source, size, total), or delete them. | +| `timenet local builds info` / `clear [--yes]` | List the builds in the local registry (dataset, version, source, size, total), or delete them. | + +A download is a release whose provenance record says `download`. A build is a release whose +record says `build`, or a release without a record. Releases can depend on each other (parent-child +relationships). In that case `downloads clear` and `builds clear` might keep releases that are +depended on, and print `Kept : needs it.` for each. Because `local clear` deletes +everything, no dependency relationships can persist, and everything is removed. A symlinked dataset +or release folder in the local registry stops every clear, `local clear` included, before it +deletes anything. Do not run a clear while a build or a download runs. `search` flags (repeat a flag to pass several values; all are ANDed across kinds): @@ -45,26 +59,66 @@ The CLI needs the `cli` extra (`timenet[cli]`); `make sync` already installs it. - `--tag` - `--limit` (default 100) -Global: `-q`/`--quiet` suppresses status output. `-r`/`--registry` picks the registry (else -`$TIMENET_REGISTRY`, else the local default). `download` takes `--storage` (else `$TIMENET_STORAGE`). +Global: `-q`/`--quiet` suppresses status output; it goes before the command. + +Sources: `list`, `search`, `info` and `download` take `--local-registry`, `--remote-registry` and +`--auto-build`/`--no-auto-build` after the command. Each flag wins over its environment variable +(see [Selecting the sources](../../../docs/cli/timenet.md#selecting-the-sources) for details). + +`list` and `search` merge every source, `info` resolves like `open()`, and `download` copies into +the local registry. Of these three flags, `publish` and the `local` commands other than +`local cache` take only `--local-registry`. The `local cache` commands take none of them. ## Python API ```python -from timenet.client import TimeNet +import timenet -tn = TimeNet() # registry: arg > $TIMENET_REGISTRY > local default +tn = timenet.TimeNet() # the default list of sources, see below tn.list() # list[DatasetMetadata] -tn.get("chengsenwang/tsqa") # Manifest (add a version or use id@version) +tn.get_manifest("chengsenwang/tsqa") # Manifest (add a version or use id@version) tn.search(domain=..., task=AnswerTask, tag="ecg", limit=50) # filters are scalar-or-list, ANDed -path = tn.download("chengsenwang/tsqa") # -> local /// dir +path = tn.download("chengsenwang/tsqa") # -> /// dir with tn.open("chengsenwang/tsqa") as reader: # TimeFReader; values stay in storage dataset = reader.read() dataset = tn.load("chengsenwang/tsqa") # every value read into memory ``` -- `TimeNet(registry=None, *, storage_path=None)`. `registry` accepts a `BaseRegistry`, a URL, a - `file://` URI, or a local path. +- `timenet.TimeNet(sources=None)`. `sources` is an ordered list. Each item is a registry object, a + URL, a `file://` URI, a local path, or `timenet.InstalledConnectors()`. Pass a list even for one + registry: `timenet.TimeNet(sources=[".timenet-registry"])`. The order of the list is the read + order: a release that two sources hold comes from the first one, so put the local registry first + to read offline. An exact version comes from the first source that has it, no version gives the + newest version of the first source that serves the dataset, and `@latest` is the highest version + of any source. +- The first `timenet.LocalRegistry(path)` of the list that is not `read_only=True` gets the + downloads and builds. A path in the list opens such a registry too. A list with + `InstalledConnectors()` needs a writable local registry, and only such a list builds on a read. + A remote registry reads only its own files. +- `TimeNet()` uses `timenet.client.default_sources()`: the local registry `$TIMENET_LOCAL_REGISTRY`, + then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()` unless `TIMENET_AUTO_BUILD=false`. + The first entry is always the local registry; `default_local_registry()` returns it alone. +- `list()` and `search()` merge every source of the list, connector cards included: one row per + dataset at the highest version, sorted by ID. `search()` filters these merged rows and stops at + `limit` matches. It reads a manifest for `task` or `time_series_spec` only for a row whose card + matches. A connector never matches these two filters. A registry without a catalog (S3) is left + out with a `CatalogWarning`, and a list without any catalog raises `TimeNetNoCatalogError`. A card + that cannot be read is left out with a `ConnectorCardWarning`. To ask one registry alone, use + `open_registry("timenet://").search(DatasetFilter.of(...))`, with `DatasetFilter` from + `timenet.types`. +- `get_manifest()` returns the manifest of the release that `open()` reads, without a read of other + files. A registry that lists a release without serving it gives its listed manifest. A release + that only a connector provides raises `TimeNetNotBuiltError` (no manifest before a build). The + error holds the card of the connector as `error.card`. + `get_card()` returns the card, also from a connector. +- `download()` needs a writable local registry in the list. It resolves like `open()`. The release + and each payload source that the local registry lacks or holds damaged come from the first + registry other than the local registry that provides it, so they can come from different + registries. If the local registry holds a complete copy of the release, only the missing payload + sources are copied. It never builds: a release that only a connector provides raises + `TimeNetNotBuiltError`, whose hint names `timenet.build()`. `force=True` copies again, and + `progress=` takes the events of `timenet.progress`. The hint of a read of a damaged copy names the + `download()` or the build that replaces it. - `open(...)` returns a `TimeFReader`; close it or use it in a `with` block. `reader.read()` returns a `TimeFDataset` whose values load only when asked: `record.signals[i].to_numpy()` / `.to_arrow()`. `reader.get(AnswerTask, task_id)` reads one object by its int ID. @@ -78,12 +132,12 @@ dataset = tn.load("chengsenwang/tsqa") # every value read int ## End-to-end recipe ```python -from timenet.client import TimeNet +import timenet from timenet.types import AnswerTask -tn = TimeNet() +tn = timenet.TimeNet() hits = tn.search(task=AnswerTask, limit=10) # find candidates -tn.get(hits[0].dataset_id) # inspect the manifest +tn.get_manifest(hits[0].dataset_id) # inspect the manifest with tn.open(hits[0].dataset_id) as reader: # values stay in storage ds = reader.read() ds.describe() # identity, counts, per-spec columns, preview @@ -95,14 +149,23 @@ CLI equivalent: `uv run timenet search --task answer` then `timenet info ` t ## Configuration -All local state lives under `TIMENET_HOME` (default `~/.cache/timenet`). Env vars, prefix `TIMENET_`: - -- `TIMENET_HOME` relocates everything. -- `TIMENET_REGISTRY` selects the registry (URL or path). -- `TIMENET_STORAGE` where downloads are cached (default `/storage`). -- `TIMENET_CACHE` raw source / Hub download cache (default `/cache`). - -Precedence for any value: explicit argument or CLI flag, then env var, then default. +Env vars, prefix `TIMENET_`. Each sets one thing; none holds a list: + +- `TIMENET_LOCAL_REGISTRY` the first source that a lookup checks, and the place where + downloads and builds land (default `~/.timenet/registry`). +- `TIMENET_REMOTE_REGISTRY` a remote registry, as a URL or a local path: the second source that a + lookup checks, if the local registry does not have the dataset (default `timenet://`). +- `TIMENET_AUTO_BUILD` whether the default list ends with `InstalledConnectors()` (default `true`). +- `TIMENET_CACHE` raw downloads of builds in `downloads/` and control databases in `control/` + (default `~/.cache/timenet`). +- `TIMENET_TOKEN` bearer token of a remote registry; `TIMENET_ISOLATION` (default `on`). + +Precedence for any value: CLI flag, then env var, then default. An empty value is an error. Before +`timenet local cache clear` or `timenet local clear` deletes, and before a build removes its raw +downloads, TimeNet refuses a cache that is the home folder or a folder above it, or that overlaps +the local registry. In the CLI, a registry flag that names no registry exits with code 2, and a bad +variable with code 1. A `sources` list that you pass ignores `TIMENET_LOCAL_REGISTRY`, +`TIMENET_REMOTE_REGISTRY` and `TIMENET_AUTO_BUILD`. ## Further reading diff --git a/README.md b/README.md index 3146063ba..1c6a723b1 100644 --- a/README.md +++ b/README.md @@ -22,22 +22,24 @@ TimeNet needs Python 3.11 or newer. ```bash uv add 'timenet[cli]' uv tool install timenet-connectors -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -Load the local build with Python: +The build lands in the local registry at `~/.timenet/registry/timenet/hello-world/1.0.0/`. Load it +with Python: ```python -from timenet.client import TimeNet +from timenet import TimeNet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +dataset = TimeNet().load("timenet/hello-world@1.0.0") dataset.describe() signal = dataset.records[0].signals[0] values = signal.to_numpy() ``` -This example uses a local registry, so it works without registry credentials or network access. +`TimeNet()` reads the local registry first, and the local registry holds this exact version. +So the example works without registry credentials or network access. See the [Get started guide](https://docs.timenet.ai/get-started.html) for installation options and a longer example. diff --git a/docs/architecture.md b/docs/architecture.md index dfdcc389f..3078ab469 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,8 +18,8 @@ for per-component detail. TimeNet splits into three parts. A **connector** builds a raw source into a TimeF version. The **client/SDK** reads its manifest from a **registry** and loads the data. The relational control plane is one immutable DuckDB file. The values plane can be Parquet or Zarr. Reading never runs -connector code. Against a local registry, `open` and `load` can first build a dataset that the -registry does not have from an installed connector. +connector code. If the sources of the client include the installed connectors, `open` and `load` +can first build a dataset that no registry has into the local registry. | | What it is | Ships | Used by | | --- | --- | --- | --- | @@ -37,19 +37,19 @@ There can be several registries: one public, private internal ones, or a local d PRODUCE dataset.yaml + connector │ ▼ - engine download ─► convert ─► derive_schema ─► store + engine download ─► convert ─► derive_schema ─► store_dataset │ ▼ registry │ ▼ -CONSUME SDK ─► open_version ─► TimeFReader ─► Arrow +CONSUME SDK ─► TimeNet.open ─► TimeFReader ─► Arrow ``` The compiled `manifest.json` is the entry point and commit marker. It contains the card metadata, -derived schema, counts, and file list. `open_version` returns a handle to the manifest and version -files. `TimeFReader` uses the handle to read the DuckDB hierarchy and lazy Signal values. The SDK -never imports connector code. +derived schema, counts, and file list. `TimeNet.open()` reads the manifest and opens a handle to +the files of the version. `TimeFReader` uses the handle to read the DuckDB hierarchy and lazy Signal +values. The SDK never imports connector code. --- @@ -61,7 +61,7 @@ Three producer-side pieces, each with one job: | --- | --- | --- | | **Connector** | one `BaseConnector` subclass per dataset ([connectors](connectors.md)) | the dataset-specific recipe: `download()` fetches raw files, `convert()` builds a `TimeFDataset`. Knows nothing about the engine or registry. | | **Engine** | `run_pipeline` ([build & publish](build.md)) | drives any connector through the fixed pipeline and owns caching, idempotency, and `force` / `keep_cache`. Knows no dataset specifics. | -| **Builder** | the `timenet-build` CLI ([build](build.md)) | the entry point: resolves the id to its connector and runs the engine into a registry. | +| **Builder** | the `timenet-build` CLI and `timenet.build()` ([build](build.md)) | the entry point: resolves the id to its connector and runs the engine into the local registry. | ``` timenet-build build org/name @@ -69,14 +69,14 @@ timenet-build build org/name ├─ builder discovery.resolve("org/name") -> Connector class │ datasets/// exposes CONNECTOR │ - └─ engine run_pipeline(connector, ) - metadata -> download -> convert -> derive_schema -> store - -> /org/name// + └─ engine run_pipeline(connector, ) + metadata -> download -> convert -> derive_schema -> store_dataset + -> /org/name// ``` `metadata()` reads the `dataset.yaml` card. The engine streams the converted dataset through -[`TimeFWriter`](timef-writer.md). A local output directory is also a valid registry, so the client -can read it directly. +[`TimeFWriter`](timef-writer.md) into the local registry, from where the client can +read it directly. --- @@ -115,9 +115,11 @@ can read it directly. 1. **Author** a connector under `timenet_connectors/datasets///`. Its `__init__.py` exposes `CONNECTOR`, and its `dataset.yaml` card sits beside it. -2. **Build**: `timenet-build build /` runs the engine, compiles the manifest, and writes a TimeF version. -3. **Verify** locally: point the SDK at the output directory (itself a valid local registry). -4. **Publish** the complete TimeF version to a registry. +2. **Build**: `timenet-build build /` or `timenet.build("/")` runs the engine, + compiles the manifest, and writes a TimeF version. +3. **Verify** locally: the SDK reads the local registry, so it finds the new version there. +4. **Publish** the complete TimeF version to a remote registry with `timenet publish` or + `timenet.publish()`. 5. **Consume**: use `TimeNet.open()` for lazy access, `TimeNet.load()` for a dataset with every value in memory, or `timenet download` from the CLI for a local copy. @@ -140,10 +142,10 @@ runs, the build prepares each parent release, as [Prepare parent releases](build.md#prepare-parent-releases) describes. `TimeNet.open()`, `TimeNet.load()`, `TimeNet.open_torch()`, and `TimeFReader.open()` resolve the -payload sources before the first read. A read never opens the catalog of a parent. Against a local -registry, the client reuses a complete copy of each release, copies it from `fetch_from`, or builds -it. `auto_build` is on by default. A remote client never runs connector code. See -[Open, load, or download](client.md#open-load-or-download). +payload sources before the first read. A read never opens the catalog of a parent. A client +reads each release where it is, through its ordered list of sources. If only +`InstalledConnectors()` in that list provides a release, the client builds it into the local +registry. See [Open, load, or download](client.md#open-load-or-download). The child keeps an internal map from the ID of each parent object to the ID of its copy. Through this map, `get(..., parent=)` finds a copy. Through the same map, a later import reuses a copy. The diff --git a/docs/assets/architecture.svg b/docs/assets/architecture.svg index 6a1ef93e1..8a45a9381 100644 --- a/docs/assets/architecture.svg +++ b/docs/assets/architecture.svg @@ -42,7 +42,7 @@ timenet-build build timenet/hello-world - --out .timenet-registry + --local-registry ./registry diff --git a/docs/build.md b/docs/build.md index b413c0cb3..4b33def97 100644 --- a/docs/build.md +++ b/docs/build.md @@ -14,17 +14,19 @@ version contains `manifest.json`, `control.duckdb`, and Parquet or Zarr Signal v Use `timenet-build` for normal builds: ```bash -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -The output can be a local directory or a writable registry URL. +The release lands in the local registry (default `~/.timenet/registry`), where +`TimeNet()` reads from. `--local-registry` selects another local registry. `--publish-to` publishes +the release to another registry after the build: ```bash # Publish through the hosted registry API. -timenet-build build org/name --out timenet:// +timenet-build build org/name --publish-to timenet:// # Publish to S3. -timenet-build build org/name --out s3://bucket/prefix +timenet-build build org/name --publish-to s3://bucket/prefix ``` See the [`timenet-build` reference](cli/build.md) for all options. @@ -38,150 +40,119 @@ requirements do not affect each other or your current environment. Use `--no-isolation` while developing a connector in the current environment: ```bash -timenet-build build org/name --out .timenet-registry --no-isolation +timenet-build build org/name --local-registry .timenet-registry --no-isolation ``` -`TIMENET_ISOLATION=off` provides the same setting. The manifest records the Python and package -versions used for the build. +The local and remote registries, auto-build, isolation, the cache and the token have environment +variables (see the [settings table](client.md#settings)). For example, +`$TIMENET_LOCAL_REGISTRY` & `TIMENET_ISOLATION=off`. These apply to every CLI command, and the +Python SDK also honors them. + +For each build, the used Python and package and TimeNet versions are recorded to the build's +manifest. ## Pipeline The engine runs these stages: -1. Read and validate the connector's `dataset.yaml` card. If the output registry is not local, the - card must declare `access: open`. -2. Open the parent releases that the card declares. The output registry must hold them, as +1. Read and validate the connector's `dataset.yaml` card. +2. Prepare the parent datasets that the card declares, in the local registry. Before the build + runs, the plan downloads or builds each missing parent there, as [Prepare parent releases](#prepare-parent-releases) describes. 3. Download raw artifacts into the build cache. 4. Compose the parents and the raw artifacts into a `TimeFDataset`. Record each parent in it. 5. Derive the dataset schema. -6. Write or publish the complete TimeF version. Then close the parent releases. Before a publish to - a registry that is not local, the build makes sure that each card in the lineage also declares - `access: open`. +6. Write the complete TimeF version into the local registry. 7. Remove the raw cache after success, unless `--keep-cache` is set. -`--force` rebuilds or republishes a version that already exists. Without this flag, the pipeline -returns the committed version and skips expensive work. A forced local build replaces the old -version only once the new one is complete, so a failed rebuild keeps it. - -The connector selects Parquet or Zarr by default. Override it when you need to inspect another -backend: - -```bash -timenet-build build timenet/hello-world \ - --out .timenet-registry \ - --values-backend zarr -``` +By default a build reuses an existing version already in the local registry and a status line says +so. Use `--force` to skip this behavior. A forced build replaces the old version only after the new +one is complete, so a failed build keeps the old one. A published version never changes, so with +`--publish-to`, `--force` only affects the build to the local registry. The publish then only checks +the rebuild against the already published version. ## Programmatic builds -Use `run_pipeline()` for a local directory: +Build and publish from Python with the producer functions of `timenet`: ```python -from pathlib import Path +import timenet -from timenet.engine import run_pipeline +# Build into the local registry. +timenet.build("org/name") -version_dir = run_pipeline( - connector, - Path(".timenet-registry"), - values_backend="parquet", -) -``` +# Read the new release, like any other. +dataset = timenet.TimeNet().load("org/name") -Use `publish_pipeline()` with a writable registry: - -```python -from timenet.engine import publish_pipeline -from timenet.registry import open_writable_registry - -registry = open_writable_registry("s3://bucket/prefix") -version = publish_pipeline(connector, registry) +# Copy the release and its payload sources to a registry. +timenet.publish("org/name", dest="s3://bucket/registry") ``` -Both functions accept `force`, `keep_cache`, `cache_dir`, `values_backend`, `pinned_by`, and a -writer progress callback. +`timenet.build()` builds a release with the installed connector that claims its dataset ID, in an +[isolated environment](#isolated-build-environments). It returns the release directory in the +local registry. Each argument that you leave out comes from the environment at the time of the +call: -A registry that is not local holds only releases with access declared `open`. `publish_pipeline()` -checks the card before it downloads anything. A credentialed or restricted card raises -`TimeNetAccessError` for such a registry. After the connector composes the dataset, -`publish_pipeline()` checks the dataset again with its lineage, before it writes. If the card of a -child declares `access: open` but its parent is credentialed, the second check raises the error. +- `local_registry` comes from `$TIMENET_LOCAL_REGISTRY`. +- `sources` is the default list of sources with the local registry first: the local registry, + then `$TIMENET_REMOTE_REGISTRY`, then `InstalledConnectors()`. With `TIMENET_AUTO_BUILD=false`, + the connectors stay out. -The error names the release and its effective access level, which is the strictest level of its -card and of the cards in its lineage. It also names each `access_url` that a user needs, with the -release that sets it. For a card, this is the `access_url` of the card. +So a default build lands where `TimeNet()` reads. `local_registry` takes a local directory, a +`file://` URI or a `LocalRegistry`. A remote registry or a `LocalRegistry` with `read_only=True` +raises `TimeFValidationError`, because a build writes into its local registry. To get the release +to such a registry, build it locally and publish it with `timenet.publish()`. -## Prepare parent releases - -A connector card can declare exact parent releases under `parents`. The connector reads these -releases when it composes the child dataset. Before the build downloads anything, it prepares each -parent release in the output registry. It also prepares the payload sources of each parent. A -payload source is a release whose payload files the parent reads. +`sources` lists, in order, where the parents and payload sources that the local registry lacks +come from (see [Prepare parent releases](#prepare-parent-releases)). The order of the list is the +read order: a release that two sources hold comes from the first one. -For each release, the build uses the first of these steps that works: +`force=True` builds the release again, even if the local registry holds it. With +`open_access_only=True`, the build refuses a release without open access with +`TimeNetAccessError` before any step runs. -1. Reuse a complete copy in the output registry. -2. Copy the release from the registry that `--fetch-from` names. -3. If `--build-parents` is set, build the release from its own connector. - -If no step works, the build stops before any download. The error names the missing release. The -build prepares each parent before its children. If the output already holds the requested version -and `--force` is not set, the build prepares only the missing payload sources of that version. It -uses the same steps in the same order. The build does not run the connector of the requested -dataset. +```python +import timenet -```bash -timenet-build build org/name --out .timenet-registry \ - --fetch-from timenet:// --build-parents +path = timenet.build( + "org/name@1.2.0", + local_registry=".timenet-registry", + sources=["timenet://", timenet.InstalledConnectors()], +) +print(path) ``` -A parent builds like the requested dataset, in the environment of its own connector. A parent never -builds with `--force`. If the builder backend of another installed package claims a parent, that -backend builds it. - -The build never builds over a release that the output holds. If a file of the copy in the output -is missing or has another size, the build copies the release again from `--fetch-from`. If the build -cannot copy it, the build raises `TimeNetFileError`. If a copied file does not match the size and -checksum that its manifest records, the build raises `TimeNetFileError`. - -If `--build-parents` is set, a missing payload source builds as a pinned release. The build -publishes the payload source only if it writes the files that the reading release pins. See -[Rebuild a pinned release](#rebuild-a-pinned-release) for the check and the two fixes. +`timenet.publish()` copies a complete release and the payload sources that it needs from `source`, +by default the local registry, to `dest`. Without a version in the dataset ID, it copies the latest +version of `source`. A `DatasetRef` names the exact release, so `publish()` copies it without a +lookup in `source`. It mirrors the [`timenet publish`](cli/timenet.md#commands) CLI command. -`--fetch-from` needs a local `--out`. An output that is not local holds only releases with access -declared `open`. So the build checks the access of each release before any download. A copy from a -registry that is not local also needs access declared `open`. +A published version cannot change. If `dest` already holds the same release, `publish()` copies +nothing. -`timenet_connectors.build()` and `timenet_connectors.load()` take the same options as keywords. -`fetch_from` takes a registry or its URI. `build_parents=True` builds missing parents. -```python -import timenet_connectors +A remote registry holds only releases with access declared `open`. Before +`timenet.publish()` copies anything to such a registry, it checks the card and the lineage of each +release. -dataset = timenet_connectors.load( - "org/name", - fetch_from="timenet://", - build_parents=True, -) -``` +## Prepare parent releases -## Rebuild a pinned release +A connector card can declare exact parent releases under `parents`. The connector reads these +releases when it composes the child dataset, preparing each parent release in the local registry +that it builds into. -A release can read payload files that another release stores. The reading release pins each of -these files by path, size, and checksum (see [Shared payloads](timef-format.md#shared-payloads)). -A rebuild of the stored release must write the same files. +With the default list of sources, the build uses the first of these steps that works: -Pass the pins to `run_pipeline()` or `publish_pipeline()` as `pinned_by`, keyed by the reading -release. The writer builds the release in a staging directory. Before it publishes the release, it -compares each pinned file with the file that it wrote. If a pinned file differs, the writer deletes -the staging directory and raises `TimeFFormatError`. Nothing is published. +1. Reuse a complete copy in the local registry. `--local-registry` selects it. +2. Copy the release from the remote registry. `--remote-registry` selects it, else + `$TIMENET_REMOTE_REGISTRY` (default `timenet://`). +3. Build the release from its own connector. `--no-auto-build`, or `TIMENET_AUTO_BUILD=false`, + removes this step. -The error names two fixes: +A Python build with its own list, `timenet.build(..., sources=[...])`, uses the order of that list +instead. -- Fetch the exact release from a registry that holds it. -- Rebuild the reading release with force, so that it pins the new files. The releases that pin the - old files of the reading release then fail to open. +If no step works, the build stops before any download, and the error names the missing release. ## Authoring loop @@ -195,11 +166,9 @@ The error names two fixes: The local check keeps build and load paths explicit: ```python -from timenet.client import TimeNet +import timenet -with TimeNet(".timenet-registry").open("org/name") as reader: +client = timenet.TimeNet(sources=[".timenet-registry"]) +with client.open("org/name") as reader: reader.read().describe() ``` - -See [Connectors](connectors.md) for the `download()` and `convert()` contract. See the -[`timenet.engine` API](api/engine.md) for the programmatic interface. diff --git a/docs/cli/build.md b/docs/cli/build.md index 1c817410f..f6768a3ef 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -8,7 +8,8 @@ tags: # `timenet-build` `timenet-build` is the producer CLI. It resolves an installed connector, runs the build pipeline, -and writes or publishes one TimeF version. It ships with `timenet-connectors`. +writes one TimeF version into the local registry, and can publish it. It ships with +`timenet-connectors`. ```bash uv tool install timenet-connectors @@ -20,26 +21,30 @@ author or change a connector. ## Build command ```bash -timenet-build [--quiet] build \ - [--out ] \ - [--values-backend parquet|zarr] \ - [--force] [--keep-cache] [--no-isolation] \ - [--fetch-from ] [--build-parents] +timenet-build [--quiet] build [@] \ + [--force] [--keep-cache] [--isolation | --no-isolation] \ + [--publish-to ] [--dry-run] \ + [--local-registry ] [--remote-registry ] \ + [--auto-build | --no-auto-build] ``` | Option | Default | Purpose | | --- | --- | --- | -| `` | required | Select the connector by its `org/name` ID. | -| `--out` | `$TIMENET_REGISTRY`, then the local registry | Select a local or remote output registry. | -| `--values-backend` | connector setting | Override the Parquet or Zarr values backend. | -| `--force`, `-f` | off | Rebuild or republish an existing version. | +| `[@]` | required | Select the connector by its `org/name` ID. A version must be the one that the card of the connector declares. | +| `--force`, `-f` | off | Build the release again in the local registry, even if the local registry holds it. | | `--keep-cache` | off | Keep downloaded source files after a successful build. | -| `--no-isolation` | isolation on | Use the current environment instead of a connector-specific one. | -| `--fetch-from` | off | Copy missing parent releases from the registry at this URI. `--out` must be local. | -| `--build-parents` | off | Build missing parent releases from their own connectors. | +| `--isolation`, `--no-isolation` | `$TIMENET_ISOLATION`, else on | Run this build in an environment built from the requirements of the connector, or in the current one. | +| `--publish-to` | off | After the build, publish the release and its payload sources to the registry at this URL or path. | +| `--dry-run` | off | Print the plan to stdout and stop. It copies, builds, and publishes nothing. | +| `--local-registry` | `$TIMENET_LOCAL_REGISTRY`, else `~/.timenet/registry` | The local registry, where downloads and builds land and which a lookup checks first. | +| `--remote-registry` | `$TIMENET_REMOTE_REGISTRY`, else `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have a parent. | +| `--auto-build`, `--no-auto-build` | `$TIMENET_AUTO_BUILD`, else on | Whether installed connectors build a parent that neither the local registry nor the remote registry has. | | `--quiet`, `-q` | off | Hide status output. Place this option before `build`. | -The output selector accepts: +The build lands in the local registry, where `TimeNet()` reads. + +`--local-registry` takes a local path or a `file://` URI. `--publish-to` and `--remote-registry` +accept: - a local path or `file://` URI; - `s3://bucket/prefix`; @@ -49,46 +54,75 @@ The output selector accepts: Remote publishing needs the credentials required by that registry. For example, the hosted API can use `TIMENET_TOKEN`. +## Publish after the build + +`--publish-to` gets the release to another registry in two steps: + +1. The release builds into the local registry, as without `--publish-to`. Its parents land there + too. +2. `timenet.publish()` copies the release, and each payload source that the destination lacks, + from the local registry to the destination. + +If the destination holds the version and `--force` is not set, the command builds and copies +nothing. If the local registry holds another release under that version, the command fails before +the build, and the hint names `--force`, which compares a fresh build with the published release. +While `--force` builds the release again in the local registry. It cannot overwrite a published +version. If the new build wrote the same files, the publish copies nothing. If it wrote other files, +the publish raises `TimeNetVersionConflictError`, and the change needs a new version. + ## Output -Status goes to stderr. A local build prints the committed version directory to stdout. A remote -build prints the published version string. +Status goes to stderr. Before the first step, the command shows its plan as a tree: the requested +release at the top, and below each release the releases that it needs. Each line names a release, +what the build does with it, and the source: + +```text +📋 Plan for 'physionet/ecg-qa-cot' 1.2.0: + physionet/ecg-qa-cot@1.2.0: build with installed connectors + └── physionet/ptb-xl@1.0.0: copy from registry https://registry.timenet.ai +``` + +Pass `--quiet` to hide the plan and the status lines. ```bash -VERSION_DIR=$( - timenet-build --quiet build timenet/hello-world \ - --out .timenet-registry -) +VERSION_DIR=$(timenet-build --quiet build timenet/hello-world) test -f "$VERSION_DIR/manifest.json" ``` This separation lets scripts capture the result without parsing progress messages. +`--dry-run` prints only the tree to stdout, and stops. It reads cards and manifests, but it copies, +builds, and publishes nothing. + ## Exit codes | Code | Meaning | | --- | --- | | `0` | The version was built, published, or reused. | | `1` | An expected TimeNet operation failed. | -| `2` | The command or one of its options is invalid. | +| `2` | The command or one of its options is invalid, for example an unknown dataset ID, or a registry flag that names no registry. | + +If an isolated build cannot make its environment, for example because `uv` is missing, the command +raises `TimeNetIsolationError`. Unexpected programming errors keep their traceback. ## Inspect a local build -Use the same explicit registry for the build and the consumer: +Build into a local registry of your own, and pass the same registry to the consumer: ```bash -timenet-build build timenet/hello-world --out .timenet-registry -timenet list --registry .timenet-registry +timenet-build build timenet/hello-world --local-registry .timenet-registry +timenet info timenet/hello-world --local-registry .timenet-registry ``` Or load it in Python: ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +client = timenet.TimeNet(sources=[".timenet-registry"]) +dataset = client.load("timenet/hello-world") ``` See [Build and publish](../build.md) for the pipeline and [Connectors](../connectors.md) for the diff --git a/docs/cli/index.md b/docs/cli/index.md index d09f32231..39882d26f 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -11,8 +11,8 @@ TimeNet ships two console scripts, one for each side of the workflow: | Command | Ships with | Role | Mirrors | | --- | --- | --- | --- | -| [`timenet`](timenet.md) | `timenet` | Consumer: browse a registry and download datasets. | The [client SDK](../client.md). | -| [`timenet-build`](build.md) | `timenet-connectors` | Producer: run a connector through the pipeline into a registry. | [Build & publish](../build.md). | +| [`timenet`](timenet.md) | `timenet` | Consumer: find, read, download and publish TimeNet datasets. | The [client SDK](../client.md). | +| [`timenet-build`](build.md) | `timenet-connectors` | Producer: run a connector through the pipeline into the local registry, and publish the result. | [Build & publish](../build.md). | Both tools use [Typer](https://typer.tiangolo.com). The `timenet` command needs the `cli` extra: @@ -23,12 +23,14 @@ uv add 'timenet[cli]' If you run it without the extra, it prints a one-line install hint instead of a traceback. Both tools split their output the same way: status lines go to stderr, the machine-readable result (a -path) goes to stdout. `--quiet`/`-q` silences the status. The flag belongs to the tool, not the -subcommand, so it comes first: `timenet --quiet list`, `timenet-build --quiet build `. - -Both tools honor `$TIMENET_REGISTRY`, but their final defaults differ. The consumer uses the hosted -registry when no registry is set. The build tool writes to the local registry. Pass `--registry` or -`--out` when a workflow must use a specific location. +path or a version) goes to stdout. `--quiet`/`-q` silences the status. The flag belongs to the tool, +not the subcommand, so it comes first: `timenet --quiet list`, `timenet-build --quiet build `. + +Both tools read the same [settings](../client.md#settings). The consumer copies into the local +registry, `$TIMENET_LOCAL_REGISTRY`, and the build tool writes to it. The commands that look up +datasets take `--local-registry`, `--remote-registry`, and `--auto-build`/`--no-auto-build`, which +override these settings for one command (see +[Selecting the sources](timenet.md#selecting-the-sources)). If you want to use datasets, start with [`timenet`](timenet.md). If you build a dataset, start with [`timenet-build`](build.md). diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 679e7356b..4419ad754 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -1,6 +1,6 @@ --- icon: lucide/terminal -description: "The timenet consumer CLI: browse a registry and download datasets from the shell." +description: "The timenet consumer CLI: find, read, download and publish TimeNet datasets from the shell." tags: - cli --- @@ -14,51 +14,97 @@ from the shell that you do in Python. Install it with the `cli` extra: uv add 'timenet[cli]' ``` -By default, the command uses the hosted TimeNet registry. You can also select a local or S3 -registry: +By default, the command uses the hosted TimeNet registry: ```bash timenet list timenet search --query ecg --domain cardiology --limit 10 -timenet info chengsenwang/tsqa@1.0.0 # pin a version; omit @ for the latest +timenet info chengsenwang/tsqa@1.0.0 # pin a version timenet download chengsenwang/tsqa ``` +To use a local or S3 registry instead, see [Selecting the sources](#selecting-the-sources). + ## Commands | Command | What it does | | --- | --- | -| `timenet list` | Print every dataset in the registry (id, name, license, domains), from the metadata of its latest version. | -| `timenet search [flags]` | Filter datasets. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). | -| `timenet info [@version]` | Show a dataset's [manifest](../manifest.md): metadata, schema, and counts. | -| `timenet download [@version]` | Copy a version's files into local storage and print the directory. `--storage ` picks the target (else `$TIMENET_STORAGE`, then `/storage`). If a local version already exists, it skips the copy. | -| `timenet cache info` | List downloaded datasets on disk (location, id, version, size) and the total. | -| `timenet cache clear` | Remove downloads and the raw cache. It prompts first. `-y` skips the prompt. `--all` also clears built data. | - -Every command writes its status to stderr. It writes its machine-readable result (a path) to stdout. -Therefore, you can capture `timenet download ` in a script safely. - -`--quiet`/`-q` removes that status output. It belongs to `timenet` itself. Therefore, it goes before +| `timenet list` | Print every dataset of the sources (id, name, license, domains), from the metadata of the highest version that a source lists. The cards of the installed connectors count too. See [`list()`](../client.md#browse-datasets). | +| `timenet search [flags]` | Filter the datasets of the sources, the cards of the installed connectors included. The flags map one-to-one to [`registry.search`](../registry.md#search). Repeat a flag for list values (`--spec` for `time_series_spec`, `--id` for `dataset_id`). `--limit` applies after the merge of the sources, and the search checks only the highest version of each dataset. | +| `timenet info [@version]` | Show the [manifest](../manifest.md) of the release that `open()` reads: metadata, schema, and counts. For a release that only an installed connector provides, show its card, and the `timenet-build` command that adds the schema and the counts. See [`get_manifest()`](../client.md#inspect-a-release). | +| `timenet download [@version] [--force]` | Copy a release and its data files into the local registry, and print the release directory. The release and each payload source that the local registry lacks come from the first registry that provides them. With `--force`, the release is copied again, also when the local registry holds a complete copy. It never builds. See [`download()`](../client.md#open-load-or-download). | +| `timenet publish [@version] --dest` `URL` or `PATH` | Copy a complete release and its payload sources to the registry `--dest`, and print the release. The release comes from the local registry, which `--local-registry` selects, or from the registry `--source` `URL` or `PATH`, and without a version it is the latest version there. A published version cannot change, so another release under the version in `--dest` is an error. See [`publish()`](../build.md#programmatic-builds). | +| `timenet local info` | Print the path and size of the cache. Then print one line for the downloads and one line for the builds in the local registry, such as `Downloads in : 2 releases, 1.4 GB`. | +| `timenet local clear [--yes]` | Delete the cache and every release in the local registry. The prompt counts the builds. | +| `timenet local cache info` | Print the path and size of the cache, `$TIMENET_CACHE`. | +| `timenet local cache clear [--yes]` | Delete the raw source files (`downloads/`) and the control databases (`control/`) of the cache that carry TimeNet's `CACHEDIR.TAG` file. | +| `timenet local downloads info` | List the downloads in the local registry: dataset, version, source, size, and the total. | +| `timenet local downloads clear [--yes]` | Delete the downloads in the local registry, except the ones that a release that stays needs. | +| `timenet local builds info` | List the builds in the local registry: dataset, version, source, size, and the total. | +| `timenet local builds clear [--yes]` | Delete the builds in the local registry, except the ones that a release that stays needs. | + +Every command writes its status to stderr and its machine-readable result (a path or a release) to +stdout, so you can capture `timenet download ` in a script safely. + +Most errors end with a hint that names a command or a flag that helps. + +`list` and `search` leave out a source without a catalog and a connector card that cannot be read. +For each, they print a warning line to stderr, without a Python file and line number. + +`--quiet`/`-q` removes the status output. It belongs to `timenet` itself. Therefore, it goes before the subcommand: `timenet --quiet list`, not `timenet list --quiet`. Be careful with the collision: after `search`, `-q` is the short form of `--query`. -## Selecting a registry +## Downloads and builds in the local registry + +The local registry holds two kinds of release, downloads and local builds. Releases can depend on +each other (parent-child relationships). In that case `timenet local downloads clear` and +`timenet local builds clear` might keep releases that are depended on, and print +`Kept : needs it.` for each. Because `local clear` deletes everything, no +dependency relationships can persist, and everything is removed. A symlinked dataset or release +folder in the local registry stops every clear, `local clear` included, before it deletes anything. +A provenance record that is not valid also stops them and the info commands. Its hint names +`timenet download @ --force` and `timenet-build build @ --force`, which +replace the release and its record. Do not run a clear while a build or a download runs. + +## Selecting the sources + +`list`, `search`, `info`, and `download` create a client with the default list of sources, +`[local registry, remote registry, installed connectors]` (see +[Choose the sources](../client.md#choose-the-sources)). Three flags set this list, and each flag +overrides one [variable](../client.md#settings): + +| Flag | Variable | Default | What it sets | +| --- | --- | --- | --- | +| `--local-registry PATH` | `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | The local registry, where downloads and builds land and which a lookup checks first. | +| `--remote-registry` `URL` or `PATH` | `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `--auto-build` / `--no-auto-build` | `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. | -The tool resolves the registry in this order: `--registry`, then `$TIMENET_REGISTRY`, then -`timenet://`. The last value selects the hosted registry. Local paths, `file://`, `s3://`, -`http(s)://`, and `timenet://` are valid selectors. +A flag wins over its variable, and the variable wins over its default. Flags go after the +subcommand: -The build tool has a different final default: it writes to the local registry. Pass the same explicit -path to both tools when you build and inspect a dataset locally. +```bash +timenet list --remote-registry https://registry.example.com +timenet download chengsenwang/tsqa --local-registry ./data --no-auto-build +``` + +`publish` copies out of the local registry by default. To mirror a hosted release in a registry of +your own, name the hosted registry as the source: + +```bash +timenet publish org/name --source timenet:// --dest s3://my-bucket/timenet +``` ## Pinning versions -Add the suffix `@` to an id to pin it. You can also use `@latest` (the default when you omit -the suffix): +Add the suffix `@` to an id to pin it. Without a suffix, the command reads the newest +version of the first source that serves the dataset, so a copy in the local registry reads offline. +`@latest` reads the highest version of all sources: ```bash timenet info chengsenwang/tsqa@1.0.0 # pinned -timenet download chengsenwang/tsqa # latest +timenet download chengsenwang/tsqa # newest of the first source +timenet info chengsenwang/tsqa@latest # highest of all sources ``` If you pin a version that is not committed, the tool exits with a `TimeNetDatasetNotFoundError`. diff --git a/docs/client.md b/docs/client.md index 0c70b48a6..a0a4930c8 100644 --- a/docs/client.md +++ b/docs/client.md @@ -1,6 +1,6 @@ --- icon: lucide/code -description: "Browse a registry and load TimeF datasets from Python." +description: "Find, read and download TimeNet datasets from Python." tags: - guide - client @@ -8,14 +8,14 @@ tags: # Client -`TimeNet` is the main Python entry point for dataset consumers. It wraps one -[registry](registry.md) and one local storage directory. +`TimeNet` is the main Python entry point for dataset consumers. It reads releases through an +ordered list of sources: [registries](registry.md) and the installed connectors. ```python -from timenet.client import TimeNet +import timenet from timenet.types import Domain -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) for metadata in client.search(domain=Domain.CARDIOLOGY): print(metadata.dataset_id) @@ -24,58 +24,148 @@ dataset = client.load("timenet/hello-world") values = dataset.records[0].signals[0].to_numpy() ``` -## Choose a registry +## Choose the sources -Create a client with a registry object, path, or URL: +A client takes an ordered list of sources. A source is a local or remote registry, or +`InstalledConnectors()`: ```python -from timenet.client import TimeNet +import timenet + +client = timenet.TimeNet(sources=[ + timenet.LocalRegistry(".timenet-registry"), + "https://registry.example.com", + "timenet://", + timenet.InstalledConnectors(), +]) +``` + +- `InstalledConnectors()` stands for every installed package that registers a builder backend, + such as `timenet-connectors`. If only this source provides a release, a read builds it. Without + it in the list, a client never runs connector code. +- The first `LocalRegistry` of the list that is not `read_only=True` is the local registry where + downloads and builds land. A path in the list opens such a registry too. The client only reads a + later local registry. `LocalRegistry(path, read_only=True)` is never written, for example a + mirror on a network drive. A list with `InstalledConnectors()` needs a writable local registry. +- A remote registry reads only its own files. + +The order of the list is the read order. A release that two sources hold is read from the first +one. So to read offline, put the local registry first in the list. -hosted = TimeNet() -local = TimeNet(".timenet-registry") -s3 = TimeNet("s3://my-bucket/timenet") -private = TimeNet("https://registry.example.com") +Without a version, the first source that serves the dataset gives its newest version. An exact +version comes from the first source that provides it, while `@latest` is the highest version that +any source provides. Only "not found" and "not served" fall through to the next source, and any +other error stops the call. So if a registry in the list does not answer, a request for `@latest` +fails. + +A client whose local registry is `LocalRegistry(path, read_only=True)` never writes. A list of +remote registries alone has no local registry, so it never writes either: + +```python +import timenet + +local = timenet.TimeNet(sources=[timenet.LocalRegistry(".timenet-registry", read_only=True)]) +hosted = timenet.TimeNet(sources=["timenet://"]) +s3 = timenet.TimeNet(sources=["s3://my-bucket/timenet"]) ``` -`TimeNet()` selects a registry in this order: +`TimeNet()` uses the default list of `timenet.client.default_sources()`. If no variable is set, +the list is: -1. The `registry` argument. -2. The `TIMENET_REGISTRY` environment variable. -3. The hosted registry, `timenet://`. +```python +[ + LocalRegistry("~/.timenet/registry"), + "timenet://", + InstalledConnectors(), +] +``` -The [getting-started example](get-started.md) uses an explicit local path so its build and load use -the same registry. +`TIMENET_LOCAL_REGISTRY` moves the local registry, `TIMENET_REMOTE_REGISTRY` replaces `timenet://`, +and `TIMENET_AUTO_BUILD=false` removes `InstalledConnectors()` (see [Settings](#settings)). The +arguments `local_registry`, `remote_registry` and `auto_build` of `default_sources()` override +them, and `timenet.client.default_local_registry()` returns the local registry alone. -## Local state +## Settings -TimeNet stores local state under `~/.cache/timenet` by default. +These environment variables configure TimeNet. Each variable sets one thing. | Environment variable | Default | Purpose | | --- | --- | --- | -| `TIMENET_HOME` | `~/.cache/timenet` | Parent directory for local state. | -| `TIMENET_REGISTRY` | unset | Registry URL or local path. | -| `TIMENET_STORAGE` | `/storage` | Complete versions downloaded from a registry. | -| `TIMENET_CACHE` | `/cache` | Raw source files used during builds. | +| `TIMENET_LOCAL_REGISTRY` | `~/.timenet/registry` | The local registry, as a local path or a `file://` URI. It is the first source that a lookup checks, and the place where downloads and builds land. | +| `TIMENET_REMOTE_REGISTRY` | `timenet://` | A remote registry, as a URL or a local path: the second source that a lookup checks, if the local registry does not have the dataset. | +| `TIMENET_AUTO_BUILD` | `true` | Whether installed connectors build a dataset that neither the local nor the remote registry has. If `true`, the default list ends with `InstalledConnectors()`. | +| `TIMENET_CACHE` | `~/.cache/timenet` | Raw downloads of builds in `downloads/`, and control databases of remote reads in `control/`. | | `TIMENET_TOKEN` | unset | Bearer token for a remote registry. | | `TIMENET_ISOLATION` | `on` | Whether connector builds use isolated environments. | -An explicit argument takes precedence over an environment variable. An environment variable takes -precedence over its default. - -The model for these values is `timenet.config.TimeNetSettings`. +An explicit argument, for example a CLI flag, wins over its variable, and the variable wins over its +default. ## Browse datasets -The client provides these discovery methods: +`list()` and `search()` answer for the whole list of sources. They return one row per dataset, at +the highest version that a source lists, sorted by dataset ID. The card of each installed connector +counts as a row. So `list()` also shows a dataset that only a connector provides. | Method | Result | | --- | --- | -| `list()` | Metadata for the latest version of every dataset. | -| `get(dataset_id, version=None)` | One dataset manifest. | -| `search(...)` | Metadata that matches registry filters. | +| `list()` | Metadata for the highest version of every dataset across the sources. | +| `search(...)` | Metadata that matches registry filters, across the sources. | `search()` accepts filters for free text, domains, task classes, licenses, Signal specifications, dataset IDs, and tags. See [Registry search](registry.md#search) for their matching rules. +`search()` filters the rows of `list()`, so it checks the highest version of each dataset only. It +does not find an older version that matches. A +`task` or `time_series_spec` filter reads the manifest of a row from the registry that lists it. +It reads only for a row whose card matches the other filters, and only until `limit` rows match. A +connector card has no manifest, so it never matches these two filters. + +The calls ask the sources one after another, in order. A registry without a catalog, such as an +S3 registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls raise +`TimeNetNoCatalogError`, and its hint names `open('@')`, the read of an exact release. +If two sources list a dataset at the same version, the row of the earlier source stays. A listed +version can be newer than the version that `open()` reads without a version. To query one registry +alone, open it and ask it: + +```python +from timenet.registry import open_registry +from timenet.types import DatasetFilter + +hits = open_registry("timenet://").search(DatasetFilter(query=("ecg",))) +``` + +## Inspect a release + +These methods describe the release that `open()` reads. They resolve the release through the +sources as `open()` does (see [Choose the sources](#choose-the-sources)), and copy and build +nothing. + +| Method | Result | +| --- | --- | +| `get_manifest(dataset_id, version=None)` | The [manifest](manifest.md) of the release. | +| `get_card(dataset_id, version=None)` | The card of the release. | + +`get_manifest()` reads only the manifest, and checks no other file. A registry can list a +release without serving its files, for example a credentialed release on the hosted registry. Then +`get_manifest()` returns the listed manifest, which shows the access level and the access page of +each card. A release gets its manifest when it is built. So for a release that only +`InstalledConnectors()` provides, `get_manifest()` raises `TimeNetNotBuiltError`. + +`get_card()` returns the card of the release. The card comes from the manifest, or from the +installed connector for a release that is not built yet. `TimeNetNotBuiltError` holds the release +as `release` and the card of the connector as `card`, so a caller needs no second call: + +```python +import timenet +from timenet.errors import TimeNetNotBuiltError + +client = timenet.TimeNet() +try: + card = client.get_manifest("chengsenwang/tsqa").metadata +except TimeNetNotBuiltError as error: + card = error.card +print(card.name, card.license) +``` ## Open, load, or download @@ -120,49 +210,32 @@ with client.open("timenet/hello-world") as reader: values = record.signals[0].to_numpy() ``` -For a local registry, the reader opens the version in place. A local read needs complete copies of -the version and of its payload sources. A payload source is a release whose payload files the -version reads. For each of these releases, `open()` and `load()` use the first of these steps that -works: - -1. Reuse a complete local copy. -2. Copy the release from the registry that `fetch_from` names. -3. If `auto_build` is set, build the release from its installed connector. `auto_build` is set by - default. - -If no step works, the call raises an error that names the release and the keyword that can get it. -If the call builds the version, the build prepares the declared parents of the version with the -same steps in the same order. +`open()`, `load()`, and `open_torch()` read each release where it is, and copy nothing. They ask +the sources by the rules of [Choose the sources](#choose-the-sources). So without a version, the +call reads the newest version of the first source that serves the dataset, for example the local +registry. With `@latest`, if an installed connector declares a newer version than the local +registry holds, the call builds the newer version. ```python -with TimeNet(".timenet-registry").open( - "org/name", - fetch_from="timenet://", -) as reader: +import timenet + +client = timenet.TimeNet(sources=[ + timenet.LocalRegistry(".timenet-registry"), + "timenet://", +]) +with client.open("org/name") as reader: records = reader.read().records ``` -`fetch_from` takes a registry or its URI. `open_torch()` takes `auto_build` and `fetch_from` too. -Set `auto_build=False` to report a missing local release without starting a build. - -The call builds a payload source with its own connector. The build makes the exact release that -the version pins. The build publishes this release only if it writes the pinned files. See -[Rebuild a pinned release](build.md#rebuild-a-pinned-release). +This list has no `InstalledConnectors()`, so a missing release raises an error instead of starting +a build. -A local copy is damaged if its manifest is present but a file is missing or has another size. If -`fetch_from` is set, the call repairs a damaged copy from that registry. If `fetch_from` is not set, -the call raises `TimeNetFileError`. The error names the first missing or resized file. The call -never builds a release over a damaged copy. +If only `InstalledConnectors()` provides a release, the call builds it into the local registry +first and reads it there. The build copies or builds each missing parent from the same sources, as +[Prepare parent releases](build.md#prepare-parent-releases) describes. -Remote registries never run connector code. A remote client rejects `fetch_from`, because it reads -from its own registry. - -When `open()`, `load()`, or `open_torch()` runs, an HTTP or S3 registry resolves the manifests of -the payload sources. Reads then use complete copies in `TIMENET_STORAGE` where they exist. Otherwise -they read the remote files by range, as each read needs them. The first read of Records or tasks -downloads the catalog of the version and makes sure that it matches the manifest. A read never opens -the catalog of a parent release. A read also never fetches a parent release that is not a payload -source. See [Payload sources](registry.md#payload-sources). +When `open()`, `load()`, or `open_torch()` reads from an HTTP or S3 registry, it reads the remote +files by range. A child dataset gives the objects that it imports from a parent IDs of its own. To read such a copy by the ID that the parent gives it, pass the parent to `get()` with `parent=`: @@ -180,29 +253,76 @@ with client.open("physionet/ecg-qa-cot") as reader: parent as a `ParentAlias`, or its exact release as a `DatasetRef`. Without `parent=`, the lookup uses the ID that the child gives the object. -`download()` copies the complete release and its payload sources into local storage: +`download()` copies the complete release and its payload sources into the local registry, and +returns the release directory there: ```python path = client.download("chengsenwang/tsqa@1.0.0") ``` -Use `force=True` to replace an existing local copy. A later `open()` or `load()` reuses the complete -copies. A remote client still requests the manifests. To read without a network, open the storage -directory as a local registry with `TimeNet(storage_dir)`. Then pass `auto_build=False` to `open()` -or `load()`. A remote read keeps the byte ranges that it reads in memory only. The copies of -`download()` stay on disk. +The release resolves as in `open()`. The call copies the release and each payload source that the +local registry lacks or holds damaged. Each one comes from the first other registry that provides +it, so they can come from different registries. If the local registry holds a complete copy of the +release, the call copies only the payload sources that it lacks. + +A download never builds. For a release that only `InstalledConnectors()` provides, it raises +`TimeNetNotBuiltError`. + +A published version cannot change. So if the local registry holds the release with other files or +another card than the registry that the call copies from, the call raises +`TimeNetVersionConflictError`. + +`progress=` takes a function that receives the progress events of `timenet.progress`. For each +release that the call copies, the function receives `StepStarted`, then `BytesMoved` events as the +files arrive, then `StepFinished`: + +```python +from timenet.progress import BytesMoved + +def show(event): + if isinstance(event, BytesMoved): + print(f"{event.subject}: {event.done} of {event.total} bytes") + +client.download("chengsenwang/tsqa", progress=show) +``` + +The local registry comes first in the default list, so a later `open()` or `load()` of the same +version reads the copy in the local registry. A remote read keeps the byte ranges that it reads in +memory only, while the copies of `download()` stay on disk. + +## The local registry + +The local registry where downloads and builds land keeps a provenance record for each release that +a copy or a build puts into it. The record says where the release comes from: + +```json +{ + "kind": "download", + "source": "https://registry.timenet.ai", + "at": "2026-11-03T14:02:11Z" +} +``` + +`kind` is `download` for a release copied from a registry, and `build` for a release +that an installed connector built. `source` names the registry, or the connector package and its +version. To read it, call `store.read_origin(release)` on the `LocalRegistry`. It returns an +`Origin` from `timenet.registry.origin`, whose `kind` is an `OriginKind`. + +A record that is not valid raises `TimeNetOriginError`, which names its first problem. To replace +the release and its record, get the release again: call `download(release, force=True)` if it came +from a registry, or `timenet.build(release, force=True)` if it was built. ## Pin a version Append `@` to a dataset ID: ```python -client.get("chengsenwang/tsqa@1.0.0") +client.get_manifest("chengsenwang/tsqa@1.0.0") client.load("chengsenwang/tsqa@latest") ``` -The `get`, `download`, `open`, `load`, and `open_torch` methods also accept `version=`. Do not -supply a version in both places. +The `get_manifest`, `get_card`, `download`, `open`, `load`, and `open_torch` methods +also accept `version=`. Do not supply a version in both places. ## Open for PyTorch @@ -212,10 +332,10 @@ does not have to fit in memory. The view holds an open reader. Close the view wi use it in a `with` block. ```python -from timenet.client import TimeNet +import timenet from timenet.types import read_value, read_values -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) with client.open_torch("timenet/hello-world") as view: item = view[0] @@ -239,4 +359,4 @@ workflows. --- -See the [API reference for `timenet.client`](api/client.md) for every parameter and return type. +See the [API reference for `timenet.client`](api/timenet.client.md) for every parameter and return type. diff --git a/docs/concepts.md b/docs/concepts.md index f98d2658e..918c7a5f5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -19,7 +19,7 @@ This page collects the vocabulary that appears across these docs. Follow a link | **[Signal](data-model/time-series.md)** | One named value stream with its own TimeAxis and TimeSeriesSpec. Values remain lazy until read. | | **[Manifest](manifest.md)** | The `manifest.json` entry point for a version. It contains metadata, schema, counts, checksums, and file paths. Its presence marks a committed version. | | **[Connector](connectors.md)** | One `BaseConnector` per dataset. `download()` fetches the raw source. `convert()` builds a `TimeFDataset`. It knows nothing about the engine or registry. | -| **[Engine](build.md#pipeline)** | `run_pipeline`: drives any connector through the fixed `download -> convert -> derive_schema -> store` pipeline, and owns caching and idempotency. | +| **[Engine](build.md#pipeline)** | `run_pipeline`: drives any connector through the fixed `download -> convert -> derive_schema -> store_dataset` pipeline, and owns caching and idempotency. | | **[Build](build.md)** | Running a connector through the engine to compile a dataset and publish it with the [`timenet-build`](cli/build.md) CLI. | | **[Registry](registry.md)** | A served location of compiled TimeF versions that the [SDK](client.md) reads. It serves a DuckDB control plane plus a Parquet or Zarr values plane. It never runs connector code. It can be a [local directory, S3, or a remote host](registry.md#choosing-a-registry). | | **[Client (SDK)](client.md)** | `TimeNet`: the consumer entry point to search, download, and load datasets. | diff --git a/docs/connectors.md b/docs/connectors.md index 39490ba23..812a4a8d0 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -12,8 +12,10 @@ A connector is the unit of dataset integration. Each dataset has one connector c fetches raw data and converts it into a [`TimeFDataset`](timef-dataset.md). A connector has no knowledge of the registry, the engine, or other connectors. A child connector reads its parent releases only through the readers that the engine passes to it. The consumer SDK never imports -connector code. On a local registry, `TimeNet.load` builds a missing dataset through the -`timenet.builders` entry point. Pass `auto_build=False` to turn this off. +connector code. If the [sources](client.md#choose-the-sources) of a client include +`InstalledConnectors()`, `TimeNet.load` builds a missing dataset through the `timenet.builders` +entry point. To turn this off, leave `InstalledConnectors()` out of the list, or set +`TIMENET_AUTO_BUILD=false` for the default list. `BaseConnector` is the connector contract in the `timenet` package. Concrete connectors ship in `timenet-connectors`, beside their [dataset cards](manifest.md). @@ -33,7 +35,7 @@ class MyConnector(BaseConnector[MyRawRef]): ``` The two stages stay separate. This lets the engine drive -`download -> convert -> derive_schema -> store`. +`download -> convert -> derive_schema -> store_dataset`. | Method | Nature | Contract | | --- | --- | --- | @@ -224,13 +226,15 @@ a connector that you did not check. Credentials come from the environment. For the HuggingFace Hub, a token is read from `HF_TOKEN` automatically (needed only for gated or private sources). Downloaded source files cache under -`` (see [client local state](client.md#local-state)). +`/downloads/`, by default in `~/.cache/timenet` (see +[Settings](client.md#settings)). A credentialed dataset (a PhysioNet DUA-gated one, for example) declares `access: credentialed` and an `access_url` on its card. TimeNet never hosts such data, so it is build-your-own: get access at the `access_url`, set the provider credential in the environment (the way `HF_TOKEN` already -works), and run `timenet-build build ` yourself. An `open`, `load`, or `download` of a -credentialed dataset from a hosted registry raises `TimeNetAccessError` instead of serving bytes. +works), and run `timenet-build build ` yourself. If no other source of the client provides a +credentialed dataset, an `open`, `load`, or `download` from a hosted registry raises +`TimeNetAccessError` instead of serving bytes. The error names each `access_url` that the dataset needs, with the release that sets it. A child dataset can need the access pages of its ancestors. @@ -306,10 +310,10 @@ timenet-build build chengsenwang/tsqa # live, from the Hub timenet-build build chengsenwang/tsqa --keep-cache # keep the raw sources ``` -A successful build removes the dataset's raw download cache, at `/`. Only -`convert` reads the sources, and they are often several times the size of the dataset they produce. -To keep them, pass `--keep-cache`, or `keep_cache=True` to `timenet_connectors.build()`. This helps -while you write a connector for a large source, because each rebuild downloads the source again. +A successful build removes the raw downloads of the dataset, at +`/downloads/`. To keep them, pass `--keep-cache`, or `keep_cache=True` to +`timenet.build()`. This helps while you write a connector for a large source, because each rebuild +downloads the source again. Keeping `download` and `convert` apart makes a connector testable offline. `convert` takes raw references and does not touch the network. A test can therefore pass small local inputs or synthetic @@ -334,7 +338,7 @@ The connector of a child overrides `compose(raw_refs, context)`. The default `co `convert(raw_refs)`. If the card declares parents and the connector does not override `compose`, the build raises before it opens a parent or downloads anything. -Before the connector runs, the build prepares each parent release in the output registry, as +Before the connector runs, the build prepares each parent release in the local registry, as [Prepare parent releases](build.md#prepare-parent-releases) describes. The engine then opens each parent as a `TimeFReader` and passes the readers in a `BuildContext`, by parent alias. The connector copies objects into the child with `dataset.import_from()`: @@ -393,7 +397,7 @@ that split keeps them. A child with a split of its own removes them and adds its --- -See the [API reference for `timenet.connectors`](api/connectors.md) for the full symbol listing. +See the [API reference for `timenet.connectors`](api/timenet.connectors.md) for the full symbol listing. ## Reading source rows diff --git a/docs/data-model/datasets.md b/docs/data-model/datasets.md index f94044d81..6948c499c 100644 --- a/docs/data-model/datasets.md +++ b/docs/data-model/datasets.md @@ -27,14 +27,16 @@ The `dataset_version` changes to `1.0.1`, while `source_revision` stays `v2.1`. ## Versions are immutable TimeNet commits a version atomically when its `manifest.json` lands. The version never changes after -that. A pinned `@version` gives you exactly those bytes. With no suffix (or `@latest`), you get the -newest committed version. +that. A pinned `@version` gives you exactly those bytes. With no suffix, you get the newest version +of the first source of the client that serves the dataset. With `@latest`, you get the highest +version that a source of the client provides. ```python -from timenet.client import TimeNet +import timenet -client = TimeNet() -client.load("chengsenwang/tsqa") # latest committed version +client = timenet.TimeNet() +client.load("chengsenwang/tsqa") # the newest version of the first source +client.load("chengsenwang/tsqa@latest") # the highest version of all sources client.load("chengsenwang/tsqa@1.0.0") # a pinned, immutable snapshot ``` diff --git a/docs/data-model/index.md b/docs/data-model/index.md index 4f905f955..d059f9a69 100644 --- a/docs/data-model/index.md +++ b/docs/data-model/index.md @@ -34,7 +34,7 @@ flowchart LR D["download()
I/O · fetch raw refs"] C["convert()
CPU only"] S["derive_schema()
types from data"] - W["store()
TimeFWriter"] + W["store_dataset()
TimeFWriter"] R[("registry
TimeF version")] D --> C --> S --> W --> R ``` diff --git a/docs/get-started.md b/docs/get-started.md index 37e9baf57..eb5d419e9 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -44,16 +44,16 @@ Use `pipx install timenet-connectors` if you use pipx for command-line tools. ## Build the example dataset -Build the offline `timenet/hello-world` dataset into a local registry: +Build the offline `timenet/hello-world` dataset: ```bash -timenet-build build timenet/hello-world --out .timenet-registry +timenet-build build timenet/hello-world ``` -The command needs no network access. It writes this version: +The command needs no network access. It writes this version into a local registry: ```text -.timenet-registry/timenet/hello-world/1.0.0/ +~/.timenet/registry/timenet/hello-world/1.0.0/ ├── manifest.json ├── control.duckdb └── time_series/ @@ -61,15 +61,16 @@ The command needs no network access. It writes this version: ## Load the dataset -Pass the same registry path to the client: +The default client reads the local registry first. An exact version that the local registry holds +needs no network access: ```python import pandas as pd -from timenet.client import TimeNet +import timenet -client = TimeNet(".timenet-registry") -dataset = client.load("timenet/hello-world") +client = timenet.TimeNet() +dataset = client.load("timenet/hello-world@1.0.0") dataset.describe() signal = dataset.records[0].signals[0] diff --git a/docs/manifest.md b/docs/manifest.md index 3501cd15a..10eaf3ebc 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -79,6 +79,10 @@ Direct construction and parsing with an unsupported `timef_format_version` raise and `counts` default to empty. Unknown fields are rejected. A malformed block raises Pydantic's `ValidationError` with its native field locations and error details. +Registries read a stored manifest with `read_manifest(path)` or `parse_manifest(text, where)` from +`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A file that cannot +be read raises a plain `TimeFFormatError`. + ### Serialization notes - Known units serialize to their pint names (`"hertz"`, `"millivolt"`, `"dimensionless"`). An @@ -126,8 +130,11 @@ packed text and time-offset files that this release stores itself. A build needs the parent releases. A read needs only the payload sources. `TimeNet.open()`, `TimeNet.load()`, `TimeNet.open_torch()`, and `TimeFReader.open()` resolve the payload sources before the first read. A read never opens the catalog of a parent. `download()` copies the release -and its payload sources. See [Payload sources](registry.md#payload-sources). +and its payload sources into the local registry. See [Payload sources](registry.md#payload-sources). + +`required_releases(with_parents=False)` lists the releases of the payload sources, which a read +needs. `required_releases(with_parents=True)` also lists the parents, which a build needs. --- -See the [API reference for `timenet.manifest`](api/manifest.md) for the full symbol listing. +See the [API reference for `timenet.manifest`](api/timenet.manifest.md) for the full symbol listing. diff --git a/docs/registry.md b/docs/registry.md index c2146ebce..05b74a9bc 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -35,13 +35,15 @@ registry = open_registry("./local_registry") | `http(s)://` | `RemoteRegistry` | | `timenet://` | `RemoteRegistry`, an alias for the hosted `https://registry.timenet.ai` | -An unrecognized scheme, for example `gs://` or `az://`, raises `ValueError` instead of becoming a -local path. `open_registry` expands `~` in a local path or a `file://` URI. +`open_registry()` returns a `WritableRegistry`, so you can also publish to the registry that it +opens (see [Writing to a registry](#writing-to-a-registry)). A remote registry reads only its own +files. To read a local copy first, put its `LocalRegistry` before the remote registry in the +sources. !!! note "Catalog support" - Every backend serves reads and publishes with `store` (remote publishing needs a writer token; the - S3 backend reads AWS credentials from the environment). The S3 backend has no catalog, so `list` and - `search` are unsupported there. + Every backend serves reads and takes new releases. Remote publishing needs a writer token, and + the S3 backend reads AWS credentials from the environment. The S3 backend has no catalog, so + `list` and `search` do not work there. ### Remote registries @@ -68,32 +70,42 @@ hardcoded. Install the extra: uv add 'timenet[s3]' ``` -The layout under the prefix mirrors the local one, `org/name/version/…` with `manifest.json` beside the -data files. `store` uploads to a temporary `version.tmp-` prefix and moves each object into place -(`manifest.json` last), so a failed publish never leaves a partial version. `open` reads lazily and -`load` reads every value. Both reuse complete local copies where they exist. Otherwise they read -remote payload ranges through a pyarrow `S3FileSystem`. The backend has no catalog, so `list` and -`search` are unsupported: publish and fetch by an explicit `org/name@version`. +The layout under the prefix is `datasets/org/name/version/…`, with `manifest.json` beside the data +files. `put_dataset` uploads to a temporary `version.tmp-` prefix and moves each object into +place (`manifest.json` last), so that failed publishing does not leak partial versions. +`TimeNet.open()` reads lazily and `load()` reads every value. Both read remote payload ranges +through a pyarrow `S3FileSystem` (see [Remote reads](#remote-reads)). The backend has no catalog, so +`list` and `search` are unsupported: publish and fetch by an explicit `org/name@version`. ## `BaseRegistry` -Every backend implements this contract: four data-access methods and two shared methods, -`download_version` and `search`. +Every backend implements this contract. | Method | Description | | --- | --- | | `list_datasets()` | Latest-version `DatasetMetadata` for every dataset, sorted by id. | -| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version, or `TimeFFormatError` if the stored manifest's own id disagrees with the directory it was loaded from. | +| `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeNetManifestError` if the stored manifest is not valid, or `TimeFFormatError` if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | -| `local_root` | The local registry or cache directory for complete releases. | -| `open_version(dataset_id, version=None)` | A `DatasetVersion`: the parsed manifest plus a filesystem-rooted, picklable handle to the version's files, with its [payload sources](#payload-sources) opened from this registry. This is the storage seam the [reader](timef-reader.md) reads through, so a read never re-opens the registry nor re-parses `manifest.json`. | -| `download_version(dataset_id, version, dest_dir, *, force=False, manifest=None, progress_cb=None)` | Copy a version's files into a local directory (shared implementation). It stages the files under `.tmp-*` with `manifest.json` last, then swaps the directory in with one rename. A subclass can override it: `RemoteRegistry` streams the files in parallel. | -| `search(...)` | Filter datasets (shared implementation). | +| `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Each backend must define it. | +| `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | +| `find_manifest(release)` | The manifest of an exact release, or `None` if the registry does not hold it. Raises `TimeFFormatError` if the stored manifest names another release. | +| `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | +| `open_release(reference, *, manifest=None)` | A `DatasetVersion` of one exact release without its payload sources: the parsed manifest and a handle to the files. Pass `manifest` if you already read it from this registry. A local registry raises `TimeNetFileError` for a missing or resized file. | +| `resolve(dataset_id, version)` | Whether the registry serves a release, as a [source](#sources) answer: `Found`, `NotFound` or `NotServed`. | +| `search(filter)` | Find the datasets that match a `DatasetFilter`. | + +To copy a release with its payload sources, use +[`TimeNet.download()`](client.md#open-load-or-download) for the local registry where downloads and +builds land, or [`copy_release()`](#copying-a-release) for another registry. To read a release +with its payload sources, use [`TimeNet.open()`](client.md#open-load-or-download). `LocalRegistry` serves a `///` tree. `RemoteRegistry` serves the same layout over the versioned REST contract (`GET /api/v1/datasets`, `/api/v1/datasets/{id}/{version}/manifest`, `.../download/{relpath}`); `S3Registry` serves the same layout under an `s3://bucket/prefix` root. The -S3 backend has no catalog, so `list_datasets` and `search` raise `NotImplementedError`. +S3 backend has no catalog: its `has_catalog` is `False`, and `list_datasets` and `search` raise +`NotImplementedError`. `TimeNet.list()` and `TimeNet.search()` leave a registry without a catalog +out, with a `CatalogWarning` that names it. If no source of the list has a catalog, they raise +`TimeNetNoCatalogError`. A dataset id is an `org/name` pair, for example `chengsenwang/tsqa`. The id nests one level deep on disk, at `/chengsenwang/tsqa//`. `list_datasets` finds these ids at any depth. Use @@ -103,95 +115,143 @@ lowercase ids to avoid casing clashes on case-insensitive filesystems. A version can reuse payload files that other versions store: its payload sources. Its manifest names each one and pins the files it reads from it (see -[Shared payloads](timef-format.md#shared-payloads)). A registry serves a version together with its -payload sources: - -- `open_version` opens each payload source from the same registry and checks it against the pins, - reading only manifests and, for local files, file sizes. A remote registry prefers complete - cached copies and otherwise reads remotely. See [Opening a version](timef-reader.md#opening-a-version) - for the checks and their errors. -- `download_version` downloads complete copies of the version and of each payload source, and checks - each source against the pins. If a downloaded file has another size or checksum than its manifest - records, the download raises `TimeNetFileError` and keeps no partial copy. -- `store` refuses a version whose payload sources the registry does not hold, holds in another form, - or that read payloads of the version itself. The [writer](timef-writer.md) checks this at the - first reuse of each source, and raises the errors that opening raises for a missing or differing - source. - -A version lists every payload source it reads from, so a registry resolves only those, never the -sources' own sources. - -A client of a local registry prepares the version and its payload sources before a read. For each -of these releases, the client reuses a complete copy, or copies the release from `fetch_from`. If -`auto_build` is set, the client can also build the release. If the client cannot get a release, -the read raises an error. The client builds a payload source with its own connector, at the exact -version of the source. - -If `fetch_from` is set, the client repairs a damaged copy from that registry. Without `fetch_from`, -the read raises `TimeNetFileError` for a damaged copy. A remote client never runs connector code. -It rejects `fetch_from`. See [Open, load, or download](client.md#open-load-or-download). - -### Remote reads of payload sources - -The `open_version()` method of an HTTP or S3 registry resolves the manifests of the payload sources -before it returns. Reads then use complete cached copies where they exist. Otherwise they read byte -ranges of the remote files, as each read needs them. The first read of Records or tasks downloads -the catalog of the version and makes sure that it matches the manifest. A read never opens the -catalog of a parent release. +[Shared payloads](timef-format.md#shared-payloads)). The version needs its payload sources: + +- A read through `TimeNet.open()` resolves each payload source through the list of sources, and + checks it against the pins. It reads only manifests and, for local files, file sizes. See + [Opening a version](timef-reader.md#opening-a-version) for the checks and their errors. +- `TimeNet.download()` and `copy_release()` copy complete copies of the version and of each payload + source, and check each source against the pins. If a downloaded file has another size or checksum + than its manifest records, the copy raises `TimeNetFileError` and keeps no partial copy. +- `put_dataset` refuses a version if the registry does not hold one of its payload sources. + +### Remote reads HTTP reads use the range reader and the read-ahead cache of fsspec. The object server must support byte ranges. The client requests a new signed URL after one expires. The registry token goes only with API requests. -A read of byte ranges cannot make sure that a whole file matches its checksum. `download()` and -`download_version()` copy complete files, and make sure that each file has its recorded size and -checksum. `TimeFReader.verify()` reads and hashes every file that a release reads. +A read of byte ranges cannot make sure that a whole file matches its checksum. `TimeNet.download()` +copies complete files, and makes sure that each file has its recorded size and checksum. +`TimeFReader.verify()` reads and hashes every file that a release reads. + +## Sources + +A source is something that can provide a release. `timenet.sources` defines the type +`ReleaseSource`. Each source has the method `resolve(dataset_id, version)`. `version` is a `Version` +object for an exact version, or `None` for the newest version of that source. Two kinds of source +exist: + +- A registry. Every `BaseRegistry` is a source. +- `InstalledConnectors()`: every installed package that registers a builder backend under the + `timenet.builders` entry point group. + +A source gives one of these answers: + +| Answer | Meaning | +| --- | --- | +| `FoundInRegistry(version, registry)` | The registry serves the release. | +| `FoundInConnector(builder)` | An installed connector builds the release. `version` is the version that its card declares. | +| `NotFound()` | The source does not know the release. | +| `NotServed(version, access, requirements)` | The source lists the release, but does not serve its files. `access` is the effective access of the release, and `requirements` names each card that is not open, with its access page. `access_error(dataset_id)` returns the `TimeNetAccessError` that names each page. | + +The type `Found` covers `FoundInRegistry` and `FoundInConnector`. `FoundInRegistry` and +`NotServed` also carry the `manifest` that the source read, or `None`, so that you do not need to +fetch it again. + +A `LocalRegistry` serves every release that it holds, so it never answers `NotServed`. A registry +that is not local serves the files of a release only if its effective access is `open`, and +answers `NotServed` otherwise. + +`InstalledConnectors()` answers `Found` only for the version that the card of a connector declares, +because a connector builds only that version. For any other request, it answers `NotFound`. + +`resolve_release(sources, dataset_id, version)` asks an ordered list of sources and follows four +rules: + +1. Without a version, the first source that serves the dataset gives its newest version. A source + that lists the dataset without serving it falls through to the next source. +2. An exact version comes from the first source that provides it. +3. `"latest"` is the highest version across all sources, the connectors included. The order then + decides only which source provides that version. +4. Only `NotFound` and `NotServed` fall through to the next source. Any other error of a source + stops the resolution. + +So the order of the list is the read order: a release that two sources hold comes from the first +one. + +```python +from timenet.registry import LocalRegistry, open_registry +from timenet.sources import InstalledConnectors, resolve_release + +sources = [ + LocalRegistry("~/.timenet/registry"), + open_registry("timenet://"), + InstalledConnectors(), +] +answer = resolve_release(sources, "timenet/hello-world", None) +``` + +A build uses such a list to find the parent releases that its local registry lacks. Each missing +parent comes from the first source that provides its exact version. + +`TimeNet` takes such a list as `sources`. The first `LocalRegistry` of the list that is not +`read_only=True` gets the downloads and builds, and a list with `InstalledConnectors()` needs one. +See [Choose the sources](client.md#choose-the-sources). ## Writing to a registry -A `WritableRegistry` adds one write method to the read contract. As a result, -[build](build.md) can publish into any backend, not only a local directory: +A `WritableRegistry` adds two write methods to the read contract. Every backend implements them, +so you can publish to a local directory, a remote registry, or an S3 bucket: | Method | Description | | --- | --- | -| `store(dataset, *, force=False, values_backend="parquet", progress_cb=None)` | Compile a dataset and publish it; returns the stored version. Derives the schema first if absent, skips an already-committed version unless `force`, and checks every payload source in this registry. `values_backend` picks the values plane format: `"parquet"` or `"zarr"`. | +| `put_dataset(dataset, *, force=False, values_backend="parquet", progress_cb=None)` | Compile a dataset and publish it. Returns the stored release as a `DatasetRef`. Derives the schema first if absent, skips an already-committed version unless `force`, and checks every payload source in this registry. `values_backend` picks the values plane format: `"parquet"` or `"zarr"`. On a remote or S3 registry, `force=True` raises `TimeFValidationError` and uploads nothing, because a published version never changes. | +| `put_release(from_directory)` | Publish a complete local release directory as it is; returns the published release as a `DatasetRef`. Only reads `from_directory` and never changes it. Makes sure that the directory holds a readable `manifest.json` and every file that the manifest declares with its recorded size, and raises `TimeFFormatError` or `TimeNetFileError` otherwise. Publishes only those files and `manifest.json`. | | `exists(dataset_id, version)` | Whether a committed version already exists (shared implementation). | -`LocalRegistry` implements `store` by streaming the dataset through a [`TimeFWriter`](timef-writer.md), -which stages under `.tmp-*` and publishes with a single atomic rename. With `force`, the old -version stays in place until the new one is complete. A replace is not isolated from readers that -already opened the old version, or from versions that reuse its payloads, which then fail to open. -`RemoteRegistry` runs the [publish flow](#remote-publishing) below. `S3Registry` uploads to a temporary -prefix and moves each object into place (`manifest.json` last). Each backend passes itself to the -writer, which checks every payload source in that registry while it stages the release locally. -`open_writable_registry(uri)` resolves a URI like `open_registry` but returns a `WritableRegistry`. +`LocalRegistry`, `RemoteRegistry` and `S3Registry` take `read_only=True`. A read-only registry +refuses each write with `TimeNetRegistryError`. + +`put_dataset` writes the dataset with a [`TimeFWriter`](timef-writer.md) and checks each payload +source of the new release in the registry that receives it. In a local registry, `force=True` +replaces a committed version only once the new one is complete, so a failed write keeps the old +one. A replace can break readers that already opened the old version, and versions that reuse its +payloads then fail to open. +`RemoteRegistry` publishes through the [publish flow](#remote-publishing) below. + +`put_release` publishes a release directory that already exists, without a new compile. The +directory must hold `manifest.json` and every file that it lists, with the recorded size. Only those +files reach the registry. + +`open_registry(uri)` returns a `WritableRegistry`, so you can publish to every registry that it +opens: ```python -from timenet.registry import open_writable_registry +from timenet.registry import open_registry -registry = open_writable_registry("./local_registry") -version = registry.store(dataset) # schema derived if needed, atomic commit +registry = open_registry("./local_registry") +# schema derived if needed, atomic commit +release = registry.put_dataset(dataset) ``` Every backend is a `WritableRegistry`. As a result, this type alone does not show if a backend is a directory the engine can write to or a remote stub. `local_registry_path(uri)` gives this information. It returns the directory that a `file://` URI or a plain path names. It raises -`TimeNetRegistryError` for a remote scheme. `default_registry_path()` uses `local_registry_path` to find -the default local registry. It uses `$TIMENET_REGISTRY` when its value is a local path. Otherwise, -it uses `/registry`. This is how [`timenet-build build`](cli/build.md) resolves its output -when `--out` is absent. The `timenet_connectors.builder` and `load` helpers use it too. +`TimeNetRegistryError` for a remote scheme or an empty string. ### Remote publishing -`RemoteRegistry.store` compiles the dataset locally, then hands the artifacts to the service over the -REST contract. Dataset bytes never pass through the API: the service issues a presigned PUT per file and -the SDK uploads straight to the object store. The token must be a writer token (`$TIMENET_TOKEN`); an -anonymous or read-only caller is rejected by the service. +`RemoteRegistry.put_dataset` compiles the dataset locally and then sends the files to the service. +The data does not pass through the API. The service gives a presigned URL for each file, and the SDK +uploads the file to the object store directly. You need a writer token in `$TIMENET_TOKEN`. The +service refuses an anonymous or read-only caller. `RemoteRegistry.put_release` runs steps 2 to 4 +for a release directory that already exists. 1. Compile into a temporary staging directory with a [`TimeFWriter`](timef-writer.md), deriving the - schema first if the dataset has none. An already-committed version returns early unless `force`. - The writer checks every payload source against the service, so a version whose sources the - service does not hold is refused before anything is published. + schema first if the dataset has none. An already-committed version returns early. The writer + checks every payload source against the service, so a version whose sources the service does not + hold is refused before anything is published. 2. `POST /datasets/{id}/{version}/publish` with the compiled `manifest.json` as the body. The service registers the pending version and returns the list of files it expects. 3. For each file, `POST /datasets/{id}/{version}/publish/upload-url` with `{"path": relpath}` to get a @@ -201,25 +261,50 @@ anonymous or read-only caller is rejected by the service. whether or not the publish succeeds. ```python -from timenet.registry import open_writable_registry +from timenet.registry import open_registry -registry = open_writable_registry("timenet://") # needs $TIMENET_TOKEN -version = registry.store(dataset) # compile, upload, finalize +registry = open_registry("timenet://") # needs $TIMENET_TOKEN +release = registry.put_dataset(dataset) # compile, upload, finalize ``` +### Copying a release + +`timenet.registry.copy_release(reference, source, dest, *, with_payload_sources=True, force=False, +progress=None)` copies one release from `source` to the writable registry `dest`. + +A dataset can depend on payload files stored as part of another dataset (parent/child relationship). +`copy_release` copies each payload source that the manifest of the release lists too, so that +`dest` can open the copied release. With `with_payload_sources=False`, it copies the release alone. +It copies only the releases that `dest` does not hold yet. + +`copy_release` checks every release before it copies anything, so a refused copy leaves `dest` +unchanged. + ## `search` ```python -registry.search( - query=None, domain=None, task=None, license=None, - time_series_spec=None, dataset_id=None, tag=None, limit=100, -) +from timenet.types import DatasetFilter, Domain + +registry.search(DatasetFilter.of(domain=Domain.CARDIOLOGY, tag="ecg", limit=10)) ``` -Every filter takes a scalar value or a list. The registry ignores `None` filters and combines all -non-`None` filters with AND logic. The consumer CLI, `timenet search`, mirrors this behavior -exactly. `limit` caps the number of results, with a default of 100. `limit=0` returns no results. -A negative `limit` raises `ValueError`. +A `DatasetFilter` from `timenet.types` holds the filters and the limit. Each filter is a tuple, and +an empty tuple sets no restriction. `DatasetFilter.of()` takes one value, a list, or `None` for +each filter. The registry combines the filters with AND logic. `limit` caps the number of +results, with a default of 100. `limit=0` returns no results, and a negative limit raises +`TimeFValidationError`. + +`DatasetFilter.select()` does the filtering. It takes `ListedRelease` rows, each a card and the +registry that lists it, and keeps their order. It checks the card first. A `task` or +`time_series_spec` filter then reads the manifest of a matching card from its registry. A row +without a registry never matches these two filters. `select()` stops at `limit` matches, so it +reads no more manifests than it needs. + +A registry sorts its catalog by dataset ID and gives it to `select()`, so `search()` returns the +first matches by dataset ID. `TimeNet.search()` and the consumer CLI, `timenet search`, take the +same filters as keywords. They give `select()` the merged rows of all sources (see +[Browse datasets](client.md#browse-datasets)). So they check only the highest version of each +dataset, and they do not find an older version that matches. | Filter | Matches | | --- | --- | @@ -237,4 +322,4 @@ manifest. The manifest always carries the derived schema, so the registry does n --- -See the [API reference for `timenet.registry`](api/registry.md) for the full symbol listing. +See the [API reference for `timenet.registry`](api/timenet.registry.md) for the full symbol listing. diff --git a/docs/timef-dataset.md b/docs/timef-dataset.md index e06bfea83..50493bc82 100644 --- a/docs/timef-dataset.md +++ b/docs/timef-dataset.md @@ -265,7 +265,7 @@ The stored version contains `manifest.json`, an immutable `control.duckdb`, and values plane. See [TimeF format](timef-format.md) and [TimeFWriter](timef-writer.md) for the physical layout. -See the [API reference for `timenet.dataset`](api/dataset.md) for the full symbol listing. +See the [API reference for `timenet.dataset`](api/timenet.dataset.md) for the full symbol listing. ## Importing from a parent @@ -305,6 +305,9 @@ still say `open`. The effective access level of a dataset is the strictest level of the cards in its `lineage`. A user must have this level of access to use the data of the dataset. `timenet info` shows the effective access level and the page of each card that requires access. +Before a build, a release has no lineage yet. So for a release that only a connector provides, +`timenet info` shows the level of its own card. If the card declares parents, the access row says +so, for example `open (this card only: the access of its 2 parents is known after a build)`. `effective_access()` and `access_requirements()` in `timenet.types.access` return the same information. diff --git a/docs/timef-format.md b/docs/timef-format.md index 36ef02c65..c3b939d79 100644 --- a/docs/timef-format.md +++ b/docs/timef-format.md @@ -270,11 +270,11 @@ constraints. The writer then checkpoints and closes the database, records final SQL scans, for tests and tooling. `TimeFReader.verify()` hashes every file that the release reads. A download makes sure that each -file has its recorded size and checksum before it adds the complete copy to the cache. Opening a -version makes sure that its local files exist with their recorded sizes, and hashes nothing. Reads -then use the local files, or byte ranges of the remote files. Normal reads stay lazy. Before a -reader opens a cached remote `control.duckdb` read-only, it makes sure that the file matches the -manifest. +file has its recorded size and checksum before it adds the complete copy to the local registry. +Opening a version makes sure that its local files exist with their recorded sizes, and hashes +nothing. Reads then use the local files, or byte ranges of the remote files. Normal reads stay lazy. +Before a reader opens a cached remote `control.duckdb` read-only, it makes sure that the file +matches the manifest. ## Catalogs of child datasets diff --git a/docs/timef-reader.md b/docs/timef-reader.md index 3cd120d0d..ea49c3c22 100644 --- a/docs/timef-reader.md +++ b/docs/timef-reader.md @@ -16,11 +16,11 @@ There are two everyday ways to get a reader. `TimeFReader.open()` opens a version directory: ```python -from timenet.client import TimeNet +import timenet from timenet.reader import TimeFReader # By dataset ID, through the client. -with TimeNet().open("timenet/hello-world") as reader: +with timenet.TimeNet().open("timenet/hello-world") as reader: dataset = reader.read() values = dataset.records[0].signals[0].to_arrow() @@ -159,11 +159,11 @@ source gave a recording, read the objects with `iter_records()` or `read_tasks() `metadata`. ```python -from timenet.client import TimeNet +import timenet from timenet.dataset import Signal from timenet.types import AnswerTask, read_value -with TimeNet().open("timenet/hello-world") as reader: +with timenet.TimeNet().open("timenet/hello-world") as reader: task = reader.get(AnswerTask, task_id) # Reads this one text and keeps nothing; None without a prompt. prompt = read_value(task.prompt) @@ -298,27 +298,32 @@ and decoded Zarr chunks until it closes. ## Opening a version -To read a version without the client, pass a [`DatasetVersion`](registry.md) to -`TimeFReader(version)`. `open_local_version()` and the `open_version()` method of a registry -return one. A `DatasetVersion` holds the parsed manifest and the local or remote file systems of -the version and its payload sources. A reader that you open this way skips three parts of the -client: the download cache, the access check, and `auto_build`. The two examples open a version -from a registry and from a directory: +To read a version directory without the client, pass a [`DatasetVersion`](registry.md) to +`TimeFReader(version)`. `open_local_version()` returns one. A `DatasetVersion` holds the parsed +manifest and the file systems of the version and its payload sources. A reader that you open this +way reads only from that directory and the registry root around it, without other sources or +builds: ```python from timenet.reader import TimeFReader -from timenet.registry import open_local_version, open_registry +from timenet.registry import open_local_version -registry = open_registry("~/timenet/registry") -with TimeFReader(registry.open_version("timenet/hello-world")) as reader: +with TimeFReader(open_local_version(version_dir)) as reader: dataset = reader.read() +``` -with TimeFReader(open_local_version(version_dir)) as reader: +To read from one registry, give the client a list with only that registry: + +```python +import timenet + +client = timenet.TimeNet(sources=["~/timenet/registry"]) +with client.open("timenet/hello-world") as reader: dataset = reader.read() ``` -`open_local_version()` and the `open_version()` method of a registry open a version at once, but -they read only metadata. Opening makes sure that: +`open_local_version()` opens a version at once, but it reads only metadata. Opening makes sure +that: - `manifest.json` is valid and has a supported format version. - Each file that the manifest lists, `control.duckdb` included, is present with its recorded size. @@ -340,7 +345,7 @@ releases that source reuses. | A payload source differs from its pins | `TimeFFormatError` | The source and the first differing file | | A file is missing or has another size | `TimeNetFileError` | The file, its expected size, and its actual size | -An invalid or unsupported manifest raises a validation error, and a missing one raises +An invalid or unsupported manifest raises `TimeNetManifestError`, and a missing one raises `FileNotFoundError`. ### Where payload sources are found @@ -348,8 +353,8 @@ An invalid or unsupported manifest raises a validation error, and a missing one Payload sources resolve relative to a registry root. By default, `open_local_version()` takes the root that holds the version as `///`, so a version and its sources keep working when they move to another root together. Pass `sources_root=` to find them under another -root. `TimeFReader.open()` takes `sources_root=` too. A registry's `open_version()` finds them in -that registry. +root. `TimeFReader.open()` takes `sources_root=` too. The client finds them through its list of +sources. A version without payload sources opens from any directory. A version with payload sources that is not stored as `///` needs `sources_root=`, and opening it without one @@ -397,4 +402,4 @@ Relationships return as object references: task inputs are Records, spans hold S Sources, axes, and annotations have no id. The reader does not keep the Python object identity from the writer process. -See the [API reference for `timenet.reader`](api/reader.md) for all members. +See the [API reference for `timenet.reader`](api/timenet.reader.md) for all members. diff --git a/docs/timef-writer.md b/docs/timef-writer.md index 2deaa9a21..d7547c912 100644 --- a/docs/timef-writer.md +++ b/docs/timef-writer.md @@ -79,8 +79,7 @@ therefore needs a different dataset ID or version than its sources. Its readers find the sources in the registry the release is published to, so the writer checks each source there at its first reuse, during `write()`. Pass that registry as `registry=`. It defaults to a -local registry over `root`, and `S3Registry.store()` and `RemoteRegistry.store()` pass themselves while -they stage the release locally. The writer fetches the source's manifest from the registry once and +local registry over `root`. The writer fetches the manifest of the source from the registry once and compares the record of each reused file. It refuses the write, before it writes the remaining values and with nothing published, when: diff --git a/docs/types.md b/docs/types.md index 2fb87b1bf..63ec36ebd 100644 --- a/docs/types.md +++ b/docs/types.md @@ -293,20 +293,31 @@ TimeNet raises its own exceptions from `timenet.errors`. | --- | --- | | `TimeNetError` | Base class for TimeNet failures. | | `TimeNetRegistryError` | A registry cannot complete an operation. | +| `TimeNetUnreachableError` | A request to a registry or its storage failed on the network. The original error is its cause. A `TimeNetRegistryError`. | +| `TimeNetNoCatalogError` | No source of a list or a search has a catalog. The hint names the read of an exact release. A `TimeNetRegistryError`. | | `TimeNetDatasetNotFoundError` | A dataset ID or version does not exist. | +| `TimeNetVersionUnavailableError` | The installed connector of a dataset builds another version than the requested one. A `TimeNetDatasetNotFoundError` and a `TimeFValidationError`. | +| `TimeNetVersionConflictError` | A registry already holds a release with other files or another card than a copy that would go there. `difference` is `ReleaseDifference.FILES` or `ReleaseDifference.CARD`, from `timenet.errors`, and `registry_location` is the location of that registry. For a publish, the hint asks for a new version. For `timenet-build build --publish-to` before the build, the hint names `--force`. For a copy into the local registry, the hint names the release folder to delete. | | `TimeFValidationError` | Input violates a TimeF invariant. | | `TimeFNullValuesError` | Values contain nulls that NumPy cannot hold. A `TimeFValidationError`. | | `TimeFNotFoundError` | An ID, parent alias, or parent release names nothing in the dataset, or the stored object has another kind. A `TimeFValidationError` and a `LookupError`. | | `TimeFFormatError` | A stored TimeF artifact is corrupt or unsupported. | | `TimeNetFileError` | A file of a release is missing, or does not match its manifest record. A `TimeFFormatError`. | +| `TimeNetManifestError` | A stored manifest is not a valid manifest. `where` names the file or the place in a registry, and `detail` names the first problem. `path` is the file, or `None` for a manifest that a registry served. A `TimeFFormatError`. | +| `TimeNetOriginError` | The provenance record of a release in the local registry is not valid. `release` names the release, `path` the file, and `detail` the first problem. A `TimeFFormatError`. | | `TimeFReaderError` | A reader could not complete a read, e.g. its connection was closed while the read ran. | | `TimeNetInvalidCardError` | `dataset.yaml` cannot be read or is not a mapping. | | `TimeNetConnectorError` | A connector found that an assumption about its source data, a download, or a parent does not hold. `TimeNetDownloadError` is a `TimeNetConnectorError`. | +| `TimeNetBuildError` | A build failed: the connector build, or the environment that it needs. | +| `TimeNetIsolationError` | An isolated build cannot make its environment, for example because `uv` is missing. A `TimeNetBuildError`. | + +Most TimeNet errors come with a `hint`: what you can do about the error. Pydantic `ValidationError` reports field and schema errors in a dataset card. The error includes the native field location and details. -Manifest parsing uses the same error for invalid JSON and invalid fields. +`Manifest.model_validate_json()` uses the same error for invalid JSON and invalid fields. A registry +that reads a stored manifest raises `TimeNetManifestError` instead. `TimeFValidationError` also subclasses `ValueError`. Existing input-validation handlers can catch it without losing the TimeNet-specific error type. @@ -314,4 +325,11 @@ without losing the TimeNet-specific error type. `SpanOutsideWindowWarning` reports a span outside a Signal window when the caller has chosen warning behavior. It derives from `TimeNetWarning` and `UserWarning`. -See the [API reference for `timenet.types`](api/types.md) for all symbols. +`ConnectorCardWarning` reports a connector card that cannot be read. The list of installed +connectors leaves its dataset out, so the card fails only its own dataset. + +`CatalogWarning` reports a source without a catalog, which `TimeNet.list()` and +`TimeNet.search()` leave out. Both warnings point at the first line of the call stack outside the +`timenet` package, which is usually your own call. + +See the [API reference for `timenet.types`](api/timenet.types.md) for all symbols. diff --git a/docs/usage.md b/docs/usage.md index bcc04449d..c5001face 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -14,9 +14,10 @@ Every TimeF dataset loads through the same client. Each Record contains a Source Source contains Signals. ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet(".timenet-registry").load("timenet/hello-world") +client = timenet.TimeNet(sources=[".timenet-registry"]) +dataset = client.load("timenet/hello-world") record = dataset.records[0] signal = record.signals[0] ``` @@ -25,7 +26,7 @@ signal = record.signals[0] instead. The reader's reads leave each Signal's values in storage until an array method reads them: ```python -with TimeNet(".timenet-registry").open("timenet/hello-world") as reader: +with client.open("timenet/hello-world") as reader: dataset = reader.read() values = dataset.records[0].signals[0].to_numpy() ``` @@ -96,9 +97,9 @@ Install the adapter with `uv add 'timenet[torch]'`. Then open a map-style PyTorc view holds an open reader, so use it in a `with` block or call its `close()`: ```python -from timenet.client import TimeNet +import timenet -client = TimeNet(".timenet-registry") +client = timenet.TimeNet(sources=[".timenet-registry"]) with client.open_torch("timenet/hello-world") as view: item = view[0] @@ -126,12 +127,11 @@ from pathlib import Path from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split -from timenet.client import TimeNet -import timenet_connectors +import timenet registry = Path(".timenet-registry") -timenet_connectors.build("timenet/test-mean", out=registry) -dataset = TimeNet(registry).load("timenet/test-mean") +timenet.build("timenet/test-mean", local_registry=registry) +dataset = timenet.TimeNet(sources=[registry]).load("timenet/test-mean") x, y = dataset.to_features_and_targets(output="numpy") x_train, x_test, y_train, y_test = train_test_split( @@ -156,10 +156,10 @@ The runnable version is ## Work with downloaded files -`download()` returns a local TimeF version directory: +`download()` copies a release into the local registry and returns its TimeF version directory there: ```python -path = TimeNet().download("org/name@1.0.0") +path = timenet.TimeNet().download("org/name@1.0.0") ``` The directory contains `manifest.json`, `control.duckdb`, and a Parquet or Zarr values plane. Use the diff --git a/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md b/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md index f86e0f7e1..edfb7bed6 100644 --- a/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md +++ b/packages/timenet-connectors/src/timenet_connectors/datasets/tsaqa/tsaqa_benchmark/README.md @@ -436,14 +436,15 @@ matches its row: The build downloads the three files, 387 MB, from the Hub at the pinned commit: ```bash -uv run timenet-build build tsaqa/tsaqa-benchmark --out /tmp/timenet +uv run timenet-build build tsaqa/tsaqa-benchmark --local-registry /tmp/timenet ``` On 2026-10-29, building in-process took 79 s and peaked at 4.1 GB of memory, most of it the 510,093 Records. The built dataset takes 123 MB. To load it: ```python -from timenet.client import TimeNet +import timenet -dataset = TimeNet("/tmp/timenet").load("tsaqa/tsaqa-benchmark") +client = timenet.TimeNet(sources=["/tmp/timenet"]) +dataset = client.load("tsaqa/tsaqa-benchmark") ``` diff --git a/scripts/gen_api_docs.py b/scripts/gen_api_docs.py index ae2f66a01..8fa586505 100644 --- a/scripts/gen_api_docs.py +++ b/scripts/gen_api_docs.py @@ -19,8 +19,7 @@ from __future__ import annotations import ast -from collections import Counter -from dataclasses import dataclass, replace +from dataclasses import dataclass from pathlib import Path import re @@ -62,14 +61,12 @@ class ApiModule: """A module's API page settings. Attributes: - dotted: The import path, such as ``timenet.registry``. - slug: The page name under ``docs/api/``: the last part of the import path, or the full path on a clash. + dotted: The import path, such as ``timenet.registry``. It is also the page name under ``docs/api/``. summary: The cleaned first line of the module docstring, or ``""``. show_submodules: Whether to include submodules recursively. True for a package without ``__all__``. """ dotted: str - slug: str summary: str show_submodules: bool @@ -154,9 +151,9 @@ def _discover(package: str, root: Path) -> list[ApiModule]: # A package that builds a public API via __all__ is documented at the # package level; one that doesn't keeps its surface in submodules, so # render them recursively instead of an empty page. - modules.append(ApiModule(dotted, name, _module_summary(init_file), not _has_dunder_all(init_file))) + modules.append(ApiModule(dotted, _module_summary(init_file), not _has_dunder_all(init_file))) else: - modules.append(ApiModule(dotted, name, _module_summary(child), False)) + modules.append(ApiModule(dotted, _module_summary(child), False)) return modules @@ -170,7 +167,7 @@ def _write_stub(mod: ApiModule) -> None: lines = ["---", f'description: "{_yaml_escape(desc)}"', "---", "", f"::: {mod.dotted}"] if mod.show_submodules: lines += [" options:", " show_submodules: true"] - (OUTPUT_DIR / f"{mod.slug}.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + (OUTPUT_DIR / f"{mod.dotted}.md").write_text("\n".join(lines) + "\n", encoding="utf-8") def _yaml_escape(text: str) -> str: @@ -208,7 +205,7 @@ def _write_index(by_package: dict[str, list[ApiModule]]) -> None: for package, mods in by_package.items(): parts.append(f"## `{package}`\n") for mod in mods: - bullet = f"- [`{mod.dotted}`]({mod.slug}.md)" + bullet = f"- [`{mod.dotted}`]({mod.dotted}.md)" if mod.summary: bullet += f" — {mod.summary}" parts.append(bullet) @@ -227,7 +224,7 @@ def _nav_line(by_package: dict[str, list[ApiModule]]) -> str: """ items = ['{ "Overview" = "api/index.md" }'] for mods in by_package.values(): - items += [f'{{ "{mod.dotted}" = "api/{mod.slug}.md" }}' for mod in mods] + items += [f'{{ "{mod.dotted}" = "api/{mod.dotted}.md" }}' for mod in mods] return '{ "API Reference" = [' + ", ".join(items) + "] }," @@ -261,33 +258,16 @@ def main() -> None: """Generate the API reference pages and landing page, and rewrite the nav region of ``zensical.toml``. It also deletes each page in ``docs/api/`` that no current module maps to. - - Raises: - SystemExit: If page names clash or the nav markers are missing or duplicated. """ OUTPUT_DIR.mkdir(parents=True, exist_ok=True) by_package = {pkg: _discover(pkg, root) for pkg, root in PACKAGE_ROOTS.items()} - # Both packages can have a module with the same name, such as timenet.sources and - # timenet_connectors.sources. The page of each such module takes its full dotted path. - shared = { - slug for slug, count in Counter(mod.slug for mods in by_package.values() for mod in mods).items() if count > 1 - } - by_package = { - pkg: [replace(mod, slug=mod.dotted) if mod.slug in shared else mod for mod in mods] - for pkg, mods in by_package.items() - } all_mods = [mod for mods in by_package.values() for mod in mods] - - # "index" is reserved for the landing page written by _write_index below. - slugs: dict[str, str] = {"index": "(API index landing page)"} - for mod in all_mods: - if mod.slug in slugs: - raise SystemExit(f"slug collision: {mod.dotted} and {slugs[mod.slug]} both map to {mod.slug}.md") - slugs[mod.slug] = mod.dotted + # Each page takes the full dotted path of its module, so two pages never share a name. + pages = {mod.dotted for mod in all_mods} # Drop stale stubs left by a previous run with a different module set. for stale in OUTPUT_DIR.glob("*.md"): - if stale.name != "index.md" and stale.stem not in slugs: + if stale.name != "index.md" and stale.stem not in pages: stale.unlink() for mod in all_mods: diff --git a/zensical.toml b/zensical.toml index 6342476d2..cfbe032a8 100644 --- a/zensical.toml +++ b/zensical.toml @@ -22,7 +22,7 @@ nav = [ { "Datasets" = [{ "Use datasets" = [{ "The client (SDK)" = "client.md" }, { "Registries" = "registry.md" }, { "Load into your stack" = "usage.md" }] }, { "Build datasets" = [{ "Connectors" = "connectors.md" }, { "Build & publish" = "build.md" }] }, { "Catalog" = [{ "Datasets" = "catalog/datasets.md" }, { "Benchmarks" = "catalog/benchmarks.md" }, { "Licensing" = "catalog/licensing.md" }] }] }, { "CLI" = [{ "Overview" = "cli/index.md" }, { "timenet (consumer)" = "cli/timenet.md" }, { "timenet-build (producer)" = "cli/build.md" }] }, # >>> api-nav: generated by scripts/gen_api_docs.py — do not edit >>> - { "API Reference" = [{ "Overview" = "api/index.md" }, { "timenet.builders" = "api/builders.md" }, { "timenet.client" = "api/client.md" }, { "timenet.config" = "api/config.md" }, { "timenet.connectors" = "api/connectors.md" }, { "timenet.dataset" = "api/dataset.md" }, { "timenet.engine" = "api/engine.md" }, { "timenet.errors" = "api/errors.md" }, { "timenet.format" = "api/format.md" }, { "timenet.json" = "api/json.md" }, { "timenet.manifest" = "api/manifest.md" }, { "timenet.producer" = "api/producer.md" }, { "timenet.progress" = "api/progress.md" }, { "timenet.provenance" = "api/provenance.md" }, { "timenet.reader" = "api/reader.md" }, { "timenet.registry" = "api/registry.md" }, { "timenet.sources" = "api/timenet.sources.md" }, { "timenet.storage" = "api/storage.md" }, { "timenet.types" = "api/types.md" }, { "timenet.values_backends" = "api/values_backends.md" }, { "timenet.writer" = "api/writer.md" }, { "timenet_connectors.bases" = "api/bases.md" }, { "timenet_connectors.discovery" = "api/discovery.md" }, { "timenet_connectors.download" = "api/download.md" }, { "timenet_connectors.sources" = "api/timenet_connectors.sources.md" }, { "timenet_connectors.splitting" = "api/splitting.md" }, { "timenet_connectors.time_axes" = "api/time_axes.md" }] }, + { "API Reference" = [{ "Overview" = "api/index.md" }, { "timenet.builders" = "api/timenet.builders.md" }, { "timenet.client" = "api/timenet.client.md" }, { "timenet.config" = "api/timenet.config.md" }, { "timenet.connectors" = "api/timenet.connectors.md" }, { "timenet.dataset" = "api/timenet.dataset.md" }, { "timenet.engine" = "api/timenet.engine.md" }, { "timenet.errors" = "api/timenet.errors.md" }, { "timenet.format" = "api/timenet.format.md" }, { "timenet.json" = "api/timenet.json.md" }, { "timenet.manifest" = "api/timenet.manifest.md" }, { "timenet.producer" = "api/timenet.producer.md" }, { "timenet.progress" = "api/timenet.progress.md" }, { "timenet.provenance" = "api/timenet.provenance.md" }, { "timenet.reader" = "api/timenet.reader.md" }, { "timenet.registry" = "api/timenet.registry.md" }, { "timenet.sources" = "api/timenet.sources.md" }, { "timenet.storage" = "api/timenet.storage.md" }, { "timenet.types" = "api/timenet.types.md" }, { "timenet.values_backends" = "api/timenet.values_backends.md" }, { "timenet.writer" = "api/timenet.writer.md" }, { "timenet_connectors.bases" = "api/timenet_connectors.bases.md" }, { "timenet_connectors.discovery" = "api/timenet_connectors.discovery.md" }, { "timenet_connectors.download" = "api/timenet_connectors.download.md" }, { "timenet_connectors.sources" = "api/timenet_connectors.sources.md" }, { "timenet_connectors.splitting" = "api/timenet_connectors.splitting.md" }, { "timenet_connectors.time_axes" = "api/timenet_connectors.time_axes.md" }] }, # <<< api-nav <<< ]