diff --git a/.github/workflows/reusable-publish-release-packages.yml b/.github/workflows/reusable-publish-release-packages.yml index cef4b9c..88d5dc5 100644 --- a/.github/workflows/reusable-publish-release-packages.yml +++ b/.github/workflows/reusable-publish-release-packages.yml @@ -71,8 +71,8 @@ on: default: false type: boolean outputs: - # What a caller's own PyPI job needs; see "Production PyPI" in - # docs/publishing-automation.md for the job that reads them. + # What a caller's own PyPI job needs; see + # docs/guides/publish-to-pypi.md for the job that reads them. pypi-publish: description: >- 'true' when the caller should upload to production PyPI: pypi-publish diff --git a/AGENTS.md b/AGENTS.md index fc9464e..2848bf7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,9 +1,9 @@ # AGENTS.md - Tests: `python3 -m unittest discover -s tests -v` from the repo root. Lint is `ruff check scripts/ tests/`. -- `publishing-vN` tags are immutable: a bad one is fixed with a new `vN+1`, never by moving or deleting the tag. The reusable workflows are called by that tag, not a branch ([publishing automation](docs/publishing-automation.md), [operations runbook](docs/operations-runbook.md)). +- `publishing-vN` tags are immutable: a bad one is fixed with a new `vN+1`, never by moving or deleting the tag. The reusable workflows are called by that tag, not a branch ([the release chain](docs/how-it-works/release-chain.md), [roll back a release](docs/guides/roll-back-a-release.md)). - Don't hand-edit `RELEASE_HEALTH.md` or `release-health/data.json`; the release-health workflow rewrites them on every release report. - Edit `data/repos_db.json`, never the generated markup, then run `python3 scripts/generate_sites.py`. It writes into the sibling repositories (`profile/README.md`, per-repo `.site/` trees, the pydevices ecosystem page), so run it with the repos checked out side by side ([README](README.md)). - `.cursor/environment.json` lists the repos Cursor's cloud environment clones; keep it in step by hand when a public repository is added or retired. -- Repository layout and file-naming conventions, including the `MICROPYPATH` / `PYTHONPATH` search paths: [repo-layout](docs/repo-layout.md). Doc and docstring style: [doc-style](docs/doc-style.md), [docstrings](docs/docstrings.md). +- Repository layout and file-naming conventions, including the `MICROPYPATH` / `PYTHONPATH` search paths: [repo-layout](docs/reference/repo-layout.md). Heading emoji and docstring style: [heading emoji](docs/reference/heading-emoji.md), [docstrings](docs/guides/write-docstrings.md). - Other repositories' generated files are overwritten by their sync scripts, so change the generator, not the copy ([CONTRIBUTING.md](CONTRIBUTING.md#generated-files-and-sync-targets)). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0da5ea7..e5f0e3f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,16 +10,16 @@ instructions — read that first. Org-wide standards every shipped design is measured against — starting with completeness ("any pairing the protocols permit that the design does not deliver is a defect in the design, not a feature request") — live in -[`docs/design-principles.md`](docs/design-principles.md). +[`docs/how-it-works/design-principles.md`](docs/how-it-works/design-principles.md). ## Where things live Repo directory and root-file layout (`src/`, `lib/`, `tests/`, -`requirements-dev.txt`, …): see [`docs/repo-layout.md`](docs/repo-layout.md). +`requirements-dev.txt`, …): see [`docs/reference/repo-layout.md`](docs/reference/repo-layout.md). The full repository map — every repo, grouped by tier, with what each one owns — is generated from `data/repos_db.json` and published at -[pydevices/docs/ecosystem.md](https://github.com/PyDevices/pydevices/blob/main/docs/ecosystem.md) and on the +[pydevices/docs/how-it-works/ecosystem.md](https://github.com/PyDevices/pydevices/blob/main/docs/how-it-works/ecosystem.md) and on the [org profile](https://github.com/PyDevices). Do not restate it here; it drifts. If you're not sure which repo an issue or PR belongs to, open it on @@ -62,5 +62,5 @@ regenerate; don't hand-edit generated files anywhere else. ## Doc style: emoji Docs across PyDevices use a small, fixed set of emoji as landmarks on -section headings — see [`docs/doc-style.md`](docs/doc-style.md) for the +section headings — see [`docs/reference/heading-emoji.md`](docs/reference/heading-emoji.md) for the full table and rules of thumb before adding or changing heading emoji. diff --git a/README.md b/README.md index 9c98545..1cfbc75 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,8 @@ the parts they'd otherwise duplicate. PyPI, report health) shared by seven publishing repositories. Callers pin a `publishing-vN` tag, never a branch; those tags are immutable by policy and by an active GitHub tag ruleset on this repository that blocks moving or - deleting one. See [`docs/publishing-automation.md`](docs/publishing-automation.md) - for the runbook and [`docs/workflows.md`](docs/workflows.md) for the full + deleting one. See [Release a package](docs/guides/release-a-package.md) + for the runbook and [Workflows](docs/reference/workflows.md) for the full workflow inventory. - **[`RELEASE_HEALTH.md`](RELEASE_HEALTH.md)** — one row per published distribution (version, TestPyPI/assets/MIP/PyPI outcome, last-updated, @@ -41,18 +41,16 @@ the parts they'd otherwise duplicate. landing page is published. [`scripts/generate_sites.py`](scripts/generate_sites.py) renders it into the org portal, into per-repo `.site/` trees, into [`profile/README.md`](profile/README.md), and into - [pydevices/docs/ecosystem.md](https://github.com/PyDevices/pydevices/blob/main/docs/ecosystem.md). + [pydevices/docs/how-it-works/ecosystem.md](https://github.com/PyDevices/pydevices/blob/main/docs/how-it-works/ecosystem.md). **Edit the database, not the generated markup** — run the generator by hand from a full workspace checkout (`python3 dotgithub/scripts/generate_sites.py`) after editing it; it validates the database first and is idempotent. - **Roadmap** — [`ROADMAP.md`](ROADMAP.md): what's planned for the shared automation. Every repository keeps its own. -- **History** — [`docs/history.md`](docs/history.md): how the organization - came to be, and a dated chronology of every repository. -- **Org docs** — [`docs/`](docs/) covers repo layout, doc style and - docstring conventions, how the ReadTheDocs sites build, GitHub presence - (Discussions/Issues/Pages/licensing), publishing automation, and the - operations runbook. +- **Docs** — [`docs/`](docs/index.md) covers setting up a checkout, releasing + a package, rolling back and fixing a release, building the Read the Docs + sites, the repository layout and docstring conventions, and where things + live on GitHub. Board / Detect inventory docs live in [pydevices/docs](https://github.com/PyDevices/pydevices/tree/main/docs) diff --git a/ROADMAP.md b/ROADMAP.md index cc00efa..31bc6b9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,4 +4,10 @@ Where the org's shared release automation and community files are heading. Bugs, and things you need that don't work yet, go to [issues](https://github.com/PyDevices/.github/issues). -Nothing is planned beyond what the issues track. +- **Real dependencies for MIP packages.** The index builder folds a required + package's files into the package that requires it and records no install-time + dependencies. The plan is for the index to carry real ones, so installing one + package brings in what it needs without duplicating files. +- **Type stubs for every published package.** The libraries keep types in + docstrings because annotations cost memory on a microcontroller. The plan is + to ship `.pyi` stubs so editors and type checkers on a desktop see real types. diff --git a/docs-site/install.md b/docs-site/install.md index 4d22ceb..54dbc07 100644 --- a/docs-site/install.md +++ b/docs-site/install.md @@ -2,7 +2,7 @@ How you install PyDevices depends on where your code will run. The exact commands and the system packages each channel needs are in the -[install workflows](https://github.com/PyDevices/pydevices/blob/main/docs/install-workflows.md) +[install guide](https://github.com/PyDevices/pydevices/blob/main/docs/get-started/install.md) of the `pydevices` repository; this page is the short version. ::::{tab-set} @@ -17,7 +17,7 @@ mip.install("palettes", index="https://PyDevices.github.io/mip") A board's `board_config` installs the same way and wires up the display and touch for that board. The board's page in the -[device matrix](https://github.com/PyDevices/pydevices/blob/main/docs/device-matrix.md) +[device matrix](https://github.com/PyDevices/pydevices/blob/main/docs/boards/device-matrix.md) names the package to ask for. ::: @@ -57,5 +57,5 @@ python -c "from board_config import display_drv; display_drv.fill_rect(50, 50, 1 If a window opens with a red square in it, the stack is working. If it does not, the -[troubleshooting page](https://github.com/PyDevices/pydevices/blob/main/docs/troubleshooting.md) +[troubleshooting page](https://github.com/PyDevices/pydevices/blob/main/docs/guides/troubleshooting.md) is the next stop. diff --git a/docs-site/newcomers.md b/docs-site/newcomers.md index 143248b..0ee17c5 100644 --- a/docs-site/newcomers.md +++ b/docs-site/newcomers.md @@ -30,7 +30,7 @@ Each project's manual has its own newcomer's guide. The board you own has a `board_config` that sets up its display and touch, so the program you ran on the desktop starts the same way on the board. The -[device matrix](https://github.com/PyDevices/pydevices/blob/main/docs/device-matrix.md) +[device matrix](https://github.com/PyDevices/pydevices/blob/main/docs/boards/device-matrix.md) lists the boards, and [mpftp](https://github.com/PyDevices/mpftp) copies files to them and gives you a REPL from your editor. diff --git a/docs/get-started/first-release.md b/docs/get-started/first-release.md new file mode 100644 index 0000000..e5bda74 --- /dev/null +++ b/docs/get-started/first-release.md @@ -0,0 +1,25 @@ +# Cut your first release + +You release a package by merging a pull request. Pick a repository that +publishes (see [what each repository publishes](../reference/release-workflows.md#what-each-repository-publishes)) +and start with a development version. It reaches TestPyPI, the release's +attached files and, for a repository that requests it, the library index, but not the MIP index or production PyPI. + +```bash +gh workflow run prepare-release.yml --repo PyDevices/ -f version=X.Y.Z.dev1 +``` + +Within a minute the repository has a pull request named `Release X.Y.Z.dev1` +on a `release/vX.Y.Z.dev1` branch. It changes `VERSION` and adds a section to +`CHANGELOG.md`. Check the version, read the changelog section, wait for CI, +then merge it. + +Merging tags `vX.Y.Z.dev1`, creates the GitHub Release, and starts the +publish run. When the run finishes, the package is on TestPyPI and the +release's row in [`RELEASE_HEALTH.md`](../../RELEASE_HEALTH.md) shows each +stage's outcome with a link to the run. + +The [release guide](../guides/release-a-package.md) covers a final release, +a retry and the manual fallback, and +[the release chain](../how-it-works/release-chain.md) explains why each step +exists. diff --git a/docs/get-started/set-up.md b/docs/get-started/set-up.md new file mode 100644 index 0000000..d2199de --- /dev/null +++ b/docs/get-started/set-up.md @@ -0,0 +1,43 @@ +# Set up a checkout + +Work on this repository from a clone. The scripts and tests need nothing +beyond Python 3 and `ruff`. + +```bash +git clone https://github.com/PyDevices/.github.git dotgithub +cd dotgithub +python3 -m unittest discover -s tests -v # the publishing-script tests +python3 -m pip install ruff +ruff check scripts/ tests/ # lint, as CI runs it +``` + +Those two commands are what the `unittest` and `ruff` jobs in +[`checks.yml`](../../.github/workflows/checks.yml) run. The same file lints +every workflow with actionlint, parses each reusable workflow as YAML, checks +that the reusables' `publishing-vN` references agree, and builds the +[main documentation project](../guides/build-and-publish-docs.md) with warnings +as errors, so a pull request here sees all of them. + +## Regenerate the org sites + +[`data/repos_db.json`](../../data/repos_db.json) is the one source for the +repository map: each repository's tier, description, buttons and the place its +landing page is published. Edit it, never the pages it produces, then run the +generator: + +```bash +python3 scripts/generate_sites.py +``` + +The generator writes into sibling repositories (the portal, the per-repository +`.site/` trees, the organization profile and the `pydevices` ecosystem page), +so clone the repositories it writes to (`PyDevices.github.io`, `pydevices`, +`gallery` and `mip`) side by side, and name this one `dotgithub` on disk, the +folder name the generator looks for: + +```bash +git clone https://github.com/PyDevices/.github.git dotgithub +``` + +A clean run produces no diff; `checks.yml` runs it against fresh clones and +fails on any change. diff --git a/docs/github-presence.md b/docs/github-presence.md deleted file mode 100644 index f9733ef..0000000 --- a/docs/github-presence.md +++ /dev/null @@ -1,134 +0,0 @@ -# PyDevices on GitHub — feature reference - -A map of how the [PyDevices](https://github.com/PyDevices) GitHub org is set -up: where discussions, issues, docs, and Pages sites live. Written to answer -"how do I...?" / "where do I find...?" questions later without re-deriving -the answer from scratch. - -## Quick answers - -| I want to... | Go here | -|---|---| -| Ask a question / propose an idea / show something I built / reach the maintainer | [pydevices Discussions](https://github.com/PyDevices/pydevices/discussions) — the organization's one Discussions board | -| Find or share a one-off example that's too narrow for the official examples | [pydevices Discussions](https://github.com/PyDevices/pydevices/discussions) (a *Recipes* category there, once created) | -| Report a bug or request a feature | Issues on the **specific repo** it affects (see [repo map](#repos--pages-sites)); use the Bug report / Feature request templates | -| Find contribution guidelines shared across repos | [PyDevices/.github CONTRIBUTING.md](https://github.com/PyDevices/.github/blob/main/CONTRIBUTING.md) | -| Read the org's public-facing "about us" | [PyDevices/.github profile README](https://github.com/PyDevices/.github/blob/main/profile/README.md) (rendered on [github.com/PyDevices](https://github.com/PyDevices)) | -| Read gallery documentation | [gallery/docs](https://github.com/PyDevices/gallery/tree/main/docs) | -| Read pygraphics documentation | [pygraphics.readthedocs.io](https://pygraphics.readthedocs.io) | -| Read palettes / pdwidgets docs | [palettes.readthedocs.io](https://palettes.readthedocs.io), [pdwidgets.readthedocs.io](https://pdwidgets.readthedocs.io) | -| Manage Read the Docs ↔ GitHub (org app) | [Read the Docs Community GitHub App](https://github.com/organizations/PyDevices/settings/installations) (all repos); migrate legacy projects at [RTD migrate-to-github-app](https://app.readthedocs.org/accounts/migrate-to-github-app/) | -| Try the library without installing anything | [PyScript browser demos](https://pydevices.github.io/gallery/pyscript/) | -| Hardware board/driver docs (configs, contract, drivers, matrix) | [pydevices docs/](https://github.com/PyDevices/pydevices/tree/main/docs) (markdown; the [Pages site](https://pydevices.github.io/pydevices/) is a landing page only) | -| Talk to a board from VS Code / Cursor (REPL, files, firmware) | [mpftp](https://github.com/PyDevices/mpftp) ([Pages](https://pydevices.github.io/mpftp/); agent state under `~/.mpftp/`) | - -## Getting help: Discussions vs. Issues - -- **Discussions** are enabled on **one** repository only: **pydevices** - ([github.com/PyDevices/pydevices/discussions](https://github.com/PyDevices/pydevices/discussions)). - It is the organization's contact point — questions, ideas, show-and-tell, and - reaching the maintainer. Every other repository has Discussions turned off - (2026-09-03) so there is exactly one place to look; SUPPORT.md, CONTRIBUTING.md - and the design docs all point there. -- **Issues** are enabled on every owned repo, for bugs and for things a user - needs that don't work yet. Planned enhancements live in each repo's - `ROADMAP.md` instead; a feature request that becomes planned work moves - there and its issue closes. - Default issue *templates* (`bug.yml`, `feature.yml`) live in the org's - [`.github`](https://github.com/PyDevices/.github/tree/main/.github/ISSUE_TEMPLATE) - repo and apply automatically to any repo that doesn't define its own. -- If you're not sure which repo an issue belongs to, open it on - [gallery](https://github.com/PyDevices/gallery/issues) — per - CONTRIBUTING.md, that's the routing point. -- **Recipes** ([pydevices Discussions](https://github.com/PyDevices/pydevices/discussions), *Recipes* category once created, - open-ended format, not answerable) is for one-off examples/how-tos that - answer a specific question well but are too narrow to promote into - `src/examples/`. If one gets enough traction, promote it into a real - example later — this category is meant to be a low-friction incubator, - not a permanent home for everything filed into it. -- **Wikis** are disabled org-wide — reference content lives in each repo's - `README.md`/`AGENTS.md`, in - [gallery documentation](https://github.com/PyDevices/gallery/tree/main/docs), or on a Pages site, - not in a wiki page. -- **Projects** are disabled org-wide (org boards and per-repo Projects). Public - work tracking is Issues + Discussions; there is no org kanban. -- **GitHub Packages** is not used. Installables go to TestPyPI / PyPI (CPython) - and [micropython-lib](https://github.com/PyDevices/mip) via `mip` - (MicroPython) — not GHCR or other GitHub package registries. - -## Org profile - -- **About / description / website**: set on the org (`https://pydevices.github.io/`). - -## Read the Docs - -Sphinx documentation for the organization, **pygraphics**, **palettes**, and **pdwidgets** -publishes to Read the Docs as one site (see [building-docs.md](building-docs.md)). GitHub integration uses the org-installed -[Read the Docs Community](https://github.com/apps/read-the-docs-community) -app (**all repositories** — see the org's -[installed GitHub Apps](https://github.com/organizations/PyDevices/settings/installations)). - -With the GitHub App, RTD receives push/PR events directly — no per-repo webhook -is required. The docs repos (**pygraphics**, **palettes**, **pdwidgets**) use this app; legacy `api/v2/webhook/...` hooks are gone. New -projects: import from the RTD dashboard (repo list comes from the app). If an -older project still needs migrating: -[Migrate to GitHub App](https://app.readthedocs.org/accounts/migrate-to-github-app/). - -Build config in each docs repo is `.readthedocs.yaml` + `docs/conf.py` (no secrets). - -## Repos & Pages sites - -Every product and module repo below ships a GitHub Pages site sharing the same -chrome (dark-default theme, light toggle in the header, PyDevices brand linking -to the org root) from -[PyDevices.github.io](https://github.com/PyDevices/PyDevices.github.io)'s -`assets/chrome/site.css` and `assets/chrome/site-chrome.js` — themselves -sync targets of this repo's `assets/css/site.css` and `assets/js/site-chrome.js`, -which are the master copies. Pages mount empty -`#pydevices-site-header` / `#pydevices-site-footer` elements; the script injects -identical markup (nav: GitHub, Core Stack, DisplayIF, Workbench, Gallery) and -`theme-toggle.js` binds `#theme-toggle`. The shared org site is the exception: -[PyDevices/.github](https://github.com/PyDevices/.github) -provides org metadata and docs, but it does not publish a product Pages site of -its own. Per-repo Pages heroes use marks under `assets/img/products/*.svg` (or -the repo's own `web/img/product.svg`); the org landing cards use compact inline -icons. Header brand stays on the shared org logo. - -| Repo | Role | Pages site | -|---|---|---| -| [gallery](https://github.com/PyDevices/gallery) | Examples, tutorials, and PyScript showcase for the PyDevices product stack | [pydevices.github.io/gallery](https://pydevices.github.io/gallery/) (+ [PyScript demos](https://pydevices.github.io/gallery/pyscript/)) | -| [palettes](https://github.com/PyDevices/palettes) | Color palette toolkit for PyDevices | [pydevices.github.io/palettes](https://pydevices.github.io/palettes/) | -| [pdwidgets](https://github.com/PyDevices/pdwidgets) | Cross-platform widget toolkit for PyDevices | [pydevices.github.io/pdwidgets](https://pydevices.github.io/pdwidgets/) | -| [omnipython-lvgl](https://github.com/PyDevices/omnipython-lvgl) | LVGL for MicroPython, CircuitPython and CPython: the binding generator and its three ports | [docs/](https://github.com/PyDevices/omnipython-lvgl/tree/main/docs) (no Pages site yet) | -| [micropython-pydevices](https://github.com/PyDevices/micropython-pydevices) | MicroPython with PyDevices' C modules (`displayif`, `audioif`, `cameraif`, `usbhost`, …) built in | [pydevices.github.io/micropython-pydevices](https://pydevices.github.io/micropython-pydevices/) (the old `displayif`, `audioif`, `cameraif` and `usbif` pages stay up as pointers) | -| [audiodsp](https://github.com/PyDevices/audiodsp) | CircuitPython-compatible audio for MicroPython plus `pydevices-audiodsp` CPython/Android/Pyodide wheels | [pydevices.github.io/audiodsp](https://pydevices.github.io/audiodsp/) · [TestPyPI](https://test.pypi.org/project/pydevices-audiodsp/) | -| [pygraphics](https://github.com/PyDevices/pygraphics) | Native + pure-Python FrameBuffer/Area (`import pygraphics`); docs on [RTD](https://pygraphics.readthedocs.io) | [pydevices.github.io/pygraphics](https://pydevices.github.io/pygraphics/) | -| [android-template](https://github.com/PyDevices/android-template) | Native Android APK application template for PyDevices | [pydevices.github.io/android-template](https://pydevices.github.io/android-template/) | -| [pyscript-template](https://github.com/PyDevices/pyscript-template) | Minimal installable PyScript application template | [pydevices.github.io/pyscript-template](https://pydevices.github.io/pyscript-template/) | -| [PyDevices.github.io](https://github.com/PyDevices/PyDevices.github.io) | Org landing + shared chrome | [pydevices.github.io](https://pydevices.github.io/) | -| [.github](https://github.com/PyDevices/.github) | This repo — org profile README, issue templates, CONTRIBUTING.md, `docs/github-presence.md`, `docs/repo-layout.md` | *(no Pages site — org metadata only)* | -| [micropython-lib](https://github.com/PyDevices/mip) | Fork carrying PyDevices' micropython-lib packages for `mip install` | *(no dedicated marketing site — it's a package index fork, not a product)* | -| [pydevices](https://github.com/PyDevices/pydevices) | Canonical reusable product: cross-interpreter libraries, board configs, and hardware drivers published for pip and MIP | [pydevices.github.io/pydevices](https://pydevices.github.io/pydevices/) | -| [mpftp](https://github.com/PyDevices/mpftp) | VS Code / Cursor / CLI board tools (REPL, file transfer, mip/circup, firmware) | [pydevices.github.io/mpftp](https://pydevices.github.io/mpftp/) | - -## Licensing - -All owned repos are **MIT**, using GitHub's standard single-author template -(so it's correctly detected as MIT rather than `NOASSERTION`/`Other`). A -handful of files in **gallery** that still carry code from other authors -(`src/utils/tft_text.py`, `tft_write.py`, `tft_bitmap.py`, and the -`polygon()` function in `PyDevices/pygraphics`'s `lib/pygraphics/_shapes.py`, tracing back through -Russ Hughes' st7789_mpy driver to Ivan Belokobylskiy's st7789py_mpy) keep -their own self-contained MIT header with that attribution — those in-file -notices govern those specific files/functions; the root `LICENSE` governs -everything else. Keep any such pointer *out* of the root `LICENSE` file -itself: GitHub's license detector does a similarity match against the exact -template, and extra text (even a short explanatory paragraph) can drop it -below the confidence threshold and flip the repo back to `NOASSERTION`. - -## Topics - -Each repo has GitHub topics set for discoverability (e.g. `lvgl`, -`micropython`, `circuitpython`, `user-c-modules`) — check a repo's sidebar on -github.com rather than duplicating the list here, since topics change more -often than this doc will be updated. diff --git a/docs/building-docs.md b/docs/guides/build-and-publish-docs.md similarity index 86% rename from docs/building-docs.md rename to docs/guides/build-and-publish-docs.md index f76b1c8..7f9d38c 100644 --- a/docs/building-docs.md +++ b/docs/guides/build-and-publish-docs.md @@ -1,22 +1,22 @@ -# Building and publishing documentation +# Build and publish documentation How to preview a PyDevices documentation project on your machine and how it reaches Read the Docs. One Sphinx configuration, in the `pydevices-docs` package -([docs-config/](../docs-config/)), gives every project the same theme, the +([docs-config/](../../docs-config/)), gives every project the same theme, the same top bar linking every project to every other, and the same Markdown and docstring setup. The organization overview is the main project, built from -[docs-site/](../docs-site/); each library's manual is a subproject. +[docs-site/](../../docs-site/); each library's manual is a subproject. | Project | Source | Site | |---|---|---| -| PyDevices (the main project) | [docs-site/](../docs-site/) here | `pydevices.readthedocs.io` | +| PyDevices (the main project) | [docs-site/](../../docs-site/) here | `pydevices.readthedocs.io` | | palettes | [palettes/docs](https://github.com/PyDevices/pydevices/tree/main/palettes/docs) | `/projects/palettes/` under the main site | | pdwidgets | [pdwidgets/docs](https://github.com/PyDevices/pydevices/tree/main/pdwidgets/docs) | `/projects/pdwidgets/` | | pygraphics | [docs/](https://github.com/PyDevices/pygraphics/tree/main/docs) in its repo | `/projects/pygraphics/` | -Docstring conventions for the generated API pages: [docstrings.md](docstrings.md). +Docstring conventions for the generated API pages: [write docstrings](write-docstrings.md). ## Preview locally @@ -31,7 +31,7 @@ python3 -m venv .venv-docs `requirements.txt` installs the tagged `pydevices-docs`. To work on the shared package itself, install your checkout instead: -`.venv-docs/bin/pip install -e ../.github/docs-config`. +`.venv-docs/bin/pip install -e /docs-config`. Open `_build/html/index.html`, or serve the folder (`python3 -m http.server -d palettes/docs/_build/html`). Live demos need http, @@ -47,14 +47,14 @@ reference map to the Python and MicroPython manuals. | File | Role | |------|------| -| `docs/conf.py` | Five lines: the project's name and where its code is, passed to `pydevices_docs.configure()` | +| `docs/conf.py` | A few lines: the project's name and where its code is, passed to `pydevices_docs.configure()` | | `docs/requirements.txt` | The pinned `pydevices-docs` release | | `.readthedocs.yaml` | Read the Docs build settings | | `docs/reference/` | API pages, written at build time from the package's modules (git-ignored) | Hand-authored pages are Markdown (MyST) in `docs/`. Cards and tabs come from `sphinx-design`, diagrams from `mermaid`, and the live browser demos are raw -HTML (see the [pydevices-docs README](../docs-config/README.md)). +HTML (see the [pydevices-docs README](../../docs-config/README.md)). ### Troubleshooting @@ -64,7 +64,7 @@ builtins) in `mock_imports` in `conf.py`. **A docstring warning fails the build** -- fix it in the **docstring**, not the signature: these packages run on MicroPython and CircuitPython, where -annotations cost bytecode and RAM. See [docstrings.md](docstrings.md). +annotations cost bytecode and RAM. See [write docstrings](write-docstrings.md). **An API page is missing a name** -- the page documents what the module defines. For a package whose public names live in private modules (pygraphics), diff --git a/docs/guides/fix-a-failed-release.md b/docs/guides/fix-a-failed-release.md new file mode 100644 index 0000000..9a999cc --- /dev/null +++ b/docs/guides/fix-a-failed-release.md @@ -0,0 +1,38 @@ +# Fix a release that failed + +Start with two places. `RELEASE_HEALTH.md` at the root of this repository +shows the last publish outcome for each distribution, updated by every publish +run, pass or fail, with a link to the run. The run's own log names the failing +job. + +Two situations are worth recognising before you read the table: + +- **A second publication lost, or the index went backwards.** Both known + causes are fixed (the queue checks out the branch tip, and the lockfile + commit retries after a rebase). A recurrence means a new instance of the same + kind: look for a workflow step that checks out without an explicit `ref:`. +- **A package moved repositories.** The MIP lockfile names the repository each + profile may be published from, and refuses a dispatch from another. The + releasing repository's `request-mip-publication` job reports success; only + `mip`'s own run goes red. Tag in the new repository, then change `repository` + and `ref` for the profile in `pydevices-lock.json` on `mip`'s `PyDevices` + branch, then rerun the failed `mip` runs. + +| Symptom | Meaning and response | +|---|---| +| No publication run after pushing a tag | A tag alone isn't the trigger. Publish the GitHub Release. | +| `VERSION contains ... but the release tag is ...` | Fix and commit `VERSION`, then make a new tag and version with a fresh "Prepare release" run. Don't move a published tag. | +| TestPyPI `invalid-publisher` | The job tried OIDC Trusted Publishing without a matching publisher. TestPyPI uses `__token__` and `TESTPYPI_API_TOKEN`. | +| TestPyPI authentication failure | Check that `TESTPYPI_API_TOKEN` exists for that repository, has permission for the project, and hasn't expired or been revoked. | +| TestPyPI duplicate-file response | Retry with a current coordinator, which sets `skip-existing: true`; otherwise publish a new version. | +| Production PyPI upload didn't run | Check four things: `pypi-publish: true` on the caller, the caller's own `pypi` job ([production PyPI](publish-to-pypi.md)), a final version (not `.devN` or `{a,b,rc}N`), and the `pypi` environment's reviewers approving the run. | +| PyPI `invalid-publisher` naming `PyDevices/.github` in `job_workflow_ref` | The upload ran inside the reusable workflow, which PyPI can't match. Move it to the caller's `pypi` job on a publishing tag that has `pypi-gate`. | +| A MIP request is queued | Expected: one concurrency group processes publication requests in order. | +| MIP validation sees `.publication-sources/manifest.py` | The shared sync job didn't remove its temporary checkouts. Start a fresh run on a current `publishing-v*` tag. | +| Pages setup or an action download returns 429, 503 or 504 | Usually transient. Retry the failed MIP job and check the live index afterwards. | +| The source workflow passed but MIP is still old | Look at the matching run in `PyDevices/mip` and wait for Pages to deploy. Success in the source repository only confirms the dispatch. | +| A native wheel platform is missing | Fix the wheel matrix and publish a new version. TestPyPI can't take a replacement for an existing filename. | +| `RELEASE_HEALTH.md` shows a stale row | The reporting job is `continue-on-error`, so a real failure elsewhere doesn't block it, but an outage of this repository or a missing App installation would. Open the linked run. | + +To retry or correct a release, see [release a package](release-a-package.md). +To pull a version back, see [roll back a release](roll-back-a-release.md). diff --git a/docs/guides/maintenance-schedule.md b/docs/guides/maintenance-schedule.md new file mode 100644 index 0000000..9d35ce6 --- /dev/null +++ b/docs/guides/maintenance-schedule.md @@ -0,0 +1,58 @@ +# Keep to the maintenance schedule + +Most upkeep runs by itself. This page lists what runs when, so you know which +red run is routine and which job is yours. + +## Continuous + +- **Release Health.** Every publish run reports, pass or fail, to + [`RELEASE_HEALTH.md`](../../RELEASE_HEALTH.md). +- **This repository's checks.** Every push and pull request runs actionlint, + the YAML and tag-reference checks, the site generator's idempotency check, + `ruff`, the script tests and the docs build + ([`checks.yml`](../../.github/workflows/checks.yml)). +- **Each repository's own CI**, on every push. + +## Daily + +- **Hero check** (06:41 UTC, + [`hero-check.yml`](../../.github/workflows/hero-check.yml)). Loads the + published demos in a browser and opens an issue when one stops running. The + page can render while its demo is dead, so nothing else notices. + +## Weekly, Mondays + +- **Font drift** in `omnipython-lvgl` (06:17 UTC). Regenerates the font + binaries and fails when they differ from the committed ones. +- **Overlay compatibility** in `micropython-pydevices` (06:43 UTC). Applies + every profile to the pinned upstream and fails when one no longer applies. +- **Firmware build** in `micropython-pydevices` (05:17 UTC). Builds every + firmware image without publishing anything, which keeps the caches warm and + shows the week an upstream pin moves and a board stops building. + +On Wednesdays (04:41 UTC) `micropython-pydevices` also rebuilds each +dev container from nothing, as a codespace would. + +Dependabot opens one grouped pull request a week per repository for pinned +action versions. + +## Monthly, by hand + +- Merge the month's Dependabot pull requests. They arrive grouped. +- Skim the TestPyPI project pages for anything you don't recognise. +- Check upstream MicroPython releases. A new stable release is information, + not an obligation: moving a pin (the MicroPython base, the LVGL submodule, + PyScript, the CircuitPython oracle) is deliberate work with its own + verification. + +## Quarterly + +- **Cold-eyes review** ([`cold-eyes-review.yml`](../../.github/workflows/cold-eyes-review.yml)). + On the first of January, April, July and October it opens an issue naming + three repositories from a rotating list, with four checks to re-run on each: + a clean build from the README, the README to a working example on a first + try, a `.devN` release through the whole chain, and a walk through the org's + front door (portal, cards, links, a 404). File what you find as issues on + the affected repositories. +- Review whether the CircuitPython version `audiodsp` pins should move. It + stays put until moving it is its own piece of work. diff --git a/docs/guides/publish-to-pypi.md b/docs/guides/publish-to-pypi.md new file mode 100644 index 0000000..525f337 --- /dev/null +++ b/docs/guides/publish-to-pypi.md @@ -0,0 +1,73 @@ +# Publish to production PyPI + +Every release reaches TestPyPI. Production PyPI is a decision you make per +package: set `pypi-publish: true` on the package's publish call. The only +repository that sets it today is `audiodsp`. + +Three things turn it on for a package, and missing one makes the run fail at +the PyPI end or skip the upload, never publish somewhere unintended: + +1. The `pypi-publish: true` input on the call to + `reusable-publish-release-packages.yml`. +2. Two jobs in the package's own `publish-release-packages.yml`, shown below. +3. A matching Trusted Publisher on the PyPI project: owner `PyDevices`, the + repository, workflow `publish-release-packages.yml`, environment `pypi`. + +## Why the upload is in the caller + +PyPI matches a Trusted Publisher against the workflow that runs the upload +job. Inside a reusable workflow that is +`PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml` at +a publishing tag, which no repository's publisher can name, so an upload from +there fails with `invalid-publisher` (see +[pypa/gh-action-pypi-publish#166](https://github.com/pypa/gh-action-pypi-publish/issues/166) +and [PyPI's note on reusable +workflows](https://docs.pypi.org/trusted-publishers/troubleshooting/#reusable-workflows-on-github)). +So the reusable's `pypi-gate` job only decides: it sets the `pypi-publish` +output to `true` for a final release whose build and TestPyPI upload +succeeded, and the caller uploads. + +## The two caller jobs + +```yaml + pypi: + needs: publish + if: >- + !cancelled() && needs.publish.outputs.pypi-publish == 'true' + runs-on: ubuntu-latest + environment: pypi + permissions: + id-token: write + steps: + - uses: actions/download-artifact@v8 + with: + name: ${{ needs.publish.outputs.dist-artifact }} + path: dist + - uses: pypa/gh-action-pypi-publish@release/v1 + with: + skip-existing: true + + report-pypi: + needs: [publish, pypi] + if: >- + !cancelled() && needs.publish.outputs.pypi-publish == 'true' + uses: PyDevices/.github/.github/workflows/reusable-report-pypi-result.yml@publishing-vN + with: + distribution-name: + version: ${{ needs.publish.outputs.version }} + pypi-result: ${{ needs.pypi.result }} + secrets: inherit +``` + +The `pypi` job runs in the repository's `pypi` GitHub Environment, which +needs a required reviewer and deployments limited to `v*` tags. Check it with +`gh api repos/PyDevices//environments/pypi`. `report-pypi` fills in +the PyPI outcome on the release's Release Health row, which shows `pending` +until it runs. + +Because the environment accepts only `v*` tags, a `workflow_dispatch` retry +from `main` can't reach it. A retry that needs PyPI dispatches on the tag, and +so runs the caller file as it was at that tag. + +TestPyPI is unaffected: it authenticates with the `TESTPYPI_API_TOKEN` secret, +not Trusted Publishing, and its upload stays in the reusable workflow. diff --git a/docs/guides/release-a-package.md b/docs/guides/release-a-package.md new file mode 100644 index 0000000..c6315d7 --- /dev/null +++ b/docs/guides/release-a-package.md @@ -0,0 +1,153 @@ +# Release a package + +A published GitHub Release named `vX.Y.Z` is the release event. The tag picks +the exact files to build and `X.Y.Z` becomes the package version. A manual +workflow run only retries an existing tag; it never publishes an untagged +branch or a different commit. + +The chain, from your side: + +```text +Prepare release (workflow_dispatch, optional version) + -> the workflow opens a release PR carrying VERSION + CHANGELOG.md + -> you confirm or edit the version on that PR, then merge it + -> Tag release creates the vX.Y.Z tag and GitHub Release + -> Publish release packages: build -> TestPyPI -> files attached to the Release + -> MIP index (final releases only) and library index -> health report +``` + +Choosing the version is the one step the automation leaves to you. + +## 1. Dispatch "Prepare release" + +```bash +gh workflow run prepare-release.yml --repo PyDevices/ +# or with a version: +gh workflow run prepare-release.yml --repo PyDevices/ -f version=X.Y.Z +``` + +The `version` input is a suggestion. Left empty, the workflow takes the +highest existing `vX.Y.Z` tag and adds one to the patch number. Neither that +value nor one you pass is committed until you confirm it on the pull request. +The workflow then: + +1. Checks the version against `X.Y.Z[{a|b|rc}N|.devN]` and refuses a tag that + exists. +2. Writes it to `VERSION` and puts a `## vX.Y.Z (date)` section at the top of + `CHANGELOG.md`, built from `git log` since the last tag. +3. Opens or updates a pull request named `Release X.Y.Z` from the branch + `release/vX.Y.Z`. + +## 2. Review and merge the release PR + +Confirm the version (push a commit to the branch to change `VERSION`), read +the changelog section, check that CI is green, and merge. Closing the pull +request without merging publishes nothing. + +## 3. Tag release runs by itself + +On every push to `main` that changes `VERSION`, the tag workflow creates the +`vX.Y.Z` tag and the GitHub Release at that commit. It marks the Release as a +pre-release when the version ends in `aN`, `bN`, `rcN` or `.devN`. + +It uses a GitHub App token, not the default `GITHUB_TOKEN`, on purpose: a +Release created with `GITHUB_TOKEN` doesn't trigger `on: release` workflows, +so the publish run would never start. + +If GitHub never delivered the push event for a merge, run the repository's tag +workflow by hand from `main`. A caller that also triggers on +`workflow_dispatch` and passes its `release` and `dry-run` inputs through (the +shape is in the header comment of +[`reusable-tag-on-release-merge.yml`](../../.github/workflows/reusable-tag-on-release-merge.yml)) +tags the commit that brought `VERSION` to that version. Leave `release` blank +to tag `VERSION` as it stands, or give a version or the merge's SHA. Set +`dry-run` to see what it would tag. + +## 4. Publish release packages runs on the Release + +The publish workflow resolves and checks the tag, builds, uploads to TestPyPI, +attaches the built files to the Release, requests MIP and library-index +publication, and reports to the [Release Health +dashboard](../how-it-works/release-chain.md#the-release-health-dashboard) +whatever the outcome. A final release can also go to production PyPI; see +[publish to production PyPI](publish-to-pypi.md). Each job is described in +[the release chain](../how-it-works/release-chain.md). + +## The manual fallback + +A release pull request is the default, but a hand-made tag and Release are valid +for a hotfix or a recovery. The one rule: the tag and the committed `VERSION` +must match exactly, because the build jobs reject a mismatch. + +```bash +git switch main && git pull --ff-only +version=X.Y.Z +printf '%s\n' "$version" > VERSION +git add VERSION CHANGELOG.md +git commit -m "Release ${version}" +git push origin main + +gh release create "v${version}" \ + --repo PyDevices/ \ + --target "$(git rev-parse HEAD)" \ + --title "v${version}" \ + --generate-notes +``` + +`gh release create` with your own token starts the publish run normally, and +publishes immediately unless you pass `--draft`. In the GitHub web UI, choose +the exact version commit, create `vX.Y.Z` and press **Publish release**: +pushing a tag, or saving a draft, doesn't start the publish run. + +## Retry an interrupted publication + +Never move the tag or rebuild a version from a different commit. Run the +publish workflow again with the existing tag: + +```bash +gh workflow run publish-release-packages.yml \ + --repo PyDevices/ \ + -f release-ref=vX.Y.Z +``` + +The workflow checks out that tag and validates it again, and TestPyPI uploads +use `skip-existing: true`, so a partial upload completes without replacing +files TestPyPI already took. + +If only the MIP run or its Pages deployment failed, rerun its failed jobs: + +```bash +gh run rerun --repo PyDevices/mip --failed +``` + +A rerun can stay pinned to the reusable-workflow commit the first attempt +used. If the reusable's ref changed since, start a new exact-tag retry from the +source repository, or dispatch the MIP queue by hand: + +```bash +gh workflow run process-mip-publication-request.yml \ + --repo PyDevices/mip \ + -f source-repository=PyDevices/ \ + -f source-ref=vX.Y.Z \ + -f version=X.Y.Z \ + -f profile= +``` + +Dispatch the queue directly only to recover, after you've confirmed the tag +and the profile. A repository's own release is the normal way in. + +## Correct a bad release + +Published TestPyPI files and released tags are immutable in practice. Deleting +a GitHub Release doesn't remove registry files, and moving a tag would make +the release impossible to reproduce. + +1. Fix the source on `main`. +2. Run "Prepare release" again for a new version. +3. Repeat the flow above. +4. Yank an unusable TestPyPI version only when you must, and say why. Don't + upload replacement files under the same version. + +Publishing the corrected version replaces the package's `latest` entry in the +MIP index. To pull a bad release back sooner, see +[roll back a release](roll-back-a-release.md). diff --git a/docs/guides/roll-back-a-release.md b/docs/guides/roll-back-a-release.md new file mode 100644 index 0000000..b3be267 --- /dev/null +++ b/docs/guides/roll-back-a-release.md @@ -0,0 +1,47 @@ +# Roll back a release + +Pick the registry the bad version reached. In every case the rule is to fix +forward with a new version number rather than rewrite history. + +**A bad version on TestPyPI.** Pre-releases are invisible to a default `pip +install`. Yank a final release from the project page: yank, don't delete, so +the history stays honest and installs stop. + +**A bad version on production PyPI.** Yank it the same way. The version number +is gone for good: never reuse it. + +**A bad publication on the MIP index.** Revert the lockfile commit +(`pydevices-lock.json`) on the `PyDevices` branch of `PyDevices/mip`, then +dispatch `deploy-mip-index-to-pages.yml`. It rebuilds every locked source from +the lockfile at the branch tip and redeploys. Check +`package///latest.json` for the version, and spot-check content too +(the bytes behind one file hash), because GitHub Pages caches for about ten +minutes and the version field alone can mislead. + +**A bad GitHub Release.** `gh release edit --draft` hides it at once and +deletes nothing, so you can decide at leisure. + +**A bad `publishing-vN` tag.** It can't change silently: a tag ruleset on this +repository blocks moving or deleting one. The fix is a new `publishing-v(N+1)` +cut with `scripts/cut_publishing_tag.sh`; the old tags stay. + +## When the release bot fails + +The release automation is a GitHub App. It mints every cross-repository token +(`create-github-app-token`, in the reusable workflows), and a failure names the +missing permission or repository in the log. The organization's owners manage +the App's installation and permissions. + +- **A red `Tag release` run** usually means the App lost access to the + repository. The release can proceed by hand: create the tag and the Release + with `gh release create` as yourself, and publication starts the same way. +- **A stuck dashboard.** The receiver, `release-health.yml`, runs only on + `repository_dispatch`, so a stuck dashboard means the App can't reach this + repository. Rows are additive JSON in `release-health/data.json`; you can + edit it by hand in a pinch and the next event regenerates the page. +- **A stuck MIP queue.** Runs serialize in the + `pydevices-mip-publication-queue` concurrency group. Cancel the stuck run, + then dispatch `process-mip-publication-request.yml` again with the profile, + repository, ref and version inputs. + +For triage when a release fails, see [fix a failed release](fix-a-failed-release.md). diff --git a/docs/docstrings.md b/docs/guides/write-docstrings.md similarity index 82% rename from docs/docstrings.md rename to docs/guides/write-docstrings.md index 9411ae8..e37ec66 100644 --- a/docs/docstrings.md +++ b/docs/guides/write-docstrings.md @@ -1,9 +1,9 @@ -# Docstring conventions +# Write docstrings Public docstrings in the three library projects — `palettes`, `pdwidgets`, and `pygraphics` — are rendered on Read the Docs by Sphinx's `autodoc` with the `napoleon` extension. Follow these rules so the generated reference pages stay -accurate. See [building-docs.md](building-docs.md) for the build itself. +accurate. See [build and publish documentation](build-and-publish-docs.md) for the build itself. ## Style @@ -25,7 +25,7 @@ One-line summary of the module's role. Optionally link to narrative docs: """ displaydev display drivers. -See also: https://github.com/PyDevices/pydevices/blob/main/docs/displaydev.md +See also: https://github.com/PyDevices/pydevices/blob/main/docs/how-it-works/displaydev-internals.md """ ``` @@ -48,7 +48,7 @@ See also: https://github.com/PyDevices/pydevices/blob/main/docs/displaydev.md ## Repository-specific notes - **`Area` returns:** Drawing helpers that return dirty regions use `pygraphics.Area` (`x`, `y`, `w`, `h`). -- **App:** Document poll/subscribe patterns; link to [Events concept](https://github.com/PyDevices/pydevices/blob/main/docs/appdev.md). +- **App:** Document poll/subscribe patterns; link to [The app loop](https://github.com/PyDevices/pydevices/blob/main/docs/guides/the-app-loop.md). - **Private API:** Names starting with `_` are excluded from the API pages; minimal or no docstrings are fine. Document public methods that live on private implementation bases when they surface via inheritance (set `:inherited-members:` on the page for that class). @@ -60,9 +60,8 @@ See also: https://github.com/PyDevices/pydevices/blob/main/docs/displaydev.md Type information goes in the **docstring**, not the signature. These packages run on MicroPython, where annotations cost bytecode and RAM. -Stubs are the planned answer for pip consumers who want real types — see -[Planned: type stubs](publishing-automation.md#planned-type-stubs-for-every-published-package). -Do not add `py.typed` to a package: it tells a type checker to trust inline +Type stubs for pip users who want real types are on the +[roadmap](../../ROADMAP.md). Do not add `py.typed` to a package: it tells a type checker to trust inline annotations that deliberately are not there. ## Verification @@ -83,5 +82,5 @@ A docstring problem is a warning, and `-W` makes it a failure. Fix it before mer | P1 | `displaybuf`, `console`, other `utils` | | P2 | Sibling packages document their own APIs: [pygraphics](https://pygraphics.readthedocs.io), [pdwidgets](https://pdwidgets.readthedocs.io), [palettes](https://palettes.readthedocs.io) | -See [CONTRIBUTING.md](../CONTRIBUTING.md) for the PR workflow, and -[building-docs.md](building-docs.md) for how the generated API pages are built. +See [CONTRIBUTING.md](../../CONTRIBUTING.md) for the PR workflow, and +[build and publish documentation](build-and-publish-docs.md) for how the generated API pages are built. diff --git a/docs/history.md b/docs/history.md deleted file mode 100644 index 6d0036e..0000000 --- a/docs/history.md +++ /dev/null @@ -1,61 +0,0 @@ -# PyDevices Organization History - -*Written 2026-09-06 and brought up to date through 2026-10-22 from GitHub's repository records, releases and merged pull requests, and the repositories' own histories; every creation date in the chronology was re-checked against GitHub on the day of each update. Update the chronology when a repository is created, transferred, renamed, archived or published.* - -## Concise history - -PyDevices began before the organization itself. The oldest project lineage starts on 2023-11-10 as Brad Barnett’s `mpdisplay` work: board configurations, display drivers, graphics, input events, and examples accumulated in one repository. CircuitPython support arrived in April 2024, followed by package extraction and broader desktop support. - -On 2024-09-02 the code was rebranded “PyDevices”; GitHub records the organization’s creation the following day, 2024-09-03. On 2024-11-19 the original repository moved from `bdbarnett/mpdisplay` to `PyDevices/pydisplay`. Three supporting repositories followed: the imported LVGL MicroPython lineage, a `micropython-lib` fork, and `cmods`. - -The organization assumed its current shape during June–August 2026. The large `pydisplay` repository was deliberately decomposed into display, LVGL, graphics, widget, palette, hardware, tooling, and examples repositories. An organization portal and shared publishing machinery were established in July; on 2026-08-11 the old flagship was refocused and renamed `gallery`, while the extracted hardware repository became the new `pydevices` flagship. - -Late August and early September broadened the project substantially: Android and PyScript application paths, a browser/mobile Workbench, the CircuitPython-compatible audio stack, VST3 hosting, a versioned MicroPython overlay, native USB host/device support, and MIPI-CSI cameras. The audio component split shipped on 2026-09-03; `usbif` and `cameraif` were published on 2026-09-04. - -September was a month of consolidation around the audio and timing layers. `mpvst` shipped its first installable release on 2026-09-11 and took that name the same day. On 2026-09-21 the audio core was renamed `audiodsp`, and the name it left, `audioif`, went to the audio hardware layer that sits under the audio pump, published that day. On 2026-09-23 `cmods` was archived: `micropython-pydevices`, created on 2026-08-29, had taken over the presets, variants and patch series that firmware builds needed. `pydevices` added Bluetooth Low Energy support on 2026-09-25 and, on 2026-09-26 in 0.6.0, the redesigned `multimer` timing layer, one timer and dispatcher per host across MicroPython, CircuitPython and CPython. `mpftp` learned to reach boards over Wi-Fi and Bluetooth on 2026-09-24 and 2026-09-25, and examples for casting a board's display and sound to a computer or a Roku TV followed on 2026-09-28. - -October moved the native code and the packaging. On 2026-10-05 `micropython-pydevices` gained one build command, `build_mp.py`, that builds firmware by module name, and on 2026-10-08 it began building CircuitPython-compatible firmware as well as MicroPython. The media modules `jpegio`, `h264enc`, `tsmux`, `ppa` and `pngio` landed there between 2026-10-06 and 2026-10-07, along with HLS casting in `castif`. In `audiocomponents`, the rebuilt effects shipped in v0.4.0 on 2026-10-15, and v0.4.1 on 2026-10-22 added SoundFont instruments, drum kits that answer pad keys 36 to 51, and a MIDI dispatcher. On 2026-10-19 the four interface repositories were folded in and archived: the C modules of `usbif`, `audioif`, `cameraif` and `displayif` moved into `micropython-pydevices` under `modules/`, and the Python parts of `usbif` moved into `pydevices` as the `usb-*` packages. The same day `pydevices` 0.7.6 was released as a final, with MIDI ports on `adafruit_midi`, Bluetooth MIDI and HID on `adafruit_midi` and `adafruit_hid`, USB on `usb.core`, the camera under `espcamera`, and I2S capture. On 2026-10-20 `palettes` and `pdwidgets` moved into `pydevices` and were archived, and a package-library builder (`tools/libbuild.py`) was merged there; the `lib` repository began serving its output as a `mip` index on 2026-10-21, and board configurations and drivers became installable packages of that library in the same days. On 2026-10-22 `micropython-pydevices` gained dev-container configurations for building firmware in a codespace, and `pydevices` 0.7.7, the first final to attach desktop and WebAssembly interpreters, was released. - -The 2014 history in `mip`, the 2018 history in `lvgl-micropython`, and the 2024 history in `workbench` are inherited upstream histories—not evidence that PyDevices itself existed at those dates. - -## Repository chronology - -Dates are UTC calendar dates. “Created” is GitHub’s repository-object date, which survives transfers, forks, and renames. - -| Repository | Created | Public / first verifiable public | Major milestones | -|---|---:|---:|---| -| [gallery](https://github.com/PyDevices/gallery) | 2023-11-10 | 2023-11-10* | Original monorepo; rebranded PyDevices 2024-09-02; transferred as `PyDevices/pydisplay` 2024-11-19; packages extracted through 2026; renamed/refocused as examples and PyScript gallery 2026-08-11; added casting, audio-rig, spectrum-meter and Google Photos examples and an “Open in Workbench” button on gallery pages in September–October. | -| [lvgl-micropython](https://github.com/PyDevices/lvgl-micropython) | 2024-11-17 | 2024-11-17* | Imported LVGL binding history reaching back to 2018; split into MicroPython-only glue in June 2026; adopted current name 2026-08-11; `display_driver` moved to `pydevices` 2026-10-07; Windows builds with LVGL fixed 2026-10-19. | -| [mip](https://github.com/PyDevices/mip) | 2024-11-22 | 2024-11-22* | Fork with inherited `micropython-lib` history from 2014; became PyDevices’ generated package mirror and MIP index; renamed from `micropython-lib` 2026-08-14; lists `midiports` and the `usb-*` packages from `pydevices` 0.7.6 (2026-10-19). | -| [cmods](https://github.com/PyDevices/cmods) | 2024-12-03 | 2024-12-03* | Evolved into the optional build/integration workspace for MicroPython, CircuitPython, native modules, LVGL, USB, and JPEG support; retired and archived 2026-09-23 (‡), once `micropython-pydevices` and each repository's own build glue had taken over, with a README that says where things went. | -| [displayif](https://github.com/PyDevices/displayif) | 2026-06-01 | ≤2026-06-25† | Native display-interface extraction, initially associated with `pydisplay_cmods`; standardized the DotClock/display API in July; added portable JPEG decoding in September; `jpegio` moved to `micropython-pydevices` 2026-10-06; archived 2026-10-19 (‡) when its C modules moved to `micropython-pydevices` under `modules/displayif`. | -| [lvgl-bindings](https://github.com/PyDevices/lvgl-bindings) | 2026-06-18 | 2026-06-18* | Shared header-to-C generator; CPython target added 2026-06-19; automated LVGL 9.x release chain reached v9.5.22 by 2026-08-29 and v9.5.23 by 2026-10-09; CPython bindings declared GIL-free for free-threaded builds 2026-10-15. | -| [lvgl-circuitpython](https://github.com/PyDevices/lvgl-circuitpython) | 2026-06-18 | 2026-06-18* | CircuitPython LVGL integration and build glue; portable build and display-driver work; renamed from `lv_circuitpython_mod` 2026-08-11; builds as a CircuitPython 11 user C module from 2026-10-08. | -| [lvgl-python](https://github.com/PyDevices/lvgl-python) | 2026-06-19 | 2026-06-19* | Native CPython extension, later covering Android and WebAssembly; first release 2026-06-30; reached v9.5.44 by 2026-08-29 and v9.5.52 on 2026-10-15; wheels for CPython 3.15 and free-threaded 3.15t added 2026-10-15. | -| [pygraphics](https://github.com/PyDevices/pygraphics) | 2026-07-05 | 2026-07-05* | Unified native and pure-Python graphics across MP/CP/CPython; package renamed to `pygraphics` 2026-07-26; v0.0.38 on 2026-08-29, v0.1.0 on 2026-10-08 and v0.1.1 on 2026-10-22; builds as a CircuitPython user C module from 2026-10-08. | -| [android-template](https://github.com/PyDevices/android-template) | 2026-07-06 | 2026-07-06* | Imported Android demo and python-for-android recipes from `usdl2`; produced the Launcher APK path; became the reusable application template; its recipes carry `pydevices` 0.7.6 as of 2026-10-20. | -| [.github](https://github.com/PyDevices/.github) | 2026-07-12 | 2026-07-12* | Organization profile and community-health files; centralized releases in August; reusable publishing contract advanced from `publishing-v1` to `publishing-v9`, reaching `publishing-v10` on 2026-09-22 and `publishing-v17` on 2026-10-22. | -| [PyDevices.github.io](https://github.com/PyDevices/PyDevices.github.io) | 2026-07-12 | 2026-07-12* | Organization portal; consolidated project pages; introduced direct MicroPython WebAssembly/PyScript apps in August and the `pydevices.com` landing page in September; added a firmware page and a one-line build installer in October; pages for the archived interface repositories point to their new homes (2026-10-19). | -| [pdwidgets](https://github.com/PyDevices/pdwidgets) | 2026-07-14 | 2026-07-14* | Widget toolkit extracted from the monorepo; first release 2026-07-15; reached v0.0.23 on 2026-08-29 and v0.0.24 on 2026-09-24; moved into the `pydevices` repository, in `pdwidgets/`, on 2026-10-20, with v0.0.25 released there on 2026-10-22; archived 2026-10-20 (‡). | -| [palettes](https://github.com/PyDevices/palettes) | 2026-07-15 | 2026-07-15* | Palette library extracted from `pydisplay` add-ons; first release immediately after extraction; v0.0.13 by 2026-08-29 and v0.0.14 on 2026-09-24; moved into the `pydevices` repository, in `palettes/`, on 2026-10-20, with v0.0.15 released there on 2026-10-22; archived 2026-10-20 (‡). | -| [mpftp](https://github.com/PyDevices/mpftp) | 2026-07-17 | 2026-07-17* | VS Code/Cursor MicroPython tooling for REPL, transfer, packages, and firmware; v0.0.1 on 2026-07-26 and v0.0.4 on 2026-08-29; reached boards over Wi-Fi (2026-09-24) and Bluetooth (2026-09-25); v0.1.0 on 2026-10-08 added a browser interface and a bundled `mpremote`; v0.1.3 on 2026-10-22. | -| [pydevices](https://github.com/PyDevices/pydevices) | 2026-07-25 | 2026-07-25* | Began as the hardware/board-driver extraction from `pydisplay`; became the flagship portable package family in August; reached v0.3.8 on 2026-08-29; added `bledev` (2026-09-25), the redesigned `multimer` timing layer (0.6.0, 2026-09-26) and many board configurations and drivers; `midiports` and the `usb-*` packages on the upstream `adafruit_midi`, `adafruit_hid` and `usb.core` APIs in 0.7.6, a final released 2026-10-19; `palettes`, `pdwidgets` and the package-library builder merged 2026-10-20; v0.7.7 on 2026-10-22 was the first final to attach desktop and WebAssembly interpreters. | -| [pyscript-template](https://github.com/PyDevices/pyscript-template) | 2026-08-12 | 2026-08-12* | Installable offline PWA template for browser-hosted PyDevices applications; repaired and standardized deployment contract on 2026-08-29; works offline after the first visit (2026-09-24) and gained a Web Workers kit (2026-10-15). | -| [android-runner](https://github.com/PyDevices/android-runner) | 2026-08-17 | 2026-08-17* | Reproducible Android CPython runtime build engine; first Runner APK release, v0.1.0, on 2026-08-29; reached v0.2.8 on 2026-10-22, carrying `pydevices` 0.7.6 since 2026-10-20. | -| [audiodsp](https://github.com/PyDevices/audiodsp) | 2026-08-25 | 2026-08-25* | CircuitPython audio APIs ported to MicroPython and CPython; cross-platform parity gates; v0.2.0 core-only split shipped with 21 wheels on 2026-09-03; renamed from `audioif` on 2026-09-21, its first published release under the new name being v0.5.1 on 2026-09-22; added Ogg Vorbis playback (2026-10-12), wheels for CPython 3.15 (2026-10-15) and the `audiograph` runner (2026-10-21); v0.6.10 on 2026-10-22. | -| [workbench](https://github.com/PyDevices/workbench) | 2026-08-25 | 2026-08-25* | Public ViperIDE fork retaining history from 2024; rebranded with PyDevices WASM simulator and examples; v0.7.0 released 2026-08-29; boots the simulator with the PyDevices stack frozen into its WebAssembly runtime (2026-10-05); opens gallery examples by link (2026-10-11); v0.7.4 on 2026-10-22. | -| [mpvst](https://github.com/PyDevices/mpvst) | 2026-08-27 | 2026-08-27* | Dedicated MicroPython VST3 instrument/effect engine; Windows and Linux byte-identical PCM, scripted synths, MIDI control, and REAPER validation; renamed from `micropython-vst3` and first released, as v0.1.0 with Windows and Linux installers, on 2026-09-11; v0.3.3 on 2026-10-22. | -| [micropython-pydevices](https://github.com/PyDevices/micropython-pydevices) | 2026-08-29 | 2026-08-29* | Versioned MicroPython patch queue, runtime profiles, provenance, WebAssembly pieces, and restricted VST3-engine profile; added `build_mp.py` (2026-10-05), CircuitPython-compatible builds (2026-10-08), `castif` (2026-09-28), the media modules `jpegio`, `h264enc`, `tsmux`, `ppa` and `pngio` (2026-10-06 to 2026-10-07), the C modules of `usbif`, `audioif`, `cameraif` and `displayif` (2026-10-19), and dev-container configurations for building in a codespace (2026-10-22). | -| [usbif](https://github.com/PyDevices/usbif) | 2026-08-30 | 2026-09-04 | Developed privately, audited, and history-cleaned before publication; proved USB audio, MIDI, CDC, HID, storage, hub, UAC, and UVC paths in host/device roles; archived 2026-10-19 (‡). Its Python packages moved to `pydevices` as `usb-core` and the `usb-host-*` and `usb-device-*` packages; its C module became `usbhost` in `micropython-pydevices`. | -| [audiocomponents](https://github.com/PyDevices/audiocomponents) | 2026-09-01 | 2026-09-03 | Seeded from the audio work begun 2026-08-24; separated 53 instruments and effects from `audiodsp`; public v0.2.0 split shipped 2026-09-03; v0.3.0 on 2026-09-23; v0.4.0 on 2026-10-15 shipped the rebuilt effects; v0.4.1 on 2026-10-22 added SoundFont instruments, drum kits that answer pad keys 36 to 51 and a MIDI dispatcher. | -| [cameraif](https://github.com/PyDevices/cameraif) | 2026-09-04 | 2026-09-04 | Same-day bring-up and publication: board-agnostic camera API, MIPI-CSI capture, hardware JPEG, zero-copy frames, scaling, examples, and an upstream OV5647 report; added an IMX219 driver and auto-exposure; archived 2026-10-19 (‡) when it moved to `micropython-pydevices` under `modules/cameraif`, where it answers to `espcamera`. | -| [audioif](https://github.com/PyDevices/audioif) | 2026-09-18 | 2026-09-21 | Developed privately as the audio pump's platform drivers (ESP32 I2S and task, unix, Windows) and a CircuitPython-shaped `audiobusio.I2SOut`; published with v0.1.0 on 2026-09-21 under the name the audio core had just given up; v0.1.2 on 2026-10-10; archived 2026-10-19 (‡) when it moved to `micropython-pydevices` under `modules/audioif`. | -| [lib](https://github.com/PyDevices/lib) | 2026-10-21 | 2026-10-21* | The `mip` package index served from GitHub Pages: each package is built from a release of the repository it lives in, and board configurations and drivers are published from their merge. | - -\* Best-supported conclusion is that the repository was public from creation; GitHub exposes creation and current visibility but not a general historical “public since” field. I found no contrary visibility transition in the available history. - -† `displayif` was definitely public under its earlier naming by 2026-06-25; the precise visibility-change instant between its 2026-06-01 creation and that public activity is unavailable. The organization audit-log endpoint that could settle this was inaccessible. - -‡ GitHub exposes whether a repository is archived but not when. The archive date given is the day of the repository's final commit, the notice that it had moved or was retired, which is also the day of its last update on GitHub; where the README gives a date, it matches. - -No files, branches, indexes, or repository configuration were changed while researching this history. diff --git a/docs/design-principles.md b/docs/how-it-works/design-principles.md similarity index 86% rename from docs/design-principles.md rename to docs/how-it-works/design-principles.md index 4bdfe44..0199eaf 100644 --- a/docs/design-principles.md +++ b/docs/how-it-works/design-principles.md @@ -34,10 +34,8 @@ What this demands in practice: stranger could watch. A matrix cell nobody has exercised is a claim, not a capability. -This standard was first written down in a Bluetooth-interface vision -(2026-08-31, a project since retired) -and applies retroactively to everything the organization ships: when a -gap in an existing surface is found, it is triaged as a defect. +This standard applies retroactively to everything the organization ships: +when a gap in an existing surface is found, it is triaged as a defect. ## 2. Portability by tier @@ -47,7 +45,7 @@ makes sense, and translated — not duplicated — where a sibling ecosystem already has its own excellent idioms. Exceptions are recorded in the design document, not assumed. Every platform claim carries its proof tier — bench-proven, CI-proven, or community-verified — as defined in -[`platform-support-tiers.md`](platform-support-tiers.md). +[platform support tiers](platform-support-tiers.md). ## 3. Identity varies with capability, deterministically @@ -63,10 +61,9 @@ always yields the same identity — so a host's cached association stays correct across reboots and re-enumerations. Fifteen configurations, fifteen identities, the same fifteen every time. -Proven in usbif's runtime function selection (2026-08-31), where -TinyUSB's own source warns of exactly this; the obligation is inherited -by every PyDevices surface that lets Python vary a device's capability -set. +It was proven in the USB host and device work, where TinyUSB's own source +warns of exactly this; the obligation is inherited by every PyDevices surface +that lets Python vary a device's capability set. --- diff --git a/docs/how-it-works/overview.md b/docs/how-it-works/overview.md new file mode 100644 index 0000000..43a11fd --- /dev/null +++ b/docs/how-it-works/overview.md @@ -0,0 +1,38 @@ +# What this repository is + +`PyDevices/.github` doesn't ship a product. It is what every other PyDevices +repository points at for the parts they'd otherwise duplicate. GitHub treats a +repository with this name specially: its community health files are the +defaults for any repository in the organization that doesn't have its own. + +## What lives here + +| Piece | Where | Read more | +|---|---|---| +| Community health defaults: contributing guide, code of conduct, security policy, support, pull request and issue templates | the root and `.github/ISSUE_TEMPLATE/` | [CONTRIBUTING.md](../../CONTRIBUTING.md) | +| The organization profile shown on github.com/PyDevices | `profile/README.md` | [where things live](../reference/github-org.md) | +| The reusable release workflows and the tag contract | `.github/workflows/reusable-*.yml`, `scripts/` | [the release chain](release-chain.md) | +| The release dashboard, one row per distribution | `RELEASE_HEALTH.md`, `release-health/data.json` | [fix a failed release](../guides/fix-a-failed-release.md) | +| The repository database and site generator | `data/repos_db.json`, `scripts/generate_sites.py` | [set up a checkout](../get-started/set-up.md) | +| The shared Sphinx configuration and the organization overview site | `docs-config/`, `docs-site/` | [build and publish documentation](../guides/build-and-publish-docs.md) | +| Shared standards | `docs/` | [design principles](design-principles.md), [platform support tiers](platform-support-tiers.md), [repository layout](../reference/repo-layout.md) | + +`RELEASE_HEALTH.md` and `release-health/data.json` are written by the +`release-health` workflow on every release report, so don't edit them by hand. +Edit `data/repos_db.json` and run the generator rather than changing the pages +it produces. Other repositories' generated files are overwritten by their sync +scripts: change the generator, not the copy +([CONTRIBUTING.md](../../CONTRIBUTING.md#generated-files-and-sync-targets)). + +## Where to start reading + +To release a package, read [the release chain](release-chain.md) first, then +[release a package](../guides/release-a-package.md). To add a repository to the +organization's pages, edit the database and regenerate. To change how every +repository's documentation builds, start with the +[build guide](../guides/build-and-publish-docs.md). + +A new maintainer reads [CONTRIBUTING.md](../../CONTRIBUTING.md), then the +README of the repository they will touch (each states its role and build path), +then [release a package](../guides/release-a-package.md) before their first +release. diff --git a/docs/platform-support-tiers.md b/docs/how-it-works/platform-support-tiers.md similarity index 97% rename from docs/platform-support-tiers.md rename to docs/how-it-works/platform-support-tiers.md index 963b30a..ca6c8c7 100644 --- a/docs/platform-support-tiers.md +++ b/docs/how-it-works/platform-support-tiers.md @@ -1,7 +1,7 @@ # Platform Support Tiers How PyDevices states what a platform claim is worth. Companion to -[`design-principles.md`](design-principles.md): the completeness standard +[design principles](design-principles.md): the completeness standard demands that every claim be proven or recorded with its honest cause — these tiers are the vocabulary for doing that consistently, org-wide. diff --git a/docs/how-it-works/release-chain.md b/docs/how-it-works/release-chain.md new file mode 100644 index 0000000..9432567 --- /dev/null +++ b/docs/how-it-works/release-chain.md @@ -0,0 +1,212 @@ +# The release chain + +This is how a package version gets from a merged pull request to TestPyPI, +the MIP index, the library index and, for some packages, production PyPI. The +organization-wide release machinery lives in this repository because it +governs every publishing repository: the reusable workflows, the tag contract +and the shared credentials. Each repository's own publishing notes cover only +what its release produces, such as `pydevices`'s dynamically discovered +package set. They add build and test detail; they don't replace what's here. + +To run a release, see [release a package](../guides/release-a-package.md). For +the tables of inputs, secrets and tags, see +[release workflows](../reference/release-workflows.md). + +## The release event + +A published GitHub Release named `vX.Y.Z` is the authoritative event. The tag +selects the exact files to build and `X.Y.Z` becomes the package version. A +manual run only retries an existing tag. + +Choosing the version is the organization's one deliberate exception to full +automation. The chain removes the manual tag, push and release steps around +that choice: + +1. **Prepare release** (`workflow_dispatch`) opens a pull request that carries + `VERSION` and `CHANGELOG.md`. +2. A person confirms or edits the version on that pull request and merges it. +3. **Tag release** sees the merged `VERSION` change on `main` and creates the + tag and the GitHub Release, using the GitHub App's token. The token matters: + a Release made with the default `GITHUB_TOKEN` doesn't trigger + `on: release` workflows, so the publish run would never start. +4. **Publish release packages** runs on the Release event. + +## How the publish run is wired + +`reusable-publish-release-packages.yml` is the whole per-repository chain, as +jobs inside one reusable workflow. It resolves the tag, runs exactly one build +(`build-pure-python`, `build-native-and-wasm` or `build-pydevices-multi`, +chosen by `build-kind`), uploads to TestPyPI, attaches the built files to the +Release, requests MIP and library-index publication, decides whether +production PyPI should follow, and reports to the Release Health dashboard. +It replaced five near-identical copies of the same three jobs. The production +PyPI upload is the one step that has to live in the caller +([why](../guides/publish-to-pypi.md#why-the-upload-is-in-the-caller)). + +Each publishing repository has thin callers (`prepare-release.yml`, +`tag-release.yml`, `publish-release-packages.yml`) that each call one reusable +workflow and supply only what differs. +[Workflows](../reference/workflows.md) lists the callers and what each passes. + +Some repositories wrap the chain in more jobs. `audiodsp` runs parity gates +and a credentials check first, and an `expected-wheel-count` that has to follow +`requires-python` by hand. `audiocomponents` runs its own gates against the +core it pins, then two `pure-python` builds from `lib/audioinstruments` and +`lib/audioeffects`; both MIP entries ride on the first job's comma-separated +`mip-profile`, since both come from one tag. + +Changing a reusable workflow's contract is an automation rollout, not a +package release. Validate it on its own, cut a new publishing tag, move the +callers deliberately, and only then use that tag for package versions. + +## The tag contract + +A caller pins a reusable workflow at a tag, not a branch: + +```yaml +uses: PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml@publishing-vN +``` + +A change here therefore doesn't alter any release until a caller moves its +tag. The tags are immutable by policy and by an active tag ruleset that blocks +moving or deleting one: a bad tag is fixed by a new one. + +The coordinator also calls four sibling reusables, and those calls name the +same tag by full path. A local `./` path doesn't work: in a called workflow it +resolves against the consumer's repository, so the consumer's run dies at +startup with no jobs. Three reusables also check out this repository's +`scripts/` at `publishing-tools-ref`, so that default has to name the same tag +too. Keeping every one of those references equal to the tag being cut is +mechanical work, which is why `scripts/cut_publishing_tag.sh` is the only way +a tag is cut: it rewrites them all, commits that one change and tags the same +commit. The `sibling-refs` job in `checks.yml` refuses a tree where any +disagree or use `./`. + +## What the `pydevices` release contains + +The `pydevices` build keeps no inclusion list. The distribution is exactly +`lib/`, though the two ecosystems package it differently: + +- **One package in each.** On PyPI it's the `pydevices` distribution; on MIP + the `pydevices` package. A package per component was tried and dropped: + micropython-lib resolves `require()` by inclusion at build time rather than + as a dependency edge, so each package that required `events.py` carried its + own copy and installing two wrote the file twice. +- **A microcontroller gets only what can run on one.** `displaydev`, `audiodev` + and `multimer` each carry platform backends, many of them desktop, browser or + Android code. The source repository's `mip-split.toml` names the host-only + modules; they ship in `pydevices-desktop` instead, and `pydevices` lists its + files one by one so the install list is the manifest. A name in the file that + no longer exists fails the sync, and so does a device-side module that + imports a host-only one at module scope. A new backend nobody classified + ships to the device, so forgetting costs one extra file on a board, never a + missing import on a host. +- **A package can publish to MIP on its own.** A `mip-split.toml` section with + `own-package = true` leaves the `pydevices` package and becomes a sibling at + `micropython/pydevices/`, with its own `requires` as `require()` lines. + `bledev` is one: a board without a radio doesn't carry BLE. On PyPI the code + stays in the `pydevices` wheel, and `pypi-extras` makes an extra such as + `pip install "pydevices[ble]"`. +- **A top-level directory can publish under its own name.** A section that is + only `top-level = true` publishes `//` to + `micropython/pydevices/`, with the directory's `manifest.py` as the + single source for its description, `require()` lines and files. The sync + fails if a listed name isn't a directory with a `manifest.py`. This is + available from `publishing-v14`. +- Every publishable entry under `utils/`, and everything publishable in + `board_configs/desktop/`, is bundled into `pydevices-desktop`, which depends + on `pydevices`. All artifacts from one release share a version. +- Board `package.json` installers aren't copied into the central MIP index; + install them from their documented raw GitHub paths. + +Adding or removing a publishable entry under `lib/` or `utils/` therefore +changes the next release automatically, so review the generated package set +when you prepare a release. Committed root TOML files such as +`pydevices-desktop.toml` and `gallery.toml` are PyScript filesystem mappings, +not package metadata; their CI validation has to pass before a release. + +## The LVGL release + +`omnipython-lvgl` carries the binding generator, its three ports and the +`pydevices-lvgl` wheel. Its `release.yml` is the release entry point. It +validates an exact commit SHA or `vX.Y.Z` tag, runs the Linux integration +matrix, and in `release` mode writes the next version to `VERSION` on `main` +and publishes a GitHub Release with an App token, for the reason given above. +That Release starts the ordinary `native-and-wasm` publish for +`pydevices-lvgl`, with no `mip-profile`. Mode `dry-run` runs every gate and +changes nothing. + +## The MIP publication queue + +Publishing to the MIP index is centralized in `PyDevices/mip` and serialized. +`process-mip-publication-request.yml` runs in the +`pydevices-mip-publication-queue` concurrency group with +`cancel-in-progress: false`, so two releases that both want MIP entries never +race; the second waits. + +**Request.** `reusable-request-mip-publication.yml` mints an App token scoped +to `mip` and sends one `repository_dispatch` per profile in the +`mip-profile` input. The MIP leg runs for final releases only: for a +`.devN`, `rcN`, `aN` or `bN` it's skipped, so a pre-release never reaches the +MIP index. A repository whose profile the MIP lockfile doesn't know (the +`palettes` and `pdwidgets` folders of `pydevices`) sets `legacy-index: false` +and sends only the library leg. + +**Sync.** The queue consumer, `reusable-synchronize-mip-package.yml`: + +1. Checks out `mip` at the `PyDevices` branch tip, not the SHA frozen into the + dispatch. A dispatch freezes `github.sha` when it's created, so the second + of two queued publications would otherwise start from a tree that predates + the first one's lockfile commit and lose its push. +2. Records the release in `pydevices-lock.json`, the source of truth for which + version of each package is live. Every publishable package needs an entry + there, added deliberately. The entry's `repository` key is also a guard: a + dispatch from a different source repository is refused. +3. Re-synchronizes every locked source, not only the one released. `build.py` + compiles every `manifest.py` in the tree at once, so one package's index + entry can't be rebuilt in isolation. +4. Removes the temporary `.publication-sources` checkouts, builds the complete + latest-only index, and verifies that every expected manifest produced an + entry and that each locked package's index version matches the lockfile. +5. Commits the lockfile with rebase-and-retry: up to three tries of `git push`, + then `git pull --rebase` on rejection. +6. Assembles and uploads the Pages artifact, which a separate job in the caller + deploys with `actions/deploy-pages`. + +`deploy-mip-index-to-pages.yml` rebuilds the same way on a push to the +`PyDevices` branch or by hand. + +## The library index + +`PyDevices/lib` serves the library index and is built beside the MIP index, not +from it. Its `publish.yml` takes the same `publish-package-release` dispatch. +A `record` job commits the release's profile and tag to `pins.json` at once and +outside the queue, because GitHub's concurrency group holds one running and one +pending job and a newer pending job replaces an older one, which would lose the +older request's pin. Only the build and the deploy queue, in the +`omnipython-lib-publication-queue` group. Each build rebuilds every package +from the pins with `tools/libbuild.py` from the `pydevices` checkout and +deploys with `actions/deploy-pages`. + +A merge to `pydevices` that touches `board_configs/` or `drivers/` sends the +same dispatch with the `boards` profile, so a board or driver is published when +it merges; the next `pydevices` release clears that pin. A profile is refused +when its repository isn't the one `pins.json` names for it. To rebuild without +a release, run `publish.yml` by hand with the inputs empty; to try a library +commit before it's released, give `test-root-ref`, which deploys the build and +records nothing. + +## The Release Health dashboard + +Every publish run, successful or not, ends with a `report-release-health` job. +It's `continue-on-error`, so a reporting failure never fails the release. It +sends a `repository_dispatch` of type `release-health` to this repository with +the outcome of each stage: TestPyPI, assets, MIP and PyPI. A release headed for +production PyPI reports `pypi` as `pending`, because that upload runs later in +the caller; the caller's `report-pypi` job then sends a partial report that +fills in the outcome on the same row. + +`release-health.yml` folds the payload into +[`release-health/data.json`](../../release-health/data.json) and regenerates +[`RELEASE_HEALTH.md`](../../RELEASE_HEALTH.md): one row per distribution, +linked to the run that produced it. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..1ec5363 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,35 @@ +# .github docs + +This repository holds the pieces every PyDevices repository shares: the +community health files, the reusable release workflows, the org portal's +repository database and the shared documentation setup. These pages are for +people who maintain those pieces or release a package with them. + +**Get started.** [Set up a checkout](get-started/set-up.md) and run its +checks, then [cut your first release](get-started/first-release.md) of a +package. + +**Guides.** [Release a package](guides/release-a-package.md), +[publish to production PyPI](guides/publish-to-pypi.md), +[roll back a release](guides/roll-back-a-release.md), +[fix a release that failed](guides/fix-a-failed-release.md), +[build and publish documentation](guides/build-and-publish-docs.md), +[write docstrings](guides/write-docstrings.md) and +[keep to the maintenance schedule](guides/maintenance-schedule.md). + +**Reference.** [Workflows](reference/workflows.md), the +[release workflows' inputs, secrets and tags](reference/release-workflows.md), +[repository layout](reference/repo-layout.md), +[heading emoji](reference/heading-emoji.md) and +[where things live on GitHub](reference/github-org.md). + +**How it works.** [What this repository is](how-it-works/overview.md), +[the release chain](how-it-works/release-chain.md), +[design principles](how-it-works/design-principles.md) and +[platform support tiers](how-it-works/platform-support-tiers.md). + +**Changelog.** This repository has no versions of its own; the reusable +workflows are versioned by `publishing-vN` tags, listed under +[tags](https://github.com/PyDevices/.github/tags). What is planned is in +[ROADMAP.md](../ROADMAP.md). To contribute, read +[CONTRIBUTING.md](../CONTRIBUTING.md). diff --git a/docs/maintenance-calendar.md b/docs/maintenance-calendar.md deleted file mode 100644 index 1cb2bd8..0000000 --- a/docs/maintenance-calendar.md +++ /dev/null @@ -1,31 +0,0 @@ -# Recurring maintenance calendar - -## Continuous (automated) -- Release Health dashboard: every publish run reports, pass or fail. -- dotgithub self-checks (actionlint, YAML, generator idempotency, ruff): - every push/PR here. -- Per-repo publish/test CI: every push per repo. - -## Weekly (automated, Mondays) -- lvgl-bindings font drift (06:17 UTC) — regenerates fonts/*.bin, fails - on divergence. -- micropython-pydevices overlay compatibility (06:43 UTC) — every - profile applies clean to the pinned upstream. - -## Monthly (manual, ~first weekend) -- Merge the month's dependabot PRs (they arrive grouped; batch-merge is - routine). -- Skim TestPyPI project pages for anything unexpected. -- Check upstream MicroPython releases: a new stable is INFORMATION, not - an obligation — moving any pin (upstream v1.28.0, lvgl submodule, - PyScript, CircuitPython oracle) is deliberate work with its own - verification, never routine. - -## Quarterly (scheduled → issue) -- The standing cold-eyes review: `cold-eyes-review.yml` opens an issue - here each quarter naming a rotating sample of repositories and the - scenarios to re-run (S1 clean build, S4 stranger's first try, S5 - release health, S6 the org walk). The bar decays without enforcement; - this is the enforcement. -- CircuitPython oracle stance review: 10.2.1 stays pinned until moving - it is chartered as its own re-port (audiodsp's call, not routine). diff --git a/docs/operations-runbook.md b/docs/operations-runbook.md deleted file mode 100644 index e36cdd5..0000000 --- a/docs/operations-runbook.md +++ /dev/null @@ -1,69 +0,0 @@ -# Operations runbook - -Companion to [publishing-automation.md](publishing-automation.md) (the -release runbook). This file covers what to do when things go wrong, and -how new maintainers and agents come aboard. - -## Rollback - -Exercised twice in production during the modernization program; the -patterns are proven. - -**A bad version reached TestPyPI**: pre-releases are invisible to default -pip; final releases can be yanked from the project page (yank, don't -delete — it preserves honest history and stops installs). Production -PyPI: same, plus the version number is burned forever — never reuse it. - -**A bad publication reached the MIP index**: revert the lockfile commit -in PyDevices/mip, then dispatch `deploy-mip-index-to-pages.yml` (it -rebuilds every locked source from the branch-tip lockfile and redeploys). -Verify: `package///latest.json` version AND a content spot -check (a file hash's bytes), not just the version field — Pages caches -for ~10 minutes. - -**A bad GitHub Release**: `gh release edit --draft` hides it -immediately without deleting anything; decide at leisure. (This rescued -the ViperIDE mis-release.) - -**A bad publishing-vN tag**: cannot happen silently — the tag ruleset on -this repo blocks moves and deletions. The fix is always a v(N+1); v3–v6 -history stays. - -## Incident triage - -1. `workflow_attention.sh` at the workspace root: current failures and - active runs, org-wide. -2. `RELEASE_HEALTH.md` in this repo: last publish outcome per - distribution, updated by every publish run, pass or fail. -3. A red `Tag release` run usually means the App lost repo access or - permissions (installation settings) — the release can always proceed - manually (tag + `gh release create` as a human; publication triggers - identically). -4. A "second publication lost" or "index went backwards" event is the - frozen-SHA class: both known holes are fixed (branch-tip checkouts + - rebase-retry), so a recurrence means a NEW instance of the class — - look for a checkout without an explicit `ref:` in whatever workflow - wrote last. - -## Bot-failure recovery - -- The release-automation App mints all cross-repo tokens - (create-github-app-token in the reusables). Failures name the missing - permission or repo in the log. The org owner holds the App's installation - and permissions; agents never hold its key (roadmap rule 3). -- The dashboard receiver (`release-health.yml`) is repository_dispatch - only; a stuck dashboard means the App can't reach this repo. Rows are - additive JSON in `release-health/data.json` — safe to hand-edit in a - pinch, the renderer regenerates the page on the next event. -- MIP queue stuck: runs serialize in the - `pydevices-mip-publication-queue` concurrency group; cancel the stuck - run, re-dispatch `process-mip-publication-request.yml` with the - profile/repo/ref/version inputs (all profiles are in the choice list). - -## Onboarding - -- **A human maintainer**: CONTRIBUTING.md here, then the repo they'll - touch (every repo states its role, status, and build path in its - README), then publishing-automation.md before their first release. -- **An agent**: [AGENTS.md](../AGENTS.md) here, then the `AGENTS.md` of the - repository it works in. diff --git a/docs/publishing-automation.md b/docs/publishing-automation.md deleted file mode 100644 index bcc6dc4..0000000 --- a/docs/publishing-automation.md +++ /dev/null @@ -1,800 +0,0 @@ -# Releasing and publishing PyDevices packages - -This is the organization-wide runbook for publishing a new package version, and -it **lives here because it is org-wide**: it governs `palettes`, `pdwidgets`, -`pygraphics`, `audiodsp`, `audiocomponents`, `pydevices`, `lvgl-python`, and -`mpftp`, and it documents the reusable workflows and shared credentials that -this repository owns. - -Repository-specific publishing documents cover only what their own release -produces — for example -[pydevices/docs/publishing.md](https://github.com/PyDevices/pydevices/blob/main/docs/publishing.md) -describes that repo's dynamically discovered package set. They add build or -test details; they do not restate or replace the procedure below. - -A **published GitHub Release named `vX.Y.Z` is the authoritative release -event**. The source tag selects the exact files to build, and `X.Y.Z` becomes -the package version. A manual workflow run is only a retry of an existing -exact tag; it is not a way to publish an untagged branch or a different -commit. - -## The tag contract - -Every publishing repository's coordinators pin the reusable workflows in this -repository at a tag, not a branch, so a change here does not alter a release -until the tag moves: - -```yaml -uses: PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml@publishing-vN -``` - -**Never pin `publishing-v10`.** It was cut 2026-09-22 with the coordinator -calling its siblings by `./` path, which resolves against the *consumer's* -repository: a consumer's publish run dies at startup with zero jobs -(pydevices v0.5.0, [.github#47](https://github.com/PyDevices/.github/issues/47)). -`publishing-v11` is the fix: the siblings are named by full path at the tag's -own name, and [`scripts/cut_publishing_tag.sh`](../scripts/cut_publishing_tag.sh) -is the only way a tag is cut from now on — it rewrites those refs *and* the -`publishing-tools-ref` input defaults (through which three reusables check -out this repository's `scripts/`) and tags the same commit, and the -`sibling-refs` check refuses a tree where any of them disagree or use `./`. -(v11 itself still carries v6 tools-ref defaults — harmless, the scripts are -identical — and is the last tag cut before the script covered them.) - -**There is no single "current" pin — a repository is on whatever tag it was -last moved to.** As of 2026-10-07: - -| Reusable | Pinned at | By | -|---|---|---| -| every other reusable it calls | `publishing-v12` | `pydevices` | -| `reusable-publish-release-packages` | `publishing-v8` | `audiodsp`, `audiocomponents` | -| `reusable-publish-release-packages` | `publishing-v6` | `palettes`, `pdwidgets`, `pygraphics`, `lvgl-python`, `mpftp` | -| `reusable-prepare-release-pr` | `publishing-v13` | every publishing repository (moved 2026-10-07 for the release PR's description) | -| `reusable-tag-on-release-merge` | `publishing-v6` | every publishing repository except `pydevices` | -| `reusable-validate-pyscript-filesystem-toml` | `publishing-v6` | `palettes`, `pdwidgets`, `pygraphics` | -| `reusable-synchronize-mip-package` (and its `publishing-tools-ref`) | `publishing-v11` | `mip` (moved 2026-09-22, mip#2, so the MCU split reaches the index at the next publication) | - -What `publishing-v11` carries beyond `publishing-v6`, with input contracts -unchanged for every reusable a consumer calls: the VERSION grammar guard -before tagging, the `## Unreleased` heading converted rather than orphaned by -the release PR (proven on pydevices v0.5.0's release PR), a release-health -push that re-folds instead of losing a report, and the MCU mip split -(host-only modules routed to `pydevices-desktop`). - -Every tag from `publishing-v1` still exists, so a release cut before a -contract change can still be retried against the contract it was built with. - -**A tag does not yet contain the workflows it runs.** The coordinator pins its -own four sibling workflows by tag as well, and those pins have to be bumped by -hand when a tag is cut — which has been missed twice. `publishing-v7` shipped a -macOS wheel matrix that never ran, costing `audioif` v0.1.0 its tag; and -`publishing-v9`'s copy of the coordinator still names `@publishing-v8`, so a -caller on v9 runs v8's builders today. [`.github#26`](https://github.com/PyDevices/.github/issues/26) -tried to fix that with local `./` paths, and #47 showed they resolve against -the consumer. The fix that holds is procedural: `scripts/cut_publishing_tag.sh` -writes the tag's own name into the sibling refs in the commit it tags, so a -tag cut that way cannot skew. For a retry against v9 or earlier, **check the -nested refs first**. - -Publishing tags are **immutable by policy, and that policy is enforced**, not -just documented: this repository has a tag ruleset named "publishing tags are -immutable" (`target: tag`, `enforcement: active`) that blocks moving or -deleting a `publishing-v*` tag. Cut a new tag for a new contract; never force- -push an existing one. - -## What each repository publishes - -| Source repository | TestPyPI | PyDevices MIP index | -|---|---|---| -| `palettes` | `pydevices-palettes` wheel and sdist | `palettes` | -| `pdwidgets` | `pydevices-pdwidgets` wheel and sdist | `pdwidgets` | -| `pygraphics` | `pydevices-pygraphics` Linux, Windows, Android, and WASM wheels | Pure-Python `pygraphics` | -| `audiodsp` | `pydevices-audiodsp` Linux, Windows, macOS, Android, and WASM wheels | Nothing — the native modules are firmware, built from the usermod source | -| `audiocomponents` | `pydevices-audioinstruments` and `pydevices-audioeffects` (pure-Python) — two distributions from one repository | `audioinstruments` and `audioeffects` | -| `pydevices` | `pydevices` (all of `lib/`) and `pydevices-desktop` | One package per `lib/` component, plus `pydevices` and `pydevices-desktop` | -| `lvgl-python` | `pydevices-lvgl` Linux, Windows, Android, and WASM wheels | Nothing | -| `mpftp` | `pydevices-mpftp` wheel and sdist | Nothing | - -The pip distribution names are prefixed with `pydevices-`; MIP package names -are not. `pydevices` and `pydevices-desktop` keep the same name in both -systems. `mpftp` additionally has its own `publish-vsix.yml`, unrelated to -this chain, which publishes the VS Code extension to the VS Marketplace and -Open VSX using `VSCE_PAT` / `OVSX_PAT` (both optional; the steps skip when -unset). - -### The dynamic `pydevices` release - -The `pydevices` build does not maintain an inclusion list. **The distribution -is exactly `lib/`, nothing more** — but the two ecosystems package it -differently, because their constraints differ: - -- **Both ecosystems ship `lib/` as a single package.** On PyPI that is the - `pydevices` distribution; on MIP the `pydevices` package. -- Publishing a package per component was tried and dropped. micropython-lib - resolves `require()` by **inclusion at build time**, not as a dependency - edge, so the index materialised every required component's files into each - package that required it -- `displaydev` and `appdev` each carried their own - copy of `events.py`, and installing both wrote it twice. The granularity was - an illusion, and it cost a package per component to maintain. -- The trade is real but bounded: a display-only board installs about 232 KiB - more source than `displaydev` alone used to pull. Revisit if that starts to - matter on a target. -- **A microcontroller gets only what can run on one.** `displaydev`, `audiodev` - and `multimer` each carry a full set of platform backends, and about twenty of - them are desktop, browser or Android code. The source repository's - `mip-split.toml` names those per package; they ship in `pydevices-desktop` - instead, and `pydevices` names its files one by one so the install list is the - manifest. An MCU install went from 51 files to 31, and `pydevices-desktop` - stayed at 58 — byte for byte the same set, because it already requires - `pydevices` (pydevices#30). - Two rules keep that file from rotting: a name in it that no longer exists - fails the sync, and so does an MCU-side module that imports a host-only one at - module scope. A *new* backend nobody classifies ships to the device, so the - cost of forgetting is one file too many on a board, never a missing import on - a host. -- **A package can publish to MIP on its own.** A `mip-split.toml` section - with `own-package = true` leaves the `pydevices` MIP package and becomes a - sibling (`micropython/pydevices/`), with its own `requires` as - `require()` lines, which the index resolves by including them. `bledev` is - the first: a board without a radio shouldn't carry BLE, and - `mip.install("bledev")` brings aioble with it. On PyPI nothing changes: the - package stays inside the `pydevices` wheel, and a section's - `pypi-extras = { ble = ["bleak>=1.0"] }` becomes - `pip install "pydevices[ble]"`. An unknown key in a section fails the sync. -- **A directory at the repository root can publish to MIP under its own - name.** A `mip-split.toml` section that is only `top-level = true` publishes - `//` to `micropython/pydevices/`. pydevices' `midiports` - and `usb-*` packages are the first, eleven of them plus `midiports` - (pydevices#226). The directory's own `manifest.py` is the single source for - its description, `require()` lines and files, since a firmware build freezes - the same file; the sync copies the directory without `package.json` and - `README.md` and sets only the manifest's `version` to the release's, so the - index and the lockfile agree. A section takes no other keys. The sync fails - if a listed name isn't a directory with a `manifest.py`, or if a listed - package `require()`s a sibling directory the file doesn't list. The reverse, - a directory nobody listed, is the source repository's own test to catch - (pydevices' `tests/test_mip_split.py`). Available from `publishing-v14`. - mip's `scripts/update_package_list.py` expects these packages once the - lockfile's pydevices ref is 0.7.6 or later (`PROFILE_PACKAGES_SINCE`). -- Every publishable entry under `utils/`, plus everything publishable in - `board_configs/desktop/`, is bundled into `pydevices-desktop`. -- `pydevices-desktop` depends on `pydevices`, so one install gets the complete - desktop stack. -- All artifacts from one `pydevices` release share a version. -- MIP meta-package dependencies resolve `latest` from the latest-only index. -- Only `pydevices` and `pydevices-desktop` declare `pypi_publish`; the MIP leaf - packages deliberately do not, because they have no PyPI distribution of their - own. -- Board `package.json` installers are not copied into the central MIP index. - Install them from their documented raw GitHub paths; the desktop board - library files themselves remain part of `pydevices-desktop`. - -Adding or removing a publishable entry under `lib/` or `utils/` therefore -changes the next release automatically. Review the generated package set as -part of release preparation. Committed root TOML files (such as -`pydevices-desktop.toml` and `gallery.toml`) are PyScript -filesystem mappings of the repository payloads, not package metadata; their -CI validation must pass before release. - -## Standard release procedure: the release-PR flow - -Every one of the seven publishing repositories now uses the same three- -workflow chain instead of hand-writing `VERSION` and tagging locally. It -exists to keep the human choice of version — the org's one deliberate -exception to full automation — while removing the manual tag/push/release -sequence that used to precede it. - -```text -Prepare release (workflow_dispatch, optional version) - -> bot opens a release PR carrying VERSION + CHANGELOG.md - -> a human confirms or edits the version on that PR - -> merge to main - -> Tag release creates the vX.Y.Z tag and GitHub Release (via the App) - -> Publish release packages: build -> TestPyPI -> assets attached - -> MIP index (final releases only) and library index (every release) -> health report -``` - -### 1. Dispatch "Prepare release" - -```bash -gh workflow run prepare-release.yml --repo PyDevices/ -# or with a specific version: -gh workflow run prepare-release.yml --repo PyDevices/ -f version=X.Y.Z -``` - -This runs `reusable-prepare-release-pr.yml`. The `version` input is optional -and is only ever a **suggestion**: leaving it empty computes the highest -existing `vX.Y.Z` tag plus one patch, but that computed value — and any -explicitly requested value — is not committed until a human confirms it. The -workflow: - -1. Resolves the version (requested value, or the computed suggestion), - rejecting anything that doesn't match - `X.Y.Z[{a|b|rc}N|.devN]` or a tag that already exists. -2. Writes that version to `VERSION` and prepends a new `## vX.Y.Z (date)` - section to `CHANGELOG.md`, built from `git log` since the last tag. -3. Opens (or updates) a PR titled `Release X.Y.Z` from branch - `release/vX.Y.Z`. Its description reads: "Release X.Y.Z: sets VERSION and - adds this release's CHANGELOG section. Merging tags the version in VERSION - and publishes it. Closing publishes nothing." - -### 2. Review and merge the release PR - -This is the human checkpoint. Confirm the suggested version is correct (or -push a commit to the PR branch changing `VERSION` to the right one), -review the generated changelog section, confirm CI is green, then merge. -Closing the PR without merging publishes nothing. - -### 3. Tag release fires automatically - -`reusable-tag-on-release-merge.yml` runs on every push to `main` that touches -`VERSION`. It: - -1. Diffs `VERSION` against the previous commit; does nothing if unchanged or - if the tag already exists. -2. Mints a GitHub App token (`PYDEVICES_APP_ID` / `PYDEVICES_APP_PRIVATE_KEY`) - and creates the `vX.Y.Z` tag and GitHub Release at that commit, marking it - `--prerelease` when the version has a `{a|b|rc}N` or `.devN` suffix. - -The App token is required here, not incidental: **a Release created with the -default `GITHUB_TOKEN` does not trigger `on: release` workflows**, so the -publish chain below would silently never fire. - -If GitHub never delivers the push event for a merge (it has happened: a merge -that ran no workflow at all), run the repository's tag workflow by hand from -`main`. A caller that also triggers on `workflow_dispatch` and passes its -`release` and `dry-run` inputs through (the shape is in the header comment of -`reusable-tag-on-release-merge.yml`) gets a manual path that tags the commit -that brought `VERSION` to the version, checks the same version grammar as the -push path, and does nothing if the tag exists. Leave `release` blank to tag -`VERSION` as it stands, or give a version or that merge's SHA. Set `dry-run` to -see what it would tag without tagging anything. - -### 4. Publish release packages runs from the Release event - -`reusable-publish-release-packages.yml` resolves and validates the `vX.Y.Z` -tag, builds (pure-Python, native-and-wasm, or pydevices-multi, per caller), -publishes to TestPyPI, attaches the built artifacts to the GitHub Release, -requests MIP publication for final releases, tells a caller that opted in to -upload to production PyPI (the upload runs in the caller's workflow), and reports to the Release Health dashboard regardless of outcome. -See "How the release chain is wired" below for the per-job detail. - -### Manual tag + release: the fallback - -The release-PR flow is the default, but a manual tag and Release remain a -valid fallback — for a hotfix, a recovery, or any release where opening a PR -is impractical. The one hard rule does not change: **the tag and the -committed `VERSION` file must match exactly**, because the build jobs -(`reusable-build-pure-python-distribution.yml`, -`reusable-build-native-and-wasm-wheels.yml`, -`reusable-build-pydevices-distributions.yml`) reject a mismatch outright. - -```bash -cd / -git switch main && git pull --ff-only - -version=X.Y.Z -printf '%s\n' "$version" > VERSION -git add VERSION CHANGELOG.md -git commit -m "Release ${version}" -git push origin main - -release_commit="$(git rev-parse HEAD)" -gh release create "v${version}" \ - --repo PyDevices/ \ - --target "$release_commit" \ - --title "v${version}" \ - --generate-notes -``` - -`gh release create` with a user token triggers `on: release` normally (only -the App-token requirement above is specific to the automated tag step, which -runs with a plain `GITHUB_TOKEN` context). `gh release create` publishes -immediately unless `--draft` is supplied; if a release is prepared in the -GitHub web UI, select the exact version commit, create `vX.Y.Z`, and -**Publish release** — merely pushing a tag or saving a draft does not trigger -the coordinator. - -## Automation and credentials - -Each publishing repository has three thin coordinators -(`prepare-release.yml`, `tag-release.yml`, `publish-release-packages.yml`), -each calling one reusable workflow from this repository and supplying only -what differs: - -```yaml -jobs: - publish: - # The tag each repository is on differs; see The tag contract above. - uses: PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml@publishing-v8 - with: - build-kind: pure-python # or native-and-wasm, pydevices-multi - distribution-name: pydevices-palettes - import-name: palettes - mip-profile: palettes # comma-separate for several, as audiocomponents does; omit to skip MIP - release-ref: ${{ inputs.release-ref }} - secrets: inherit -``` - -Two repositories wrap the reusable chain in more than one job. -`audiodsp`'s `publish-release-packages.yml` runs a `parity` gate (the -`tests/parity/verify_*.py` scripts and every DSP probe) and a `credentials` -check, then one `native-and-wasm` build for `pydevices-audiodsp` with no -`mip-profile`, and an `expected-wheel-count` that has to follow -`requires-python` by hand — it did not once, and the release that found out -published nothing (audiodsp#122). `audiocomponents` runs its own gates against -the core it pins, then two `pure-python` builds (`pydevices-audioinstruments`, -`pydevices-audioeffects`) from `lib/audioinstruments` and `lib/audioeffects`; -both MIP entries ride on the first job's comma-separated -`mip-profile: audioinstruments,audioeffects`, since both come from one tag. - -Required secrets: - -| Secret | Where needed | Purpose | -|---|---|---| -| `PYDEVICES_APP_ID`, `PYDEVICES_APP_PRIVATE_KEY` | Every publishing repository, `PyDevices/mip` | Mint short-lived GitHub App installation tokens: to open release PRs, to create tags/Releases (so `on: release` fires), to dispatch the MIP queue, and to dispatch the Release Health report | -| `TESTPYPI_API_TOKEN` | Every repository that uploads to TestPyPI | API token for the `bdbarnett` TestPyPI account | -| `VSCE_PAT`, `OVSX_PAT` | **`mpftp` only** (`publish-vsix.yml`, unrelated to this chain) | VS Marketplace and Open VSX. Both optional; the steps skip when unset | - -TestPyPI still uses token authentication (`user: __token__`, `password: -${{ secrets.TESTPYPI_API_TOKEN }}`, `repository-url: -https://test.pypi.org/legacy/`, `skip-existing: true`) — it is not Trusted -Publishing, and that is deliberate: TestPyPI Trusted Publishing has its own -per-repo/workflow/environment registration that has not been set up. - -### Production PyPI: opt-in, protected, Trusted Publishing - -Production PyPI (as opposed to TestPyPI, which every release reaches) is an -explicit per-package decision: `pypi-publish: true` on the coordinator call. -`audiodsp` is the only repository that sets it. - -**The upload runs in the caller's own workflow, not in the reusable one.** -PyPI matches a Trusted Publisher against the workflow that runs the upload job -(`job_workflow_ref`). Inside a called workflow that is -`PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml` -at a publishing tag, which no repository's publisher can name, so an upload -from there always fails with `invalid-publisher` (audiodsp v0.6.1; -[pypa/gh-action-pypi-publish#166](https://github.com/pypa/gh-action-pypi-publish/issues/166), -[PyPI's note on reusable workflows](https://docs.pypi.org/trusted-publishers/troubleshooting/#reusable-workflows-on-github)). -From the tag that carries this change, the coordinator only decides: its -`pypi-gate` job sets the `pypi-publish` output to `true` for a final release -whose build and TestPyPI upload succeeded, and the caller uploads: - -```yaml - pypi: - needs: publish - if: >- - !cancelled() && needs.publish.outputs.pypi-publish == 'true' - runs-on: ubuntu-latest - environment: pypi - permissions: - id-token: write - steps: - - uses: actions/download-artifact@v8 - with: - name: ${{ needs.publish.outputs.dist-artifact }} - path: dist - - uses: pypa/gh-action-pypi-publish@release/v1 - with: - skip-existing: true - - report-pypi: - needs: [publish, pypi] - if: >- - !cancelled() && needs.publish.outputs.pypi-publish == 'true' - uses: PyDevices/.github/.github/workflows/reusable-report-pypi-result.yml@publishing-vN - with: - distribution-name: pydevices-audiodsp - version: ${{ needs.publish.outputs.version }} - pypi-result: ${{ needs.pypi.result }} - secrets: inherit -``` - -The `pypi` job runs through the repository's `pypi` **GitHub Environment** -(a required maintainer review, deployment limited to `v*` tags; check with -`gh api repos/PyDevices//environments/pypi`) and authenticates with -**Trusted Publishing** (OIDC, no stored PyPI credential). The PyPI project's -publisher is owner `PyDevices`, the repository, workflow -`publish-release-packages.yml`, environment `pypi`. - -Because the `v*` tag rule applies, a `workflow_dispatch` retry from `main` -cannot reach the `pypi` environment; a retry that needs PyPI dispatches on the -tag, and so runs the caller file as it was at that tag. - -Turning on `pypi-publish` for a package therefore takes three things: the -input, the two caller jobs above, and a matching Trusted Publisher on the PyPI -project. Miss one and the run fails at the PyPI end or skips the upload, never -silently publishes somewhere else. - -TestPyPI is unaffected: it authenticates with `TESTPYPI_API_TOKEN`, not -Trusted Publishing, so its upload stays in the coordinator. - -## The LVGL model - -`lvgl-python`, `lvgl-micropython`, and `lvgl-circuitpython` all consume -generated bindings from `lvgl-bindings`, which is the **single writer** of -the generated C, the `.pyi` stub, and the sync-relevant support files -(`lv_conf.h`, `fs_driver.py`, the `lvgl` submodule pointer). Hand-editing -any of that in a consumer repo is overwritten by the next sync — see -CONTRIBUTING.md. `display_driver.py` is not in that set: it lives in -pydevices `lib/` and publishes with the pydevices wheel and mip package. - -The trigger is explicit dispatch, not a webhook chain: - -```text -lvgl-bindings: Release bindings (workflow_dispatch, bindings-ref, publish=true) - -> validates the exact ref, runs the linux-integration matrix, - runs the non-mutating release_dry_run.sh gate - -> if publish=true: mints a GitHub App token scoped to lvgl-python - and dispatches lvgl-python's sync-and-release.yml - -f lvgl_bindings_ref= -f mode=release - -lvgl-python: Sync and release (workflow_dispatch, lvgl_bindings_ref, mode) - -> ./scripts/sync_from_lvgl_bindings.sh --ref - -> writes/commits LVGL_BINDINGS_COMMIT, VERSION, generated/lvgl_python.c, - generated/lvgl.pyi, lv_conf.h, fs_driver.py, lvgl - -> builds + runs its test suite + the lvgl-bindings smoke suite - -> mode=release: mints an App token, publishes vX.Y.Z (App token, same - reasoning as tag-release.yml -- GITHUB_TOKEN would not trigger - publish-release-packages.yml) -``` - -`mode` also accepts `dry-run` (validate only, nothing committed) and `sync` -(commit the sync, no release) for exercising the pipeline without publishing. -`lvgl-micropython` and `lvgl-circuitpython` each carry their own -`scripts/sync_from_lvgl_bindings.sh` and an `LVGL_BINDINGS_COMMIT` file -recording what they last synced, but neither has a `.github/workflows/` -directory of its own — their sync is run by hand, not on a schedule or -dispatch chain. - -`lvgl-bindings/check-dispatch-token.yml` is a read-only diagnostic (does the -App token reach `lvgl-python`?) kept until the App path has enough track -record to retire it; it does not trigger anything. - -The previously documented `trigger-lvgl-python-release.yml` and -`LVCPYTHON_MOD_DISPATCH_TOKEN` (a PAT-based dispatch mechanism) are -**retired** — `lvgl-bindings` has no such workflow and no such secret -reference today. The App-token dispatch above replaced it. - -Once `lvgl-python`'s Release publishes, `publish-release-packages.yml` there -runs the ordinary `native-and-wasm` build for `pydevices-lvgl` with no -`mip-profile` — `lvgl-python` does not publish to the MIP index. - -## The MIP publication queue - -Publishing to the MIP index is centralized in `PyDevices/mip` and -**serialized**: `process-mip-publication-request.yml` runs in the -`pydevices-mip-publication-queue` concurrency group with -`cancel-in-progress: false`, so two releases that both want MIP entries never -race — the second waits for the first to finish. - -### Request → queue → sync - -`reusable-request-mip-publication.yml` (called from a publishing repository -with `secrets: inherit`) mints an App token scoped to `mip` and dispatches -one `repository_dispatch` per profile in the (possibly comma-separated) -`mip-profile` input — `audiocomponents` sends two in the same release. That -dispatch happens for **final releases only**: the request's `prerelease` input -is true for a `.devN`/`rcN`/`aN`/`bN`, and then the MIP leg is skipped, so a -pre-release never reaches the MIP index (mip installs its `latest.json`). -Since `publishing-v15` the same request also dispatches each profile to the -library index, `PyDevices/lib`, **for every release, pre-releases included** -(see [The library index](#the-library-index)). A repository whose profile the -MIP lockfile doesn't know (palettes and pdwidgets, now folders of `pydevices`) -sets `legacy-index: false` and sends only the library leg. - -The queue consumer, `reusable-synchronize-mip-package.yml`: - -1. Checks out `mip` at the **`PyDevices` branch tip**, not the SHA frozen at - dispatch time. This is deliberate: a `repository_dispatch` payload freezes - `github.sha` at creation time, so the second of two queued publications - (audiocomponents' two profiles) would otherwise check out a tree that predates - the first one's lockfile commit and lose its push as a non-fast-forward — - this happened in practice and left `audioinstruments` a release behind - `audioeffects` in the live index for two days. -2. Records the release in `pydevices-lock.json` — the source of truth for - "what version of each package is currently live." Every publishable - package must already have an entry in the lockfile; a new profile is - added there deliberately, not auto-created. The entry's `repository` key - is also a **guard**: a dispatch whose source repository disagrees is - refused (`profile 'audioinstruments' is locked to PyDevices/audioif, not - PyDevices/audiocomponents`). - - That guard is what you meet when a package **moves repositories**, and it - fails in a place you are not watching: the releasing repository's - `request-mip-publication` job reports success, and only `mip`'s own run - goes red. The order is: tag in the new repository, then repoint - `repository` and `ref` in `pydevices-lock.json` on `mip`'s `PyDevices` - branch, then re-run the failed `mip` runs. -3. **Re-synchronizes every locked source**, not just the one being released: - it clones each `{repository, ref}` pair from the (now-updated) lockfile - and runs `synchronize_mip_package.py` for each, because `build.py` - compiles every `manifest.py` in the tree at once — there is no way to - rebuild one package's index entry in isolation. -4. Removes the temporary `.publication-sources` checkouts, builds the - complete latest-only index, and verifies every expected manifest produced - an index entry and that each locked package's index version matches the - lockfile. -5. Commits the lockfile with **rebase-and-retry**, not fail-fast: up to 3 - attempts of `git push` → on rejection, `git pull --rebase` → retry. The - concurrency group already serializes same-repo runs, but this covers a - push that lands from elsewhere between checkout and push. -6. Assembles and uploads the GitHub Pages artifact; a separate - `deploy-complete-mip-index` job in the caller workflow deploys it via - `actions/deploy-pages@v4`. - -### The library index - -`PyDevices/lib` serves the library index (`https://pydevices.github.io/lib`, -`lib.omnipython.org` once that domain points at it) and is built beside the MIP -index, not from it. Its `publish.yml` takes the same `publish-package-release` -dispatch, queues in the `omnipython-lib-publication-queue` group (no -cancel-in-progress), records the release's profile and tag in `pins.json`, and -rebuilds every package from the pins with `tools/libbuild.py` from the -`pydevices` checkout: `pydevices`, `pydevices-desktop`, `bledev`, `midiports`, -the `usb-*` packages, every board config and driver, `appkit`, `palettes` and -`pdwidgets` from the `pydevices` tag (a `palettes-vX.Y.Z` or `pdwidgets-vX.Y.Z` -tag replaces just that folder), plus `pygraphics`, `audioinstruments` and -`audioeffects`. It deploys with `actions/deploy-pages`, then commits `pins.json` -so the file says what is live. A profile is refused if its repository isn't the -one `pins.json` names for it. To rebuild without a release, run `publish.yml` -by hand with the inputs empty; to try a library commit before it is released, -give `test-root-ref`, which deploys the build and records nothing. - -### Manual dispatch - -`process-mip-publication-request.yml` also accepts `workflow_dispatch` for -recovery, with `profile` as a fixed choice input covering every current -profile: - -```bash -gh workflow run process-mip-publication-request.yml \ - --repo PyDevices/mip \ - -f source-repository=PyDevices/ \ - -f source-ref=vX.Y.Z \ - -f version=X.Y.Z \ - -f profile= -``` - -Use a direct central dispatch only for recovery after confirming the source -tag and profile are correct. The normal entry point is always a publishing -repository's own release, not this fallback. - -## Retrying an interrupted publication - -Never move the tag or rebuild the same version from a different commit. -Retry the coordinator with the exact existing tag: - -```bash -gh workflow run publish-release-packages.yml \ - --repo PyDevices/ \ - -f release-ref=vX.Y.Z -``` - -The workflows check out that tag, revalidate it, and use -`skip-existing: true` for TestPyPI. This safely completes a partial upload -without replacing files that TestPyPI already accepted. - -If only the current MIP run or Pages deployment failed, retry its failed -jobs: - -```bash -gh run rerun --repo PyDevices/mip --failed -``` - -If the shared reusable-workflow ref changed after the failed run, GitHub's -**rerun** operation may remain pinned to the reusable workflow commit -selected by the original attempt. Start a fresh source exact-tag retry, or -use the manual MIP dispatch above. - -## Correcting a bad release - -Published TestPyPI files and released source tags are immutable in practice. -Deleting a GitHub Release does not remove registry files, and moving a tag -would make the release irreproducible (and the tag ruleset would refuse the -move on this repository's own `publishing-v*` tags in any case). - -To correct a release: - -1. Fix the source on `main`. -2. Run "Prepare release" again for a new version. -3. Repeat the release-PR flow (or the manual fallback). -4. Yank an unusable TestPyPI version only when necessary and document why; do - not attempt to upload replacement files under the same version. - -Publishing the corrected MIP version replaces the affected package's `latest` -entry when the central latest-only index is deployed. - -## The Release Health dashboard - -Every `publish-release-packages` run — success or failure — ends with a -`report-release-health` job (`continue-on-error: true`, so a reporting -failure never fails the release itself) that dispatches a -`repository_dispatch` of type `release-health` to `PyDevices/.github`, with -the run's outcome for each stage (TestPyPI, assets, MIP, PyPI). A release -headed for production PyPI reports `pypi` as `pending` there, because that -upload runs later, in the caller; the caller's `report-pypi` job then sends a -partial report (`reusable-report-pypi-result.yml`) that fills in the outcome on -the same row. This -repository's `release-health.yml` folds that payload into -[`release-health/data.json`](../release-health/data.json) and regenerates -[`RELEASE_HEALTH.md`](../RELEASE_HEALTH.md) at the repo root — one row per -distribution, linking back to the run that produced it. - -## Troubleshooting - -| Symptom | Meaning and response | -|---|---| -| No publication run after pushing a tag | A tag alone is not the trigger. Publish the GitHub Release. | -| `VERSION contains ... but the release tag is ...` | Fix and commit `VERSION`, then create a **new** tag/version via a fresh "Prepare release" run. Do not move a published tag. | -| TestPyPI `invalid-publisher` | The job attempted OIDC Trusted Publishing without a matching publisher. TestPyPI still uses `__token__` + `TESTPYPI_API_TOKEN`. | -| TestPyPI authentication failure | Confirm that `TESTPYPI_API_TOKEN` exists in that source repository, belongs to `bdbarnett`, has permission for the project, and has not expired or been revoked. | -| TestPyPI duplicate-file response | Retry with the current coordinator, which sets `skip-existing: true`; otherwise publish a new version. | -| Production PyPI publish did not run | Check four things: `pypi-publish: true` on the caller, the caller's own `pypi` job (see "Production PyPI" above), the release is a final version (not `.devN`/`{a,b,rc}N`), and the `pypi` environment's required reviewers approved the run. | -| PyPI `invalid-publisher` with `job_workflow_ref` naming `PyDevices/.github` | The upload ran inside the reusable workflow, which PyPI cannot match. Move it to the caller's `pypi` job on a publishing tag that has `pypi-gate` ("Production PyPI" above). | -| MIP request is queued | Expected — the central concurrency group processes publication requests serially. | -| MIP validation sees `.publication-sources/manifest.py` | The shared synchronization job did not remove temporary checkouts; start a fresh run on a current `publishing-v*` tag. | -| Pages setup or action download returns 429/503/504 | Usually transient GitHub infrastructure trouble. Retry the failed MIP job and verify the live index afterward. | -| Source workflow succeeds but MIP is still old | Inspect the downstream `PyDevices/mip` run and wait for the Pages deployment; source success only confirms dispatch. | -| A native wheel platform is missing | Fix the wheel matrix and publish a new version. TestPyPI cannot accept a replacement with an existing filename. | -| `RELEASE_HEALTH.md` shows a stale row | The reporting job runs `continue-on-error`; a real failure elsewhere in the chain does not block it, but a `PyDevices/.github` outage or a missing App installation would. Check the linked run URL directly. | - -## Planned: real MIP dependencies instead of `require()` inclusion - -**Not started.** Cross-repository `require()`s were removed in the meantime, so -MIP packages currently have **no** dependencies at all — see "Current state" -below. - -### The finding - -`mip`'s package format supports install-time dependencies, and the client -resolves them. `utils/mip.py` walks `package_json["deps"]` and recursively -installs each entry, which is how board-config `package.json` files work -(`"deps": [["pydevices", "latest"]]`). - -`build.py` never emits that field. The string `deps` appears in it exactly -once, in a comment on the format: - -``` -# "deps": [ <-- not used by micropython-lib packages -``` - -`require()` in a `manifest.py` is a **build-time include**, inherited from the -manifest system used to freeze modules into firmware. It folds the required -package's files into the requiring package and emits one flat file list. - -So the three layers disagree: the format supports deps, the client resolves -deps, the builder never writes them. - -### What that produced - -Before removal, `pdwidgets` required `pydevices`, `pygraphics`, and `palettes` -— and its index entry was **212 files with `deps: null`**, carrying a complete -copy of all three. A board installing `pdwidgets` and `pydevices` wrote every -`lib/` file twice, and `pdwidgets` could not be updated without re-fetching -everything it had absorbed. - -### Two ways to fix it - -| Approach | Scope | Cost | -|---|---|---| -| **Patch `build.py`** so `require()` emits a `deps` entry rather than inlining | The whole index | Changes behaviour for the ~25 inherited upstream packages too — `aioble` and friends require each other — and is a permanent fork divergence to carry against upstream | -| **Emit hand-written `package.json`** for PyDevices packages, as board configs already do, bypassing the manifest path | PyDevices packages only | Leaves upstream untouched, but means maintaining a second publishing path alongside `synchronize_mip_package.py` | - -The second is narrower and does not touch inherited files, which -[repo-layout.md](repo-layout.md) protects. The first is more honest if the -index is to behave like a package index generally. - -Either way, verify against a real device install before switching: `deps` -resolution is client-side, so an older `mip` on a board may not honour it. - -### Current state - -Cross-repository `require()`s are removed from every profile. Consequences: - -- **MIP:** `mip.install("pdwidgets")` installs *only* `pdwidgets`. Its - dependencies must be installed by hand — the README and `docs/index.md` now - say so explicitly. -- **pip:** unaffected. `pdwidgets/pyproject.toml` keeps real dependencies and - pip resolves them normally. -- `pydevices-desktop` still requires `pydevices`. That one is intra-repository - — both are generated from the same source tree — so it is inclusion of code - that ships together anyway, not another repo's package. - -## Planned: type stubs for every published package - -**Not started. Recorded so the reasoning is not re-derived.** - -Intent: ship `.pyi` stubs for all published packages so pip consumers get real -type information. - -### Why stubs rather than annotations - -These packages run on MicroPython, where annotations cost bytecode and RAM. -That is why the convention is types in docstrings, not signatures. A `.pyi` -file carries the same information for a type checker, costs nothing at -runtime, and never reaches a device — MIP installs `.py` files only, verified -against the live index. - -The org already does this elsewhere: `lvgl-bindings` generates `lvgl.pyi` and -syncs it into `lvgl-python`, and `tools/typings/` in the archived pydevices-examples repo holds a -tree of `.pyi` for MicroPython builtins. - -### The `py.typed` constraint - -Maintainer preference is **no `py.typed` files**. PEP 561 makes that a real -constraint rather than a formatting choice: a type checker ignores `.pyi` -bundled inside a regular package unless `py.typed` is present. Stubs dropped -into the `pydevices` wheel without the marker would ship and never be read. - -Two mechanisms avoid the marker: - -| Mechanism | pip-installable | Needs `py.typed` | -|---|---|---| -| **GitHub Release assets** — `.pyi` attached per tag, consumer points `mypy_path` / `stubPath` at them | no, manual setup | no — not a distribution | -| **A `pydevices-stubs` distribution** — importable dirs named `multimer-stubs/`, `displaydev-stubs/`, … | yes | no — PEP 561 exempts stub-only packages | - -Maintainer picked release assets when asked. The `-stubs` route was surfaced -afterwards and satisfies both constraints more completely; decide between them -before starting. - -### Scope - -Roughly 633 functions across `appdev`, `audiodev`, `displaydev`, and -`multimer`. `stubgen` (mypy) is the starting point, but on unannotated source -it emits mostly `Any`, which is worth nothing to a consumer — the work is -hand-writing signatures. Treat generation as scaffolding, not output. - -`displaydev` is partly annotated already (101 of 310 functions), clustered in -the host backends — `sdldisplay` 19/42, `pgdisplay` 17/33 — while MCU-facing -files such as `windisplay` are at 0/27. Those inline annotations are a -reasonable seed for its stubs, and `busdisplay` at 16/26 is worth a look on -its own, since that is MCU code carrying a cost the convention says it should -not. - -### What prompted this - -`lib/multimer/py.typed` was removed in the 0.1.4 cycle. It claimed the package -shipped inline types while annotating 0 of 151 functions, which makes checker -inference strictly worse than no marker: `py.typed` tells a checker to trust -inline types and stop looking for stubs. - -## How the release chain is wired - -`reusable-publish-release-packages.yml` is the whole per-repository release -chain: it resolves the tag, dispatches the matching build (`build-pure-python` -/ `build-native-and-wasm` / `build-pydevices-multi`, exactly one of which runs -per invocation, selected by `build-kind`), publishes to TestPyPI, attaches -release assets, requests MIP publication, decides whether production PyPI -should follow, and reports to Release Health — all as jobs inside that one -reusable workflow, not separate coordinators. The production PyPI upload -itself is the one step that has to live in the caller (see "Production PyPI"). It replaced five near-identical copies of -the same three jobs. - -Changing a reusable workflow contract is an automation rollout, not a package -release. Validate it independently, create a new stable publishing tag when -the contract changes, update coordinators deliberately, and only then use -that tag for future package versions. - -## Shared implementation reference - -For the workflows *outside* this release chain — the Pages `deploy.yml` -pipeline, test and lint jobs, and validators — see [workflows.md](workflows.md). - -| Workflow | Responsibility | -|---|---| -| `reusable-prepare-release-pr.yml` | Open the release PR: compute the version suggestion, write `VERSION` + `CHANGELOG.md`, push, open/update the PR | -| `reusable-tag-on-release-merge.yml` | On a merged `VERSION` change, create the `vX.Y.Z` tag and GitHub Release with the App token | -| `reusable-publish-release-packages.yml` | The whole release chain: resolve the tag, build, publish to TestPyPI, attach assets, request MIP publication, tell the caller whether to publish to PyPI, report health | -| `reusable-report-pypi-result.yml` | Report a caller-side PyPI upload's outcome to Release Health | -| `reusable-build-pure-python-distribution.yml` | Build, check, clean-install, and upload one wheel/sdist artifact | -| `reusable-build-native-and-wasm-wheels.yml` | Build Linux, Windows, Android, and WASM (Pyodide) wheels into one validated artifact | -| `reusable-build-pydevices-distributions.yml` | Discover `pydevices/lib` leaves and `utils` desktop payload; build every exact-version distribution | -| `reusable-request-mip-publication.yml` | Dispatch the exact repository, ref, version, and profile(s) to the central MIP queue | -| `reusable-synchronize-mip-package.yml` | Re-synchronize every locked source, validate the complete latest-only index, create one lockfile commit, and stage the Pages artifact | -| `reusable-validate-pyscript-filesystem-toml.yml` | Reject stale generated PyScript filesystem mappings | -| `release-health.yml` | Fold a `release-health` dispatch into `release-health/data.json` and regenerate `RELEASE_HEALTH.md` | - -The shared scripts use descriptive operation names and filesystem discovery; -new publishable sources are picked up without an include list. PyScript TOMLs -contain runtime filesystem mappings, not package metadata. diff --git a/docs/reference/github-org.md b/docs/reference/github-org.md new file mode 100644 index 0000000..9d8ef17 --- /dev/null +++ b/docs/reference/github-org.md @@ -0,0 +1,90 @@ +# Where things live on GitHub + +How the [PyDevices](https://github.com/PyDevices) organization is set up: +where to ask, where bugs go, and which pieces live in this repository. The +full list of repositories, grouped by tier, is generated from +`data/repos_db.json` and published as the +[ecosystem page](https://github.com/PyDevices/pydevices/blob/main/docs/how-it-works/ecosystem.md) +and on the [organization profile](https://github.com/PyDevices); this page +doesn't repeat it. + +## Quick answers + +| I want to... | Go here | +|---|---| +| Ask a question, propose an idea, show something I built, or reach the maintainer | [pydevices Discussions](https://github.com/PyDevices/pydevices/discussions), the organization's one board | +| Share a one-off example too narrow for the official examples | The *Recipes* category in [pydevices Discussions](https://github.com/PyDevices/pydevices/discussions) | +| Report a bug or ask for something that doesn't work yet | Issues on the repository it affects; the Bug report and Feature request templates apply by default | +| See what is planned | The `ROADMAP.md` at the root of each repository | +| Read the shared contribution guidelines | [CONTRIBUTING.md](../../CONTRIBUTING.md) | +| Read the organization's profile | [profile/README.md](../../profile/README.md), shown on [github.com/PyDevices](https://github.com/PyDevices) | +| Read a library's manual | The project's Read the Docs site, for example [pygraphics.readthedocs.io](https://pygraphics.readthedocs.io); the [documentation overview](https://pydevices.readthedocs.io) links the rest | +| Try the libraries without installing | The [PyScript gallery](https://pydevices.github.io/gallery/pyscript/) | +| Find board and driver documentation | [pydevices `docs/`](https://github.com/PyDevices/pydevices/tree/main/docs) | + +## Discussions and issues + +**Discussions** are on in one repository, `pydevices`: questions, ideas, +show-and-tell, and reaching the maintainer. Every other repository has them +off, so there is one place to look; `SUPPORT.md`, `CONTRIBUTING.md` and the +design docs all point there. Its *Recipes* category is a low-friction home for +one-off how-tos; one that gets traction can be promoted into a real example. + +**Issues** are for bugs and for things you need that don't work yet. Planned +enhancements go in each repository's `ROADMAP.md` instead; a request that +becomes planned work moves there and its issue closes with a link. The default +issue templates (`bug.yml`, `feature.yml`) live in this repository's +[`.github/ISSUE_TEMPLATE`](../../.github/ISSUE_TEMPLATE) and apply to any +repository that doesn't define its own. If you aren't sure where an issue +belongs, open it on [pydevices](https://github.com/PyDevices/pydevices/issues) +and it will be routed. + +**GitHub Projects** are off, so public work tracking is Issues, Discussions and +the roadmaps. **GitHub Packages** isn't used: installables go to TestPyPI and +PyPI for CPython, and to the `mip` and `lib` indexes for MicroPython. + +## Read the Docs + +Sphinx documentation for the organization overview, `pygraphics`, `palettes` +and `pdwidgets` publishes to Read the Docs as one site; see [build and publish +documentation](../guides/build-and-publish-docs.md). Read the Docs connects +through its [Community GitHub App](https://github.com/apps/read-the-docs-community), +installed on all of the organization's repositories (see the +[installed apps](https://github.com/organizations/PyDevices/settings/installations)), +so push and pull-request events arrive without a per-repository webhook. A new +project is imported from the Read the Docs dashboard, which lists repositories +through the app. A project still on a legacy webhook can +[migrate to the app](https://app.readthedocs.org/accounts/migrate-to-github-app/). +Each project's build is `.readthedocs.yaml` plus `docs/conf.py`, with no +secrets. + +## Pages sites + +Product repositories publish a landing page at `pydevices.github.io//` +from the portal repository, [PyDevices.github.io](https://github.com/PyDevices/PyDevices.github.io). +This repository doesn't serve a Pages site of its own. The pages share one +chrome: `assets/css/site.css` and `assets/js/site-chrome.js` here are the +master copies, and the portal's `assets/chrome/` files are their sync targets. +Each page mounts empty `#pydevices-site-header` and `#pydevices-site-footer` +elements and the script fills them in; `theme-toggle.js` binds +`#theme-toggle`. Which pages exist, and where each is served, comes from +`data/repos_db.json` through `scripts/generate_sites.py` +([set up a checkout](../get-started/set-up.md#regenerate-the-org-sites)). + +## Licensing + +The organization's repositories are MIT, using GitHub's standard +single-author template, so GitHub detects it as MIT rather than +`NOASSERTION`. Keep the root `LICENSE` as the plain template: GitHub matches +it against the exact text, and even a short added paragraph can drop the match +below the confidence threshold. A few files carry code from other authors, for +example `gallery`'s `lib/utils/tft_text.py`, `tft_write.py` and `tft_bitmap.py`. +Those keep their own MIT header with the attribution, and the header governs +the file. New outside code is credited as described under [code from other +projects](repo-layout.md#code-from-other-projects). + +## Topics + +Each repository sets GitHub topics for discoverability (for example `lvgl`, +`micropython`, `circuitpython`). Read them in a repository's sidebar rather +than from a list here, which would go stale. diff --git a/docs/doc-style.md b/docs/reference/heading-emoji.md similarity index 80% rename from docs/doc-style.md rename to docs/reference/heading-emoji.md index 7b969b5..63f6661 100644 --- a/docs/doc-style.md +++ b/docs/reference/heading-emoji.md @@ -1,4 +1,4 @@ -# Doc style: emoji +# Heading emoji Docs across PyDevices use a small, fixed set of emoji as landmarks on section headings — not decoration. Each emoji has exactly one meaning and is @@ -29,18 +29,12 @@ Rules of thumb: tag none of them rather than picking one arbitrarily — repeating the same emoji on adjacent headings reads as noise, not a landmark. -## Where this applies (amended 2026-08-29, Gate 4 decision) - -Practice across the organization settled differently from this document's -original ambition, and the convention below now describes what is real -rather than what was hoped: +## Where this applies - **READMEs are emoji-free.** A README renders on GitHub first, where - restraint reads better. This is the prescribed convention (a handful of - READMEs still carry stray emoji; Phase 4 removes them as each repo gets - its pass). + restraint reads better. This is the prescribed convention (remove a stray one when you edit a README). - **Docs-site pages** (Read the Docs and portal pages) may use the landmark set - above, sparingly, on headings only — the original rules (one per + above, sparingly, on headings only — the rules above (one per heading, only the fixed set, never in body text or code) apply there. - A repo using none at all is compliant. The failure mode this document guards against is *decorative scatter* — emoji in body text, list diff --git a/docs/reference/release-workflows.md b/docs/reference/release-workflows.md new file mode 100644 index 0000000..008fab8 --- /dev/null +++ b/docs/reference/release-workflows.md @@ -0,0 +1,108 @@ +# Release workflows: inputs, secrets and tags + +The release chain's moving parts, as tables. How they fit together is in +[the release chain](../how-it-works/release-chain.md). + +## What each repository publishes + +| Source repository | TestPyPI | MIP index | +|---|---|---| +| `pydevices` | `pydevices` (all of `lib/`) and `pydevices-desktop` | `pydevices` | +| `pygraphics` | `pydevices-pygraphics`: Linux, Windows, Android and WASM wheels | pure-Python `pygraphics` | +| `audiodsp` | `omnipython-audiodsp`: Linux, Windows, macOS, Android and WASM wheels | nothing; the native modules are firmware, built from the usermod source | +| `audiocomponents` | `pydevices-audioinstruments` and `pydevices-audioeffects`, pure Python, from one repository | `audioinstruments` and `audioeffects` | +| `omnipython-lvgl` | `pydevices-lvgl`: Linux, Windows, Android and WASM wheels | nothing | +| `mpftp` | `pydevices-mpftp` wheel and sdist | nothing | + +The `palettes` and `pdwidgets` folders of `pydevices` publish separately, on +`palettes-vX.Y.Z` and `pdwidgets-vX.Y.Z` tags, to TestPyPI and the library +index. The library index (`PyDevices/lib`) takes every release of every +repository that asks it, pre-releases included. + +pip distribution names carry a prefix (`pydevices-` or `omnipython-`); MIP +package names don't. `pydevices` and `pydevices-desktop` keep the same name in +both. `mpftp` also publishes its VS Code extension with `publish-vsix.yml`, +outside this chain. + +## Inputs of `reusable-publish-release-packages.yml` + +| Input | Meaning | +|---|---| +| `build-kind` | `pure-python`, `native-and-wasm` or `pydevices-multi`; exactly one build runs | +| `distribution-name` | The TestPyPI distribution name | +| `import-name` | `pure-python` only: the module the smoke test imports | +| `wheel-filename-prefix` | `native-and-wasm` only, for example `pydevices_pygraphics` | +| `expected-wheel-count` | `native-and-wasm` only; `0` accepts any non-empty wheel set | +| `install-dependencies` | `pure-python` only: install dependencies before the smoke test | +| `checkout-submodules` | Check out submodules before building | +| `mip-profile` | A synchronization profile, or a comma-separated list when a repository publishes several packages; empty skips MIP and the library index | +| `release-ref` | An exact `vX.Y.Z` tag, for a `workflow_dispatch` retry; empty on a release event | +| `working-directory` | `pure-python` only: the directory with `pyproject.toml` | +| `pypi-publish` | Mark final releases for production PyPI; see [publish to production PyPI](../guides/publish-to-pypi.md) | + +A typical call: + +```yaml +jobs: + publish: + uses: PyDevices/.github/.github/workflows/reusable-publish-release-packages.yml@publishing-vN + with: + build-kind: pure-python # or native-and-wasm, pydevices-multi + distribution-name: pydevices-palettes + import-name: palettes + mip-profile: palettes # omit to skip MIP + release-ref: ${{ inputs.release-ref }} + secrets: inherit +``` + +## Secrets + +| Secret | Where | Purpose | +|---|---|---| +| `PYDEVICES_APP_ID`, `PYDEVICES_APP_PRIVATE_KEY` | Every publishing repository and `PyDevices/mip` | Mint short-lived GitHub App tokens: to open release PRs, create tags and Releases (so `on: release` fires), dispatch the MIP queue and send the Release Health report | +| `TESTPYPI_API_TOKEN` | Every repository that uploads to TestPyPI | API token for TestPyPI | +| `VSCE_PAT`, `OVSX_PAT` | `mpftp` only (`publish-vsix.yml`) | VS Marketplace and Open VSX; both optional, the steps skip when unset | + +TestPyPI uses token authentication (`user: __token__`, the token as the +password, `repository-url: https://test.pypi.org/legacy/`, +`skip-existing: true`), not Trusted Publishing. Production PyPI uses Trusted +Publishing; see [publish to production PyPI](../guides/publish-to-pypi.md). + +## Tags + +`publishing-vN` tags carry the reusable workflows. They're immutable: a tag +ruleset named "publishing tags are immutable" blocks moving or deleting one. +Cut a new one with `scripts/cut_publishing_tag.sh N` from a clean `main`. + +There's no single current tag; each caller sits on the tag it was last moved +to. To see which, in a clone of a repository: + +```bash +grep -rn "PyDevices/\.github.*@publishing" .github/workflows +``` + +`git ls-remote --tags https://github.com/PyDevices/.github 'publishing-v*'` +lists every tag. `mip` keeps its tools-ref in two places +(`process-mip-publication-request.yml` and `deploy-mip-index-to-pages.yml`); +move them together, because the deploy workflow rebuilds every package and an +older tools ref overwrites what a publication wrote. + +## Tag history to know + +- **Never pin `publishing-v10`.** It called its sibling workflows by `./` + path, which resolves against the consumer's repository, so a consumer's run + died at startup with no jobs ([.github#47](https://github.com/PyDevices/.github/issues/47)). + `publishing-v11` and later name the siblings by full path at the tag's own + name. +- **`publishing-v9` and earlier** name an older tag in their own sibling + references, so a caller on one runs that older tag's builders. For a retry + against v9 or earlier, check the nested references first. +- **`publishing-v11`** carries the version-grammar guard before tagging, the + `## Unreleased` heading converted by the release PR, a release-health push + that re-folds instead of losing a report, and the MCU split of the MIP + packages. Its `publishing-tools-ref` defaults still name v6, which is + harmless because the scripts are identical. +- **`publishing-v14`** added `top-level = true` sections to `mip-split.toml`. + **`publishing-v15`** added the library-index request. Every tag from + `publishing-v1` still exists, so a release cut before a contract change can + be retried against the contract it was built with. diff --git a/docs/repo-layout.md b/docs/reference/repo-layout.md similarity index 96% rename from docs/repo-layout.md rename to docs/reference/repo-layout.md index d4d9364..3cd6071 100644 --- a/docs/repo-layout.md +++ b/docs/reference/repo-layout.md @@ -48,10 +48,9 @@ Long-form notes and guides belong under `docs/`. written for users, one bullet of at most three lines each. Bugs and unmet needs stay in issues. A repository with nothing planned says so in a line, and its README links the file. -- **Do not add `RELEASE_NOTES.md`.** All three that existed carried a bare - `## Unreleased` heading, never recorded a shipped version, and were linked from - nowhere; they were deleted in August 2026. Releases use `--generate-notes`, so - a hand-maintained file is a second source of truth that goes stale immediately. +- **Do not add `RELEASE_NOTES.md`.** Releases use `--generate-notes` and + `CHANGELOG.md`, so a hand-maintained notes file is a second source of truth + that goes stale immediately. - Name narrative and topic documents under `docs/` with lowercase kebab-case, for example `building-wheels.md`, `build-and-flash.md`, and `soft-reset-and-bring-up.md`. @@ -145,7 +144,7 @@ There is one way to credit outside code. Vendored code goes in `third_party//`, unchanged where possible, with the upstream license file and an `ORIGIN.txt` naming the upstream URL, the commit, and every change made. A root `NOTICE` lists each one, and the root `LICENSE` stays the -plain MIT template (see [Licensing](github-presence.md#licensing)), because a +plain MIT template (see [Licensing](github-org.md#licensing)), because a license file that points elsewhere is not one tools can read. A file that ports upstream code line by line keeps the upstream copyright diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md new file mode 100644 index 0000000..e25dc9c --- /dev/null +++ b/docs/reference/workflows.md @@ -0,0 +1,140 @@ +# GitHub Actions workflows + +What exists, in which repository, and why. The procedures are in +[release a package](../guides/release-a-package.md) and +[build and publish documentation](../guides/build-and-publish-docs.md); the +design of the release chain is in [the release chain](../how-it-works/release-chain.md). +Inputs, secrets and tags are in [release workflows](release-workflows.md). + +## Reusable workflows in this repository + +| Workflow | Purpose | +|---|---| +| `reusable-prepare-release-pr.yml` | Open the release PR: compute a version suggestion, write `VERSION` and `CHANGELOG.md`, open or update the PR | +| `reusable-tag-on-release-merge.yml` | On a merged `VERSION` change, create the `vX.Y.Z` tag and GitHub Release with the App token | +| `reusable-publish-release-packages.yml` | The whole release chain: resolve the tag, build, publish to TestPyPI, attach assets, request MIP publication, tell the caller whether to publish to PyPI, report health | +| `reusable-report-pypi-result.yml` | Report a caller-side PyPI upload to Release Health ([why the upload is caller-side](../guides/publish-to-pypi.md)) | +| `reusable-build-pure-python-distribution.yml` | sdist and wheel for a pure-Python package | +| `reusable-build-native-and-wasm-wheels.yml` | cibuildwheel: Linux, Windows, Android and Pyodide wasm32 | +| `reusable-build-pydevices-distributions.yml` | The `pydevices` and `pydevices-desktop` distributions, derived from `lib/` and `utils/` | +| `reusable-request-mip-publication.yml` | Dispatch into the publication queues of `mip` and `lib`, once per profile | +| `reusable-synchronize-mip-package.yml` | Record the release in `pydevices-lock.json`, re-sync every locked source, rebuild and validate the index, stage the Pages artifact | +| `reusable-validate-pyscript-filesystem-toml.yml` | Check a `*.toml` filesystem map against the tree it claims to mirror | + +Callers name a reusable by tag (`@publishing-vN`), never by branch. The build +workflows that need this repository's scripts check it out at +`publishing-tools-ref`, which defaults to the tag the workflow itself is at. + +## Workflows that run here + +| Workflow | Trigger | What it does | +|---|---|---| +| `checks.yml` | push to `main`, pull request | actionlint over the workflows; a YAML parse of each; the site generator's idempotency check against fresh clones of the repositories it writes to; a check that the reusables' references agree on one tag; `ruff` over `scripts/` and `tests/`; the script tests; the main documentation build | +| `release-health.yml` | `repository_dispatch` from a publish run | Fold the report into `release-health/data.json` and regenerate `RELEASE_HEALTH.md` | +| `hero-check.yml` | daily, and by hand | Load each published demo in a browser; open or update an issue when one is dead | +| `cold-eyes-review.yml` | quarterly, and by hand | Open the [review issue](../guides/maintenance-schedule.md#quarterly) | + +## Releasing + +Six repositories release through the reusable chain. Five of them +(`pydevices`, `pygraphics`, `audiodsp`, `audiocomponents` and `mpftp`) carry a +`prepare-release.yml`, `tag-release.yml` and `publish-release-packages.yml` +trio. The first two are one-job callers; `publish-release-packages.yml` supplies +what differs, the build kind, the distribution name and the MIP profile: + +| Repository | `build-kind` | Distribution | MIP profile | +|---|---|---|---| +| `pydevices` | `pydevices-multi` | `pydevices` (and `pydevices-desktop`) | `pydevices` | +| `pygraphics` | `native-and-wasm` | `pydevices-pygraphics` | `pygraphics` | +| `audiodsp` | `native-and-wasm` | `omnipython-audiodsp` | none | +| `audiocomponents` | `pure-python`, twice | `pydevices-audioinstruments`, `pydevices-audioeffects` | `audioinstruments,audioeffects` | +| `mpftp` | `pure-python` | `pydevices-mpftp` | none | +| `omnipython-lvgl` | `native-and-wasm` | `pydevices-lvgl` | none | + +`publish-release-packages.yml` triggers on a published GitHub Release, or on +`workflow_dispatch` with an exact `vX.Y.Z` tag to retry. `prepare-release.yml` +triggers only on `workflow_dispatch`, with an optional `version` input. +`tag-release.yml` triggers on a push to `main` that touches `VERSION`. +`omnipython-lvgl` has no prepare or tag workflow: its `release.yml` does both +(below). + +`audiodsp` runs a `parity` job (the `tests/parity/verify_*.py` gates and every +DSP probe) and a `credentials` job (confirms `TESTPYPI_API_TOKEN` is visible) +before its publish job. `audiocomponents` runs the same kind of gates and +credentials check before its two. + +Releases that don't use the trio: + +| Repository | Workflow | Role | +|---|---|---| +| `omnipython-lvgl` | `release.yml` | By hand: with `mode=dry-run`, run every gate and change nothing; with `mode=release`, also write the next version to `VERSION` on `main` and publish a GitHub Release, which starts `publish-release-packages.yml` | +| `pydevices` | `tag-package-release.yml`, `publish-package-release.yml` | Release the `palettes` and `pdwidgets` folders on their own `palettes-vX.Y.Z` and `pdwidgets-vX.Y.Z` tags: build the distribution from the folder at the tag and upload it to TestPyPI, then ask the library index for it | +| `pydevices` | `publish-lib-boards.yml` | Ask the library index for board configurations and drivers when they merge | +| `mpftp` | `publish-vsix.yml` | Build the VS Code extension and publish it to VS Marketplace and Open VSX (`VSCE_PAT`, `OVSX_PAT`, both optional) | +| `android-runner` | `release_apk.yml` | Build the APK that `pydevices/bin/android.py --install-apk` downloads | + +> A Release created with `GITHUB_TOKEN` doesn't trigger `on: release` +> workflows. That is why `reusable-tag-on-release-merge.yml` and +> `omnipython-lvgl`'s `release.yml` create the Release with an App token. + +## The MIP and library indexes + +| Repository | Workflow | Role | +|---|---|---| +| `mip` | `process-mip-publication-request.yml` | Queue consumer (`pydevices-mip-publication-queue` concurrency group, `cancel-in-progress: false`): synchronize a package, rebuild the index, deploy. Accepts `repository_dispatch` (from `reusable-request-mip-publication.yml`) or `workflow_dispatch` with `profile` as a fixed choice | +| `mip` | `deploy-mip-index-to-pages.yml` | Rebuild every locked package and publish the index. Triggers on the **`PyDevices`** branch, not `main` | +| `mip` | `validate-mip-package-index.yml`, `refresh-package-list.yml` | Check index integrity; refresh the committed package list from each deployed index | +| `mip` | `tests.yml`, `ruff.yml`, `commit-formatting.yml` | Inherited from micropython-lib; leave them alone | +| `lib` | `publish.yml`, `test.yml` | Build the library index from `pins.json` and deploy it; the builder's tests | + +## Testing and validation + +| Repository | Workflows | +|---|---| +| `pydevices` | `tests.yml`; `palettes.yml` and `pdwidgets.yml` for those folders; `appkit.yml`; `validate-pyscript-filesystem-toml.yml` | +| `pygraphics` | `tests.yml` (pure Python, then the native extension with `PYGRAPHICS_TEST_NATIVE=1`); `validate-pyscript-filesystem-toml.yml` | +| `audiodsp` | `test-cpython.yml`, `clean-build.yml`, `lint.yml`, `android-wheels.yml`; the parity gates also rerun at release time | +| `audiocomponents` | `tests.yml`, `lint.yml` | +| `omnipython-lvgl` | `generator-checks.yml`, `linux-integration.yml`, `font-drift.yml` | +| `gallery` | `manifests.yml` (install-manifest freshness) | +| `.github` (this repository) | `checks.yml` | + +## Pages + +| Repository | What it serves | +|---|---| +| `PyDevices.github.io` | The portal and every repository's landing page at `//`. Serves from `main` with no workflow | +| `gallery` | `deploy.yml`: the PyScript gallery | +| `mip` | `deploy-mip-index-to-pages.yml`: the package index | +| `lib` | `publish.yml`: the library index | + +Every other landing page is generated by +[`scripts/generate_sites.py`](../../scripts/generate_sites.py) into the portal +repository from [`data/repos_db.json`](../../data/repos_db.json). The entry's +`page` field says where a page goes: `portal-root`, `portal-subdir`, `self`, +`self-subpath` or `none`. A landing page is one `index.html` that references +`/assets/chrome/` and `/assets/img/logo.svg` at the portal root. + +## Documentation + +Read the Docs builds the main project, `palettes`, `pdwidgets` and `pygraphics` +on push through the organization's GitHub App, with `fail_on_warning: true`. +Repositories with a `docs.yml` build the same way in CI (`sphinx-build -W`), +so a warning fails the pull request before it can fail the published build. + +## Dependabot + +Each repository with workflows has a `dependabot.yml` that watches pinned +action versions, in one grouped weekly pull request. + +## Conventions + +- Pin actions to a major version (`actions/checkout@v7`). +- Logic shared by more than one repository belongs in a `reusable-*.yml` + here, called at a pinned `publishing-v*` tag. +- Path filters on `push` and `pull_request`; `workflow_dispatch` on anything + you might need to run by hand. +- A workflow that writes to another repository needs a token from `secrets`. + `GITHUB_TOKEN` can't reach across repositories and can't start another + workflow even within one, hence the App-token pattern for tagging, MIP + dispatch and the LVGL release. diff --git a/docs/workflows.md b/docs/workflows.md deleted file mode 100644 index bfc0f15..0000000 --- a/docs/workflows.md +++ /dev/null @@ -1,156 +0,0 @@ -# GitHub Actions workflows - -What exists, in which repository, and why. Procedures live elsewhere: -[publishing-automation.md](publishing-automation.md) for releases, -[building-docs.md](building-docs.md) for documentation. - -The inventory includes the native `audiodsp` CPython release caller. - -## Reusable workflows (`PyDevices/.github`) - -Callers pin a tag, not a branch, so a change here does not alter a release -until the tag moves. **`publishing-v9` is the newest tag, but there is no one -"current" pin** — repositories sit on v6, v8 and v9, and -[publishing-automation.md](publishing-automation.md#the-tag-contract) has the -map and the nested-ref trap that goes with it. Earlier tags all remain, so a -release cut before a contract change can still be retried against the exact -contract it was built with. -`publishing-v*` tags are immutable by policy and by an active tag ruleset on -this repository ("publishing tags are immutable") that blocks moving or -deleting them. - -| Workflow | Purpose | -|---|---| -| `reusable-prepare-release-pr.yml` | Open the release PR: compute a version suggestion, write `VERSION` + `CHANGELOG.md`, open/update the PR | -| `reusable-tag-on-release-merge.yml` | On a merged `VERSION` change, create the `vX.Y.Z` tag and GitHub Release with the App token | -| `reusable-publish-release-packages.yml` | The whole release chain: resolve the tag, build, publish to TestPyPI, attach assets, request MIP publication, tell the caller whether to publish to PyPI, report health | -| `reusable-report-pypi-result.yml` | Report a caller-side PyPI upload to Release Health ([why the upload is caller-side](publishing-automation.md#production-pypi-opt-in-protected-trusted-publishing)) | -| `reusable-build-pure-python-distribution.yml` | sdist + wheel for a pure-Python package | -| `reusable-build-native-and-wasm-wheels.yml` | cibuildwheel: Linux, Windows, Android, Pyodide wasm32 | -| `reusable-build-pydevices-distributions.yml` | The `pydevices` and `pydevices-desktop` distributions, derived from `lib/` and `utils/` | -| `reusable-request-mip-publication.yml` | Dispatch into `mip`'s serialized publication queue (one dispatch per profile) | -| `reusable-synchronize-mip-package.yml` | Record the release in `pydevices-lock.json`, re-sync every locked source, rebuild and validate the index, stage the Pages artifact | -| `reusable-validate-pyscript-filesystem-toml.yml` | Check a `*.toml` filesystem map against the tree it claims to mirror | -| `release-health.yml` (not reusable; own trigger) | Fold a `release-health` `repository_dispatch` into `release-health/data.json` and regenerate `RELEASE_HEALTH.md` | - -The build workflows that need shared scripts also check this repository out, -pinned by `publishing-tools-ref` — which defaults to `publishing-v6`, so -workflows and scripts always come from the same tag. - -## Releasing - -Each of seven repositories has a `prepare-release.yml` + `tag-release.yml` + -`publish-release-packages.yml` trio. The first two are near-identical -one-job callers; `publish-release-packages.yml` supplies only what differs — -the build kind, the distribution name, and the MIP profile: - -| Repository | `build-kind` | Distribution | MIP profile | -|---|---|---|---| -| `pydevices` | `pydevices-multi` | `pydevices` | `pydevices` | -| `palettes` | `pure-python` | `pydevices-palettes` | `palettes` | -| `pdwidgets` | `pure-python` | `pydevices-pdwidgets` | `pdwidgets` | -| `pygraphics` | `native-and-wasm` | `pydevices-pygraphics` | `pygraphics` | -| `audiodsp` | `native-and-wasm` + 2× `pure-python` | `pydevices-audiodsp`, `pydevices-audioinstruments`, `pydevices-audioeffects` | `audioinstruments,audioeffects` (on the native job only) | -| `lvgl-python` | `native-and-wasm` | `pydevices-lvgl` | — none | -| `mpftp` | `pure-python` | `pydevices-mpftp` | — none | - -`publish-release-packages.yml` triggers on a published GitHub Release, or on -`workflow_dispatch` with an exact `vX.Y.Z` tag to retry one. -`prepare-release.yml` triggers only on `workflow_dispatch` (optional -`version` input). `tag-release.yml` triggers on `push` to `main` touching -`VERSION`. - -`audiodsp` additionally runs a `parity` job (the four -`tests/parity/verify_*.py` acceptance/effects/streaming/biquad gates) and a -`credentials` job (confirms `TESTPYPI_API_TOKEN` is visible) before any of -its three publish jobs. - -Cross-repository release plumbing: - -| Repository | Workflow | Role | -|---|---|---| -| `lvgl-bindings` | `release.yml` | On explicit `workflow_dispatch` with `publish=true`, mint an App token scoped to `lvgl-python` and dispatch its `sync-and-release.yml` with the exact bindings commit | -| `lvgl-bindings` | `check-dispatch-token.yml` | Manual, read-only diagnostic that the App token can reach `lvgl-python` | -| `lvgl-python` | `sync-and-release.yml` | Sync generated bindings (`LVGL_BINDINGS_COMMIT`, `generated/*`, `lv_conf.h`, drivers), commit, and (`mode=release`) publish a Release with the App token | -| `mpftp` | `publish-vsix.yml` | Build the VSIX and publish to VS Marketplace / Open VSX (`VSCE_PAT` / `OVSX_PAT`, both optional) | -| `android-runner` | `release_apk.yml` | Build the APK that `pydevices/bin/android.py --install-apk` downloads | - -> `sync-and-release.yml` mints its own App token to create `lvgl-python`'s -> Release, which looks redundant and is not: **a Release created with -> `GITHUB_TOKEN` does not trigger `on: release` workflows**, so -> `publish-release-packages.yml` there would never fire and nothing would -> reach TestPyPI. The same reasoning is why `reusable-tag-on-release-merge.yml` -> mints an App token for every other repository's tag/Release step too. - -`lvgl-micropython` and `lvgl-circuitpython` each carry a -`scripts/sync_from_lvgl_bindings.sh` and their own `LVGL_BINDINGS_COMMIT`, -but neither has a `.github/workflows/` directory — their sync from -`lvgl-bindings` is run by hand, not dispatched by a workflow. - -## MIP index (`mip`) - -| Workflow | Role | -|---|---| -| `process-mip-publication-request.yml` | Serialized queue consumer (`pydevices-mip-publication-queue` concurrency group, `cancel-in-progress: false`): synchronize a package, rebuild the index, deploy. Accepts `repository_dispatch` (from `reusable-request-mip-publication.yml`) or `workflow_dispatch` with `profile` as a fixed choice of every current profile | -| `deploy-mip-index-to-pages.yml` | Compile and publish the index. Triggers on the **`PyDevices`** branch, not `main` | -| `validate-mip-package-index.yml` | Check index integrity | -| `tests.yml`, `ruff.yml`, `commit-formatting.yml` | Inherited from micropython-lib upstream — leave them alone | - -## Testing and validation - -| Repository | Workflow | Notes | -|---|---|---| -| `pydevices` | `tests.yml` | Also validates board MIP installers | -| `palettes`, `pdwidgets` | `tests.yml` | Sparse-checkout of sibling repos onto `PYTHONPATH` (`pdwidgets` needs four) | -| `pygraphics` | `tests.yml` | Runs the suite twice — pure Python, then the native extension via `PYGRAPHICS_TEST_NATIVE=1` | -| `audiodsp` | `tests.yml` | Also the source of the `parity` gate `publish-release-packages.yml` reruns at release time | -| `gallery` | `tests.yml`, `manifests.yml` | Gallery and install-manifest freshness | -| `pydevices`, `pygraphics` | `validate-pyscript-filesystem-toml.yml` | Standalone callers of the reusable validator. `pdwidgets` and `palettes` call the same reusable as a job inside `tests.yml` | -| `.github` (this repo) | `checks.yml` | actionlint over `.github/workflows`, a YAML-parse smoke test of every reusable, a site-generator idempotency check against checked-out siblings, and `ruff` over `scripts/` | - -## Pages - -**Two repositories publish Pages**, plus the portal: - -| Repository | What it serves | -|---|---| -| `PyDevices.github.io` | The portal **and every repository's landing page**, at `//`. Serves from `main`, no workflow | -| `gallery` | `deploy.yml` — the PyScript gallery: manifest audits and symlink replacement | -| `mip` | `deploy-mip-index-to-pages.yml` — the package index | - -Every other landing page is generated by -[`scripts/generate_sites.py`](../scripts/generate_sites.py) into the portal -repository from [`data/repos_db.json`](../data/repos_db.json). Where a page goes -is the entry's `page` field: `portal-root`, `portal-subdir`, `self`, -`self-subpath`, or `none`. There are no per-repo `deploy.yml` files and no -`gh-pages` branches; thirteen of each were removed once the portal took over -those paths. - -> A landing page is a single `index.html` referencing `/assets/chrome/` -> and `/assets/img/logo.svg` at the portal root. - -## Documentation - -ReadTheDocs builds the main project, `palettes`, `pdwidgets`, and `pygraphics` -on push via the org GitHub App, with `fail_on_warning: true`. Each repo's CI -builds its docs the same way (`sphinx-build -W`, from the repository root with -the paths in `.readthedocs.yaml`), so a warning fails the pull request before it -can fail the published build. - -## Maintenance - -`dependabot.yml` watches pinned action versions in the repositories that have -workflows, grouped into one weekly PR each. `mip` is excluded — its -workflows come from upstream. - -## Conventions - -- Pin actions to a major version (`actions/checkout@v7`). -- Logic shared by more than one repository belongs in a `reusable-*.yml` here, - called at a pinned `publishing-v*` tag. -- Path filters on `push` and `pull_request`; `workflow_dispatch` on anything you - might need to re-run by hand. -- A workflow that writes to another repository needs a token in `secrets`; - `GITHUB_TOKEN` cannot reach across repositories, and cannot trigger another - workflow even within one — hence the App-token pattern used for tagging, - MIP dispatch, and the LVGL sync dispatch. diff --git a/scripts/synchronize_mip_package.py b/scripts/synchronize_mip_package.py index 9693748..dcfb03c 100755 --- a/scripts/synchronize_mip_package.py +++ b/scripts/synchronize_mip_package.py @@ -36,8 +36,7 @@ class Profile: # by inclusion at build time, so these three shipped all of pydevices, # pygraphics, and palettes inside the pdwidgets package -- 212 files # for a widget toolkit. Removed until build.py can emit real install - # time deps; see "Planned: real MIP dependencies" in - # docs/publishing-automation.md. + # time deps; see ROADMAP.md. # # The PyPI side keeps proper dependencies in pdwidgets/pyproject.toml; # only MIP loses them.