Skip to content

ci: EXPKITS-1512 add nightly validation and health reports - #288

Open
nyari-kovacs-arm wants to merge 15 commits into
mainfrom
feature/EXPKITS-1512/nightly_health_check
Open

nyari-kovacs-arm wants to merge 15 commits into
mainfrom
feature/EXPKITS-1512/nightly_health_check

Conversation

@nyari-kovacs-arm

Copy link
Copy Markdown
Contributor

Goal

Detect nightly build, packaging and runtime regressions, and publish a readable YAML health report for nightly and release runs. This supersedes #273 and includes its EXPKITS-1514 nightly validation work together with EXPKITS-1512 health reporting.

Change

  • Add scheduled runs at 01:00 UTC and manual dispatch, using reusable development, image and package workflows. Reuse the checksum-pinned released ExecuTorch DEB and packaged image thumbnails.
  • Exercise installed packages, extracted archives and the deployment entrypoint, requiring seven face detections and seven camera-contact classifications from the ONNX → ExecuTorch cascade.
  • Publish a YAML report with unsuccessful jobs and links, test counts, Valgrind results, native/Python coverage, release download counts, recent scheduled-nightly success rates and container footprints. Link the artifact from the run summary and add a nightly workflow badge.
  • Enforce tracked coverage minima and per-architecture deployment image limits after publishing the report. Treat incomplete container measurements as unavailable so their failure does not prevent report publication.
  • Retain logs and reports for 14 days. Package uploads are optional and disabled by default. Release publication remains independent of the additional health checks.
  • Build OR conformance tests only when the existing or_runtime option is enabled; ordinary test builds do not require the OR submodule. Keep the writer-error test independent of GStreamer message ordering.

Testing

The nightly workflow covers both x86-64 and ARM64:

  • Development, documentation, Cairn runtime and deployment image builds.
  • Separate clean debug and release product builds with ExecuTorch and Python Ops enabled, followed by Meson tests.
  • Runtime/demo DEB builds and architecture tarball generation; install or extract each format into a clean Debian container and test plugin discovery and runtime pipelines. Include the packaged launcher for DEBs and the existing archive checks for PythonScript, the Python SDK wheel and opksink.
  • ONNX → ExecuTorch cascade execution for both package formats, plus the deployment image's real entrypoint, with assertions on serialized output counts.

Additional quality jobs run Valgrind, formatting, documentation validation, Python tool/script coverage, and a separate ARM64 native coverage build with Python Ops enabled and ExecuTorch/MLWL disabled.

Local verification during the signed interactive rebase passed actionlint and applicable tests at each resulting commit, including SDK-backed cascade tests. The Python coverage suite passed at each health-report commit. Meson configuration checks confirmed OR-disabled test builds omit the conformance targets and interface requirement, while OR-enabled builds retain them.

The pre-rebase nightly passed. The nightly on the rebased branch, at 3ecaa46, is in progress when this PR is opened; full native/container validation of the new base is pending that run.

Cairn interoperability and comprehensive packaged SDK consumer tests are deferred. Reports initially use expiring Actions artifacts; durable history is deferred. The scheduled trigger and optional package uploads have not been separately verified.

@nyari-kovacs-arm
nyari-kovacs-arm force-pushed the feature/EXPKITS-1512/nightly_health_check branch from 3ecaa46 to 6bb1e7c Compare November 13, 2026 14:00
@nyari-kovacs-arm
nyari-kovacs-arm marked this pull request as ready for review November 16, 2026 08:22
@nyari-kovacs-arm
nyari-kovacs-arm requested review from a team as code owners November 16, 2026 08:22
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright 2026 Arm Limited and/or its affiliates

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow the header agreed in the repository. (Check in other existing files)

Task: EXPKITS-1512

Normal OPK test builds should not require the optional OR interface or
build its conformance suite while or_runtime is disabled. Gate both the
interface dependency and conformance targets on the existing option.

Keep all three conformance tests available for explicit OR-enabled builds.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

Use one checksum-verified download helper for the release and nightly builds so
both consume the same architecture-specific SDK packages. Cover both pins,
unsupported architectures and checksum failures with focused tests.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

The Cairn build stage configures the full OPK Meson project, which requires the
checked-in generated SDK and configuration schemas. Copy those inputs from the
existing filtered source stage so image builds can configure without relying on
the host checkout.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

A streaming source can report an error before opkcomm reports a failed live
writer reopen. Wait for the error from opkcomm within a fixed deadline and
verify that it identifies the requested invalid destination.

Cover both message orders and the case where the expected error is absent.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

Build product images and packages on native x86-64 and ARM64 runners,
and test clean debug and release development builds with the released
ExecuTorch SDK and Python Ops enabled.

Introduce separate reusable development, image and package workflows from
the outset. Schedule the orchestrator and allow manual dispatch, retaining
logs with package uploads disabled unless explicitly requested. Skip
demo-video downloads in build jobs; runtime package smoke follows as a
separate capability.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

Install runtime and demo DEBs and separately extract the architecture
archive in clean Debian containers. Check plugin loading, pipelines and
packaged consumers, then run the UltraFace ONNX to NITEC ExecuTorch cascade
on the bundled face thumbnail.

Use the existing model download owner for archive inputs and require
seven face detections and seven camera-contact classifications in one
frame. Introduce the checker and smoke scripts with their final names,
media policy and output assertions, with focused failure tests.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514

Run a finite face detection and camera-contact pipeline through the
deployment image entrypoint on both architectures. Reuse package smoke
models and result assertions, retain outputs and verify failure cleanup.

Point the launcher at the packaged plugins so the ExecuTorch operation is
discoverable.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Measure repository Python scripts and tools with their existing tests,
and collect native line coverage from a separate ARM64 debug test build.
Retain machine-readable coverage artifacts for nightly and release runs.

Keep the coverage producers independently runnable before adding their
health-report consumer. Pin the Python coverage dependency and document
its licence.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Reuse the existing documentation publication validator to compile and
check the public documentation site during nightly and release runs.
Inject the source commit metadata and validate without publishing a site.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Collect unsuccessful workflow jobs and steps, test and Valgrind results,
coverage, release download counts and recent nightly success rates in a
YAML artifact linked from the run summary.

Consume the existing coverage and documentation jobs, retain the Valgrind
summary, and include existing formatting and Valgrind quality checks.
Retain reports for 14 days and expose overall nightly status with the
README badge. Durable history and SDK consumer coverage remain deferred.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Record minimum native and Python line coverage based on the latest
successful nightly. Compare exact counts and publish the report before
failing the health check for low or unavailable coverage.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Measure fresh development images and deployment smoke containers on both
architectures, separating image, writable-layer and mounted-model bytes.
Enforce deployment image limits based on the measured baseline plus 5%.

Read each artifact layout correctly and keep publishing the health report
when an incomplete measurement must be marked unavailable.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Generate the clean-package and deployment-entrypoint pipelines from one
finite fixture so their image input, opchain and output stages cannot drift.
Pass each environment's paths through the existing image-pipeline helper
and retain the generated deployment pipeline with the smoke logs.

Check that direct execution and JSON generation use the same pipeline,
including paths with spaces, and preserve the seven-result assertions.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Collect debug and release Meson logs once under explicit configuration
directories, including failed runs, instead of uploading the active-build
symlink as another result. Read container measurements from the resulting
artifact layout.

Validate threshold settings once and give each metric its own section,
using explicit branches for deployment-limit results.

Upload the YAML without a ZIP wrapper using archive: false. Include the
run kind and attempt in the filename to preserve separate rerun reports.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512

Use the repository's combined SPDX and Apache notice in the ExecuTorch
downloader and other branch-added files with abbreviated headers.
Preserve the existing copyright information and executable content.

Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
@nyari-kovacs-arm
nyari-kovacs-arm force-pushed the feature/EXPKITS-1512/nightly_health_check branch from 997403a to 22cf556 Compare November 16, 2026 10:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants