Repository navigation
ci: EXPKITS-1512 add nightly validation and health reports - #288
Open
nyari-kovacs-arm wants to merge 15 commits into
Open
nyari-kovacs-arm wants to merge 15 commits into
nyari-kovacs-arm wants to merge 15 commits into
Conversation
nyari-kovacs-arm
force-pushed
the
feature/EXPKITS-1512/nightly_health_check
branch
from
November 13, 2026 14:00
3ecaa46 to
6bb1e7c
Compare
nyari-kovacs-arm
marked this pull request as ready for review
November 16, 2026 08:22
msandor93
reviewed
Nov 16, 2026
| @@ -0,0 +1,25 @@ | |||
| #!/usr/bin/env bash | |||
| # SPDX-FileCopyrightText: Copyright 2026 Arm Limited and/or its affiliates | |||
Contributor
There was a problem hiding this comment.
Please follow the header agreed in the repository. (Check in other existing files)
Task: EXPKITS-1512 Normal OPK test builds should not require the optional OR interface or build its conformance suite while or_runtime is disabled. Gate both the interface dependency and conformance targets on the existing option. Keep all three conformance tests available for explicit OR-enabled builds. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 Use one checksum-verified download helper for the release and nightly builds so both consume the same architecture-specific SDK packages. Cover both pins, unsupported architectures and checksum failures with focused tests. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 The Cairn build stage configures the full OPK Meson project, which requires the checked-in generated SDK and configuration schemas. Copy those inputs from the existing filtered source stage so image builds can configure without relying on the host checkout. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 A streaming source can report an error before opkcomm reports a failed live writer reopen. Wait for the error from opkcomm within a fixed deadline and verify that it identifies the requested invalid destination. Cover both message orders and the case where the expected error is absent. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 Build product images and packages on native x86-64 and ARM64 runners, and test clean debug and release development builds with the released ExecuTorch SDK and Python Ops enabled. Introduce separate reusable development, image and package workflows from the outset. Schedule the orchestrator and allow manual dispatch, retaining logs with package uploads disabled unless explicitly requested. Skip demo-video downloads in build jobs; runtime package smoke follows as a separate capability. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 Install runtime and demo DEBs and separately extract the architecture archive in clean Debian containers. Check plugin loading, pipelines and packaged consumers, then run the UltraFace ONNX to NITEC ExecuTorch cascade on the bundled face thumbnail. Use the existing model download owner for archive inputs and require seven face detections and seven camera-contact classifications in one frame. Introduce the checker and smoke scripts with their final names, media policy and output assertions, with focused failure tests. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1514 Run a finite face detection and camera-contact pipeline through the deployment image entrypoint on both architectures. Reuse package smoke models and result assertions, retain outputs and verify failure cleanup. Point the launcher at the packaged plugins so the ExecuTorch operation is discoverable. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Measure repository Python scripts and tools with their existing tests, and collect native line coverage from a separate ARM64 debug test build. Retain machine-readable coverage artifacts for nightly and release runs. Keep the coverage producers independently runnable before adding their health-report consumer. Pin the Python coverage dependency and document its licence. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Reuse the existing documentation publication validator to compile and check the public documentation site during nightly and release runs. Inject the source commit metadata and validate without publishing a site. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Collect unsuccessful workflow jobs and steps, test and Valgrind results, coverage, release download counts and recent nightly success rates in a YAML artifact linked from the run summary. Consume the existing coverage and documentation jobs, retain the Valgrind summary, and include existing formatting and Valgrind quality checks. Retain reports for 14 days and expose overall nightly status with the README badge. Durable history and SDK consumer coverage remain deferred. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Record minimum native and Python line coverage based on the latest successful nightly. Compare exact counts and publish the report before failing the health check for low or unavailable coverage. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Measure fresh development images and deployment smoke containers on both architectures, separating image, writable-layer and mounted-model bytes. Enforce deployment image limits based on the measured baseline plus 5%. Read each artifact layout correctly and keep publishing the health report when an incomplete measurement must be marked unavailable. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Generate the clean-package and deployment-entrypoint pipelines from one finite fixture so their image input, opchain and output stages cannot drift. Pass each environment's paths through the existing image-pipeline helper and retain the generated deployment pipeline with the smoke logs. Check that direct execution and JSON generation use the same pipeline, including paths with spaces, and preserve the seven-result assertions. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Collect debug and release Meson logs once under explicit configuration directories, including failed runs, instead of uploading the active-build symlink as another result. Read container measurements from the resulting artifact layout. Validate threshold settings once and give each metric its own section, using explicit branches for deployment-limit results. Upload the YAML without a ZIP wrapper using archive: false. Include the run kind and attempt in the filename to preserve separate rerun reports. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
Task: EXPKITS-1512 Use the repository's combined SPDX and Apache notice in the ExecuTorch downloader and other branch-added files with abbreviated headers. Preserve the existing copyright information and executable content. Signed-off-by: Nyari-Kovacs, David Tamas <david.nyari-kovacs@arm.com>
nyari-kovacs-arm
force-pushed
the
feature/EXPKITS-1512/nightly_health_check
branch
from
November 16, 2026 10:24
997403a to
22cf556
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Detect nightly build, packaging and runtime regressions, and publish a readable YAML health report for nightly and release runs. This supersedes #273 and includes its EXPKITS-1514 nightly validation work together with EXPKITS-1512 health reporting.
Change
or_runtimeoption is enabled; ordinary test builds do not require the OR submodule. Keep the writer-error test independent of GStreamer message ordering.Testing
The nightly workflow covers both x86-64 and ARM64:
opksink.Additional quality jobs run Valgrind, formatting, documentation validation, Python tool/script coverage, and a separate ARM64 native coverage build with Python Ops enabled and ExecuTorch/MLWL disabled.
Local verification during the signed interactive rebase passed actionlint and applicable tests at each resulting commit, including SDK-backed cascade tests. The Python coverage suite passed at each health-report commit. Meson configuration checks confirmed OR-disabled test builds omit the conformance targets and interface requirement, while OR-enabled builds retain them.
The pre-rebase nightly passed. The nightly on the rebased branch, at
3ecaa46, is in progress when this PR is opened; full native/container validation of the new base is pending that run.Cairn interoperability and comprehensive packaged SDK consumer tests are deferred. Reports initially use expiring Actions artifacts; durable history is deferred. The scheduled trigger and optional package uploads have not been separately verified.