Skip to content

AUS-1012: Publish the benchmarks from the cpu-performance job, regenerated from data - #172

Merged
Astn merged 2 commits into
masterfrom
aus-1012-publish-job
Sep 28, 2026
Merged

Astn merged 2 commits into
masterfrom
aus-1012-publish-job

Conversation

@Astn

@Astn Astn commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Linear: AUS-1012 (parent AUS-1001; supersedes AUS-985). Built on #171 (AUS-1011), now merged.

The published benchmarks now come from one Hugging Face Jobs cpu-performance run (AMD EPYC 7R13, 32 cores, whole host) instead of the desktop, and republishing them is a pipeline rather than a transcription.

Pipeline

  • Every benchmark mode writes a JSON file when JSONRPC_BENCH_RESULTS is set (TestServer_Console/BenchResults.cs; benchmarks/Baseline has its own writer). Rows carry a stable id, throughput and the CPU usage from AUS-1011: CPU-utilisation axis per benchmark row, derived machine description, Hugging Face Jobs runner #171.
  • benchmarks/hf/run.sh <ref> publish runs the README's policy on the job host: the scaling gate once, two rounds of --sync, --async 3 1, --async 3 N, --kestrel, --kestrel async and --compare, three legacy t runs, two baseline runs and five --sweep 2 runs, every run and every JSON file framed on stdout. fetch.py extracts them.
  • benchmarks/charts/ingest.py <dir> folds the JSON into benchmarks.json (measured fields only; ids, labels, notes and groups stay hand-maintained; idempotent) and copies the sweep files.
  • render.py renders the SVGs, the explorer and the README tables between <!-- benchmarks:<set> --> markers (README.md and Json-Rpc/README.md); render.py --check fails on a stale block, alt text or chart.
  • The sweep and sync ladders reach the core count (powers of two up to Environment.ProcessorCount).

Republishing is: launch.sh <ref> cpu-performance publish, fetch.py, ingest.py, render.py, commit.

Data

Job 6ab9b3936b030d633f69b0fb (https://huggingface.co/jobs/astnh/6ab9b3936b030d633f69b0fb), commit 6c3c272, 2026-09-28. Gate 12.2 / 12.1 / 13.1 at 32 workers, pass. Sync 44.2 M to 45.7 M at 32 threads; Kestrel TCP 18.5 M at 32 clients; the 2.0 string API 9.8 M to 11.1 M against 1.2.3 at 1.29 M to 2.41 M (the two baseline runs differ widely; the ratios use the medians). The WebAssembly table stays a browser measurement on the desktop and says so.

Checks

render.py --check, the 43 chart tests, the request-path checker, the site build and the solution build and tests (1220 on net8.0 and net10.0) pass; a second ingest.py on the job directory reports no change.

@Astn
Astn merged commit fa2bdd4 into master Sep 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant