Skip to content

AUS-1011: CPU-utilisation axis per benchmark row, derived machine description, Hugging Face Jobs runner - #171

Merged
Astn merged 4 commits into
masterfrom
aus-1011-cpu-axis
Sep 27, 2026
Merged

Astn merged 4 commits into
masterfrom
aus-1011-cpu-axis

Conversation

@Astn

@Astn Astn commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Linear: https://linear.app/austins-test/issue/AUS-1011/200-8a-of-10-harness-cpu-utilisation-axis-per-row-cgroup-aware-machine (parent AUS-1001, related AUS-985). Harness, charts and documentation only; nothing under Json-Rpc/ or the serializer packages changes.

Why

The sweep chart raised the question whether the harness's own client loops saturate the box and stop feeding the server. Every timed row now records CPU during its window as a percent of the cores available to the process, split three ways: the whole machine, the harness process, and (for the TCP rows, whose clients are dedicated threads) the client threads. A high client share with the machine pegged is the saturation signature; a flat series with idle cores is not.

What

  • TestServer_Console/CpuMeter.cs: counters read only at the edges of a timed window. Machine figure from the cgroup's cpu.stat against its cpu.max quota inside a container, /proc/stat on bare Linux, GetSystemTimes on Windows; process figure from TotalProcessorTime; each tcp-client-<n> thread reads its own CPU time (GetThreadTimes / clock_gettime(CLOCK_THREAD_CPUTIME_ID)). Unmeasurable figures are null and print as n/a.
  • Every timed run (TcpRun, HttpRun, the gRPC runs, PipeRun, ProxyRun, RunSync, the --async and --scale rows) returns its usage; every table row and --sweep cell line ends with cpu: system 87 %, process 85 %, clients 31 %.
  • Sweep JSON: top-level cores, and per series cpuSystem, cpuProcess, cpuClients parallel to rpcPerSec (null where not measured). render.py folds them into per-point medians when present and validates their length; the explorer tables gain the three columns, blank for the five committed runs, which carry no arrays. The committed SVGs are unchanged; explorer.html is regenerated.
  • Machine description derived at run time (CPU model, Environment.ProcessorCount plus of N host CPUs, cgroup quota when a quota applies, OS, runtime, GC mode), printed as the first line of every mode and written into the sweep JSON; JSONRPC_BENCH_MACHINE overrides it. A --machine mode prints only that line.
  • benchmarks/hf/: run.sh (runs inside mcr.microsoft.com/dotnet/sdk:10.0: clone at a ref, build, the release list of modes, then --sweep 2 k times with each JSON framed on stdout), launch.sh <ref> <flavor> [runs] (the hf jobs run line), fetch.py <job id> <out dir> (reads the log as UTF-8 and writes the JSON files; needs no token).
  • README: the CPU text on the --kestrel, --compare and --sweep paragraphs, the Charts section, and a short "On Hugging Face Jobs" paragraph with the flavors and the shared-vCPU caveat. CHANGELOG Added line.

Verification

  • dotnet build -c Release AustinHarris.JsonRpc.sln -nodeReuse:false: 0 errors, only the pre-existing warnings. dotnet test: 1220/1220 on net8.0 and net10.0.
  • --kestrel 1, --compare 1, --sweep 1, --sync 1, --async 1 2, --scale 1 4 1.0: every row carries the CPU text; process never exceeds system; the one-thread sync row reports one sixteenth of the machine; JSONRPC_BENCH_MACHINE lands in the JSON; thread-pool series have null client arrays.
  • python -m unittest benchmarks/charts/test_render.py (21 tests), render.py --check, the request-path checker (23 listed uses, none unlisted), site/build.py, bash -n on both scripts. A sweep file with a short CPU array is rejected with a clear message.
  • Smoke job on cpu-basic (id 6ab9887452d0dbd7f1d9e0e7): the SDK image, clone and build work; the container reports 16 host CPUs under a two-core quota, which is why the machine figure uses the cgroup counters.

Sample from a busy 16-thread box, --sweep 1, JSON-RPC.Net over TCP at 16 connections: 12.2 M RPC/s, cpu: system 100 %, process 73 %, clients 14.6 %.

…o the harness

Every timed benchmark row now reports CPU during its window as a percent of
the cores available to the process: the whole machine (cgroup counters inside
a container, /proc/stat or GetSystemTimes otherwise), the harness process, and
the dedicated TCP client threads. The figures trail every table row and sweep
cell line and go into the sweep JSON as per-series arrays that render.py folds
into the explorer tables; the five committed runs stay valid without them.

The machine description is derived at run time (with the cgroup quota when one
applies) and can be overridden with JSONRPC_BENCH_MACHINE; a --machine mode
prints it alone. benchmarks/hf/ holds the Hugging Face Jobs runner: run.sh for
the .NET 10 SDK image, launch.sh for the hf jobs run line, fetch.py to pull the
framed sweep files out of the job log.
The hf CLI parses -l as its own --label option, so the container received
bash with the script as a file name and every job failed with exit 127.
git fetch refuses a short hash, which is what the first launch passed. The
runner now fetches when it can and otherwise checks the ref out from the
full clone; the README and the usage lines say what a ref may be.
On Windows the hf CLI wrote the job log with the console code page and
failed on the harness's box-drawing characters before fetch.py saw a byte.
@Astn
Astn merged commit 46148d6 into master Sep 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant