Repository navigation
AUS-1011: CPU-utilisation axis per benchmark row, derived machine description, Hugging Face Jobs runner - #171
Merged
Merged
Conversation
…o the harness Every timed benchmark row now reports CPU during its window as a percent of the cores available to the process: the whole machine (cgroup counters inside a container, /proc/stat or GetSystemTimes otherwise), the harness process, and the dedicated TCP client threads. The figures trail every table row and sweep cell line and go into the sweep JSON as per-series arrays that render.py folds into the explorer tables; the five committed runs stay valid without them. The machine description is derived at run time (with the cgroup quota when one applies) and can be overridden with JSONRPC_BENCH_MACHINE; a --machine mode prints it alone. benchmarks/hf/ holds the Hugging Face Jobs runner: run.sh for the .NET 10 SDK image, launch.sh for the hf jobs run line, fetch.py to pull the framed sweep files out of the job log.
The hf CLI parses -l as its own --label option, so the container received bash with the script as a file name and every job failed with exit 127.
git fetch refuses a short hash, which is what the first launch passed. The runner now fetches when it can and otherwise checks the ref out from the full clone; the README and the usage lines say what a ref may be.
On Windows the hf CLI wrote the job log with the console code page and failed on the harness's box-drawing characters before fetch.py saw a byte.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: https://linear.app/austins-test/issue/AUS-1011/200-8a-of-10-harness-cpu-utilisation-axis-per-row-cgroup-aware-machine (parent AUS-1001, related AUS-985). Harness, charts and documentation only; nothing under
Json-Rpc/or the serializer packages changes.Why
The sweep chart raised the question whether the harness's own client loops saturate the box and stop feeding the server. Every timed row now records CPU during its window as a percent of the cores available to the process, split three ways: the whole machine, the harness process, and (for the TCP rows, whose clients are dedicated threads) the client threads. A high client share with the machine pegged is the saturation signature; a flat series with idle cores is not.
What
TestServer_Console/CpuMeter.cs: counters read only at the edges of a timed window. Machine figure from the cgroup'scpu.statagainst itscpu.maxquota inside a container,/proc/staton bare Linux,GetSystemTimeson Windows; process figure fromTotalProcessorTime; eachtcp-client-<n>thread reads its own CPU time (GetThreadTimes/clock_gettime(CLOCK_THREAD_CPUTIME_ID)). Unmeasurable figures are null and print asn/a.TcpRun,HttpRun, the gRPC runs,PipeRun,ProxyRun,RunSync, the--asyncand--scalerows) returns its usage; every table row and--sweepcell line ends withcpu: system 87 %, process 85 %, clients 31 %.cores, and per seriescpuSystem,cpuProcess,cpuClientsparallel torpcPerSec(null where not measured).render.pyfolds them into per-point medians when present and validates their length; the explorer tables gain the three columns, blank for the five committed runs, which carry no arrays. The committed SVGs are unchanged;explorer.htmlis regenerated.Environment.ProcessorCountplusof N host CPUs, cgroup quotawhen a quota applies, OS, runtime, GC mode), printed as the first line of every mode and written into the sweep JSON;JSONRPC_BENCH_MACHINEoverrides it. A--machinemode prints only that line.benchmarks/hf/:run.sh(runs insidemcr.microsoft.com/dotnet/sdk:10.0: clone at a ref, build, the release list of modes, then--sweep 2k times with each JSON framed on stdout),launch.sh <ref> <flavor> [runs](thehf jobs runline),fetch.py <job id> <out dir>(reads the log as UTF-8 and writes the JSON files; needs no token).--kestrel,--compareand--sweepparagraphs, the Charts section, and a short "On Hugging Face Jobs" paragraph with the flavors and the shared-vCPU caveat. CHANGELOG Added line.Verification
dotnet build -c Release AustinHarris.JsonRpc.sln -nodeReuse:false: 0 errors, only the pre-existing warnings.dotnet test: 1220/1220 on net8.0 and net10.0.--kestrel 1,--compare 1,--sweep 1,--sync 1,--async 1 2,--scale 1 4 1.0: every row carries the CPU text; process never exceeds system; the one-thread sync row reports one sixteenth of the machine;JSONRPC_BENCH_MACHINElands in the JSON; thread-pool series have null client arrays.python -m unittest benchmarks/charts/test_render.py(21 tests),render.py --check, the request-path checker (23 listed uses, none unlisted),site/build.py,bash -non both scripts. A sweep file with a short CPU array is rejected with a clear message.cpu-basic(id 6ab9887452d0dbd7f1d9e0e7): the SDK image, clone and build work; the container reports 16 host CPUs under a two-core quota, which is why the machine figure uses the cgroup counters.Sample from a busy 16-thread box,
--sweep 1, JSON-RPC.Net over TCP at 16 connections: 12.2 M RPC/s,cpu: system 100 %, process 73 %, clients 14.6 %.