From 6c3c2729f8119ed2babe13cf00634d8f05d7db4e Mon Sep 17 00:00:00 2001 From: Austin Harris Date: Sun, 27 Sep 2026 18:23:39 -0600 Subject: [PATCH 1/2] AUS-1012: JSON results from every benchmark mode, the publish profile, ingest and generated README tables --- Json-Rpc/README.md | 3 + README.md | 32 +- TestServer_Console/AsyncBenchmark.cs | 73 ++- TestServer_Console/BenchResults.cs | 96 ++++ TestServer_Console/Benchmark.cs | 43 +- TestServer_Console/CompareBenchmark.cs | 73 ++- TestServer_Console/KestrelBenchmark.cs | 24 + TestServer_Console/Program.cs | 17 +- benchmarks/Baseline/Program.cs | 159 ++++++ benchmarks/charts/benchmarks.json | 118 +++-- .../charts/compare-connections-dark.svg | 4 +- benchmarks/charts/compare-connections.svg | 4 +- .../charts/compare-streamjsonrpc-dark.svg | 4 +- benchmarks/charts/compare-streamjsonrpc.svg | 4 +- benchmarks/charts/explorer.html | 8 +- benchmarks/charts/explorer_template.html | 2 +- benchmarks/charts/headline-1x-vs-2-dark.svg | 4 +- benchmarks/charts/headline-1x-vs-2.svg | 4 +- benchmarks/charts/ingest.py | 477 ++++++++++++++++++ benchmarks/charts/inprocess-paths-dark.svg | 4 +- benchmarks/charts/inprocess-paths.svg | 4 +- benchmarks/charts/kestrel-transports-dark.svg | 4 +- benchmarks/charts/kestrel-transports.svg | 4 +- benchmarks/charts/render.py | 345 +++++++++++-- benchmarks/charts/sync-threads-dark.svg | 4 +- benchmarks/charts/sync-threads.svg | 4 +- benchmarks/charts/test_render.py | 236 ++++++++- .../charts/testdata/ingest/async-w1-1.json | 140 +++++ .../charts/testdata/ingest/async-w4-1.json | 140 +++++ .../charts/testdata/ingest/baseline-1.json | 47 ++ .../charts/testdata/ingest/compare-1.json | 106 ++++ .../charts/testdata/ingest/kestrel-1.json | 64 +++ .../testdata/ingest/kestrel-async-1.json | 74 +++ .../charts/testdata/ingest/legacy-1.json | 46 ++ .../charts/testdata/ingest/scale-1.json | 206 ++++++++ .../charts/testdata/ingest/sweep-1.json | 218 ++++++++ benchmarks/charts/testdata/ingest/sync-1.json | 53 ++ benchmarks/charts/testdata/ingest/sync-2.json | 53 ++ benchmarks/charts/wasm-interop-dark.svg | 4 +- benchmarks/charts/wasm-interop.svg | 4 +- benchmarks/hf/README.md | 28 + benchmarks/hf/fetch.py | 86 +++- benchmarks/hf/launch.sh | 10 +- benchmarks/hf/run.sh | 107 +++- 44 files changed, 2933 insertions(+), 207 deletions(-) create mode 100644 TestServer_Console/BenchResults.cs create mode 100644 benchmarks/charts/ingest.py create mode 100644 benchmarks/charts/testdata/ingest/async-w1-1.json create mode 100644 benchmarks/charts/testdata/ingest/async-w4-1.json create mode 100644 benchmarks/charts/testdata/ingest/baseline-1.json create mode 100644 benchmarks/charts/testdata/ingest/compare-1.json create mode 100644 benchmarks/charts/testdata/ingest/kestrel-1.json create mode 100644 benchmarks/charts/testdata/ingest/kestrel-async-1.json create mode 100644 benchmarks/charts/testdata/ingest/legacy-1.json create mode 100644 benchmarks/charts/testdata/ingest/scale-1.json create mode 100644 benchmarks/charts/testdata/ingest/sweep-1.json create mode 100644 benchmarks/charts/testdata/ingest/sync-1.json create mode 100644 benchmarks/charts/testdata/ingest/sync-2.json create mode 100644 benchmarks/hf/README.md diff --git a/Json-Rpc/README.md b/Json-Rpc/README.md index 90e0de1..8632837 100644 --- a/Json-Rpc/README.md +++ b/Json-Rpc/README.md @@ -140,6 +140,7 @@ The 2.0 runs used the built-in serializer on an AMD Ryzen 7 7800X3D with .NET 10 ![JSON-RPC.Net 1.2.3 and 2.0 throughput through the string and byte entry points](https://raw.githubusercontent.com/Astn/JSON-RPC.NET/master/benchmarks/charts/headline-1x-vs-2.svg) + | Path | RPC/s | Against 1.2.3 | | --- | ---: | ---: | | 1.2.3, `Task Process(string)`, thread pool, best batch size | 3.08 M | | @@ -147,6 +148,8 @@ The 2.0 runs used the built-in serializer on an AMD Ryzen 7 7800X3D with .NET 10 | 2.0, `Process(bytes)`, 16 dedicated threads | 31.7 M | 10.3× | | 2.0, `ProcessAsync(bytes)`, 16 awaited workers | 32.1 M | 10.4× | + + These measurements cover the library without a transport. The asynchronous byte row uses methods that complete inline. For Kestrel TCP with 256 requests in flight per connection and `EnableAsyncMethods = false`, diff --git a/README.md b/README.md index 94fe1d6..d5c9769 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ Version 2.0 rebuilt the pipeline around UTF-8 bytes and made the JSON serializer JSON-RPC.Net 1.2.3 and 2.0 on one machine, one run per row on 2026-09-25: 3.08 M requests per second through the 1.2.3 string API, 13.3 M through the same API on 2.0, and 31.7 M and 32.1 M through the 2.0 byte entry points + | Path | RPC/s | Against 1.2.3 | | --- | ---: | ---: | | 1.2.3, `Task Process(string)`, thread pool, best batch size | 3.08 M | | @@ -22,6 +23,8 @@ Version 2.0 rebuilt the pipeline around UTF-8 bytes and made the JSON serializer | 2.0, `Process(bytes)`, 16 dedicated threads | 31.7 M | 10.3× | | 2.0, `ProcessAsync(bytes)`, 16 awaited workers | 32.1 M | 10.4× | + + The request path uses a span tokenizer over the UTF-8 bytes and invokers compiled against the concrete reader and writer. It writes the response straight into a pooled buffer, with no `Task`, result string or continuation per request. A numeric request never touches the GC. Over Kestrel TCP, the AspNetCore package holds 14.3 M to 16.5 M with 256 requests in flight per connection. [Benchmarks](#benchmarks) has the full tables, conditions and the ranges over each day's runs. The [explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) has the exact values. `benchmarks/Baseline` re-runs the 1.2.3 row from NuGet with the same loop as the 2.0 harness. @@ -600,6 +603,7 @@ The `jsonrpc` member policy (`Lenient` by default) is a compatibility setting, n ## Benchmarks + | What | API and mode | RPC/s | Details | | --- | --- | ---: | --- | | Library alone, 16 threads | `Process(bytes)`, dedicated threads | 31.7 M to 36.4 M | [Sync](#sync-the-library-alone) | @@ -612,6 +616,8 @@ The `jsonrpc` member policy (`Lenient` by default) is a compatibility setting, n | Kestrel HTTP, one request per POST | `EnableAsyncMethods = false` | 123 k to 182 k | HTTP/1.1 round trips dominate | | Legacy string API, thread pool | `Task Process(string)`, batches of 6,000 | 12.4 M to 13.3 M | [Legacy](#legacy-string-api-scheduled-synchronous-execution); the 1.x overloads, not the byte path | + + The Sync, Async, Legacy and Kestrel results were measured on 2026-09-25 on an idle AMD Ryzen 7 7800X3D (8 cores / 16 threads, 4.2 GHz), 64 GB, Windows 11, .NET 10, Release, Server GC, with the built-in serializer. Throughput ranges are the low and high over that day's runs: three runs for Sync, Legacy and default-mode Kestrel, two for the `EnableAsyncMethods = true` rows. Each Async throughput cell is one 3 s run; the summary's inline Async spread is across registrations, not runs. Each Sync ns figure is the reported cost from one of that row's runs. The StreamJsonRpc and gRPC comparison and its in-process rows are from 2026-09-23, with the WSL virtual machine shut down. Their ranges cover two runs; the comparison's single throughput figure is the value both runs rounded to. The WebAssembly results are also from 2026-09-23: one interpreter run and the better of two AOT runs per row. The connection sweep is a separate session with WSL and other work active, so its absolute figures are not comparable with the tables. A single benchmark thread on this machine varies with whatever else lands on its core's SMT sibling, so the 1-thread rows are from runs on an idle core. `TestServer_Console` is the benchmark harness. It binds one service with five small methods (`add`, `addInt`, `NullableFloatToNullableFloat`, `Test2`, `StringMe`), drives the same five requests through the server, checks every response is a `result` rather than an error, and ends each mode with a bar chart of RPC/s. For one-request timings with an allocation column, the numbers to check before merging a change to the dispatch path, see [benchmarks/Micro](benchmarks/Micro/README.md). @@ -622,7 +628,7 @@ dotnet run -c Release --project TestServer_Console -- --async 3 16 # ProcessAsy dotnet run -c Release --project TestServer_Console -- --scale 3 16 4.0 # release gate: ProcessAsync must scale at least 4x from 1 to 16 workers, then per-serializer diagnostics (--no-diagnostics skips them) dotnet run -c Release --project TestServer_Console -- --kestrel 3 # through the AspNetCore package, HTTP and TCP (add `async` for EnableAsyncMethods = true) dotnet run -c Release --project TestServer_Console -- --compare 3 # the same calls through StreamJsonRpc and gRPC for .NET, side by side -dotnet run -c Release --project TestServer_Console -- --sweep 2 benchmarks/charts/sweep.json # every library and transport at 1, 2, 4, 8, 16 connections; one file per run +dotnet run -c Release --project TestServer_Console -- --sweep 2 benchmarks/charts/sweep.json # every library and transport at 1, 2, 4, ... connections up to the core count; one file per run dotnet run -c Release --project TestServer_Console # menu: Enter = Process(bytes), a = ProcessAsync(bytes), t = legacy string API, k = Kestrel, x = compare, q = quit dotnet run -c Release --project benchmarks/Baseline # the last 1.x release (1.2.3 from NuGet) through the same loop as the t entry dotnet run --project samples/WasmHost # browser: "Run benchmark" on the page @@ -630,7 +636,7 @@ dotnet run --project samples/WasmHost # browser: " The three modes measure different things and are named for the entry point they call. `Process(bytes), dedicated threads` measures the library alone. `ProcessAsync(bytes), awaited workers` measures the entry point an asynchronous host calls. `Legacy Process(string), scheduled synchronous work` measures the 1.x string overloads through the thread pool. The Kestrel rows say whether `EnableAsyncMethods` was on. -On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs]` (`` is a branch, tag or commit) and retrieve JSON with `benchmarks/hf/fetch.py `; `cpu-upgrade` is 8 vCPU ($0.03/h), `cpu-xl` is 16 vCPU ($1/h), and `cpu-performance` is 32 vCPU ($1.90/h), while `cpu-basic` is 2 vCPU for smoke runs only. These are shared vCPUs, so each job's figures belong to its own machine row and are never merged into the reference-box tables. +On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs|publish]` (`` is a branch, tag or commit) and retrieve JSON with `benchmarks/hf/fetch.py `; `cpu-upgrade` is 8 vCPU ($0.03/h), `cpu-xl` is 16 vCPU ($1/h), and `cpu-performance` is 32 vCPU ($1.90/h), while `cpu-basic` is 2 vCPU for smoke runs only. These are shared vCPUs, so each job's figures belong to its own machine row and are never merged into the reference-box tables. ### Sync: the library alone @@ -641,6 +647,7 @@ On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs]` (` + | Threads | RPC/s | ns per request per thread | Allocations per request | | ---: | ---: | ---: | --- | | 1 | 4.6 M to 5.0 M | 214 | 0 bytes for numeric shapes, one string for `StringMe` | @@ -649,12 +656,15 @@ On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs]` (` + Per-thread cost rises with thread count because the 16 threads share 8 physical cores. ### Async: ProcessAsync, awaited workers `--async [seconds] [workers]` calls the byte-level `JsonRpcProcessor.ProcessAsync` from `Task.Run` workers that await each call. It uses the same five requests and loop shape as `--sync`. Workers start behind a barrier, stop on a shared flag and walk the inputs with an index. Worker startup is reported separately. Each row registers the five methods with one return shape and one `RpcContextFlow`. The `yieldsOnce` rows use one method that awaits `Task.Yield()`, giving a real suspension per request. Responses are checked before timing. The bytes column reports allocations per request at one worker, including the service method's own allocations. The inline rows use exact per-thread counts, and the `yieldsOnce` rows use the process-wide counter. + | Registration | 1 worker | 16 workers | B per request | | --- | ---: | ---: | ---: | | synchronous methods, None | 3.47 M | 24.6 M | 6 | @@ -665,6 +675,8 @@ Per-thread cost rises with thread count because the 16 threads share 8 physical | `yieldsOnce`, Flow | 915 k | 6.89 M | 741 | | `yieldsOnce`, None | 1.14 M | 8.96 M | 559 | + + The 16-worker rows are an equal-weight mix of the five requests, except `yieldsOnce`, which is one request. The table below gives bytes per request for each request shape at one worker, including the method's own allocations. The harness prints these lines before each row. | Registration | `add` | `addInt` | nullable float | decimal | `StringMe` | @@ -683,6 +695,7 @@ Before 2.0.0, the `ProcessAsync` path was capped near 4 M RPC/s at every worker The `t` menu entry submits batches through the 1.x `Task Process(string)` overload from every core at once. This compatibility string API pays for transcoding, a thread-pool hop, a `Task`, a result string and a continuation per request. An asynchronous host uses the byte entry points measured in the Sync and Async tables above. Each batch size is repeated for at least half a second after a one-second warm-up. Throughput peaks once a batch is large enough to keep every core busy and falls off again when hundreds of thousands of requests are queued at once: + | Batch size | RPC/s | | ---: | ---: | | 50 | 1.6 M to 2.0 M | @@ -692,6 +705,8 @@ The `t` menu entry submits batches through the 1.x `Task Process(string) | 252,000 | 7.5 M to 7.8 M | | 2,016,000 | 7.5 M to 8.0 M | + + This mode is slower than the byte modes because it measures the cost of the .NET thread pool and a `Task`, a string and a continuation per request as much as the library itself. The AspNetCore host does not use that string path or allocate a result string. It awaits transport reads and flushes without a thread per connection, as the next table shows. ### Kestrel: through the AspNetCore package @@ -703,6 +718,7 @@ This mode is slower than the byte modes because it measures the cost of the .NET JSON-RPC.Net by transport: HTTP single, HTTP batch of 100 and TCP pipelined, as low-to-high intervals on a log axis, with the in-process figure for scale + | Transport | RPC/s | Note | | --- | ---: | --- | | in-process, 16 threads | 33.8 M to 37.1 M | | @@ -712,6 +728,8 @@ This mode is slower than the byte modes because it measures the cost of the .NET | TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that complete inline | 15.0 M to 15.4 M | `--kestrel 3 async`, two runs; inside the spread of the `false` row | | TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that suspend once | 1.25 M to 1.29 M | five `async Task` methods awaiting `Task.Yield()` | + + The TCP client keeps 256 requests in flight per connection and refills from a precomputed ring of request bytes with one `Send` per refill. On the server, `JsonFramer` feeds the same `Process` call the HTTP endpoint makes. With `EnableAsyncMethods = true`, the connection handler processes each connection's documents one at a time, in order. So 256 pipelined requests are 256 sequential invocations, and a method that suspends pays that cost per request. Concurrency comes from the 16 connections. The last row is a release-required regression row: it is measured with `--kestrel 3 async` on the reference machine before every release and compared with the previous release's published figure, a drop larger than the paired-run spread blocks the release, and it has no absolute floor. ### Versus StreamJsonRpc and gRPC @@ -723,6 +741,7 @@ The TCP client keeps 256 requests in flight per connection and refills from a pr JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET at 16 connections: low-to-high intervals on a log axis, grouped by library, with the in-process paths under a rule + | Library and path | RPC/s | | --- | ---: | | JSON-RPC.Net over Kestrel TCP, raw documents | 13.7 M to 14.6 M | @@ -732,6 +751,8 @@ The TCP client keeps 256 requests in flight per connection and refills from a pr | gRPC for .NET, unary calls over HTTP/2 (Grpc.Net.Client, 16 channels × 256 in flight) | 192 k to 198 k | | gRPC for .NET, one bidirectional stream per channel, 256 in flight, batched writes | 200 k to 209 k | + + StreamJsonRpc 2.25.29, defaults apart from the formatter and framing named in each row. It is a full bidirectional RPC framework (client proxies, cancellation, progress, marshaled objects, events). The comparison is of the server side answering the same five requests; on that measure JSON-RPC.Net is about 10× faster on the same connections. JSON-RPC.Net ran its built-in serializer, and the fastest StreamJsonRpc rows use System.Text.Json, so this is a comparison of whole server paths, not of one JSON library against itself; running JSON-RPC.Net with the System.Text.Json serializer is a separate measurement and is not in this table. gRPC for .NET 2.84.0 with default settings apart from Kestrel's `MaxStreamsPerConnection` (raised to 256 so the pipeline depth is not capped at 100). protobuf has no `decimal`, so `Test2` carries the units/nanos `DecimalValue` message the gRPC docs recommend; nullable values use proto3 `optional`. The gRPC rows are a different kind of measurement from the rows above them: there is no cheap raw client for HTTP/2 + protobuf, so the client is Grpc.Net.Client on the same 8 cores as the server, and the figure is what a .NET caller and a .NET service get end to end. One channel alone reaches about 130 k unary calls per second; sixteen channels do not scale much further because client and server compete for the same cores. The streaming row batches its writes the way the TCP client does (BufferHint on every message but the last of a refill), and the server flushes only when its input runs dry, the same once-per-read-group flush `JsonRpcConnectionHandler` does. @@ -744,12 +765,15 @@ gRPC for .NET 2.84.0 with default settings apart from Kestrel's `MaxStreamsPerCo In-process paths: JSON-RPC.Net direct call, StreamJsonRpc Pipe pair and StreamJsonRpc typed proxy, as low-to-high intervals on a log axis + | Path | RPC/s | | --- | ---: | | JSON-RPC.Net in-process, 1 thread (direct call, bytes in, bytes out) | 2.6 M to 3.6 M | | StreamJsonRpc in-process, 1 client over a `Pipe` pair, newline framing, System.Text.Json formatter, 256 pipelined | 117 k to 142 k | | StreamJsonRpc typed proxy, sequential `await` per call, in-process pipes | 96 k to 97 k (10 µs per round trip) | + + StreamJsonRpc's server side has no "document in, document out" call, so its in-process row is a pair of `System.IO.Pipelines` pipes, the closest it has to a direct call; the proxy row is one call at a time, so it measures a round trip, not throughput. The direct-call row is from the comparison session and sits below the sync table's newer 1-thread figure. @@ -759,7 +783,7 @@ StreamJsonRpc's server side has no "document in, document out" call, so its in-p Every library and transport by client connections, 1 to 16: three panels on a shared log axis, one per library, with a marker shape and dash per setting and whiskers spanning five runs -`--sweep` runs every one of those paths at 1, 2, 4, 8 and 16 client connections (gRPC: channels) and writes one JSON file per run; the chart above is five 2 s runs per point, the marker at the median and the whisker from the lowest to the highest run. New sweep files also record the available core count and per-cell system, process and dedicated TCP-client CPU percentages; a high client share can mean the harness is competing with the server for CPU. It is a separate session from the tables: the WSL virtual machine was running and other work was active, so its absolute figures sit below the table rows (JSON-RPC.Net over TCP 10.9 M to 13.0 M at 16 connections against 13.7 M to 14.6 M in the table), and its gRPC unary figure runs higher (360 k to 404 k against 192 k to 198 k; the cause is not pinned down, and the table keeps the `--compare` figure). What the sweep adds is the shape: JSON-RPC.Net over TCP and batched HTTP climb almost linearly with connections, StreamJsonRpc gains 8 to 10× from one connection to sixteen, and gRPC's .NET client is flat from two channels on because it competes with the server for the same eight cores. +`--sweep` runs every one of those paths at each power of two from 1 to the core count of client connections (gRPC: channels) and writes one JSON file per run; the chart above is five 2 s runs per point, the marker at the median and the whisker from the lowest to the highest run. New sweep files also record the available core count and per-cell system, process and dedicated TCP-client CPU percentages; a high client share can mean the harness is competing with the server for CPU. It is a separate session from the tables: the WSL virtual machine was running and other work was active, so its absolute figures sit below the table rows (JSON-RPC.Net over TCP 10.9 M to 13.0 M at 16 connections against 13.7 M to 14.6 M in the table), and its gRPC unary figure runs higher (360 k to 404 k against 192 k to 198 k; the cause is not pinned down, and the table keeps the `--compare` figure). What the sweep adds is the shape: JSON-RPC.Net over TCP and batched HTTP climb almost linearly with connections, StreamJsonRpc gains 8 to 10× from one connection to sixteen, and gRPC's .NET client is flat from two channels on because it competes with the server for the same eight cores. The [benchmark explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) is the same data as an interactive page: toggle series, hover or tab to a point for the exact low, median, high and every run, switch the axis between log and linear, and download the data. It is one self-contained HTML file, [benchmarks/charts/explorer.html](benchmarks/charts/explorer.html), so it also works saved to disk. @@ -814,7 +838,7 @@ The 1.x projects that 2.0 does not build (`AustinHarris.JsonRpc.Client`, `Austin ### Charts -The charts, the explorer page and the benchmark figures in this README come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json): the tables transcribed with their published precision and conditions, plus the `--sweep` run files. `python benchmarks/charts/render.py` (plain Python, no packages) renders every chart in a light and a dark variant, which the README picks between with a `` element, and `render.py --check` fails if a committed chart is stale or a figure in this README no longer matches the data; the pull-request build runs it. GitHub serves README images through a proxy as plain ``, so the SVGs carry no scripts, hover text or links, and every range is drawn as an interval with its figures beside it; the interactive parts live on the explorer page. The explorer tables show median system, process and dedicated TCP-client CPU percentages when the sweep files contain them; a high client share can limit the requests the harness feeds to the server. +The charts, the explorer page and the benchmark figures in this README come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json): the tables transcribed with their published precision and conditions, plus the `--sweep` run files. `python benchmarks/charts/render.py` (plain Python, no packages) renders every chart in a light and a dark variant, which the README picks between with a `` element, and `render.py --check` fails if a committed chart is stale or a figure in this README no longer matches the data; the pull-request build runs it. To republish, run `benchmarks/hf/launch.sh cpu-performance publish`, then `python benchmarks/hf/fetch.py ` to extract each run's JSON and text. `python benchmarks/charts/ingest.py --source --conditions "..."` folds the JSON into benchmarks.json and the sweep files, and `render.py` regenerates the charts, the explorer and the README tables between `` markers. `render.py --check` fails when any of them is stale. GitHub serves README images through a proxy as plain ``, so the SVGs carry no scripts, hover text or links, and every range is drawn as an interval with its figures beside it; the interactive parts live on the explorer page. The explorer tables show median system, process and dedicated TCP-client CPU percentages when the sweep files contain them; a high client share can limit the requests the harness feeds to the server. ## License diff --git a/TestServer_Console/AsyncBenchmark.cs b/TestServer_Console/AsyncBenchmark.cs index d141dcd..8f0a598 100644 --- a/TestServer_Console/AsyncBenchmark.cs +++ b/TestServer_Console/AsyncBenchmark.cs @@ -3,6 +3,7 @@ using System.Diagnostics; using System.Linq; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -47,6 +48,7 @@ internal static async Task RunAsync(Action print, double seconds, int wo { if (seconds <= 0 || workers <= 0) throw new ArgumentOutOfRangeException(nameof(seconds)); var summary = new List(); + var resultRows = new JsonArray(); foreach (var (shape, flow) in Rows) { string session = "async-benchmark-" + shape + "-" + flow; @@ -54,26 +56,45 @@ internal static async Task RunAsync(Action print, double seconds, int wo { Register(session, shape, flow); var rowInputs = Inputs(shape); - await ValidateAndReportAllocations(print, session, shape, flow, rowInputs); + var perShape = await ValidateAndReportAllocations(print, session, shape, flow, rowInputs); bool yield = shape == "yield"; var m = await Measure(session, rowInputs, workers, seconds, processWide: yield); string label = yield ? "one request (yieldsOnce)" : "equal-weight mix of five requests"; print($"{shape} / {flow}: {m.Count:N0} RPCs, {workers} workers, {m.Seconds:F3} s, {m.RpcPerSec:N0} RPC/s; worker startup {m.StartupSeconds * 1000:F1} ms; " + $"{(yield ? "process-wide" : "worker-thread")} allocation total {m.AllocatedBytes:N0} B ({m.BytesPerRpc:F0} B/RPC, including method and harness allocations); {label}; {m.Cpu.TrailingText}."); summary.Add(new BenchmarkRunner.ChartRow($"{shape} / {flow}", m.RpcPerSec, $"{m.Count,12:N0} RPCs {m.BytesPerRpc,6:F0} B/RPC {m.Cpu.TrailingText}")); + var resultRow = BenchResults.Row(RowId(workers == 1 ? "async1" : "asyncn", shape, flow), m.RpcPerSec, m.Count, m.Cpu); + resultRow.Insert(1, "shape", shape); + resultRow.Insert(2, "flow", flow.ToString()); + resultRow.Insert(5, "bytesPerRpc", Math.Round(m.BytesPerRpc)); + resultRow.Insert(6, "startupMs", Math.Round(m.StartupSeconds * 1000, 1)); + if (perShape != null) resultRow["bytesPerShape"] = BenchResults.Numbers(perShape); + resultRows.Add(resultRow); } finally { Handler.DestroySession(session); } } BenchmarkRunner.PrintBarChart($"ProcessAsync(bytes), awaited workers - {Config.Serializer.Name} - {workers} worker{(workers == 1 ? "" : "s")} - RPC/s by registration", "Registration", summary); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("async", seconds); + doc["workers"] = workers; + doc["rows"] = resultRows; + BenchResults.Write($"async-w{workers}", doc, print); + } } + /// The results id of a registration row: <prefix>-<shape>-<flow> in lower case, as benchmarks.json names the series. + internal static string RowId(string prefix, string shape, RpcContextFlow flow) => $"{prefix}-{shape.ToLowerInvariant()}-{flow.ToString().ToLowerInvariant()}"; + /// /// The scaling gate: the inline None rows at 1, 2 and workers, /// paired runs, medians, and the 2/1 and N/1 ratios. Returns false when any N/1 ratio is below /// . A process-wide serialization point on the async path holds the ratio near 1.3 - /// on any core count; the per-thread cache measured about 6.8 on 16 threads. + /// on any core count; the per-thread cache measured about 6.8 on 16 threads. When is given, + /// the gate cells (median, the paired runs, CPU), the ratios and the verdict are added to it for the results file. /// - internal static async Task ScaleAsync(Action print, double seconds, int workers, double threshold, int runs = 3) + internal static async Task ScaleAsync(Action print, double seconds, int workers, double threshold, int runs = 3, JsonObject report = null) { if (workers < 2) throw new ArgumentOutOfRangeException(nameof(workers)); var counts = new[] { 1, 2, workers }.Distinct().ToArray(); @@ -101,6 +122,8 @@ internal static async Task ScaleAsync(Action print, double seconds } bool pass = true; + var cells = new JsonArray(); + var gate = new JsonArray(); print(""); print($"| Registration | 1 worker | 2 workers | {workers} workers | 2/1 | {workers}/1 | gate | CPU at {workers} workers |"); print("| --- | ---: | ---: | ---: | ---: | ---: | --- | --- |"); @@ -113,11 +136,32 @@ internal static async Task ScaleAsync(Action print, double seconds bool ok = many / one >= threshold; pass &= ok; print($"| {shape} / {flow} | {one:N0} | {two:N0} | {many:N0} | {two / one:F2} | {many / one:F2} | {(ok ? "pass" : "FAIL")} | {cpu.TrailingText} |"); + foreach (int w in counts) + { + var list = results[(shape, flow, w)]; + var cell = BenchResults.Row(RowId("scale", shape, flow), Median(list.Select(m => m.RpcPerSec).ToList()), list.Sum(m => m.Count), MedianCpu(list)); + cell.Insert(1, "shape", shape); + cell.Insert(2, "flow", flow.ToString()); + cell.Insert(3, "workers", w); + cell.Insert(5, "values", BenchResults.Numbers(list.Select(m => Math.Round(m.RpcPerSec)))); + cells.Add(cell); + } + gate.Add(new JsonObject { ["shape"] = shape, ["flow"] = flow.ToString(), ["twoOverOne"] = Math.Round(two / one, 2), ["nOverOne"] = Math.Round(many / one, 2), ["pass"] = ok }); } print(""); print(pass ? $"Scaling check passed: every inline row scales at least {threshold:0.0#}x from 1 to {workers} workers." : $"Scaling check FAILED: an inline row scales less than {threshold:0.0#}x from 1 to {workers} workers; look for a process-wide serialization point on the ProcessAsync path."); print("The 2/1 ratio is diagnostic (the lock was already visible at two workers, about 1.3); it is not gated."); + if (report != null) + { + report["workers"] = workers; + report["threshold"] = threshold; + report["runs"] = runs; + report["counts"] = BenchResults.Numbers(counts.Select(c => (double)c)); + report["rows"] = cells; + report["gate"] = gate; + report["verdict"] = pass ? "pass" : "fail"; + } return pass; } @@ -128,12 +172,14 @@ internal static async Task ScaleAsync(Action print, double seconds /// per request at one worker as reports them, so a regression in one serializer's path or in the /// suspending path shows up on its own. The process-wide serializer is switched with /// for each block and restored afterwards. A failure is printed - /// after the rows measured so far and is not thrown, so it cannot change the gate's exit code. + /// after the rows measured so far and is not thrown, so it cannot change the gate's exit code. When + /// is given, the rows measured and any failure are added to it for the results file. /// - internal static async Task ScaleDiagnosticsAsync(Action print, double seconds, int workers) + internal static async Task ScaleDiagnosticsAsync(Action print, double seconds, int workers, JsonObject report = null) { var serializers = new Func[] { () => JsmnSerializer.Instance, () => new SystemTextJsonRpcSerializer(), () => new NewtonsoftJsonRpcSerializer() }; var rows = new List(); + var diagnostics = new JsonArray(); string where = "start"; Exception failure = null; var previous = Config.Serializer; @@ -164,6 +210,12 @@ internal static async Task ScaleDiagnosticsAsync(Action print, double se var many = await Measure(session, rowInputs, workers, seconds, processWide: yield); print($" {where}, {workers,2} workers: {many.RpcPerSec,14:N0} RPC/s ({many.Count:N0} RPCs in {many.Seconds:F3} s); {many.Cpu.TrailingText}"); rows.Add($"| {shape} / {flow} | {one.RpcPerSec:N0} | {many.RpcPerSec:N0} | {many.RpcPerSec / one.RpcPerSec:F2} | {one.BytesPerRpc:F0} | {many.Cpu.TrailingText} |"); + diagnostics.Add(new JsonObject + { + ["id"] = RowId("scale-diag-" + name, shape, flow), ["serializer"] = name, ["shape"] = shape, ["flow"] = flow.ToString(), + ["one"] = Math.Round(one.RpcPerSec), ["many"] = Math.Round(many.RpcPerSec), ["ratio"] = Math.Round(many.RpcPerSec / one.RpcPerSec, 2), + ["bytesPerRpc"] = Math.Round(one.BytesPerRpc), ["cpu"] = BenchResults.Cpu(many.Cpu), + }); } finally { Handler.DestroySession(session); } } @@ -179,6 +231,11 @@ internal static async Task ScaleDiagnosticsAsync(Action print, double se print(""); if (failure != null) print($"Diagnostics stopped at {where}: {failure.GetType().Name}: {failure.Message}. The gate result above stands; the diagnostics are not part of it."); + if (report != null) + { + report["diagnostics"] = diagnostics; + if (failure != null) report["diagnosticsError"] = $"{where}: {failure.GetType().Name}: {failure.Message}"; + } } private static double Median(List values) @@ -204,8 +261,10 @@ private static ReadOnlyMemory[] Inputs(string shape) => shape == "yield" /// Checks every response, warms the row, and prints the per-shape allocation of an inline document. /// False checks, warms and asserts inline completion without printing the per-shape lines. - private static async Task ValidateAndReportAllocations(Action print, string session, string shape, RpcContextFlow flow, ReadOnlyMemory[] rowInputs, bool report = true) + /// Bytes per request for each input, as printed; null for the yield row, which is not measured per shape. + private static async Task ValidateAndReportAllocations(Action print, string session, string shape, RpcContextFlow flow, ReadOnlyMemory[] rowInputs, bool report = true) { + var perShape = shape == "yield" ? null : new double[rowInputs.Length]; var expected = shape == "yield" ? new[] { "{\"jsonrpc\":\"2.0\",\"result\":7,\"id\":6}" } : BenchmarkRunner.taskInputs.Select(t => JsonRpcProcessor.ProcessSync(t)).ToArray(); @@ -227,7 +286,9 @@ private static async Task ValidateAndReportAllocations(Action print, str } long totalBytes = GC.GetAllocatedBytesForCurrentThread() - before; if (report) print($"{shape} / {flow} / shape {i + 1}: {totalBytes} B total over 2000 requests ({totalBytes / 2000.0:F1} B/RPC), including method allocations."); + perShape[i] = Math.Round(totalBytes / 2000.0, 1); } + return perShape; } /// diff --git a/TestServer_Console/BenchResults.cs b/TestServer_Console/BenchResults.cs new file mode 100644 index 0000000..fa16f2a --- /dev/null +++ b/TestServer_Console/BenchResults.cs @@ -0,0 +1,96 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime; +using System.Runtime.InteropServices; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using AustinHarris.JsonRpc; + +namespace TestServer_Console; + +/// +/// Machine-readable results for benchmarks/charts/ingest.py. When JSONRPC_BENCH_RESULTS=<dir> is set, each +/// published mode writes <dir>/<stem>-<run>.json beside its text output, with the run number from +/// JSONRPC_BENCH_RUN (1 when unset) and the commit from JSONRPC_BENCH_COMMIT. Every row carries the id of the +/// benchmarks.json series it feeds (or an id ingest.py lists as unpublished), its RPC/s and the CPU of its timed window. +/// Nothing here runs inside a timed loop. +/// +internal static class BenchResults +{ + /// The output directory, or null when results are not requested. + internal static string Directory + { + get + { + var dir = Environment.GetEnvironmentVariable("JSONRPC_BENCH_RESULTS"); + return string.IsNullOrWhiteSpace(dir) ? null : dir; + } + } + + /// True when JSONRPC_BENCH_RESULTS is set. + internal static bool Enabled => Directory != null; + + /// The run number for the file name: JSONRPC_BENCH_RUN, or 1. + internal static int Run => int.TryParse(Environment.GetEnvironmentVariable("JSONRPC_BENCH_RUN"), out int run) && run > 0 ? run : 1; + + /// The runtime as the results record it: framework, build configuration and GC mode. + internal static string Runtime => +#if DEBUG + $"{RuntimeInformation.FrameworkDescription}, Debug, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; +#else + $"{RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; +#endif + + /// A results document with the fields every mode shares; the caller adds its parameters and rows. + internal static JsonObject Document(string mode, double seconds) => new JsonObject + { + ["mode"] = mode, + ["machine"] = MachineDescription.Current(), + ["runtime"] = Runtime, + ["serializer"] = Config.Serializer.Name, + ["date"] = DateTime.UtcNow.ToString("yyyy-MM-dd"), + ["seconds"] = seconds, + ["commit"] = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT") is { Length: > 0 } commit ? commit : null, + ["run"] = Run, + ["cores"] = Environment.ProcessorCount, + }; + + /// One result row: its series id, RPC/s rounded to a whole request, the request count and the CPU of the window. + internal static JsonObject Row(string id, double rpcPerSec, long rpcs, CpuUsage cpu) + { + var row = new JsonObject { ["id"] = id, ["rpcPerSec"] = Math.Round(rpcPerSec), ["rpcs"] = rpcs }; + var usage = Cpu(cpu); + if (usage.Count > 0) row["cpu"] = usage; + return row; + } + + /// The CPU percentages that were measured, to one decimal; a figure the platform cannot provide is left out. + internal static JsonObject Cpu(CpuUsage cpu) + { + var o = new JsonObject(); + if (cpu.SystemPercent is double system) o["system"] = Math.Round(system, 1); + if (cpu.ProcessPercent is double process) o["process"] = Math.Round(process, 1); + if (cpu.ClientPercent is double clients) o["clients"] = Math.Round(clients, 1); + return o; + } + + /// The path a mode's file goes to: <dir>/<stem>-<run>.json. + internal static string PathFor(string stem) => Path.Combine(Directory, $"{stem}-{Run}.json"); + + /// Writes the document as indented UTF-8 JSON without a byte order mark and prints the path. + internal static void Write(string stem, JsonObject document, Action print, string path = null) + { + path ??= PathFor(stem); + var dir = Path.GetDirectoryName(Path.GetFullPath(path)); + if (!string.IsNullOrEmpty(dir)) System.IO.Directory.CreateDirectory(dir); + var json = document.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping }); + File.WriteAllText(path, json + "\n", new UTF8Encoding(false)); + print?.Invoke($"wrote {path}"); + } + + /// A JSON array of numbers. + internal static JsonArray Numbers(IEnumerable values) => new JsonArray(values.Select(v => (JsonNode)JsonValue.Create(v)).ToArray()); +} diff --git a/TestServer_Console/Benchmark.cs b/TestServer_Console/Benchmark.cs index a004f26..a0fbc4c 100644 --- a/TestServer_Console/Benchmark.cs +++ b/TestServer_Console/Benchmark.cs @@ -3,6 +3,7 @@ using System.Diagnostics; using System.Linq; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -65,6 +66,7 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize RunSync(session, inputs, serializer, Math.Min(threads, 2), 0.5); // allocation profile per request shape (a zero here means the request never touches the GC) + var allocations = new JsonArray(); { using var w = new PooledByteBufferWriter(1024); sb.Clear(); @@ -78,15 +80,15 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize long per = (GC.GetAllocatedBytesForCurrentThread() - before) / reps; w.Clear(); JsonRpcProcessor.Process(session, inputs[k], w, null, serializer); sb.Append(" ").Append(per.ToString().PadLeft(6)).Append(" B ").Append(taskInputs[k]).Append(" -> ").Append(w.ToString()).Append('\n'); + allocations.Add(new JsonObject { ["request"] = taskInputs[k], ["bytes"] = per }); } _print(sb.ToString()); } // Sweep 1, 2, 4, ... up to the requested thread count (the count itself is always included). - var threadCounts = new List(); - for (int t = 1; t < threads; t *= 2) threadCounts.Add(t); - threadCounts.Add(threads); + var threadCounts = Doubling(threads); var rows = new List(threadCounts.Count); + var resultRows = new JsonArray(); foreach (var t in threadCounts) { @@ -95,6 +97,10 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize double mbIn = rps * bytesIn / inputs.Length / (1024.0 * 1024.0); double mbOut = rps * bytesOut / inputs.Length / (1024.0 * 1024.0); rows.Add(new ChartRow($"{t} thread{(t == 1 ? "" : "s")}", rps, $"{1e9 / rps * t:N0} ns/RPC per thread {cpu.TrailingText}")); + var resultRow = BenchResults.Row("ours-sync", rps, total, cpu); + resultRow.Insert(1, "threads", t); + resultRow.Insert(4, "nsPerRequest", Math.Round(1e9 / rps * t)); + resultRows.Add(resultRow); // The detailed boxes are only printed for the two documented points: one thread and all threads. if (t != 1 && t != threads) continue; @@ -119,6 +125,25 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize } PrintBarChart($"Sync benchmark - {serializer.Name} - RPC/s by thread count", "Threads", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("sync", seconds); + doc["threads"] = threads; + doc["allocations"] = allocations; + doc["rows"] = resultRows; + BenchResults.Write("sync", doc, _print); + } + } + + /// The published worker ladder: 1, 2, 4, ... below , then itself. + internal static List Doubling(int max) + { + if (max < 1) throw new ArgumentOutOfRangeException(nameof(max)); + var counts = new List(); + for (int t = 1; t < max; t *= 2) counts.Add(t); + counts.Add(max); + return counts; } internal static (long count, double seconds, CpuUsage cpu) RunSync(string session, ReadOnlyMemory[] inputs, JsonRpcSerializer serializer, int threads, double seconds) @@ -174,6 +199,7 @@ internal static void Benchmark(Action print = null) var iterations = 8; var cnt = 50; var results = new List(iterations); + var resultRows = new JsonArray(); var minIterationTime = TimeSpan.FromMilliseconds(500); // Warm up on the exact path the iterations use, for long enough that tiered compilation has replaced @@ -221,12 +247,23 @@ internal static void Benchmark(Action print = null) double seconds = currentStopwatch.Elapsed.TotalSeconds; long total = (long)cnt * currentBatches; results.Add(new ChartRow($"#{iteration} {cnt,11:N0}", total / seconds, $"{total,11:N0} RPCs in {seconds:F3} s {cpu.TrailingText}")); + var resultRow = BenchResults.Row("legacy-string", total / seconds, total, cpu); + resultRow.Insert(1, "batch", cnt); + resultRow.Insert(4, "seconds", Math.Round(seconds, 3)); + resultRows.Add(resultRow); } lock (renderLock) { PrintBarChart("Benchmark Summary - RPC/s by batch size", " # Batch size", results); } + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("legacy", minIterationTime.TotalSeconds); + doc["rows"] = resultRows; + BenchResults.Write("legacy", doc, _print); + } } /// diff --git a/TestServer_Console/CompareBenchmark.cs b/TestServer_Console/CompareBenchmark.cs index f0cf202..07215e7 100644 --- a/TestServer_Console/CompareBenchmark.cs +++ b/TestServer_Console/CompareBenchmark.cs @@ -8,6 +8,7 @@ using System.Net.Sockets; using System.Text; using System.Text.Json; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -78,6 +79,7 @@ internal static async Task RunAsync(Action print, double seconds = 3, in { print($"{Versions()}; Kestrel on loopback, {clients} clients, pipeline {pipeline}, {seconds:0.#} s per row\n"); var rows = new List(); + var resultRows = new JsonArray(); var session = Handler.DefaultSessionId(); // ---- in-process floors @@ -85,31 +87,37 @@ internal static async Task RunAsync(Action print, double seconds = 3, in BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, Warm); var (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, seconds); rows.Add(new BenchmarkRunner.ChartRow("JSON-RPC.Net in-process, 1 thread", total / elapsed, $"{total,12:N0} RPCs direct call, bytes in, bytes out {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row("ours-direct-1", total / elapsed, total, cpu)); print($" JSON-RPC.Net in-process done ({rows[^1].RpcPerSec:N0} RPC/s)"); PipeRun(newLine, Warm, pipeline); var (count, secs, usage) = PipeRun(newLine, seconds, pipeline); rows.Add(new BenchmarkRunner.ChartRow("StreamJsonRpc in-process, 1 client", count / secs, $"{count,12:N0} RPCs Pipe pair, newline framing, STJ formatter, {pipeline} pipelined {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("sjr-pipe", count / secs, count, usage)); print($" StreamJsonRpc in-process done ({count / secs:N0} RPC/s)"); await ProxyRun(Warm); (count, secs, usage) = await ProxyRun(seconds); rows.Add(new BenchmarkRunner.ChartRow("StreamJsonRpc proxy, sequential await", count / secs, $"{count,12:N0} RPCs {secs / count * 1e6:N1} us per round trip {usage.TrailingText}")); + var proxyRow = BenchResults.Row("sjr-proxy", count / secs, count, usage); + proxyRow["usPerRequest"] = Math.Round(secs / count * 1e6, 1); + resultRows.Add(proxyRow); print($" StreamJsonRpc proxy done ({count / secs:N0} RPC/s)"); // ---- Kestrel TCP, same client for every row - foreach (var (label, port, inputs, prefix) in new[] + foreach (var (label, id, port, inputs, prefix) in new[] { - ("JSON-RPC.Net TCP, raw documents", oursPort, raw, JsonRpcPrefix), - ("StreamJsonRpc TCP, newline + STJ", sjrNewLineStjPort, newLine, JsonRpcPrefix), - ("StreamJsonRpc TCP, Content-Length + STJ", sjrHeaderStjPort, header, (byte[])null), - ("StreamJsonRpc TCP, Content-Length + Json.NET", sjrHeaderNewtonsoftPort, header, (byte[])null), + ("JSON-RPC.Net TCP, raw documents", "ours-tcp-jsonrpc", oursPort, raw, JsonRpcPrefix), + ("StreamJsonRpc TCP, newline + STJ", "sjr-newline-stj", sjrNewLineStjPort, newLine, JsonRpcPrefix), + ("StreamJsonRpc TCP, Content-Length + STJ", "sjr-cl-stj", sjrHeaderStjPort, header, (byte[])null), + ("StreamJsonRpc TCP, Content-Length + Json.NET", "sjr-cl-newtonsoft", sjrHeaderNewtonsoftPort, header, (byte[])null), }) { Probe(port, inputs, label); KestrelBenchmark.TcpRun(port, inputs, clients, Warm, pipeline, prefix); (count, secs, usage) = KestrelBenchmark.TcpRun(port, inputs, clients, seconds, pipeline, prefix); rows.Add(new BenchmarkRunner.ChartRow(label, count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(id, count / secs, count, usage)); print($" {label} done ({count / secs:N0} RPC/s)"); } @@ -117,22 +125,40 @@ internal static async Task RunAsync(Action print, double seconds = 3, in await GrpcCompare.UnaryRun(grpcPort, clients, pipeline, Warm); (count, secs, usage) = await GrpcCompare.UnaryRun(grpcPort, clients, pipeline, seconds); rows.Add(new BenchmarkRunner.ChartRow("gRPC for .NET unary, HTTP/2", count / secs, $"{count,12:N0} RPCs {clients} channels, {pipeline} calls in flight each {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("grpc-unary", count / secs, count, usage)); print($" gRPC unary done ({count / secs:N0} RPC/s)"); await GrpcCompare.StreamRun(grpcPort, clients, pipeline, Warm); (count, secs, usage) = await GrpcCompare.StreamRun(grpcPort, clients, pipeline, seconds); rows.Add(new BenchmarkRunner.ChartRow("gRPC for .NET bidirectional stream", count / secs, $"{count,12:N0} RPCs {clients} streams, {pipeline} calls in flight each {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("grpc-stream", count / secs, count, usage)); print($" gRPC stream done ({count / secs:N0} RPC/s)"); BenchmarkRunner.PrintBarChart("JSON-RPC.Net vs StreamJsonRpc vs gRPC - RPC/s", "Library / transport", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("compare", seconds); + doc["clients"] = clients; + doc["pipeline"] = pipeline; + doc["versions"] = new JsonObject + { + ["jsonrpc"] = typeof(JsonRpcProcessor).Assembly.GetName().Version?.ToString(), + ["streamJsonRpc"] = PackageVersion(typeof(JsonRpc)), + ["grpc"] = PackageVersion(typeof(Grpc.Net.Client.GrpcChannel)), + }; + doc["rows"] = resultRows; + BenchResults.Write("compare", doc, print); + } } } /// - /// Every library and transport at 1, 2, 4, 8 and 16 client connections: the data for the README's - /// multi-series chart. Same host, same five calls, same pipeline depth per connection as - /// . Prints a table and, when is given, writes the - /// numbers as JSON for benchmarks/charts/render.py. + /// Every library and transport at 1, 2, 4, ... client connections up to the core count + /// (): the data for the README's multi-series chart. Same host, same five + /// calls, same pipeline depth per connection as . Prints a table and, when + /// is given (or JSONRPC_BENCH_RESULTS is set, as sweep-<run>.json there), + /// writes the numbers as JSON for benchmarks/charts/render.py. /// internal static async Task SweepAsync(Action print, double seconds = 2, string outputPath = null, int pipeline = 256) { @@ -143,17 +169,17 @@ internal static async Task SweepAsync(Action print, double seconds = 2, var batch = new[] { KestrelBenchmark.BuildBatch(raw, 100) }; await using var h = await StartAsync(pipeline); - int[] connections = { 1, 2, 4, 8, 16 }; - (string name, string kind, Func> run)[] series = + int[] connections = BenchmarkRunner.Doubling(Environment.ProcessorCount).ToArray(); + (string name, string id, string kind, Func> run)[] series = { - ("JSON-RPC.Net, TCP", "ours", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.OursPort, raw, c, s, pipeline, JsonRpcPrefix))), - ("JSON-RPC.Net, HTTP, batch of 100 per POST", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, batch, 100, c, s)), - ("JSON-RPC.Net, HTTP, 1 request per POST", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, raw, 1, c, s)), - ("StreamJsonRpc, TCP, newline + System.Text.Json", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrNewLineStjPort, newLine, c, s, pipeline, JsonRpcPrefix))), - ("StreamJsonRpc, TCP, Content-Length + System.Text.Json", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderStjPort, header, c, s, pipeline, null))), - ("StreamJsonRpc, TCP, Content-Length + Json.NET", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderNewtonsoftPort, header, c, s, pipeline, null))), - ("gRPC for .NET, unary", "grpc", (c, s) => GrpcCompare.UnaryRun(h.GrpcPort, c, pipeline, s)), - ("gRPC for .NET, bidirectional stream", "grpc", (c, s) => GrpcCompare.StreamRun(h.GrpcPort, c, pipeline, s)), + ("JSON-RPC.Net, TCP", "sweep-ours-tcp", "ours", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.OursPort, raw, c, s, pipeline, JsonRpcPrefix))), + ("JSON-RPC.Net, HTTP, batch of 100 per POST", "sweep-ours-http-100", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, batch, 100, c, s)), + ("JSON-RPC.Net, HTTP, 1 request per POST", "sweep-ours-http-1", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, raw, 1, c, s)), + ("StreamJsonRpc, TCP, newline + System.Text.Json", "sweep-sjr-newline-stj", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrNewLineStjPort, newLine, c, s, pipeline, JsonRpcPrefix))), + ("StreamJsonRpc, TCP, Content-Length + System.Text.Json", "sweep-sjr-cl-stj", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderStjPort, header, c, s, pipeline, null))), + ("StreamJsonRpc, TCP, Content-Length + Json.NET", "sweep-sjr-cl-newtonsoft", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderNewtonsoftPort, header, c, s, pipeline, null))), + ("gRPC for .NET, unary", "sweep-grpc-unary", "grpc", (c, s) => GrpcCompare.UnaryRun(h.GrpcPort, c, pipeline, s)), + ("gRPC for .NET, bidirectional stream", "sweep-grpc-stream", "grpc", (c, s) => GrpcCompare.StreamRun(h.GrpcPort, c, pipeline, s)), }; Probe(h.OursPort, raw, series[0].name); Probe(h.SjrNewLineStjPort, newLine, series[3].name); @@ -180,11 +206,16 @@ internal static async Task SweepAsync(Action print, double seconds = 2, for (int si = 0; si < series.Length; si++) print(" " + series[si].name.PadRight(56) + string.Join("", connections.Select((_, ci) => $"{results[si, ci],14:N0}"))); + if (outputPath == null && BenchResults.Enabled) outputPath = BenchResults.PathFor("sweep"); if (outputPath != null) { var doc = new { + mode = "sweep", machine = MachineDescription.Current(), + runtime = BenchResults.Runtime, + commit = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT") is { Length: > 0 } commit ? commit : null, + run = BenchResults.Run, cores = Environment.ProcessorCount, date = DateTime.Now.ToString("yyyy-MM-dd"), secondsPerCell = seconds, @@ -192,13 +223,15 @@ internal static async Task SweepAsync(Action print, double seconds = 2, connections, series = series.Select((s, si) => new { - s.name, s.kind, + s.name, s.id, s.kind, rpcPerSec = connections.Select((_, ci) => Math.Round(results[si, ci])).ToArray(), cpuSystem = connections.Select((_, ci) => usages[si, ci].SystemPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray(), cpuProcess = connections.Select((_, ci) => usages[si, ci].ProcessPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray(), cpuClients = connections.Select((_, ci) => usages[si, ci].ClientPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray() }).ToArray(), }; + var outputDir = Path.GetDirectoryName(Path.GetFullPath(outputPath)); + if (!string.IsNullOrEmpty(outputDir)) Directory.CreateDirectory(outputDir); File.WriteAllText(outputPath, JsonSerializer.Serialize(doc, new JsonSerializerOptions { WriteIndented = true }) + "\n"); print($"\nwrote {outputPath}"); } diff --git a/TestServer_Console/KestrelBenchmark.cs b/TestServer_Console/KestrelBenchmark.cs index 2345b23..fa0e6cd 100644 --- a/TestServer_Console/KestrelBenchmark.cs +++ b/TestServer_Console/KestrelBenchmark.cs @@ -8,6 +8,7 @@ using System.Net.Http.Headers; using System.Net.Sockets; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -68,6 +69,10 @@ internal static async Task RunAsync(Action print, double seconds = 3, in print($"Kestrel on {httpUrl} (HTTP) and 127.0.0.1:{tcpPort} (TCP), {clients} clients, pipeline {pipeline}, {seconds:0.#} s per row, EnableAsyncMethods = {(asyncMethods ? "true" : "false")}\n"); var rows = new List(); + var resultRows = new JsonArray(); + // Result ids: the EnableAsyncMethods = false rows and the two TCP rows under true feed benchmarks.json; the rest + // are kept in the file under their own ids. + string Id(string defaultId, string asyncId) => asyncMethods ? asyncId : defaultId; var session = Handler.DefaultSessionId(); var memInputs = inputs.Select(i => (ReadOnlyMemory)i).ToArray(); @@ -75,26 +80,33 @@ internal static async Task RunAsync(Action print, double seconds = 3, in BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, 0.5); // warm-up var (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, seconds); rows.Add(new BenchmarkRunner.ChartRow("in-process, 1 thread", total / elapsed, $"{total,12:N0} RPCs {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("kestrel-inproc-1", "kestrel-async-inproc-1"), total / elapsed, total, cpu)); (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow($"in-process, {clients} threads", total / elapsed, $"{total,12:N0} RPCs {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-inproc-n", "kestrel-async-inproc-n"), total / elapsed, total, cpu)); print($" in-process rows done ({rows[0].RpcPerSec:N0} / {rows[1].RpcPerSec:N0} RPC/s)"); // HTTP, one request per POST. await HttpRun(httpUrl, inputs, 1, clients, 0.5); // warm-up var (count, secs, usage) = await HttpRun(httpUrl, inputs, 1, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow("HTTP, 1 request/POST", count / secs, $"{count,12:N0} RPCs {secs / count * 1e6 * clients:N1} us/request per client {usage.TrailingText}")); + var httpRow = BenchResults.Row(Id("ours-http-1", "kestrel-async-http-1"), count / secs, count, usage); + httpRow["usPerRequest"] = Math.Round(secs / count * 1e6 * clients, 1); + resultRows.Add(httpRow); print($" HTTP single done ({count / secs:N0} RPC/s)"); // HTTP, a batch per POST. await HttpRun(httpUrl, new[] { batch }, batchSize, clients, 0.5); (count, secs, usage) = await HttpRun(httpUrl, new[] { batch }, batchSize, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow($"HTTP, batch of {batchSize}/POST", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-http-100", "kestrel-async-http-100"), count / secs, count, usage)); print($" HTTP batch done ({count / secs:N0} RPC/s)"); // TCP, pipelined. TcpRun(tcpPort, inputs, clients, 0.5, pipeline, ResultPrefix); (count, secs, usage) = TcpRun(tcpPort, inputs, clients, seconds, pipeline, ResultPrefix); rows.Add(new BenchmarkRunner.ChartRow($"TCP, {pipeline} pipelined{(asyncMethods ? ", inline methods" : "")}", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-tcp", "ours-tcp-async-inline"), count / secs, count, usage)); print($" TCP done ({count / secs:N0} RPC/s)"); if (asyncMethods) @@ -104,10 +116,22 @@ internal static async Task RunAsync(Action print, double seconds = 3, in TcpRun(tcpPort, yieldInputs, clients, 0.5, pipeline, ResultPrefix); (count, secs, usage) = TcpRun(tcpPort, yieldInputs, clients, seconds, pipeline, ResultPrefix); rows.Add(new BenchmarkRunner.ChartRow($"TCP, {pipeline} pipelined, yielding methods", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("ours-tcp-async-yield", count / secs, count, usage)); print($" TCP yielding done ({count / secs:N0} RPC/s)"); } BenchmarkRunner.PrintBarChart($"Kestrel benchmark - {Config.Serializer.Name} - EnableAsyncMethods = {(asyncMethods ? "true" : "false")} - RPC/s by transport", "Transport", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document(asyncMethods ? "kestrel-async" : "kestrel", seconds); + doc["clients"] = clients; + doc["pipeline"] = pipeline; + doc["batchSize"] = batchSize; + doc["asyncMethods"] = asyncMethods; + doc["rows"] = resultRows; + BenchResults.Write(asyncMethods ? "kestrel-async" : "kestrel", doc, print); + } } finally { diff --git a/TestServer_Console/Program.cs b/TestServer_Console/Program.cs index 4fec7ce..e638998 100644 --- a/TestServer_Console/Program.cs +++ b/TestServer_Console/Program.cs @@ -33,16 +33,22 @@ static void Main(string[] args) // `dotnet run -- --scale [seconds] [workers] [threshold]` is the release gate for the ProcessAsync path: // the inline rows at 1, 2 and N workers, three paired runs, medians; exit code 1 when N/1 is below the threshold. // After the gate it prints a per-serializer diagnostics table that never changes the exit code; a final - // `--no-diagnostics` argument skips it. + // `--no-diagnostics` argument skips it. The worker count defaults to the core count. if (args.Length > 0 && args[0] == "--scale") { double seconds = args.Length > 1 && double.TryParse(args[1], out var s) ? s : 3; - int workers = args.Length > 2 && int.TryParse(args[2], out var t) ? t : 16; + int workers = args.Length > 2 && int.TryParse(args[2], out var t) ? t : Environment.ProcessorCount; double threshold = args.Length > 3 && double.TryParse(args[3], out var r) ? r : 4.0; bool diagnostics = !(args.Length > 1 && args[args.Length - 1] == "--no-diagnostics"); - bool pass = AsyncBenchmark.ScaleAsync(Console.WriteLine, seconds, workers, threshold).GetAwaiter().GetResult(); + var report = BenchResults.Enabled ? BenchResults.Document("scale", seconds) : null; + bool pass = AsyncBenchmark.ScaleAsync(Console.WriteLine, seconds, workers, threshold, report: report).GetAwaiter().GetResult(); Environment.ExitCode = pass ? 0 : 1; - if (diagnostics) AsyncBenchmark.ScaleDiagnosticsAsync(Console.WriteLine, seconds, workers).GetAwaiter().GetResult(); + if (diagnostics) AsyncBenchmark.ScaleDiagnosticsAsync(Console.WriteLine, seconds, workers, report).GetAwaiter().GetResult(); + if (report != null) + { + report["exitCode"] = Environment.ExitCode; + BenchResults.Write("scale", report, Console.WriteLine); + } return; } @@ -75,7 +81,8 @@ static void Main(string[] args) return; } - // `dotnet run -- --sweep [seconds] [output.json]` measures every library and transport at 1, 2, 4, 8 and 16 connections. + // `dotnet run -- --sweep [seconds] [output.json]` measures every library and transport at 1, 2, 4, ... connections up to + // the core count; with JSONRPC_BENCH_RESULTS set and no output path, the file goes to /sweep-.json. if (args.Length > 0 && args[0] == "--sweep") { double seconds = args.Length > 1 && double.TryParse(args[1], out var s) ? s : 2; diff --git a/benchmarks/Baseline/Program.cs b/benchmarks/Baseline/Program.cs index f0d992e..b7506d3 100644 --- a/benchmarks/Baseline/Program.cs +++ b/benchmarks/Baseline/Program.cs @@ -1,6 +1,13 @@ using System; using System.Collections.Generic; using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Runtime; +using System.Runtime.InteropServices; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -47,21 +54,28 @@ public static int Main() while (warm.Elapsed < TimeSpan.FromSeconds(1)) RunBatch(5000); var rows = new List<(int size, double rps)>(); + var resultRows = new JsonArray(); int cnt = 50; for (int iteration = 1; iteration <= 8; iteration++) { cnt *= iteration; int batches = 0; + var cpu = CpuWindow.Start(); var sw = Stopwatch.StartNew(); do { RunBatch(cnt); batches++; } while (sw.Elapsed < TimeSpan.FromMilliseconds(500)); sw.Stop(); + var usage = cpu.Stop(); double rps = (long)cnt * batches / sw.Elapsed.TotalSeconds; rows.Add((cnt, rps)); Console.WriteLine($"#{iteration} batch {cnt,10:N0}: {rps,14:N0} RPC/s ({(long)cnt * batches,12:N0} RPCs in {sw.Elapsed.TotalSeconds:F3} s)"); + var row = new JsonObject { ["id"] = "headline-123-string", ["batch"] = cnt, ["rpcPerSec"] = Math.Round(rps), ["rpcs"] = (long)cnt * batches, ["seconds"] = Math.Round(sw.Elapsed.TotalSeconds, 3) }; + if (usage.Count > 0) row["cpu"] = usage; + resultRows.Add(row); } double peak = 0; int peakSize = 0; foreach (var (size, rps) in rows) if (rps > peak) { peak = rps; peakSize = size; } Console.WriteLine($"peak {peak:N0} RPC/s at batch {peakSize:N0}"); + WriteResults(version, resultRows); GC.KeepAlive(service); return 0; } @@ -79,5 +93,150 @@ private static void RunBatch(int count) done.Wait(); } } + + /// + /// With JSONRPC_BENCH_RESULTS set, writes baseline-<JSONRPC_BENCH_RUN>.json there in the shape TestServer_Console's + /// results use (benchmarks/charts/ingest.py reads it): one row per batch size, id headline-123-string. + /// + private static void WriteResults(Version version, JsonArray rows) + { + var dir = Environment.GetEnvironmentVariable("JSONRPC_BENCH_RESULTS"); + if (string.IsNullOrWhiteSpace(dir)) return; + int run = int.TryParse(Environment.GetEnvironmentVariable("JSONRPC_BENCH_RUN"), out int r) && r > 0 ? r : 1; + var commit = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT"); + var doc = new JsonObject + { + ["mode"] = "baseline", + ["machine"] = Machine(), + ["runtime"] = $"{RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}", + ["serializer"] = "Json.NET (1.2.3)", + ["date"] = DateTime.UtcNow.ToString("yyyy-MM-dd"), + ["seconds"] = 0.5, + ["commit"] = string.IsNullOrEmpty(commit) ? null : commit, + ["run"] = run, + ["cores"] = Environment.ProcessorCount, + ["version"] = version.ToString(), + ["rows"] = rows, + }; + Directory.CreateDirectory(dir); + var path = Path.Combine(dir, $"baseline-{run}.json"); + File.WriteAllText(path, doc.ToJsonString(new JsonSerializerOptions { WriteIndented = true }) + "\n", new UTF8Encoding(false)); + Console.WriteLine($"wrote {path}"); + } + + /// CPU model, capacity, OS and runtime: the string TestServer_Console's machine line gives on Linux. + private static string Machine() + { + var overridden = Environment.GetEnvironmentVariable("JSONRPC_BENCH_MACHINE"); + if (!string.IsNullOrWhiteSpace(overridden)) return overridden; + string model = Environment.GetEnvironmentVariable("PROCESSOR_IDENTIFIER") ?? RuntimeInformation.ProcessArchitecture.ToString(); + int hostCpus = 0; + bool named = false; + if (OperatingSystem.IsLinux()) + { + try + { + foreach (var line in File.ReadLines("/proc/cpuinfo")) + { + if (!named && line.StartsWith("model name", StringComparison.Ordinal)) { model = line.Split(':', 2)[1].Trim(); named = true; } + if (line.StartsWith("processor", StringComparison.Ordinal) && line.Contains(':')) hostCpus++; + } + } + catch (IOException) { } + } + string capacity = Environment.ProcessorCount + " cores"; + if (OperatingSystem.IsLinux() && CpuWindow.Quota() > 0 && hostCpus > 0) capacity += $" of {hostCpus} host CPUs, cgroup quota"; + return $"{model}, {capacity}, {RuntimeInformation.OSDescription}, {RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; + } + } + + /// + /// CPU over one timed window, with TestServer_Console's CpuMeter arithmetic: the process as a share of wall time times + /// the cores, the machine from cgroup cpu.stat under a quota, else /proc/stat, else GetSystemTimes. There are no client + /// threads here, so there is no client figure. + /// + internal sealed class CpuWindow + { + private readonly Stopwatch _wall = Stopwatch.StartNew(); + private readonly double _process = ProcessSeconds(); + private readonly double _quota = Quota(); + private readonly ulong _busy, _total; + private readonly bool _system; + + private CpuWindow() { _system = SystemTimes(out _busy, out _total); } + + internal static CpuWindow Start() => new CpuWindow(); + + internal JsonObject Stop() + { + double wall = _wall.Elapsed.TotalSeconds; + var o = new JsonObject(); + if (_system && SystemTimes(out ulong busy, out ulong total)) + { + // Under a quota the counters are cgroup microseconds of CPU; otherwise busy and total ticks. + double system = _quota > 0 ? Percent((busy - _busy) / 1_000_000.0, wall * _quota) : Percent(busy - _busy, total - _total); + o["system"] = Math.Round(system, 1); + } + o["process"] = Math.Round(Percent(ProcessSeconds() - _process, wall * Environment.ProcessorCount), 1); + return o; + } + + private static double Percent(double amount, double capacity) => capacity > 0 ? Math.Clamp(100 * amount / capacity, 0, 100) : 0; + + private static double ProcessSeconds() + { + using var p = Process.GetCurrentProcess(); + return p.TotalProcessorTime.TotalSeconds; + } + + /// The cgroup v2 CPU quota in cores, or 0 when there is none. + internal static double Quota() + { + try + { + if (!OperatingSystem.IsLinux() || !File.Exists("/sys/fs/cgroup/cpu.max")) return 0; + var parts = File.ReadAllText("/sys/fs/cgroup/cpu.max").Split((char[])null, StringSplitOptions.RemoveEmptyEntries); + if (parts.Length < 2 || parts[0] == "max") return 0; + double quota = double.Parse(parts[0], System.Globalization.CultureInfo.InvariantCulture); + double period = double.Parse(parts[1], System.Globalization.CultureInfo.InvariantCulture); + return quota > 0 && period > 0 ? quota / period : 0; + } + catch (Exception) { return 0; } + } + + private bool SystemTimes(out ulong busy, out ulong total) + { + busy = total = 0; + try + { + if (OperatingSystem.IsLinux() && _quota > 0) + { + var line = File.ReadLines("/sys/fs/cgroup/cpu.stat").FirstOrDefault(s => s.StartsWith("usage_usec ", StringComparison.Ordinal)); + return line != null && ulong.TryParse(line.AsSpan("usage_usec ".Length), out busy); + } + if (OperatingSystem.IsLinux()) + { + var parts = File.ReadLines("/proc/stat").First().Split((char[])null, StringSplitOptions.RemoveEmptyEntries); + if (parts.Length < 6 || parts[0] != "cpu") return false; + for (int i = 1; i < parts.Length; i++) total += ulong.Parse(parts[i]); + busy = total - ulong.Parse(parts[4]) - ulong.Parse(parts[5]); + return true; + } + if (OperatingSystem.IsWindows() && GetSystemTimes(out var idle, out var kernel, out var user)) + { + busy = kernel.Value - idle.Value + user.Value; + total = kernel.Value + user.Value; + return true; + } + } + catch (Exception) { } + return false; + } + + [StructLayout(LayoutKind.Sequential)] + private struct FileTime { internal uint Low; internal uint High; internal readonly ulong Value => ((ulong)High << 32) | Low; } + + [DllImport("kernel32.dll")] + private static extern bool GetSystemTimes(out FileTime idle, out FileTime kernel, out FileTime user); } } diff --git a/benchmarks/charts/benchmarks.json b/benchmarks/charts/benchmarks.json index 59c2ad8..9bd4e32 100644 --- a/benchmarks/charts/benchmarks.json +++ b/benchmarks/charts/benchmarks.json @@ -2,25 +2,39 @@ "schema": 1, "machine": "AMD Ryzen 7 7800X3D (8 cores / 16 threads, 4.2 GHz), 64 GB, Windows 11", "runtime": ".NET 10, Release, Server GC, built-in serializer", + "summary": [ + { "what": "Library alone, {workers} threads", "api": "`Process(bytes)`, dedicated threads", "source": "sync:ours-sync:last", "details": "[Sync](#sync-the-library-alone)" }, + { "what": "Library alone, {workers} workers", "api": "`ProcessAsync(bytes)`, awaited workers, methods that complete inline", "source": "asyncn:asyncn-sync-none,asyncn-task-flow,asyncn-task-none,asyncn-valuetask-flow,asyncn-valuetask-none:range", "details": "[Async](#async-processasync-awaited-workers); the spread is across registrations, not runs" }, + { "what": "Library alone, {workers} workers, one real suspension per request", "api": "`ProcessAsync(bytes)`, `yieldsOnce`", "source": "asyncn:asyncn-yield-none", "details": "" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-tcp", "details": "[Kestrel](#kestrel-through-the-aspnetcore-package)" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = true`, methods that complete inline", "source": "kestrel:ours-tcp-async-inline", "details": "" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = true`, methods that suspend once", "source": "kestrel:ours-tcp-async-yield", "details": "" }, + { "what": "Kestrel HTTP, batch of 100 per POST", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-http-100", "details": "" }, + { "what": "Kestrel HTTP, one request per POST", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-http-1", "details": "HTTP/1.1 round trips dominate" }, + { "what": "Legacy string API, thread pool", "api": "`Task Process(string)`, batches of {batch}", "source": "legacy:legacy-string:best", "details": "[Legacy](#legacy-string-api-scheduled-synchronous-execution); the 1.x overloads, not the byte path" } + ], "sets": { "headline": { "title": "JSON-RPC.Net 1.2.3 and 2.0 on one machine", "date": "2026-09-25", "source": "README.md, Performance table; branch async-scratch-cache", "harness": "benchmarks/Baseline (1.2.3 from NuGet), then TestServer_Console menu entry t, --sync 3 16 and --async 3 16, one session", - "workload": "the five benchmark requests; the string rows are the best batch size of the thread-pool loop, the byte rows a loop on 16 dedicated threads or 16 awaited workers", + "workload": "the five benchmark requests; the string rows are the best batch size of the thread-pool loop, the byte rows a loop on {workers} dedicated threads or {workers} awaited workers", "conditions": "idle box; one run per row", "policy": "one run per row, all in one session; the day-to-day ranges are the sync, async, legacy and kestrel sets", + "workers": 16, + "subtitle": ["The same five requests on one machine in one session, {date}: Ryzen 7 7800X3D, .NET 10, Server GC, idle box.", "String rows: the best batch size of the thread-pool loop. Byte rows: {workers} dedicated threads or {workers} awaited workers in a loop."], + "alt": "JSON-RPC.Net 1.2.3 and 2.0 on one machine, one run per row on {date}: {p:headline-123-string} requests per second through the 1.2.3 string API, {p:headline-20-string} through the same API on 2.0, and {p:headline-20-sync} and {p:headline-20-async} through the 2.0 byte entry points", "groups": [ { "heading": "1.2.3, the last 1.x release on NuGet", "rule": false, "series": ["headline-123-string"] }, { "heading": "2.0, the same string API and the same loop", "rule": false, "series": ["headline-20-string"] }, { "heading": "2.0, the byte entry points", "rule": false, "series": ["headline-20-sync", "headline-20-async"] } ], "series": [ - { "id": "headline-123-string", "label": "Task Process(string), thread pool", "note": "best batch size, 1,200; Json.NET on every request", "family": "scale", "low": 3080000, "high": 3080000, "status": "single", "published": "3.08 M" }, - { "id": "headline-20-string", "label": "Task Process(string), thread pool", "note": "best batch size, 36,000; the 1.x overloads, kept for compatibility", "family": "ours", "low": 13300000, "high": 13300000, "status": "single", "published": "13.3 M" }, - { "id": "headline-20-sync", "label": "Process(bytes), 16 dedicated threads", "note": "the library alone; 0 B per numeric request", "family": "ours", "low": 31700000, "high": 31700000, "status": "single", "published": "31.7 M" }, - { "id": "headline-20-async", "label": "ProcessAsync(bytes), 16 awaited workers", "note": "ValueTask methods, RpcContextFlow.None; 6 B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M" } + { "id": "headline-123-string", "label": "Task Process(string), thread pool", "note": "best batch size, {batch}; Json.NET on every request", "family": "scale", "low": 3080000, "high": 3080000, "status": "single", "published": "3.08 M", "row": "1.2.3, `Task Process(string)`, thread pool, best batch size", "batch": 1200 }, + { "id": "headline-20-string", "label": "Task Process(string), thread pool", "note": "best batch size, {batch}; the 1.x overloads, kept for compatibility", "family": "ours", "low": 13300000, "high": 13300000, "status": "single", "published": "13.3 M", "row": "2.0, the same string API and the same loop", "batch": 36000, "against": "4.3×" }, + { "id": "headline-20-sync", "label": "Process(bytes), {workers} dedicated threads", "note": "the library alone; 0 B per numeric request", "family": "ours", "low": 31700000, "high": 31700000, "status": "single", "published": "31.7 M", "row": "2.0, `Process(bytes)`, {workers} dedicated threads", "against": "10.3×" }, + { "id": "headline-20-async", "label": "ProcessAsync(bytes), {workers} awaited workers", "note": "ValueTask methods, RpcContextFlow.None; {bytes} B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M", "row": "2.0, `ProcessAsync(bytes)`, {workers} awaited workers", "bytes": 6, "against": "10.4×" } ] }, "sync": { @@ -31,12 +45,16 @@ "workload": "the five benchmark requests through the byte-level JsonRpcProcessor.Process in a loop; no scheduler, no transport", "conditions": "idle box; three 3 s runs of --sync 3 in one session", "policy": "low and high over three runs; ns per request is the harness's reported figure for one of them", - "x": { "name": "worker threads", "values": [1, 2, 4, 8, 16] }, + "workers": 16, + "subtitle": ["Byte entry point in a loop, no scheduler, no transport. Ryzen 7 7800X3D, .NET 10, Server GC, two sweeps on an idle box."], + "alt": "JSON-RPC.Net alone, by worker threads: aggregate requests per second as a low-to-high band, and the reported ns per request per thread", + "x": { "name": "worker threads", "values": [1, 2, 4, 8, 16], "caption": "worker threads (each step doubles; 8 physical cores, 16 with SMT)" }, "series": [ { "id": "ours-sync", "label": "JSON-RPC.Net, byte entry point", "family": "ours", + "note": "0 bytes for numeric shapes, one string for `StringMe`", "points": [ { "low": 4570000, "high": 5000000, "status": "range", "published": "4.6 M to 5.0 M", "ns": 214 }, { "low": 8500000, "high": 9100000, "status": "range", "published": "8.5 M to 9.1 M", "ns": 235 }, @@ -47,27 +65,28 @@ } ] }, - "async16": { - "title": "ProcessAsync, 16 awaited workers, by registration", + "asyncn": { + "title": "ProcessAsync, {workers} awaited workers, by registration", "date": "2026-09-25", "source": "README.md, Async table, 16-worker column; branch async-scratch-cache", "harness": "TestServer_Console --async 3 16", "workload": "the five benchmark requests (yieldsOnce: one request) through the byte-level JsonRpcProcessor.ProcessAsync from Task.Run workers that await each call", "conditions": "idle box; one 3 s run per row", "policy": "one run per row", + "workers": 16, "groups": [ - { "heading": "methods that complete inline, RpcContextFlow.None", "rule": false, "series": ["async16-sync-none", "async16-task-none", "async16-valuetask-none"] }, - { "heading": "methods that complete inline, RpcContextFlow.Flow", "rule": false, "series": ["async16-task-flow", "async16-valuetask-flow"] }, - { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["async16-yield-none", "async16-yield-flow"] } + { "heading": "methods that complete inline, RpcContextFlow.None", "rule": false, "series": ["asyncn-sync-none", "asyncn-task-none", "asyncn-valuetask-none"] }, + { "heading": "methods that complete inline, RpcContextFlow.Flow", "rule": false, "series": ["asyncn-task-flow", "asyncn-valuetask-flow"] }, + { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["asyncn-yield-none", "asyncn-yield-flow"] } ], "series": [ - { "id": "async16-sync-none", "label": "synchronous methods, None", "note": "6 B per request, the StringMe result string", "family": "ours", "low": 24600000, "high": 24600000, "status": "single", "published": "24.6 M" }, - { "id": "async16-task-none", "label": "Task, None", "note": "67 B per request, the service's own Task.FromResult", "family": "ours", "low": 30000000, "high": 30000000, "status": "single", "published": "30.0 M" }, - { "id": "async16-valuetask-none", "label": "ValueTask, None", "note": "6 B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M" }, - { "id": "async16-task-flow", "label": "Task, Flow", "note": "251 B per request", "family": "ours", "low": 22200000, "high": 22200000, "status": "single", "published": "22.2 M" }, - { "id": "async16-valuetask-flow", "label": "ValueTask, Flow", "note": "190 B per request", "family": "ours", "low": 23400000, "high": 23400000, "status": "single", "published": "23.4 M" }, - { "id": "async16-yield-none", "label": "yieldsOnce, None", "note": "559 B per request", "family": "ours", "low": 8960000, "high": 8960000, "status": "single", "published": "8.96 M" }, - { "id": "async16-yield-flow", "label": "yieldsOnce, Flow", "note": "741 B per request", "family": "ours", "low": 6890000, "high": 6890000, "status": "single", "published": "6.89 M" } + { "id": "asyncn-sync-none", "label": "synchronous methods, None", "note": "{bytes} B per request, the StringMe result string", "family": "ours", "low": 24600000, "high": 24600000, "status": "single", "published": "24.6 M", "bytes": 6 }, + { "id": "asyncn-task-flow", "label": "Task, Flow", "note": "{bytes} B per request", "family": "ours", "low": 22200000, "high": 22200000, "status": "single", "published": "22.2 M", "row": "`Task`, Flow", "bytes": 251 }, + { "id": "asyncn-task-none", "label": "Task, None", "note": "{bytes} B per request, the service's own Task.FromResult", "family": "ours", "low": 30000000, "high": 30000000, "status": "single", "published": "30.0 M", "row": "`Task`, None", "bytes": 67 }, + { "id": "asyncn-valuetask-flow", "label": "ValueTask, Flow", "note": "{bytes} B per request", "family": "ours", "low": 23400000, "high": 23400000, "status": "single", "published": "23.4 M", "row": "`ValueTask`, Flow", "bytes": 190 }, + { "id": "asyncn-valuetask-none", "label": "ValueTask, None", "note": "{bytes} B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M", "row": "`ValueTask`, None", "bytes": 6 }, + { "id": "asyncn-yield-flow", "label": "yieldsOnce, Flow", "note": "{bytes} B per request", "family": "ours", "low": 6890000, "high": 6890000, "status": "single", "published": "6.89 M", "row": "`yieldsOnce`, Flow", "bytes": 741 }, + { "id": "asyncn-yield-none", "label": "yieldsOnce, None", "note": "{bytes} B per request", "family": "ours", "low": 8960000, "high": 8960000, "status": "single", "published": "8.96 M", "row": "`yieldsOnce`, None", "bytes": 559 } ] }, "async1": { @@ -75,7 +94,7 @@ "date": "2026-09-25", "source": "README.md, Async table, 1-worker column; branch async-scratch-cache", "harness": "TestServer_Console --async 3 1", - "workload": "as async16, one worker", + "workload": "as asyncn, one worker", "conditions": "idle box; one 3 s run per row", "policy": "one run per row", "groups": [ @@ -84,13 +103,13 @@ { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["async1-yield-none", "async1-yield-flow"] } ], "series": [ - { "id": "async1-sync-none", "label": "synchronous methods, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M" }, - { "id": "async1-task-none", "label": "Task, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M" }, - { "id": "async1-valuetask-none", "label": "ValueTask, None", "family": "ours", "low": 3760000, "high": 3760000, "status": "single", "published": "3.76 M" }, - { "id": "async1-task-flow", "label": "Task, Flow", "family": "ours", "low": 2940000, "high": 2940000, "status": "single", "published": "2.94 M" }, - { "id": "async1-valuetask-flow", "label": "ValueTask, Flow", "family": "ours", "low": 3110000, "high": 3110000, "status": "single", "published": "3.11 M" }, - { "id": "async1-yield-none", "label": "yieldsOnce, None", "family": "ours", "low": 1140000, "high": 1140000, "status": "single", "published": "1.14 M" }, - { "id": "async1-yield-flow", "label": "yieldsOnce, Flow", "family": "ours", "low": 915000, "high": 915000, "status": "single", "published": "915 k" } + { "id": "async1-sync-none", "label": "synchronous methods, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M", "bytes": 6 }, + { "id": "async1-task-flow", "label": "Task, Flow", "family": "ours", "low": 2940000, "high": 2940000, "status": "single", "published": "2.94 M", "row": "`Task`, Flow", "bytes": 251 }, + { "id": "async1-task-none", "label": "Task, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M", "row": "`Task`, None", "bytes": 67 }, + { "id": "async1-valuetask-flow", "label": "ValueTask, Flow", "family": "ours", "low": 3110000, "high": 3110000, "status": "single", "published": "3.11 M", "row": "`ValueTask`, Flow", "bytes": 190 }, + { "id": "async1-valuetask-none", "label": "ValueTask, None", "family": "ours", "low": 3760000, "high": 3760000, "status": "single", "published": "3.76 M", "row": "`ValueTask`, None", "bytes": 6 }, + { "id": "async1-yield-flow", "label": "yieldsOnce, Flow", "family": "ours", "low": 915000, "high": 915000, "status": "single", "published": "915 k", "row": "`yieldsOnce`, Flow", "bytes": 741 }, + { "id": "async1-yield-none", "label": "yieldsOnce, None", "family": "ours", "low": 1140000, "high": 1140000, "status": "single", "published": "1.14 M", "row": "`yieldsOnce`, None", "bytes": 559 } ] }, "legacy": { @@ -123,31 +142,37 @@ "date": "2026-09-25", "source": "README.md, Kestrel table; branch async-scratch-cache", "harness": "TestServer_Console --kestrel 3", - "workload": "AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, 16 clients on the server's 8 cores", + "workload": "AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, {workers} clients on the server's 8 cores", "conditions": "idle box; three 3 s runs of --kestrel 3 and two of --kestrel 3 async, one session", "policy": "low and high over the runs: three for the default rows, two for the EnableAsyncMethods = true rows", + "workers": 16, + "subtitle": ["AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, {workers} clients on the server's 8 cores; three 3 s runs, two for the async rows.", "HTTP clients await one POST at a time; TCP keeps 256 requests in flight per connection."], + "alt": "JSON-RPC.Net by transport: HTTP single, HTTP batch of 100 and TCP pipelined, as low-to-high intervals on a log axis, with the in-process figure for scale", "groups": [ { "heading": "EnableAsyncMethods = false (the default)", "rule": false, "series": ["ours-http-1", "ours-http-100", "ours-tcp"] }, { "heading": "EnableAsyncMethods = true: every document through ProcessAsync", "rule": false, "series": ["ours-tcp-async-inline", "ours-tcp-async-yield"] }, - { "heading": "Reference: no transport", "rule": true, "series": ["ours-inproc-16"] } + { "heading": "Reference: no transport", "rule": true, "series": ["ours-inproc-n"] } ], "series": [ - { "id": "ours-http-1", "label": "HTTP, 1 request per POST", "note": "each client awaits one POST at a time; 88 to 130 µs per round trip", "family": "ours", "low": 123000, "high": 182000, "status": "range", "published": "123 k to 182 k" }, + { "id": "ours-inproc-n", "label": "in-process, {workers} threads", "note": "the same requests through the byte entry point", "family": "scale", "low": 33800000, "high": 37100000, "status": "range", "published": "33.8 M to 37.1 M" }, + { "id": "ours-http-1", "label": "HTTP, 1 request per POST", "note": "each client awaits one POST at a time; {us} µs per round trip", "family": "ours", "low": 123000, "high": 182000, "status": "range", "published": "123 k to 182 k", "latency_us": [88, 130], "row_note": "{us} µs per round trip per client depending on the run; HTTP/1.1 request-response is the cost, not the server" }, { "id": "ours-http-100", "label": "HTTP, batch of 100 per POST", "note": "one POST outstanding per client, 100 RPCs in it", "family": "ours", "low": 12800000, "high": 14500000, "status": "range", "published": "12.8 M to 14.5 M" }, - { "id": "ours-tcp", "label": "TCP, 256 pipelined", "note": "JsonRpcConnectionHandler, 256 requests in flight per connection", "family": "ours", "low": 14300000, "high": 16500000, "status": "range", "published": "14.3 M to 16.5 M" }, - { "id": "ours-tcp-async-inline", "label": "TCP, 256 pipelined, methods that complete inline", "note": "--kestrel 3 async", "family": "ours", "low": 15000000, "high": 15400000, "status": "range", "published": "15.0 M to 15.4 M" }, - { "id": "ours-tcp-async-yield", "label": "TCP, 256 pipelined, methods that suspend once", "note": "five async Task methods awaiting Task.Yield(); documents are sequential per connection", "family": "ours", "low": 1250000, "high": 1290000, "status": "range", "published": "1.25 M to 1.29 M" }, - { "id": "ours-inproc-16", "label": "in-process, 16 threads", "note": "the same requests through the byte entry point", "family": "scale", "low": 33800000, "high": 37100000, "status": "range", "published": "33.8 M to 37.1 M" } + { "id": "ours-tcp", "label": "TCP, 256 pipelined", "note": "JsonRpcConnectionHandler, 256 requests in flight per connection", "family": "ours", "low": 14300000, "high": 16500000, "status": "range", "published": "14.3 M to 16.5 M", "row_note": "ring-buffer clients, one thread each, streaming framer" }, + { "id": "ours-tcp-async-inline", "label": "TCP, 256 pipelined, methods that complete inline", "note": "--kestrel 3 async", "family": "ours", "low": 15000000, "high": 15400000, "status": "range", "published": "15.0 M to 15.4 M", "row": "TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that complete inline", "n": 2, "row_note": "`--kestrel 3 async`, {runs_word} runs; inside the spread of the `false` row" }, + { "id": "ours-tcp-async-yield", "label": "TCP, 256 pipelined, methods that suspend once", "note": "five async Task methods awaiting Task.Yield(); documents are sequential per connection", "family": "ours", "low": 1250000, "high": 1290000, "status": "range", "published": "1.25 M to 1.29 M", "row": "TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that suspend once", "row_note": "five `async Task` methods awaiting `Task.Yield()`" } ] }, "compare": { - "title": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections", + "title": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, {workers} connections", "date": "2026-09-23", "source": "README.md, comparison table; commit 192a97f", "harness": "TestServer_Console --compare 3", - "workload": "the same five calls on the same Kestrel; 16 connections or channels with 256 requests in flight each; every request carries \"jsonrpc\":\"2.0\"", + "workload": "the same five calls on the same Kestrel; {workers} connections or channels with 256 requests in flight each; every request carries \"jsonrpc\":\"2.0\"", "conditions": "WSL VM shut down; two 3 s runs", "policy": "observed low and high over two runs; a single figure is one value that both runs rounded to", + "workers": 16, + "subtitle": ["Same five calls, same Kestrel, {workers} connections or channels with 256 requests in flight each, two 3 s runs.", "Every request carries \"jsonrpc\":\"2.0\" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value."], + "alt": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET at {workers} connections: low-to-high intervals on a log axis, grouped by library, with the in-process paths under a rule", "versions": "StreamJsonRpc 2.25.29, gRPC for .NET 2.84.0", "groups": [ { "heading": "JSON-RPC.Net", "rule": false, "series": ["ours-tcp-jsonrpc"] }, @@ -155,12 +180,12 @@ { "heading": "gRPC for .NET 2.84: its .NET client shares the server's cores", "rule": false, "series": ["grpc-unary", "grpc-stream"] } ], "series": [ - { "id": "ours-tcp-jsonrpc", "label": "over Kestrel TCP", "note": "raw documents", "family": "ours", "low": 13700000, "high": 14600000, "status": "range", "published": "13.7 M to 14.6 M" }, - { "id": "sjr-newline-stj", "label": "over Kestrel TCP", "note": "newline framing, System.Text.Json formatter", "family": "theirs", "low": 1380000, "high": 1440000, "status": "range", "published": "1.38 M to 1.44 M" }, - { "id": "sjr-cl-stj", "label": "over Kestrel TCP", "note": "Content-Length framing, System.Text.Json formatter", "family": "theirs", "low": 1410000, "high": 1450000, "status": "range", "published": "1.41 M to 1.45 M" }, - { "id": "sjr-cl-newtonsoft", "label": "over Kestrel TCP", "note": "Content-Length framing, Json.NET formatter (default)", "family": "theirs", "low": 625000, "high": 625000, "status": "single", "published": "625 k" }, - { "id": "grpc-unary", "label": "unary calls", "note": "HTTP/2 + protobuf, 16 channels × 256 in flight", "family": "grpc", "low": 192000, "high": 198000, "status": "range", "published": "192 k to 198 k" }, - { "id": "grpc-stream", "label": "bidirectional stream", "note": "one stream per channel, 256 in flight, batched writes", "family": "grpc", "low": 200000, "high": 209000, "status": "range", "published": "200 k to 209 k" } + { "id": "ours-tcp-jsonrpc", "label": "over Kestrel TCP", "note": "raw documents", "family": "ours", "low": 13700000, "high": 14600000, "status": "range", "published": "13.7 M to 14.6 M", "row": "JSON-RPC.Net over Kestrel TCP, raw documents" }, + { "id": "sjr-newline-stj", "label": "over Kestrel TCP", "note": "newline framing, System.Text.Json formatter", "family": "theirs", "low": 1380000, "high": 1440000, "status": "range", "published": "1.38 M to 1.44 M", "row": "StreamJsonRpc over Kestrel TCP, newline framing, System.Text.Json formatter" }, + { "id": "sjr-cl-stj", "label": "over Kestrel TCP", "note": "Content-Length framing, System.Text.Json formatter", "family": "theirs", "low": 1410000, "high": 1450000, "status": "range", "published": "1.41 M to 1.45 M", "row": "StreamJsonRpc over Kestrel TCP, `Content-Length` framing, System.Text.Json formatter" }, + { "id": "sjr-cl-newtonsoft", "label": "over Kestrel TCP", "note": "Content-Length framing, Json.NET formatter (default)", "family": "theirs", "low": 625000, "high": 625000, "status": "single", "published": "625 k", "row": "StreamJsonRpc over Kestrel TCP, `Content-Length` framing, Json.NET formatter (its default)" }, + { "id": "grpc-unary", "label": "unary calls", "note": "HTTP/2 + protobuf, {workers} channels × 256 in flight", "family": "grpc", "low": 192000, "high": 198000, "status": "range", "published": "192 k to 198 k", "row": "gRPC for .NET, unary calls over HTTP/2 (Grpc.Net.Client, {workers} channels × 256 in flight)" }, + { "id": "grpc-stream", "label": "bidirectional stream", "note": "one stream per channel, 256 in flight, batched writes", "family": "grpc", "low": 200000, "high": 209000, "status": "range", "published": "200 k to 209 k", "row": "gRPC for .NET, one bidirectional stream per channel, 256 in flight, batched writes" } ] }, "inprocess": { @@ -171,13 +196,15 @@ "workload": "the same five calls with no network: a direct call, a Pipe pair, a typed proxy", "conditions": "WSL VM shut down; two 3 s runs", "policy": "observed low and high over two runs", + "subtitle": ["No network. A direct call, a Pipe pair and a typed proxy cross different boundaries, so these rows are not a", "like-for-like comparison with each other or with the transport rows. Two 3 s runs; intervals span them."], + "alt": "In-process paths: JSON-RPC.Net direct call, StreamJsonRpc Pipe pair and StreamJsonRpc typed proxy, as low-to-high intervals on a log axis", "groups": [ { "heading": "", "rule": false, "series": ["ours-direct-1", "sjr-pipe", "sjr-proxy"] } ], "series": [ - { "id": "ours-direct-1", "label": "JSON-RPC.Net, direct call, 1 thread", "note": "bytes in, bytes out, no transport", "family": "ours", "low": 2600000, "high": 3600000, "status": "range", "published": "2.6 M to 3.6 M" }, - { "id": "sjr-pipe", "label": "StreamJsonRpc, Pipe pair, 1 client", "note": "newline framing, System.Text.Json, 256 pipelined", "family": "theirs", "low": 117000, "high": 142000, "status": "range", "published": "117 k to 142 k" }, - { "id": "sjr-proxy", "label": "StreamJsonRpc, typed proxy", "note": "sequential await per call, 10 µs per round trip", "family": "theirs", "low": 96000, "high": 97000, "status": "range", "published": "96 k to 97 k" } + { "id": "ours-direct-1", "label": "JSON-RPC.Net, direct call, 1 thread", "note": "bytes in, bytes out, no transport", "family": "ours", "low": 2600000, "high": 3600000, "status": "range", "published": "2.6 M to 3.6 M", "row": "JSON-RPC.Net in-process, 1 thread (direct call, bytes in, bytes out)" }, + { "id": "sjr-pipe", "label": "StreamJsonRpc, Pipe pair, 1 client", "note": "newline framing, System.Text.Json, 256 pipelined", "family": "theirs", "low": 117000, "high": 142000, "status": "range", "published": "117 k to 142 k", "row": "StreamJsonRpc in-process, 1 client over a `Pipe` pair, newline framing, System.Text.Json formatter, 256 pipelined" }, + { "id": "sjr-proxy", "label": "StreamJsonRpc, typed proxy", "note": "sequential await per call, {us} µs per round trip", "family": "theirs", "low": 96000, "high": 97000, "status": "range", "published": "96 k to 97 k", "latency_us": [10, 10], "row": "StreamJsonRpc typed proxy, sequential `await` per call, in-process pipes", "rpc_note": "{us} µs per round trip" } ] }, "sweep": { @@ -187,6 +214,8 @@ "workload": "the same five calls on the same Kestrel, clients on the server's 8 cores; each cell warmed for 2 s, then timed", "conditions": "WSL VM running and other sessions active: a separate session from the tables, so its absolute figures are not comparable with them", "policy": "observed low, high and median over the runs; each run contributes one observation per cell", + "subtitle": ["Same five calls, same Kestrel, clients on the server's 8 cores, {date}. TCP and gRPC keep {pipeline} requests in flight", "per connection; HTTP clients await one POST at a time (1 or 100 RPCs). {runs_text}."], + "alt": "Every library and transport by client connections, 1 to {x_max}: three panels on a shared log axis, one per library, with a marker shape and dash per setting and whiskers spanning {runs_word} runs", "x": { "name": "client connections (gRPC: channels)" }, "series": [ { "id": "sweep-ours-tcp", "name": "JSON-RPC.Net, TCP", "label": "TCP", "family": "ours", "style": 0, "concurrency": "256 requests in flight per connection" }, @@ -207,6 +236,7 @@ "workload": "add(1, 2) through each interop path, 20,000 RPCs per row", "conditions": "8-core desktop; interpreter from dotnet run, AOT from dotnet publish -c Release with the wasm-tools workload", "policy": "interpreter: one run; AOT: the better of two runs. The interpreter-to-AOT distance is a deployment difference, not a range", + "subtitle": ["samples/WasmHost in Chrome 152, .NET 10, 20,000 RPCs per row. Interpreter: one run. AOT: the better of two runs.", "Batches are per RPC. The interpreter-to-AOT distance is a deployment difference, not a range."], "modes": [ { "id": "interp", "label": "interpreter", "marker": "circle", "policy": "one run" }, { "id": "aot", "label": "AOT", "marker": "diamond", "policy": "better of two runs" } diff --git a/benchmarks/charts/compare-connections-dark.svg b/benchmarks/charts/compare-connections-dark.svg index 706d09f..8089a79 100644 --- a/benchmarks/charts/compare-connections-dark.svg +++ b/benchmarks/charts/compare-connections-dark.svg @@ -1,5 +1,5 @@ - -Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision eacac12e8d55. + +Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision 2ebd3033c54f. Every library and transport, by client connections Same five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight diff --git a/benchmarks/charts/compare-connections.svg b/benchmarks/charts/compare-connections.svg index 661419a..49020d8 100644 --- a/benchmarks/charts/compare-connections.svg +++ b/benchmarks/charts/compare-connections.svg @@ -1,5 +1,5 @@ - -Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision eacac12e8d55. + +Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision 2ebd3033c54f. Every library and transport, by client connections Same five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight diff --git a/benchmarks/charts/compare-streamjsonrpc-dark.svg b/benchmarks/charts/compare-streamjsonrpc-dark.svg index 1ccb1fd..909790b 100644 --- a/benchmarks/charts/compare-streamjsonrpc-dark.svg +++ b/benchmarks/charts/compare-streamjsonrpc-dark.svg @@ -1,5 +1,5 @@ - -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision eacac12e8d55. + +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision 2ebd3033c54f. JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections Same five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. diff --git a/benchmarks/charts/compare-streamjsonrpc.svg b/benchmarks/charts/compare-streamjsonrpc.svg index c1cc29b..2a7423c 100644 --- a/benchmarks/charts/compare-streamjsonrpc.svg +++ b/benchmarks/charts/compare-streamjsonrpc.svg @@ -1,5 +1,5 @@ - -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision eacac12e8d55. + +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision 2ebd3033c54f. JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections Same five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. diff --git a/benchmarks/charts/explorer.html b/benchmarks/charts/explorer.html index 20af779..0eb7e67 100644 --- a/benchmarks/charts/explorer.html +++ b/benchmarks/charts/explorer.html @@ -5,7 +5,7 @@ JSON-RPC.NET benchmark explorer - +