diff --git a/.github/workflows/build_pull_request.yml b/.github/workflows/build_pull_request.yml index 8b50706..5566eba 100644 --- a/.github/workflows/build_pull_request.yml +++ b/.github/workflows/build_pull_request.yml @@ -46,7 +46,7 @@ jobs: # The measurement: ProcessAsync must scale from 1 to 4 workers. A process-wide serialization point holds the ratio # near 1.3 on any core count. Diagnostic on the shared runner (its core count and isolation are not promised, so - # this job is not required and continues on error); the release gate is `--scale 3 16 4.0` on the reference machine. + # this job is not required and continues on error); the release gate is `--scale 3 32 4.0` in the cpu-performance publish job. # The per-serializer diagnostics after the gate (one run per cell, never part of the exit code) go to their own summary section. scaling: runs-on: ubuntu-latest diff --git a/AustinHarris.JsonRpc.AspNetCore/README.md b/AustinHarris.JsonRpc.AspNetCore/README.md index a542e71..92c86fb 100644 --- a/AustinHarris.JsonRpc.AspNetCore/README.md +++ b/AustinHarris.JsonRpc.AspNetCore/README.md @@ -161,9 +161,9 @@ invoked. Replies already finished are flushed before the connection waits on a slow method. When the connection closes, the handler waits for the running method to finish and discards its response. - **Cost:** every document then goes through `ProcessAsync`. With methods that complete inline, the TCP row measured - 15.0 M to 15.4 M against 14.3 M to 16.5 M for the synchronous mode on the same day, inside its spread. A method that + 17 M to 17.1 M against 18.5 M for the synchronous mode in the same benchmark job, about 8 % lower by the medians. A method that suspends pays for its own async state, the library's completion state and a continuation per request; the main README's - Async table measured 559 B per request for the yielding None row in process at one worker, including the service's own + Async table measured 548 B per request for the yielding None row in process at one worker, including the service's own allocations. The main README's Kestrel table has both rows, measured with `TestServer_Console --kestrel 3 async`. A method receives the token by declaring a `[JsonRpcCancellation] CancellationToken` parameter; see diff --git a/CHANGELOG.md b/CHANGELOG.md index 43c3c48..8183bdf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,7 @@ behaviour: a breaking change to either means a new major version. - Notifications never get a wire response, whatever their outcome. - Dates and non-finite numbers are written the same way by every serializer (fraction only when non-zero, `Z`/offset/nothing by `Kind`; `NaN` and the infinities as quoted strings). - Clarified that the WebAssembly sample is not a Native AOT or full-trimming compatibility guarantee. +- The published benchmarks are measured on a Hugging Face Jobs `cpu-performance` host (AMD EPYC 7R13, 32 cores) and regenerated from the job's data (`benchmarks/hf/`, `ingest.py`, `render.py`); the desktop figures are gone from the README except the browser table. ### Deprecated diff --git a/Json-Rpc/README.md b/Json-Rpc/README.md index 90e0de1..b194642 100644 --- a/Json-Rpc/README.md +++ b/Json-Rpc/README.md @@ -135,22 +135,26 @@ The synchronous entry points do not await those methods. ## Performance -These results compare 1.2.3 and 2.0 with the same five requests on the same machine in one session. -The 2.0 runs used the built-in serializer on an AMD Ryzen 7 7800X3D with .NET 10, Release and Server GC. +These results compare 1.2.3 and 2.0 with the same five requests on the same machine in one benchmark job. +The 2.0 runs used the built-in serializer on a Hugging Face Jobs `cpu-performance` host (AMD EPYC 7R13, 32 cores) with .NET 10, Release and Server GC. +Each range is the low and high over the job's runs; the ratios are of the medians. ![JSON-RPC.Net 1.2.3 and 2.0 throughput through the string and byte entry points](https://raw.githubusercontent.com/Astn/JSON-RPC.NET/master/benchmarks/charts/headline-1x-vs-2.svg) + | Path | RPC/s | Against 1.2.3 | | --- | ---: | ---: | -| 1.2.3, `Task Process(string)`, thread pool, best batch size | 3.08 M | | -| 2.0, the same string API and the same loop | 13.3 M | 4.3× | -| 2.0, `Process(bytes)`, 16 dedicated threads | 31.7 M | 10.3× | -| 2.0, `ProcessAsync(bytes)`, 16 awaited workers | 32.1 M | 10.4× | +| 1.2.3, `Task Process(string)`, thread pool, best batch size | 1.29 M to 2.41 M | | +| 2.0, the same string API and the same loop | 9.83 M to 11.1 M | 6.0× | +| 2.0, `Process(bytes)`, 32 dedicated threads | 44.2 M to 45.7 M | 24.3× | +| 2.0, `ProcessAsync(bytes)`, 32 awaited workers | 36.3 M to 40.5 M | 20.7× | + + These measurements cover the library without a transport. The asynchronous byte row uses methods that complete inline. For Kestrel TCP with 256 requests in flight per connection and `EnableAsyncMethods = false`, -the measured range is 14.3 M to 16.5 M RPC/s on loopback, with clients and server on the same machine. +the measured figure is 18.5 M RPC/s on loopback, with clients and server on the same machine. See the [benchmark tables](https://astn.github.io/JSON-RPC.NET/#benchmarks) for conditions and the [benchmark explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) for the data. diff --git a/README.md b/README.md index 94fe1d6..f9040c3 100644 --- a/README.md +++ b/README.md @@ -4,25 +4,28 @@ JSON-RPC.Net is a [JSON-RPC 2.0](https://www.jsonrpc.org/specification) server for .NET. You give it a request document and it gives you the response document, bytes in and bytes out; the transport is yours. Host it in Kestrel, a console app, sockets, pipes, or a Blazor WebAssembly page. -Version 2.0 rebuilt the pipeline around UTF-8 bytes and made the JSON serializer pluggable. The core depends on no JSON library: Json.NET and System.Text.Json ship as separate packages, and the built-in serializer needs neither. On one core it answers a small request in about 214 ns, with no allocation for numeric parameters. The library alone answers over 30 million requests per second on an 8-core desktop, ten times what 1.2.3 does on the same machine. Over pipelined TCP, a Kestrel host answers 14.3 M to 16.5 M requests per second. [Benchmarks](#benchmarks) gives the method and the full tables. +Version 2.0 rebuilt the pipeline around UTF-8 bytes and made the JSON serializer pluggable. The core depends on no JSON library: Json.NET and System.Text.Json ship as separate packages, and the built-in serializer needs neither. On one core it answers a small request in about 315 ns, with no allocation for numeric parameters. The library alone answers 44.2 M to 45.7 M requests per second on a 32-core cloud host, 24 times what 1.2.3 does on the same machine. Over pipelined TCP, a Kestrel host answers 18.5 M requests per second. [Benchmarks](#benchmarks) gives the method and the full tables. ## Performance -2.0 answers the same five requests 4.3 times faster than 1.2.3 through the 1.x string API and 10.3 times faster through the byte entry points its hosts use. The synchronous byte path handles 31.7 M requests per second on 16 threads of one 8-core desktop, with no allocation for a numeric request. All results come from the same machine, the same requests and one session, one run per row. +2.0 answers the same five requests 6.0 times faster than 1.2.3 through the 1.x string API and 24.3 times faster through the byte entry points its hosts use (ratios of the medians). The synchronous byte path handles 44.2 M to 45.7 M requests per second on 32 threads of a 32-core Hugging Face Jobs `cpu-performance` host, with no allocation for a numeric request. All results come from the same machine, the same requests and one job; each range is the low and high over that job's runs. - JSON-RPC.Net 1.2.3 and 2.0 on one machine, one run per row on 2026-09-25: 3.08 M requests per second through the 1.2.3 string API, 13.3 M through the same API on 2.0, and 31.7 M and 32.1 M through the 2.0 byte entry points + JSON-RPC.Net 1.2.3 and 2.0 on one machine, low to high over the runs of one job on 2026-09-28: 1.29 M to 2.41 M requests per second through the 1.2.3 string API, 9.83 M to 11.1 M through the same API on 2.0, and 44.2 M to 45.7 M and 36.3 M to 40.5 M through the 2.0 byte entry points + | Path | RPC/s | Against 1.2.3 | | --- | ---: | ---: | -| 1.2.3, `Task Process(string)`, thread pool, best batch size | 3.08 M | | -| 2.0, the same string API and the same loop | 13.3 M | 4.3× | -| 2.0, `Process(bytes)`, 16 dedicated threads | 31.7 M | 10.3× | -| 2.0, `ProcessAsync(bytes)`, 16 awaited workers | 32.1 M | 10.4× | +| 1.2.3, `Task Process(string)`, thread pool, best batch size | 1.29 M to 2.41 M | | +| 2.0, the same string API and the same loop | 9.83 M to 11.1 M | 6.0× | +| 2.0, `Process(bytes)`, 32 dedicated threads | 44.2 M to 45.7 M | 24.3× | +| 2.0, `ProcessAsync(bytes)`, 32 awaited workers | 36.3 M to 40.5 M | 20.7× | -The request path uses a span tokenizer over the UTF-8 bytes and invokers compiled against the concrete reader and writer. It writes the response straight into a pooled buffer, with no `Task`, result string or continuation per request. A numeric request never touches the GC. Over Kestrel TCP, the AspNetCore package holds 14.3 M to 16.5 M with 256 requests in flight per connection. [Benchmarks](#benchmarks) has the full tables, conditions and the ranges over each day's runs. The [explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) has the exact values. `benchmarks/Baseline` re-runs the 1.2.3 row from NuGet with the same loop as the 2.0 harness. + + +The request path uses a span tokenizer over the UTF-8 bytes and invokers compiled against the concrete reader and writer. It writes the response straight into a pooled buffer, with no `Task`, result string or continuation per request. A numeric request never touches the GC. Over Kestrel TCP, the AspNetCore package holds 18.5 M with 256 requests in flight per connection. [Benchmarks](#benchmarks) has the full tables, the conditions and the ranges over the job's runs. The [explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) has the exact values. `benchmarks/Baseline` re-runs the 1.2.3 row from NuGet with the same loop as the 2.0 harness. It is a server only. There are no client proxies and no server-to-client calls. If you need a bidirectional RPC framework, look at [StreamJsonRpc](https://www.nuget.org/packages/StreamJsonRpc); the benchmarks compare the two. @@ -377,7 +380,7 @@ In the second `-32602` row, "could not convert" means the serializer refused the A method may return `Task`, `Task`, `ValueTask` or `ValueTask`. Call it through `JsonRpcProcessor.ProcessAsync`. The processor awaits the method and writes its result; `Task` and `ValueTask` answer `null`. A method that completes synchronously runs inline and the byte overloads then return `Task.CompletedTask`. -**Cost.** With `RpcContextFlow.None`, the built-in numeric fast path adds no dispatcher allocation when the invocation completes inline. The service's own allocations (a `Task.FromResult`, a result string) are separate and included in the harness figures. Flow allocates an `InvocationState` even when the call completes inline. A method that suspends may allocate its own async state plus completion state in the result writer, the request handler and the document processor. The yielding benchmark measured 559 B per request under None in a single run at one worker on 2026-09-25, including the service's allocations. A suspension also costs a continuation per request. The figures are in [Async](#async-processasync-awaited-workers). +**Cost.** With `RpcContextFlow.None`, the built-in numeric fast path adds no dispatcher allocation when the invocation completes inline. The service's own allocations (a `Task.FromResult`, a result string) are separate and included in the harness figures. Flow allocates an `InvocationState` even when the call completes inline. A method that suspends may allocate its own async state plus completion state in the result writer, the request handler and the document processor. The yielding benchmark measured 548 B per request under None at one worker (the median of two runs on 2026-09-28), including the service's allocations. A suspension also costs a continuation per request. The figures are in [Async](#async-processasync-awaited-workers). ```csharp [JsonRpcMethod("lookup")] @@ -600,29 +603,32 @@ The `jsonrpc` member policy (`Lenient` by default) is a compatibility setting, n ## Benchmarks + | What | API and mode | RPC/s | Details | | --- | --- | ---: | --- | -| Library alone, 16 threads | `Process(bytes)`, dedicated threads | 31.7 M to 36.4 M | [Sync](#sync-the-library-alone) | -| Library alone, 16 workers | `ProcessAsync(bytes)`, awaited workers, methods that complete inline | 22.2 M to 32.1 M | [Async](#async-processasync-awaited-workers); the spread is across registrations, not runs | -| Library alone, 16 workers, one real suspension per request | `ProcessAsync(bytes)`, `yieldsOnce` | 8.96 M | | -| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = false` | 14.3 M to 16.5 M | [Kestrel](#kestrel-through-the-aspnetcore-package) | -| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = true`, methods that complete inline | 15.0 M to 15.4 M | | -| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = true`, methods that suspend once | 1.25 M to 1.29 M | | -| Kestrel HTTP, batch of 100 per POST | `EnableAsyncMethods = false` | 12.8 M to 14.5 M | | -| Kestrel HTTP, one request per POST | `EnableAsyncMethods = false` | 123 k to 182 k | HTTP/1.1 round trips dominate | -| Legacy string API, thread pool | `Task Process(string)`, batches of 6,000 | 12.4 M to 13.3 M | [Legacy](#legacy-string-api-scheduled-synchronous-execution); the 1.x overloads, not the byte path | - -The Sync, Async, Legacy and Kestrel results were measured on 2026-09-25 on an idle AMD Ryzen 7 7800X3D (8 cores / 16 threads, 4.2 GHz), 64 GB, Windows 11, .NET 10, Release, Server GC, with the built-in serializer. Throughput ranges are the low and high over that day's runs: three runs for Sync, Legacy and default-mode Kestrel, two for the `EnableAsyncMethods = true` rows. Each Async throughput cell is one 3 s run; the summary's inline Async spread is across registrations, not runs. Each Sync ns figure is the reported cost from one of that row's runs. The StreamJsonRpc and gRPC comparison and its in-process rows are from 2026-09-23, with the WSL virtual machine shut down. Their ranges cover two runs; the comparison's single throughput figure is the value both runs rounded to. The WebAssembly results are also from 2026-09-23: one interpreter run and the better of two AOT runs per row. The connection sweep is a separate session with WSL and other work active, so its absolute figures are not comparable with the tables. A single benchmark thread on this machine varies with whatever else lands on its core's SMT sibling, so the 1-thread rows are from runs on an idle core. +| Library alone, 32 threads | `Process(bytes)`, dedicated threads | 44.2 M to 45.7 M | [Sync](#sync-the-library-alone) | +| Library alone, 32 workers | `ProcessAsync(bytes)`, awaited workers, methods that complete inline | 26.7 M to 40.5 M | [Async](#async-processasync-awaited-workers); the spread is across registrations, not runs | +| Library alone, 32 workers, one real suspension per request | `ProcessAsync(bytes)`, `yieldsOnce` | 2.55 M | | +| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = false` | 18.5 M | [Kestrel](#kestrel-through-the-aspnetcore-package) | +| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = true`, methods that complete inline | 17 M to 17.1 M | | +| Kestrel TCP, 256 pipelined | `EnableAsyncMethods = true`, methods that suspend once | 2.86 M to 2.87 M | | +| Kestrel HTTP, batch of 100 per POST | `EnableAsyncMethods = false` | 14.4 M to 14.8 M | | +| Kestrel HTTP, one request per POST | `EnableAsyncMethods = false` | 156 k to 162 k | HTTP/1.1 round trips dominate | +| Legacy string API, thread pool | `Task Process(string)`, batches of 6,000 | 9.83 M to 11.1 M | [Legacy](#legacy-string-api-scheduled-synchronous-execution); the 1.x overloads, not the byte path | + + + +Every table, chart and figure in this section except the WebAssembly one comes from one Hugging Face Jobs run on a `cpu-performance` host: AMD EPYC 7R13, 32 cores (a cgroup quota of all 32 of the host's CPUs), Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC, with the built-in serializer. The job ran `benchmarks/hf/run.sh publish` on 2026-09-28 (job 6ab9b3936b030d633f69b0fb, commit 6c3c272): the scaling gate once, three runs of the `t` entry, two of every other mode and of `benchmarks/Baseline`, and five of `--sweep`, one after another. Throughput ranges are the low and high over those runs, and a single figure is one that every run rounded to; the [explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) also has the median and every run. The summary's inline Async range spans the registrations as well as the runs. Each Sync ns figure is the median of the runs' reported costs. The WebAssembly results are a browser measurement on a desktop from 2026-09-23; see the sample's README. `TestServer_Console` is the benchmark harness. It binds one service with five small methods (`add`, `addInt`, `NullableFloatToNullableFloat`, `Test2`, `StringMe`), drives the same five requests through the server, checks every response is a `result` rather than an error, and ends each mode with a bar chart of RPC/s. For one-request timings with an allocation column, the numbers to check before merging a change to the dispatch path, see [benchmarks/Micro](benchmarks/Micro/README.md). ``` dotnet run -c Release --project TestServer_Console -- --sync 3 # library only, 1..N threads (add a thread count, e.g. --sync 3 1, for one row) -dotnet run -c Release --project TestServer_Console -- --async 3 16 # ProcessAsync from 16 awaited workers, one row per registration (--async 3 1 for the 1-worker column) -dotnet run -c Release --project TestServer_Console -- --scale 3 16 4.0 # release gate: ProcessAsync must scale at least 4x from 1 to 16 workers, then per-serializer diagnostics (--no-diagnostics skips them) +dotnet run -c Release --project TestServer_Console -- --async 3 32 # ProcessAsync from 32 awaited workers (the job host's core count), one row per registration (--async 3 1 for the 1-worker column) +dotnet run -c Release --project TestServer_Console -- --scale 3 32 4.0 # release gate: ProcessAsync must scale at least 4x from 1 to 32 workers (default: the core count), then per-serializer diagnostics (--no-diagnostics skips them) dotnet run -c Release --project TestServer_Console -- --kestrel 3 # through the AspNetCore package, HTTP and TCP (add `async` for EnableAsyncMethods = true) dotnet run -c Release --project TestServer_Console -- --compare 3 # the same calls through StreamJsonRpc and gRPC for .NET, side by side -dotnet run -c Release --project TestServer_Console -- --sweep 2 benchmarks/charts/sweep.json # every library and transport at 1, 2, 4, 8, 16 connections; one file per run +dotnet run -c Release --project TestServer_Console -- --sweep 2 benchmarks/charts/sweep.json # every library and transport at 1, 2, 4, ... connections up to the core count; one file per run dotnet run -c Release --project TestServer_Console # menu: Enter = Process(bytes), a = ProcessAsync(bytes), t = legacy string API, k = Kestrel, x = compare, q = quit dotnet run -c Release --project benchmarks/Baseline # the last 1.x release (1.2.3 from NuGet) through the same loop as the t entry dotnet run --project samples/WasmHost # browser: "Run benchmark" on the page @@ -630,7 +636,7 @@ dotnet run --project samples/WasmHost # browser: " The three modes measure different things and are named for the entry point they call. `Process(bytes), dedicated threads` measures the library alone. `ProcessAsync(bytes), awaited workers` measures the entry point an asynchronous host calls. `Legacy Process(string), scheduled synchronous work` measures the 1.x string overloads through the thread pool. The Kestrel rows say whether `EnableAsyncMethods` was on. -On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs]` (`` is a branch, tag or commit) and retrieve JSON with `benchmarks/hf/fetch.py `; `cpu-upgrade` is 8 vCPU ($0.03/h), `cpu-xl` is 16 vCPU ($1/h), and `cpu-performance` is 32 vCPU ($1.90/h), while `cpu-basic` is 2 vCPU for smoke runs only. These are shared vCPUs, so each job's figures belong to its own machine row and are never merged into the reference-box tables. +On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs|publish]` (`` is a branch, tag or commit) and retrieve JSON with `benchmarks/hf/fetch.py `; `cpu-upgrade` is 8 vCPU ($0.03/h), `cpu-xl` is 16 vCPU ($1/h), and `cpu-performance` is 32 vCPU ($1.90/h), while `cpu-basic` is 2 vCPU for smoke runs only. The published tables come from the `publish` profile on `cpu-performance` ([benchmarks/hf](benchmarks/hf/README.md) describes it); figures from any other flavor belong to their own machine and are not merged into them. ### Sync: the library alone @@ -641,31 +647,38 @@ On Hugging Face Jobs, run `benchmarks/hf/launch.sh [runs]` (` + | Threads | RPC/s | ns per request per thread | Allocations per request | | ---: | ---: | ---: | --- | -| 1 | 4.6 M to 5.0 M | 214 | 0 bytes for numeric shapes, one string for `StringMe` | -| 2 | 8.5 M to 9.1 M | 235 | | -| 4 | 16.6 M to 17.1 M | 234 | | -| 8 | 22.8 M to 27.7 M | 292 | | -| 16 | 31.7 M to 36.4 M | 439 | | +| 1 | 3.17 M to 3.18 M | 315 | 0 bytes for numeric shapes, one string for `StringMe` | +| 2 | 4.31 M to 4.66 M | 446 | | +| 4 | 10.7 M to 12 M | 355 | | +| 8 | 18.4 M to 19.8 M | 419 | | +| 16 | 34.5 M to 35.8 M | 456 | | +| 32 | 44.2 M to 45.7 M | 712 | | -Per-thread cost rises with thread count because the 16 threads share 8 physical cores. + + +Per-thread cost rises with the thread count, from 315 ns at one thread to 712 ns at 32, where the loop occupies every core the job has. ### Async: ProcessAsync, awaited workers `--async [seconds] [workers]` calls the byte-level `JsonRpcProcessor.ProcessAsync` from `Task.Run` workers that await each call. It uses the same five requests and loop shape as `--sync`. Workers start behind a barrier, stop on a shared flag and walk the inputs with an index. Worker startup is reported separately. Each row registers the five methods with one return shape and one `RpcContextFlow`. The `yieldsOnce` rows use one method that awaits `Task.Yield()`, giving a real suspension per request. Responses are checked before timing. The bytes column reports allocations per request at one worker, including the service method's own allocations. The inline rows use exact per-thread counts, and the `yieldsOnce` rows use the process-wide counter. -| Registration | 1 worker | 16 workers | B per request | + +| Registration | 1 worker | 32 workers | B per request | | --- | ---: | ---: | ---: | -| synchronous methods, None | 3.47 M | 24.6 M | 6 | -| `Task`, Flow | 2.94 M | 22.2 M | 251 | -| `Task`, None | 3.47 M | 30.0 M | 67 | -| `ValueTask`, Flow | 3.11 M | 23.4 M | 190 | -| `ValueTask`, None | 3.76 M | 32.1 M | 6 | -| `yieldsOnce`, Flow | 915 k | 6.89 M | 741 | -| `yieldsOnce`, None | 1.14 M | 8.96 M | 559 | +| synchronous methods, None | 2.51 M to 2.52 M | 26.7 M to 28.7 M | 6 | +| `Task`, Flow | 1.98 M | 27 M to 27.6 M | 251 | +| `Task`, None | 2.63 M to 2.66 M | 34.4 M to 35.5 M | 67 | +| `ValueTask`, Flow | 2.05 M to 2.06 M | 28.6 M to 31.4 M | 190 | +| `ValueTask`, None | 2.77 M to 2.82 M | 36.3 M to 40.5 M | 6 | +| `yieldsOnce`, Flow | 563 k to 589 k | 2.74 M to 2.78 M | 740 | +| `yieldsOnce`, None | 544 k to 565 k | 2.55 M | 548 | + + -The 16-worker rows are an equal-weight mix of the five requests, except `yieldsOnce`, which is one request. The table below gives bytes per request for each request shape at one worker, including the method's own allocations. The harness prints these lines before each row. +The 32-worker rows are an equal-weight mix of the five requests, except `yieldsOnce`, which is one request. The table below gives bytes per request for each request shape at one worker, including the method's own allocations. The harness prints these lines before each row. | Registration | `add` | `addInt` | nullable float | decimal | `StringMe` | | --- | ---: | ---: | ---: | ---: | ---: | @@ -675,66 +688,77 @@ The 16-worker rows are an equal-weight mix of the five requests, except `yieldsO | `ValueTask`, Flow | 184 | 184 | 184 | 184 | 216 | | `ValueTask`, None | 0 | 0 | 0 | 0 | 32 | -With `RpcContextFlow.None`, the dispatcher adds no allocation to a method that completes inline. Allocations in the `Task` None row come from the service's own `Task.FromResult` (`Task` for 8 comes from the runtime's cache). The 32 bytes of `StringMe` are its result string. Flow allocates the `InvocationState` and the execution-context bridge on every call, inline or not. A real suspension allocates the method's own async state plus completion state in the result writer, the request handler and the document processor. The `yieldsOnce` None row measured 559 B per request at one worker, including the method's own allocations. The 7 to 10 M target for a hosted server applies to methods that complete inline. A method that suspends also costs a continuation per request. +With `RpcContextFlow.None`, the dispatcher adds no allocation to a method that completes inline. Allocations in the `Task` None row come from the service's own `Task.FromResult` (`Task` for 8 comes from the runtime's cache). The 32 bytes of `StringMe` are its result string. Flow allocates the `InvocationState` and the execution-context bridge on every call, inline or not. A real suspension allocates the method's own async state plus completion state in the result writer, the request handler and the document processor. The `yieldsOnce` None row measured 548 B per request at one worker, including the method's own allocations. The 7 to 10 M target for a hosted server applies to methods that complete inline. A method that suspends also costs a continuation per request. -Before 2.0.0, the `ProcessAsync` path was capped near 4 M RPC/s at every worker count because each document took one lock on the shared scratch pool. No single-threaded benchmark could see this limit. Each thread now caches one scratch in front of that pool. `--scale [seconds] [workers] [threshold]` is the gate that catches the next such limit. It measures the inline None rows at 1, 2 and 16 workers in three paired runs and takes the medians. It exits non-zero when any 16/1 ratio is below 4.0. After the gate it prints a diagnostics table that is never gated: the three inline None rows and the `yieldsOnce` None row at 1 and 16 workers under each serializer (`jsmn`, `stj`, `newtonsoft`), one run per cell, with the 16/1 ratio and the bytes per request at one worker, so that a regression in one serializer's path or in the suspending path shows up on its own. A final `--no-diagnostics` argument skips the table. The lock gave 1.3; the cache gives 7.1 to 7.3 on the idle reference machine (the gate's medians, 2026-09-25). Run it on the reference machine before a release and paste its gate table into the release notes. The same release run measures the `--kestrel 3 async` TCP row with methods that suspend once and compares it with the previous release's published figure: a drop larger than the paired-run spread is a release blocker, and the row has no absolute floor (see [Kestrel](#kestrel-through-the-aspnetcore-package)). The pull-request build runs a diagnostic `--scale 3 4 2.0` on the shared runner. It also checks that every `lock`, `Interlocked`, `Volatile.Write`, thread-static and writable static field on the request-path files of the core and both companion serializers is listed in `.github/request-path-sync.allowlist` with a reason (per-thread, miss-path, registration-only, read-only-after-init). +Before 2.0.0, the `ProcessAsync` path was capped near 4 M RPC/s at every worker count because each document took one lock on the shared scratch pool. No single-threaded benchmark could see this limit. Each thread now caches one scratch in front of that pool. `--scale [seconds] [workers] [threshold]` is the gate that catches the next such limit. It measures the inline None rows at 1, 2 and N workers (N defaults to the core count) in three paired runs and takes the medians. It exits non-zero when any N/1 ratio is below the threshold (4.0 in the release gate). After the gate it prints a diagnostics table that is never gated: the three inline None rows and the `yieldsOnce` None row at 1 and N workers under each serializer (`jsmn`, `stj`, `newtonsoft`), one run per cell, with the N/1 ratio and the bytes per request at one worker, so that a regression in one serializer's path or in the suspending path shows up on its own. A final `--no-diagnostics` argument skips the table. The lock gave 1.3; the cache gives 12.1 to 13.1 at 32 workers on the job host (the gate's medians, 2026-09-28). Before a release, run `benchmarks/hf/launch.sh cpu-performance publish`, which runs the gate first, and paste its gate table into the release notes. The same release job measures the `--kestrel 3 async` TCP row with methods that suspend once and compares it with the previous release's figure from the same flavor: a drop larger than the spread of its runs blocks the release, and the row has no absolute floor (see [Kestrel](#kestrel-through-the-aspnetcore-package)). The pull-request build runs a diagnostic `--scale 3 4 2.0` on the shared runner. It also checks that every `lock`, `Interlocked`, `Volatile.Write`, thread-static and writable static field on the request-path files of the core and both companion serializers is listed in `.github/request-path-sync.allowlist` with a reason (per-thread, miss-path, registration-only, read-only-after-init). ### Legacy string API: scheduled synchronous execution The `t` menu entry submits batches through the 1.x `Task Process(string)` overload from every core at once. This compatibility string API pays for transcoding, a thread-pool hop, a `Task`, a result string and a continuation per request. An asynchronous host uses the byte entry points measured in the Sync and Async tables above. Each batch size is repeated for at least half a second after a one-second warm-up. Throughput peaks once a batch is large enough to keep every core busy and falls off again when hundreds of thousands of requests are queued at once: + | Batch size | RPC/s | | ---: | ---: | -| 50 | 1.6 M to 2.0 M | -| 300 | 8.0 M to 9.2 M | -| 6,000 | 12.4 M to 13.3 M | -| 36,000 | 11.6 M to 13.3 M | -| 252,000 | 7.5 M to 7.8 M | -| 2,016,000 | 7.5 M to 8.0 M | +| 50 | 1.08 M to 1.44 M | +| 100 | 1.43 M to 1.51 M | +| 300 | 3.72 M to 4.1 M | +| 1,200 | 7.55 M to 7.8 M | +| 6,000 | 9.83 M to 11.1 M | +| 36,000 | 10.3 M to 10.8 M | +| 252,000 | 5.96 M to 6.78 M | +| 2,016,000 | 3.4 M to 6.15 M | + + This mode is slower than the byte modes because it measures the cost of the .NET thread pool and a `Task`, a string and a continuation per request as much as the library itself. The AspNetCore host does not use that string path or allocate a result string. It awaits transport reads and flushes without a thread per connection, as the next table shows. ### Kestrel: through the AspNetCore package -`--kestrel [seconds]` starts a real Kestrel on loopback with `MapJsonRpc` and `JsonRpcConnectionHandler`, then drives it with 16 clients on the same machine. Every figure is therefore an upper bound for one box talking to itself. The in-process rows show the same requests through the byte entry point with no transport, to give a sense of scale. The `EnableAsyncMethods = false` rows are the default, where the host calls `Process`. `--kestrel 3 async` runs the same host with `EnableAsyncMethods = true` and sends every document through `ProcessAsync`. It also adds a TCP row whose five methods await `Task.Yield()` before answering. The trailing text on each `--kestrel` row reports system, harness-process and, for TCP, dedicated-client CPU during that row's timed window as a share of the available cores. +`--kestrel [seconds]` starts a real Kestrel on loopback with `MapJsonRpc` and `JsonRpcConnectionHandler`, then drives it with one client per core (32 on the job host) on the same machine. Every figure is therefore an upper bound for one box talking to itself. The in-process rows show the same requests through the byte entry point with no transport, to give a sense of scale. The `EnableAsyncMethods = false` rows are the default, where the host calls `Process`. `--kestrel 3 async` runs the same host with `EnableAsyncMethods = true` and sends every document through `ProcessAsync`. It also adds a TCP row whose five methods await `Task.Yield()` before answering. The trailing text on each `--kestrel` row reports system, harness-process and, for TCP, dedicated-client CPU during that row's timed window as a share of the available cores. JSON-RPC.Net by transport: HTTP single, HTTP batch of 100 and TCP pipelined, as low-to-high intervals on a log axis, with the in-process figure for scale + | Transport | RPC/s | Note | | --- | ---: | --- | -| in-process, 16 threads | 33.8 M to 37.1 M | | -| HTTP, 1 request per POST | 123 k to 182 k | 88 to 130 µs per round trip per client depending on the run; HTTP/1.1 request-response is the cost, not the server | -| HTTP, batch of 100 per POST | 12.8 M to 14.5 M | | -| TCP, 256 pipelined | 14.3 M to 16.5 M | ring-buffer clients, one thread each, streaming framer | -| TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that complete inline | 15.0 M to 15.4 M | `--kestrel 3 async`, two runs; inside the spread of the `false` row | -| TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that suspend once | 1.25 M to 1.29 M | five `async Task` methods awaiting `Task.Yield()` | +| in-process, 32 threads | 47.3 M to 49.6 M | | +| HTTP, 1 request per POST | 156 k to 162 k | 197 to 205 µs per round trip per client depending on the run; HTTP/1.1 request-response is the cost, not the server | +| HTTP, batch of 100 per POST | 14.4 M to 14.8 M | | +| TCP, 256 pipelined | 18.5 M | ring-buffer clients, one thread each, streaming framer | +| TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that complete inline | 17 M to 17.1 M | `--kestrel 3 async`, two runs | +| TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that suspend once | 2.86 M to 2.87 M | five `async Task` methods awaiting `Task.Yield()` | + + -The TCP client keeps 256 requests in flight per connection and refills from a precomputed ring of request bytes with one `Send` per refill. On the server, `JsonFramer` feeds the same `Process` call the HTTP endpoint makes. With `EnableAsyncMethods = true`, the connection handler processes each connection's documents one at a time, in order. So 256 pipelined requests are 256 sequential invocations, and a method that suspends pays that cost per request. Concurrency comes from the 16 connections. The last row is a release-required regression row: it is measured with `--kestrel 3 async` on the reference machine before every release and compared with the previous release's published figure, a drop larger than the paired-run spread blocks the release, and it has no absolute floor. +The TCP client keeps 256 requests in flight per connection and refills from a precomputed ring of request bytes with one `Send` per refill. On the server, `JsonFramer` feeds the same `Process` call the HTTP endpoint makes. With `EnableAsyncMethods = true`, the connection handler processes each connection's documents one at a time, in order. So 256 pipelined requests are 256 sequential invocations, and a method that suspends pays that cost per request. Concurrency comes from the connections, one per core. The last row is a release-required regression row: it is measured by the publish job on `cpu-performance` before every release and compared with the previous release's figure from the same flavor; a drop larger than the spread of its runs blocks the release, and it has no absolute floor. ### Versus StreamJsonRpc and gRPC -[StreamJsonRpc](https://www.nuget.org/packages/StreamJsonRpc) is Microsoft's JSON-RPC library, the one behind Visual Studio and the language-server stack. `--compare` hosts both libraries on the same Kestrel TCP listener and drives them with the same pipelining client (16 connections, 256 requests in flight each), so the only variable is the library answering. StreamJsonRpc requires the `jsonrpc` member, so every request in this mode carries `"jsonrpc":"2.0"`, which is why the JSON-RPC.Net rows are a little below the other tables. The same mode also hosts [gRPC for .NET](https://learn.microsoft.com/aspnet/core/grpc/) (HTTP/2, protobuf) on the same Kestrel, answering the same five calls from [calculator.proto](TestServer_Console/Protos/calculator.proto), driven by its own client with the same shape: 16 channels, 256 calls in flight each. Each row was run twice for 3 s; both results are shown. The trailing text on each `--compare` row reports system, harness-process and, for TCP, dedicated-client CPU during that row's timed window as a share of the available cores. +[StreamJsonRpc](https://www.nuget.org/packages/StreamJsonRpc) is Microsoft's JSON-RPC library, the one behind Visual Studio and the language-server stack. `--compare` hosts both libraries on the same Kestrel TCP listener and drives them with the same pipelining client (one connection per core, 32 on the job host, 256 requests in flight each), so the only variable is the library answering. StreamJsonRpc requires the `jsonrpc` member, so every request in this mode carries `"jsonrpc":"2.0"`, which is why the JSON-RPC.Net rows are a little below the other tables. The same mode also hosts [gRPC for .NET](https://learn.microsoft.com/aspnet/core/grpc/) (HTTP/2, protobuf) on the same Kestrel, answering the same five calls from [calculator.proto](TestServer_Console/Protos/calculator.proto), driven by its own client with the same shape: one channel per core, 256 calls in flight each. Each row was run twice for 3 s; both results are shown. The trailing text on each `--compare` row reports system, harness-process and, for TCP, dedicated-client CPU during that row's timed window as a share of the available cores. - JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET at 16 connections: low-to-high intervals on a log axis, grouped by library, with the in-process paths under a rule + JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET at 32 connections: low-to-high intervals on a log axis, grouped by library, with the in-process paths under a rule + | Library and path | RPC/s | | --- | ---: | -| JSON-RPC.Net over Kestrel TCP, raw documents | 13.7 M to 14.6 M | -| StreamJsonRpc over Kestrel TCP, newline framing, System.Text.Json formatter | 1.38 M to 1.44 M | -| StreamJsonRpc over Kestrel TCP, `Content-Length` framing, System.Text.Json formatter | 1.41 M to 1.45 M | -| StreamJsonRpc over Kestrel TCP, `Content-Length` framing, Json.NET formatter (its default) | 625 k | -| gRPC for .NET, unary calls over HTTP/2 (Grpc.Net.Client, 16 channels × 256 in flight) | 192 k to 198 k | -| gRPC for .NET, one bidirectional stream per channel, 256 in flight, batched writes | 200 k to 209 k | +| JSON-RPC.Net over Kestrel TCP, raw documents | 16.4 M to 16.8 M | +| StreamJsonRpc over Kestrel TCP, newline framing, System.Text.Json formatter | 847 k to 879 k | +| StreamJsonRpc over Kestrel TCP, `Content-Length` framing, System.Text.Json formatter | 1.06 M to 1.15 M | +| StreamJsonRpc over Kestrel TCP, `Content-Length` framing, Json.NET formatter (its default) | 560 k to 583 k | +| gRPC for .NET, unary calls over HTTP/2 (Grpc.Net.Client, 32 channels × 256 in flight) | 158 k to 166 k | +| gRPC for .NET, one bidirectional stream per channel, 256 in flight, batched writes | 703 k to 733 k | -StreamJsonRpc 2.25.29, defaults apart from the formatter and framing named in each row. It is a full bidirectional RPC framework (client proxies, cancellation, progress, marshaled objects, events). The comparison is of the server side answering the same five requests; on that measure JSON-RPC.Net is about 10× faster on the same connections. JSON-RPC.Net ran its built-in serializer, and the fastest StreamJsonRpc rows use System.Text.Json, so this is a comparison of whole server paths, not of one JSON library against itself; running JSON-RPC.Net with the System.Text.Json serializer is a separate measurement and is not in this table. + -gRPC for .NET 2.84.0 with default settings apart from Kestrel's `MaxStreamsPerConnection` (raised to 256 so the pipeline depth is not capped at 100). protobuf has no `decimal`, so `Test2` carries the units/nanos `DecimalValue` message the gRPC docs recommend; nullable values use proto3 `optional`. The gRPC rows are a different kind of measurement from the rows above them: there is no cheap raw client for HTTP/2 + protobuf, so the client is Grpc.Net.Client on the same 8 cores as the server, and the figure is what a .NET caller and a .NET service get end to end. One channel alone reaches about 130 k unary calls per second; sixteen channels do not scale much further because client and server compete for the same cores. The streaming row batches its writes the way the TCP client does (BufferHint on every message but the last of a refill), and the server flushes only when its input runs dry, the same once-per-read-group flush `JsonRpcConnectionHandler` does. +StreamJsonRpc 2.25.29, defaults apart from the formatter and framing named in each row. It is a full bidirectional RPC framework (client proxies, cancellation, progress, marshaled objects, events). The comparison is of the server side answering the same five requests; on that measure JSON-RPC.Net is about 15× faster than StreamJsonRpc's fastest row on the same connections. JSON-RPC.Net ran its built-in serializer, and the fastest StreamJsonRpc rows use System.Text.Json, so this is a comparison of whole server paths, not of one JSON library against itself; running JSON-RPC.Net with the System.Text.Json serializer is a separate measurement and is not in this table. + +gRPC for .NET 2.84.0 with default settings apart from Kestrel's `MaxStreamsPerConnection` (raised to 256 so the pipeline depth is not capped at 100). protobuf has no `decimal`, so `Test2` carries the units/nanos `DecimalValue` message the gRPC docs recommend; nullable values use proto3 `optional`. The gRPC rows are a different kind of measurement from the rows above them: there is no cheap raw client for HTTP/2 + protobuf, so the client is Grpc.Net.Client on the same cores as the server, and the figure is what a .NET caller and a .NET service get end to end. In the sweep below, one channel reaches 136 k to 143 k unary calls per second and eight reach 338 k to 357 k; more channels lose ground because client and server compete for the same cores. The streaming row batches its writes the way the TCP client does (BufferHint on every message but the last of a refill), and the server flushes only when its input runs dry, the same once-per-read-group flush `JsonRpcConnectionHandler` does.
In-process paths: a direct call, a Pipe pair and a typed proxy (different boundaries, not comparable with the rows above) @@ -744,22 +768,25 @@ gRPC for .NET 2.84.0 with default settings apart from Kestrel's `MaxStreamsPerCo In-process paths: JSON-RPC.Net direct call, StreamJsonRpc Pipe pair and StreamJsonRpc typed proxy, as low-to-high intervals on a log axis + | Path | RPC/s | | --- | ---: | -| JSON-RPC.Net in-process, 1 thread (direct call, bytes in, bytes out) | 2.6 M to 3.6 M | -| StreamJsonRpc in-process, 1 client over a `Pipe` pair, newline framing, System.Text.Json formatter, 256 pipelined | 117 k to 142 k | -| StreamJsonRpc typed proxy, sequential `await` per call, in-process pipes | 96 k to 97 k (10 µs per round trip) | +| JSON-RPC.Net in-process, 1 thread (direct call, bytes in, bytes out) | 2.97 M to 3.01 M | +| StreamJsonRpc in-process, 1 client over a `Pipe` pair, newline framing, System.Text.Json formatter, 256 pipelined | 133 k to 154 k | +| StreamJsonRpc typed proxy, sequential `await` per call, in-process pipes | 53.7 k to 56.2 k (18 to 19 µs per round trip) | + + -StreamJsonRpc's server side has no "document in, document out" call, so its in-process row is a pair of `System.IO.Pipelines` pipes, the closest it has to a direct call; the proxy row is one call at a time, so it measures a round trip, not throughput. The direct-call row is from the comparison session and sits below the sync table's newer 1-thread figure. +StreamJsonRpc's server side has no "document in, document out" call, so its in-process row is a pair of `System.IO.Pipelines` pipes, the closest it has to a direct call; the proxy row is one call at a time, so it measures a round trip, not throughput. The direct-call row comes from `--compare` and sits a little below the Sync table's 1-thread figure.
- Every library and transport by client connections, 1 to 16: three panels on a shared log axis, one per library, with a marker shape and dash per setting and whiskers spanning five runs + Every library and transport by client connections, 1 to 32: three panels on a shared log axis, one per library, with a marker shape and dash per setting and whiskers spanning five runs -`--sweep` runs every one of those paths at 1, 2, 4, 8 and 16 client connections (gRPC: channels) and writes one JSON file per run; the chart above is five 2 s runs per point, the marker at the median and the whisker from the lowest to the highest run. New sweep files also record the available core count and per-cell system, process and dedicated TCP-client CPU percentages; a high client share can mean the harness is competing with the server for CPU. It is a separate session from the tables: the WSL virtual machine was running and other work was active, so its absolute figures sit below the table rows (JSON-RPC.Net over TCP 10.9 M to 13.0 M at 16 connections against 13.7 M to 14.6 M in the table), and its gRPC unary figure runs higher (360 k to 404 k against 192 k to 198 k; the cause is not pinned down, and the table keeps the `--compare` figure). What the sweep adds is the shape: JSON-RPC.Net over TCP and batched HTTP climb almost linearly with connections, StreamJsonRpc gains 8 to 10× from one connection to sixteen, and gRPC's .NET client is flat from two channels on because it competes with the server for the same eight cores. +`--sweep` runs every one of those paths at each power of two from 1 to the core count of client connections (gRPC: channels) and writes one JSON file per run; the chart above is five 2 s runs per point, up to 32 connections on the job host, the marker at the median and the whisker from the lowest to the highest run. Each sweep file also records the available core count and per-cell system, process and dedicated TCP-client CPU percentages; a high client share can mean the harness is competing with the server for CPU. The sweep ran in the same job as the tables, with a 2 s warm-up and 2 s of timing per cell, so its cells sit near the table rows rather than on them (JSON-RPC.Net over TCP 17.3 M to 17.8 M at 32 connections against 16.4 M to 16.8 M in the comparison table). Its gRPC figures run higher than the `--compare` rows: unary 269 k to 290 k at 32 channels against 158 k to 166 k, and the stream 893 k to 913 k against 703 k to 733 k; the cause is not pinned down, and the table keeps the `--compare` figures. What the sweep adds is the shape: JSON-RPC.Net over TCP and batched HTTP climb with every doubling of connections, from 1.58 M and 744 k at one connection to 17.5 M and 14.6 M at 32 (medians); StreamJsonRpc gains 5 to 6× from one connection to 32, and its two `Content-Length` rows level off after 16; gRPC's unary calls peak at eight channels, while its stream keeps climbing to 902 k at 32. The [benchmark explorer](https://astn.github.io/JSON-RPC.NET/benchmarks/charts/explorer.html) is the same data as an interactive page: toggle series, hover or tab to a point for the exact low, median, high and every run, switch the axis between log and linear, and download the data. It is one self-contained HTML file, [benchmarks/charts/explorer.html](benchmarks/charts/explorer.html), so it also works saved to disk. @@ -773,11 +800,11 @@ simdjson was evaluated as a fourth parser and not adopted: through the only main ### History -The charts, the explorer page and the figures in this file come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json); how they are rendered and checked is under [Building](#charts). +The charts, the explorer page and the figures in this file come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json); how they are rendered and checked is under [Building](#charts). Until 2026-09-28 the published figures were measured on a desktop, an AMD Ryzen 7 7800X3D (8 cores / 16 threads), and the dated figures below come from it. On 2026-09-25 the `ProcessAsync` path was found capped near 4 M RPC/s at every worker count. Every document took one lock on the shared scratch pool, which the single-threaded micro-benchmarks could not detect. A one-slot per-thread cache in front of the pool took the inline rows to 22.2 M to 32.1 M at 16 workers (one run per registration), against 31.7 M for the synchronous entry point in the same session. The `--scale` gate and the request-path allowlist exist to catch the next such limit before a release. -The Performance section at the top reports one session on 2026-09-25. The last 1.x release on NuGet, 1.2.3, was driven by the same loop as the `t` entry ([benchmarks/Baseline](benchmarks/Baseline/Program.cs)). It reached 3.08 M at its best batch size of 1,200 and fell to 1.5 M at two million. In the same session, 2.0 peaked at 13.3 M through the same string API, and the byte entry points ran at 31.7 M and 32.1 M. +The Performance section reported one desktop session on 2026-09-25. The last 1.x release on NuGet, 1.2.3, was driven by the same loop as the `t` entry ([benchmarks/Baseline](benchmarks/Baseline/Program.cs)). It reached 3.08 M at its best batch size of 1,200 and fell to 1.5 M at two million. In the same session, 2.0 peaked at 13.3 M through the same string API, and the byte entry points ran at 31.7 M and 32.1 M. The 2026-09-23 performance pass (compiled invokers that read the tokens and write the pooled buffer through direct calls instead of virtual, delegate and interface calls; a tokenizer that keeps its scanner state in locals; a last-session cache; envelope keys matched by length; a flat method table) was measured A/B in one session: the same seven runs of `--sync 2 1` went from 3.2 M to 4.1 M (median 3.6 M) before to 4.0 M to 4.8 M (median 4.4 M) after, about 20 to 25 % more on one thread. The transport rows are bound by the loopback round trips rather than by the library and moved less. @@ -814,7 +841,7 @@ The 1.x projects that 2.0 does not build (`AustinHarris.JsonRpc.Client`, `Austin ### Charts -The charts, the explorer page and the benchmark figures in this README come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json): the tables transcribed with their published precision and conditions, plus the `--sweep` run files. `python benchmarks/charts/render.py` (plain Python, no packages) renders every chart in a light and a dark variant, which the README picks between with a `` element, and `render.py --check` fails if a committed chart is stale or a figure in this README no longer matches the data; the pull-request build runs it. GitHub serves README images through a proxy as plain ``, so the SVGs carry no scripts, hover text or links, and every range is drawn as an interval with its figures beside it; the interactive parts live on the explorer page. The explorer tables show median system, process and dedicated TCP-client CPU percentages when the sweep files contain them; a high client share can limit the requests the harness feeds to the server. +The charts, the explorer page and the benchmark figures in this README come from one data file, [benchmarks/charts/benchmarks.json](benchmarks/charts/benchmarks.json): the publish job's results folded in by `ingest.py`, plus the `--sweep` run files. `python benchmarks/charts/render.py` (plain Python, no packages) renders every chart in a light and a dark variant, which the README picks between with a `` element, and `render.py --check` fails if a committed chart is stale or a figure in this README no longer matches the data; the pull-request build runs it. To republish, run `benchmarks/hf/launch.sh cpu-performance publish`, then `python benchmarks/hf/fetch.py ` to extract each run's JSON and text. `python benchmarks/charts/ingest.py --source --conditions "..."` folds the JSON into benchmarks.json and the sweep files, and `render.py` regenerates the charts, the explorer and the README tables between `` markers. `render.py --check` fails when any of them is stale. GitHub serves README images through a proxy as plain ``, so the SVGs carry no scripts, hover text or links, and every range is drawn as an interval with its figures beside it; the interactive parts live on the explorer page. The explorer tables show median system, process and dedicated TCP-client CPU percentages when the sweep files contain them; a high client share can limit the requests the harness feeds to the server. ## License diff --git a/TestServer_Console/AsyncBenchmark.cs b/TestServer_Console/AsyncBenchmark.cs index d141dcd..8f0a598 100644 --- a/TestServer_Console/AsyncBenchmark.cs +++ b/TestServer_Console/AsyncBenchmark.cs @@ -3,6 +3,7 @@ using System.Diagnostics; using System.Linq; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -47,6 +48,7 @@ internal static async Task RunAsync(Action print, double seconds, int wo { if (seconds <= 0 || workers <= 0) throw new ArgumentOutOfRangeException(nameof(seconds)); var summary = new List(); + var resultRows = new JsonArray(); foreach (var (shape, flow) in Rows) { string session = "async-benchmark-" + shape + "-" + flow; @@ -54,26 +56,45 @@ internal static async Task RunAsync(Action print, double seconds, int wo { Register(session, shape, flow); var rowInputs = Inputs(shape); - await ValidateAndReportAllocations(print, session, shape, flow, rowInputs); + var perShape = await ValidateAndReportAllocations(print, session, shape, flow, rowInputs); bool yield = shape == "yield"; var m = await Measure(session, rowInputs, workers, seconds, processWide: yield); string label = yield ? "one request (yieldsOnce)" : "equal-weight mix of five requests"; print($"{shape} / {flow}: {m.Count:N0} RPCs, {workers} workers, {m.Seconds:F3} s, {m.RpcPerSec:N0} RPC/s; worker startup {m.StartupSeconds * 1000:F1} ms; " + $"{(yield ? "process-wide" : "worker-thread")} allocation total {m.AllocatedBytes:N0} B ({m.BytesPerRpc:F0} B/RPC, including method and harness allocations); {label}; {m.Cpu.TrailingText}."); summary.Add(new BenchmarkRunner.ChartRow($"{shape} / {flow}", m.RpcPerSec, $"{m.Count,12:N0} RPCs {m.BytesPerRpc,6:F0} B/RPC {m.Cpu.TrailingText}")); + var resultRow = BenchResults.Row(RowId(workers == 1 ? "async1" : "asyncn", shape, flow), m.RpcPerSec, m.Count, m.Cpu); + resultRow.Insert(1, "shape", shape); + resultRow.Insert(2, "flow", flow.ToString()); + resultRow.Insert(5, "bytesPerRpc", Math.Round(m.BytesPerRpc)); + resultRow.Insert(6, "startupMs", Math.Round(m.StartupSeconds * 1000, 1)); + if (perShape != null) resultRow["bytesPerShape"] = BenchResults.Numbers(perShape); + resultRows.Add(resultRow); } finally { Handler.DestroySession(session); } } BenchmarkRunner.PrintBarChart($"ProcessAsync(bytes), awaited workers - {Config.Serializer.Name} - {workers} worker{(workers == 1 ? "" : "s")} - RPC/s by registration", "Registration", summary); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("async", seconds); + doc["workers"] = workers; + doc["rows"] = resultRows; + BenchResults.Write($"async-w{workers}", doc, print); + } } + /// The results id of a registration row: <prefix>-<shape>-<flow> in lower case, as benchmarks.json names the series. + internal static string RowId(string prefix, string shape, RpcContextFlow flow) => $"{prefix}-{shape.ToLowerInvariant()}-{flow.ToString().ToLowerInvariant()}"; + /// /// The scaling gate: the inline None rows at 1, 2 and workers, /// paired runs, medians, and the 2/1 and N/1 ratios. Returns false when any N/1 ratio is below /// . A process-wide serialization point on the async path holds the ratio near 1.3 - /// on any core count; the per-thread cache measured about 6.8 on 16 threads. + /// on any core count; the per-thread cache measured about 6.8 on 16 threads. When is given, + /// the gate cells (median, the paired runs, CPU), the ratios and the verdict are added to it for the results file. /// - internal static async Task ScaleAsync(Action print, double seconds, int workers, double threshold, int runs = 3) + internal static async Task ScaleAsync(Action print, double seconds, int workers, double threshold, int runs = 3, JsonObject report = null) { if (workers < 2) throw new ArgumentOutOfRangeException(nameof(workers)); var counts = new[] { 1, 2, workers }.Distinct().ToArray(); @@ -101,6 +122,8 @@ internal static async Task ScaleAsync(Action print, double seconds } bool pass = true; + var cells = new JsonArray(); + var gate = new JsonArray(); print(""); print($"| Registration | 1 worker | 2 workers | {workers} workers | 2/1 | {workers}/1 | gate | CPU at {workers} workers |"); print("| --- | ---: | ---: | ---: | ---: | ---: | --- | --- |"); @@ -113,11 +136,32 @@ internal static async Task ScaleAsync(Action print, double seconds bool ok = many / one >= threshold; pass &= ok; print($"| {shape} / {flow} | {one:N0} | {two:N0} | {many:N0} | {two / one:F2} | {many / one:F2} | {(ok ? "pass" : "FAIL")} | {cpu.TrailingText} |"); + foreach (int w in counts) + { + var list = results[(shape, flow, w)]; + var cell = BenchResults.Row(RowId("scale", shape, flow), Median(list.Select(m => m.RpcPerSec).ToList()), list.Sum(m => m.Count), MedianCpu(list)); + cell.Insert(1, "shape", shape); + cell.Insert(2, "flow", flow.ToString()); + cell.Insert(3, "workers", w); + cell.Insert(5, "values", BenchResults.Numbers(list.Select(m => Math.Round(m.RpcPerSec)))); + cells.Add(cell); + } + gate.Add(new JsonObject { ["shape"] = shape, ["flow"] = flow.ToString(), ["twoOverOne"] = Math.Round(two / one, 2), ["nOverOne"] = Math.Round(many / one, 2), ["pass"] = ok }); } print(""); print(pass ? $"Scaling check passed: every inline row scales at least {threshold:0.0#}x from 1 to {workers} workers." : $"Scaling check FAILED: an inline row scales less than {threshold:0.0#}x from 1 to {workers} workers; look for a process-wide serialization point on the ProcessAsync path."); print("The 2/1 ratio is diagnostic (the lock was already visible at two workers, about 1.3); it is not gated."); + if (report != null) + { + report["workers"] = workers; + report["threshold"] = threshold; + report["runs"] = runs; + report["counts"] = BenchResults.Numbers(counts.Select(c => (double)c)); + report["rows"] = cells; + report["gate"] = gate; + report["verdict"] = pass ? "pass" : "fail"; + } return pass; } @@ -128,12 +172,14 @@ internal static async Task ScaleAsync(Action print, double seconds /// per request at one worker as reports them, so a regression in one serializer's path or in the /// suspending path shows up on its own. The process-wide serializer is switched with /// for each block and restored afterwards. A failure is printed - /// after the rows measured so far and is not thrown, so it cannot change the gate's exit code. + /// after the rows measured so far and is not thrown, so it cannot change the gate's exit code. When + /// is given, the rows measured and any failure are added to it for the results file. /// - internal static async Task ScaleDiagnosticsAsync(Action print, double seconds, int workers) + internal static async Task ScaleDiagnosticsAsync(Action print, double seconds, int workers, JsonObject report = null) { var serializers = new Func[] { () => JsmnSerializer.Instance, () => new SystemTextJsonRpcSerializer(), () => new NewtonsoftJsonRpcSerializer() }; var rows = new List(); + var diagnostics = new JsonArray(); string where = "start"; Exception failure = null; var previous = Config.Serializer; @@ -164,6 +210,12 @@ internal static async Task ScaleDiagnosticsAsync(Action print, double se var many = await Measure(session, rowInputs, workers, seconds, processWide: yield); print($" {where}, {workers,2} workers: {many.RpcPerSec,14:N0} RPC/s ({many.Count:N0} RPCs in {many.Seconds:F3} s); {many.Cpu.TrailingText}"); rows.Add($"| {shape} / {flow} | {one.RpcPerSec:N0} | {many.RpcPerSec:N0} | {many.RpcPerSec / one.RpcPerSec:F2} | {one.BytesPerRpc:F0} | {many.Cpu.TrailingText} |"); + diagnostics.Add(new JsonObject + { + ["id"] = RowId("scale-diag-" + name, shape, flow), ["serializer"] = name, ["shape"] = shape, ["flow"] = flow.ToString(), + ["one"] = Math.Round(one.RpcPerSec), ["many"] = Math.Round(many.RpcPerSec), ["ratio"] = Math.Round(many.RpcPerSec / one.RpcPerSec, 2), + ["bytesPerRpc"] = Math.Round(one.BytesPerRpc), ["cpu"] = BenchResults.Cpu(many.Cpu), + }); } finally { Handler.DestroySession(session); } } @@ -179,6 +231,11 @@ internal static async Task ScaleDiagnosticsAsync(Action print, double se print(""); if (failure != null) print($"Diagnostics stopped at {where}: {failure.GetType().Name}: {failure.Message}. The gate result above stands; the diagnostics are not part of it."); + if (report != null) + { + report["diagnostics"] = diagnostics; + if (failure != null) report["diagnosticsError"] = $"{where}: {failure.GetType().Name}: {failure.Message}"; + } } private static double Median(List values) @@ -204,8 +261,10 @@ private static ReadOnlyMemory[] Inputs(string shape) => shape == "yield" /// Checks every response, warms the row, and prints the per-shape allocation of an inline document. /// False checks, warms and asserts inline completion without printing the per-shape lines. - private static async Task ValidateAndReportAllocations(Action print, string session, string shape, RpcContextFlow flow, ReadOnlyMemory[] rowInputs, bool report = true) + /// Bytes per request for each input, as printed; null for the yield row, which is not measured per shape. + private static async Task ValidateAndReportAllocations(Action print, string session, string shape, RpcContextFlow flow, ReadOnlyMemory[] rowInputs, bool report = true) { + var perShape = shape == "yield" ? null : new double[rowInputs.Length]; var expected = shape == "yield" ? new[] { "{\"jsonrpc\":\"2.0\",\"result\":7,\"id\":6}" } : BenchmarkRunner.taskInputs.Select(t => JsonRpcProcessor.ProcessSync(t)).ToArray(); @@ -227,7 +286,9 @@ private static async Task ValidateAndReportAllocations(Action print, str } long totalBytes = GC.GetAllocatedBytesForCurrentThread() - before; if (report) print($"{shape} / {flow} / shape {i + 1}: {totalBytes} B total over 2000 requests ({totalBytes / 2000.0:F1} B/RPC), including method allocations."); + perShape[i] = Math.Round(totalBytes / 2000.0, 1); } + return perShape; } /// diff --git a/TestServer_Console/BenchResults.cs b/TestServer_Console/BenchResults.cs new file mode 100644 index 0000000..fa16f2a --- /dev/null +++ b/TestServer_Console/BenchResults.cs @@ -0,0 +1,96 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime; +using System.Runtime.InteropServices; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using AustinHarris.JsonRpc; + +namespace TestServer_Console; + +/// +/// Machine-readable results for benchmarks/charts/ingest.py. When JSONRPC_BENCH_RESULTS=<dir> is set, each +/// published mode writes <dir>/<stem>-<run>.json beside its text output, with the run number from +/// JSONRPC_BENCH_RUN (1 when unset) and the commit from JSONRPC_BENCH_COMMIT. Every row carries the id of the +/// benchmarks.json series it feeds (or an id ingest.py lists as unpublished), its RPC/s and the CPU of its timed window. +/// Nothing here runs inside a timed loop. +/// +internal static class BenchResults +{ + /// The output directory, or null when results are not requested. + internal static string Directory + { + get + { + var dir = Environment.GetEnvironmentVariable("JSONRPC_BENCH_RESULTS"); + return string.IsNullOrWhiteSpace(dir) ? null : dir; + } + } + + /// True when JSONRPC_BENCH_RESULTS is set. + internal static bool Enabled => Directory != null; + + /// The run number for the file name: JSONRPC_BENCH_RUN, or 1. + internal static int Run => int.TryParse(Environment.GetEnvironmentVariable("JSONRPC_BENCH_RUN"), out int run) && run > 0 ? run : 1; + + /// The runtime as the results record it: framework, build configuration and GC mode. + internal static string Runtime => +#if DEBUG + $"{RuntimeInformation.FrameworkDescription}, Debug, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; +#else + $"{RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; +#endif + + /// A results document with the fields every mode shares; the caller adds its parameters and rows. + internal static JsonObject Document(string mode, double seconds) => new JsonObject + { + ["mode"] = mode, + ["machine"] = MachineDescription.Current(), + ["runtime"] = Runtime, + ["serializer"] = Config.Serializer.Name, + ["date"] = DateTime.UtcNow.ToString("yyyy-MM-dd"), + ["seconds"] = seconds, + ["commit"] = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT") is { Length: > 0 } commit ? commit : null, + ["run"] = Run, + ["cores"] = Environment.ProcessorCount, + }; + + /// One result row: its series id, RPC/s rounded to a whole request, the request count and the CPU of the window. + internal static JsonObject Row(string id, double rpcPerSec, long rpcs, CpuUsage cpu) + { + var row = new JsonObject { ["id"] = id, ["rpcPerSec"] = Math.Round(rpcPerSec), ["rpcs"] = rpcs }; + var usage = Cpu(cpu); + if (usage.Count > 0) row["cpu"] = usage; + return row; + } + + /// The CPU percentages that were measured, to one decimal; a figure the platform cannot provide is left out. + internal static JsonObject Cpu(CpuUsage cpu) + { + var o = new JsonObject(); + if (cpu.SystemPercent is double system) o["system"] = Math.Round(system, 1); + if (cpu.ProcessPercent is double process) o["process"] = Math.Round(process, 1); + if (cpu.ClientPercent is double clients) o["clients"] = Math.Round(clients, 1); + return o; + } + + /// The path a mode's file goes to: <dir>/<stem>-<run>.json. + internal static string PathFor(string stem) => Path.Combine(Directory, $"{stem}-{Run}.json"); + + /// Writes the document as indented UTF-8 JSON without a byte order mark and prints the path. + internal static void Write(string stem, JsonObject document, Action print, string path = null) + { + path ??= PathFor(stem); + var dir = Path.GetDirectoryName(Path.GetFullPath(path)); + if (!string.IsNullOrEmpty(dir)) System.IO.Directory.CreateDirectory(dir); + var json = document.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping }); + File.WriteAllText(path, json + "\n", new UTF8Encoding(false)); + print?.Invoke($"wrote {path}"); + } + + /// A JSON array of numbers. + internal static JsonArray Numbers(IEnumerable values) => new JsonArray(values.Select(v => (JsonNode)JsonValue.Create(v)).ToArray()); +} diff --git a/TestServer_Console/Benchmark.cs b/TestServer_Console/Benchmark.cs index a004f26..a0fbc4c 100644 --- a/TestServer_Console/Benchmark.cs +++ b/TestServer_Console/Benchmark.cs @@ -3,6 +3,7 @@ using System.Diagnostics; using System.Linq; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -65,6 +66,7 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize RunSync(session, inputs, serializer, Math.Min(threads, 2), 0.5); // allocation profile per request shape (a zero here means the request never touches the GC) + var allocations = new JsonArray(); { using var w = new PooledByteBufferWriter(1024); sb.Clear(); @@ -78,15 +80,15 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize long per = (GC.GetAllocatedBytesForCurrentThread() - before) / reps; w.Clear(); JsonRpcProcessor.Process(session, inputs[k], w, null, serializer); sb.Append(" ").Append(per.ToString().PadLeft(6)).Append(" B ").Append(taskInputs[k]).Append(" -> ").Append(w.ToString()).Append('\n'); + allocations.Add(new JsonObject { ["request"] = taskInputs[k], ["bytes"] = per }); } _print(sb.ToString()); } // Sweep 1, 2, 4, ... up to the requested thread count (the count itself is always included). - var threadCounts = new List(); - for (int t = 1; t < threads; t *= 2) threadCounts.Add(t); - threadCounts.Add(threads); + var threadCounts = Doubling(threads); var rows = new List(threadCounts.Count); + var resultRows = new JsonArray(); foreach (var t in threadCounts) { @@ -95,6 +97,10 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize double mbIn = rps * bytesIn / inputs.Length / (1024.0 * 1024.0); double mbOut = rps * bytesOut / inputs.Length / (1024.0 * 1024.0); rows.Add(new ChartRow($"{t} thread{(t == 1 ? "" : "s")}", rps, $"{1e9 / rps * t:N0} ns/RPC per thread {cpu.TrailingText}")); + var resultRow = BenchResults.Row("ours-sync", rps, total, cpu); + resultRow.Insert(1, "threads", t); + resultRow.Insert(4, "nsPerRequest", Math.Round(1e9 / rps * t)); + resultRows.Add(resultRow); // The detailed boxes are only printed for the two documented points: one thread and all threads. if (t != 1 && t != threads) continue; @@ -119,6 +125,25 @@ internal static void BenchmarkSync(Action print = null, JsonRpcSerialize } PrintBarChart($"Sync benchmark - {serializer.Name} - RPC/s by thread count", "Threads", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("sync", seconds); + doc["threads"] = threads; + doc["allocations"] = allocations; + doc["rows"] = resultRows; + BenchResults.Write("sync", doc, _print); + } + } + + /// The published worker ladder: 1, 2, 4, ... below , then itself. + internal static List Doubling(int max) + { + if (max < 1) throw new ArgumentOutOfRangeException(nameof(max)); + var counts = new List(); + for (int t = 1; t < max; t *= 2) counts.Add(t); + counts.Add(max); + return counts; } internal static (long count, double seconds, CpuUsage cpu) RunSync(string session, ReadOnlyMemory[] inputs, JsonRpcSerializer serializer, int threads, double seconds) @@ -174,6 +199,7 @@ internal static void Benchmark(Action print = null) var iterations = 8; var cnt = 50; var results = new List(iterations); + var resultRows = new JsonArray(); var minIterationTime = TimeSpan.FromMilliseconds(500); // Warm up on the exact path the iterations use, for long enough that tiered compilation has replaced @@ -221,12 +247,23 @@ internal static void Benchmark(Action print = null) double seconds = currentStopwatch.Elapsed.TotalSeconds; long total = (long)cnt * currentBatches; results.Add(new ChartRow($"#{iteration} {cnt,11:N0}", total / seconds, $"{total,11:N0} RPCs in {seconds:F3} s {cpu.TrailingText}")); + var resultRow = BenchResults.Row("legacy-string", total / seconds, total, cpu); + resultRow.Insert(1, "batch", cnt); + resultRow.Insert(4, "seconds", Math.Round(seconds, 3)); + resultRows.Add(resultRow); } lock (renderLock) { PrintBarChart("Benchmark Summary - RPC/s by batch size", " # Batch size", results); } + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("legacy", minIterationTime.TotalSeconds); + doc["rows"] = resultRows; + BenchResults.Write("legacy", doc, _print); + } } /// diff --git a/TestServer_Console/CompareBenchmark.cs b/TestServer_Console/CompareBenchmark.cs index f0cf202..07215e7 100644 --- a/TestServer_Console/CompareBenchmark.cs +++ b/TestServer_Console/CompareBenchmark.cs @@ -8,6 +8,7 @@ using System.Net.Sockets; using System.Text; using System.Text.Json; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -78,6 +79,7 @@ internal static async Task RunAsync(Action print, double seconds = 3, in { print($"{Versions()}; Kestrel on loopback, {clients} clients, pipeline {pipeline}, {seconds:0.#} s per row\n"); var rows = new List(); + var resultRows = new JsonArray(); var session = Handler.DefaultSessionId(); // ---- in-process floors @@ -85,31 +87,37 @@ internal static async Task RunAsync(Action print, double seconds = 3, in BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, Warm); var (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, seconds); rows.Add(new BenchmarkRunner.ChartRow("JSON-RPC.Net in-process, 1 thread", total / elapsed, $"{total,12:N0} RPCs direct call, bytes in, bytes out {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row("ours-direct-1", total / elapsed, total, cpu)); print($" JSON-RPC.Net in-process done ({rows[^1].RpcPerSec:N0} RPC/s)"); PipeRun(newLine, Warm, pipeline); var (count, secs, usage) = PipeRun(newLine, seconds, pipeline); rows.Add(new BenchmarkRunner.ChartRow("StreamJsonRpc in-process, 1 client", count / secs, $"{count,12:N0} RPCs Pipe pair, newline framing, STJ formatter, {pipeline} pipelined {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("sjr-pipe", count / secs, count, usage)); print($" StreamJsonRpc in-process done ({count / secs:N0} RPC/s)"); await ProxyRun(Warm); (count, secs, usage) = await ProxyRun(seconds); rows.Add(new BenchmarkRunner.ChartRow("StreamJsonRpc proxy, sequential await", count / secs, $"{count,12:N0} RPCs {secs / count * 1e6:N1} us per round trip {usage.TrailingText}")); + var proxyRow = BenchResults.Row("sjr-proxy", count / secs, count, usage); + proxyRow["usPerRequest"] = Math.Round(secs / count * 1e6, 1); + resultRows.Add(proxyRow); print($" StreamJsonRpc proxy done ({count / secs:N0} RPC/s)"); // ---- Kestrel TCP, same client for every row - foreach (var (label, port, inputs, prefix) in new[] + foreach (var (label, id, port, inputs, prefix) in new[] { - ("JSON-RPC.Net TCP, raw documents", oursPort, raw, JsonRpcPrefix), - ("StreamJsonRpc TCP, newline + STJ", sjrNewLineStjPort, newLine, JsonRpcPrefix), - ("StreamJsonRpc TCP, Content-Length + STJ", sjrHeaderStjPort, header, (byte[])null), - ("StreamJsonRpc TCP, Content-Length + Json.NET", sjrHeaderNewtonsoftPort, header, (byte[])null), + ("JSON-RPC.Net TCP, raw documents", "ours-tcp-jsonrpc", oursPort, raw, JsonRpcPrefix), + ("StreamJsonRpc TCP, newline + STJ", "sjr-newline-stj", sjrNewLineStjPort, newLine, JsonRpcPrefix), + ("StreamJsonRpc TCP, Content-Length + STJ", "sjr-cl-stj", sjrHeaderStjPort, header, (byte[])null), + ("StreamJsonRpc TCP, Content-Length + Json.NET", "sjr-cl-newtonsoft", sjrHeaderNewtonsoftPort, header, (byte[])null), }) { Probe(port, inputs, label); KestrelBenchmark.TcpRun(port, inputs, clients, Warm, pipeline, prefix); (count, secs, usage) = KestrelBenchmark.TcpRun(port, inputs, clients, seconds, pipeline, prefix); rows.Add(new BenchmarkRunner.ChartRow(label, count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(id, count / secs, count, usage)); print($" {label} done ({count / secs:N0} RPC/s)"); } @@ -117,22 +125,40 @@ internal static async Task RunAsync(Action print, double seconds = 3, in await GrpcCompare.UnaryRun(grpcPort, clients, pipeline, Warm); (count, secs, usage) = await GrpcCompare.UnaryRun(grpcPort, clients, pipeline, seconds); rows.Add(new BenchmarkRunner.ChartRow("gRPC for .NET unary, HTTP/2", count / secs, $"{count,12:N0} RPCs {clients} channels, {pipeline} calls in flight each {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("grpc-unary", count / secs, count, usage)); print($" gRPC unary done ({count / secs:N0} RPC/s)"); await GrpcCompare.StreamRun(grpcPort, clients, pipeline, Warm); (count, secs, usage) = await GrpcCompare.StreamRun(grpcPort, clients, pipeline, seconds); rows.Add(new BenchmarkRunner.ChartRow("gRPC for .NET bidirectional stream", count / secs, $"{count,12:N0} RPCs {clients} streams, {pipeline} calls in flight each {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("grpc-stream", count / secs, count, usage)); print($" gRPC stream done ({count / secs:N0} RPC/s)"); BenchmarkRunner.PrintBarChart("JSON-RPC.Net vs StreamJsonRpc vs gRPC - RPC/s", "Library / transport", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document("compare", seconds); + doc["clients"] = clients; + doc["pipeline"] = pipeline; + doc["versions"] = new JsonObject + { + ["jsonrpc"] = typeof(JsonRpcProcessor).Assembly.GetName().Version?.ToString(), + ["streamJsonRpc"] = PackageVersion(typeof(JsonRpc)), + ["grpc"] = PackageVersion(typeof(Grpc.Net.Client.GrpcChannel)), + }; + doc["rows"] = resultRows; + BenchResults.Write("compare", doc, print); + } } } /// - /// Every library and transport at 1, 2, 4, 8 and 16 client connections: the data for the README's - /// multi-series chart. Same host, same five calls, same pipeline depth per connection as - /// . Prints a table and, when is given, writes the - /// numbers as JSON for benchmarks/charts/render.py. + /// Every library and transport at 1, 2, 4, ... client connections up to the core count + /// (): the data for the README's multi-series chart. Same host, same five + /// calls, same pipeline depth per connection as . Prints a table and, when + /// is given (or JSONRPC_BENCH_RESULTS is set, as sweep-<run>.json there), + /// writes the numbers as JSON for benchmarks/charts/render.py. /// internal static async Task SweepAsync(Action print, double seconds = 2, string outputPath = null, int pipeline = 256) { @@ -143,17 +169,17 @@ internal static async Task SweepAsync(Action print, double seconds = 2, var batch = new[] { KestrelBenchmark.BuildBatch(raw, 100) }; await using var h = await StartAsync(pipeline); - int[] connections = { 1, 2, 4, 8, 16 }; - (string name, string kind, Func> run)[] series = + int[] connections = BenchmarkRunner.Doubling(Environment.ProcessorCount).ToArray(); + (string name, string id, string kind, Func> run)[] series = { - ("JSON-RPC.Net, TCP", "ours", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.OursPort, raw, c, s, pipeline, JsonRpcPrefix))), - ("JSON-RPC.Net, HTTP, batch of 100 per POST", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, batch, 100, c, s)), - ("JSON-RPC.Net, HTTP, 1 request per POST", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, raw, 1, c, s)), - ("StreamJsonRpc, TCP, newline + System.Text.Json", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrNewLineStjPort, newLine, c, s, pipeline, JsonRpcPrefix))), - ("StreamJsonRpc, TCP, Content-Length + System.Text.Json", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderStjPort, header, c, s, pipeline, null))), - ("StreamJsonRpc, TCP, Content-Length + Json.NET", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderNewtonsoftPort, header, c, s, pipeline, null))), - ("gRPC for .NET, unary", "grpc", (c, s) => GrpcCompare.UnaryRun(h.GrpcPort, c, pipeline, s)), - ("gRPC for .NET, bidirectional stream", "grpc", (c, s) => GrpcCompare.StreamRun(h.GrpcPort, c, pipeline, s)), + ("JSON-RPC.Net, TCP", "sweep-ours-tcp", "ours", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.OursPort, raw, c, s, pipeline, JsonRpcPrefix))), + ("JSON-RPC.Net, HTTP, batch of 100 per POST", "sweep-ours-http-100", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, batch, 100, c, s)), + ("JSON-RPC.Net, HTTP, 1 request per POST", "sweep-ours-http-1", "ours", (c, s) => KestrelBenchmark.HttpRun(h.HttpUrl, raw, 1, c, s)), + ("StreamJsonRpc, TCP, newline + System.Text.Json", "sweep-sjr-newline-stj", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrNewLineStjPort, newLine, c, s, pipeline, JsonRpcPrefix))), + ("StreamJsonRpc, TCP, Content-Length + System.Text.Json", "sweep-sjr-cl-stj", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderStjPort, header, c, s, pipeline, null))), + ("StreamJsonRpc, TCP, Content-Length + Json.NET", "sweep-sjr-cl-newtonsoft", "theirs", (c, s) => Task.FromResult(KestrelBenchmark.TcpRun(h.SjrHeaderNewtonsoftPort, header, c, s, pipeline, null))), + ("gRPC for .NET, unary", "sweep-grpc-unary", "grpc", (c, s) => GrpcCompare.UnaryRun(h.GrpcPort, c, pipeline, s)), + ("gRPC for .NET, bidirectional stream", "sweep-grpc-stream", "grpc", (c, s) => GrpcCompare.StreamRun(h.GrpcPort, c, pipeline, s)), }; Probe(h.OursPort, raw, series[0].name); Probe(h.SjrNewLineStjPort, newLine, series[3].name); @@ -180,11 +206,16 @@ internal static async Task SweepAsync(Action print, double seconds = 2, for (int si = 0; si < series.Length; si++) print(" " + series[si].name.PadRight(56) + string.Join("", connections.Select((_, ci) => $"{results[si, ci],14:N0}"))); + if (outputPath == null && BenchResults.Enabled) outputPath = BenchResults.PathFor("sweep"); if (outputPath != null) { var doc = new { + mode = "sweep", machine = MachineDescription.Current(), + runtime = BenchResults.Runtime, + commit = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT") is { Length: > 0 } commit ? commit : null, + run = BenchResults.Run, cores = Environment.ProcessorCount, date = DateTime.Now.ToString("yyyy-MM-dd"), secondsPerCell = seconds, @@ -192,13 +223,15 @@ internal static async Task SweepAsync(Action print, double seconds = 2, connections, series = series.Select((s, si) => new { - s.name, s.kind, + s.name, s.id, s.kind, rpcPerSec = connections.Select((_, ci) => Math.Round(results[si, ci])).ToArray(), cpuSystem = connections.Select((_, ci) => usages[si, ci].SystemPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray(), cpuProcess = connections.Select((_, ci) => usages[si, ci].ProcessPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray(), cpuClients = connections.Select((_, ci) => usages[si, ci].ClientPercent is double v ? Math.Round(v, 1) : (double?)null).ToArray() }).ToArray(), }; + var outputDir = Path.GetDirectoryName(Path.GetFullPath(outputPath)); + if (!string.IsNullOrEmpty(outputDir)) Directory.CreateDirectory(outputDir); File.WriteAllText(outputPath, JsonSerializer.Serialize(doc, new JsonSerializerOptions { WriteIndented = true }) + "\n"); print($"\nwrote {outputPath}"); } diff --git a/TestServer_Console/KestrelBenchmark.cs b/TestServer_Console/KestrelBenchmark.cs index 2345b23..fa0e6cd 100644 --- a/TestServer_Console/KestrelBenchmark.cs +++ b/TestServer_Console/KestrelBenchmark.cs @@ -8,6 +8,7 @@ using System.Net.Http.Headers; using System.Net.Sockets; using System.Text; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -68,6 +69,10 @@ internal static async Task RunAsync(Action print, double seconds = 3, in print($"Kestrel on {httpUrl} (HTTP) and 127.0.0.1:{tcpPort} (TCP), {clients} clients, pipeline {pipeline}, {seconds:0.#} s per row, EnableAsyncMethods = {(asyncMethods ? "true" : "false")}\n"); var rows = new List(); + var resultRows = new JsonArray(); + // Result ids: the EnableAsyncMethods = false rows and the two TCP rows under true feed benchmarks.json; the rest + // are kept in the file under their own ids. + string Id(string defaultId, string asyncId) => asyncMethods ? asyncId : defaultId; var session = Handler.DefaultSessionId(); var memInputs = inputs.Select(i => (ReadOnlyMemory)i).ToArray(); @@ -75,26 +80,33 @@ internal static async Task RunAsync(Action print, double seconds = 3, in BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, 0.5); // warm-up var (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, 1, seconds); rows.Add(new BenchmarkRunner.ChartRow("in-process, 1 thread", total / elapsed, $"{total,12:N0} RPCs {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("kestrel-inproc-1", "kestrel-async-inproc-1"), total / elapsed, total, cpu)); (total, elapsed, cpu) = BenchmarkRunner.RunSync(session, memInputs, Config.Serializer, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow($"in-process, {clients} threads", total / elapsed, $"{total,12:N0} RPCs {cpu.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-inproc-n", "kestrel-async-inproc-n"), total / elapsed, total, cpu)); print($" in-process rows done ({rows[0].RpcPerSec:N0} / {rows[1].RpcPerSec:N0} RPC/s)"); // HTTP, one request per POST. await HttpRun(httpUrl, inputs, 1, clients, 0.5); // warm-up var (count, secs, usage) = await HttpRun(httpUrl, inputs, 1, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow("HTTP, 1 request/POST", count / secs, $"{count,12:N0} RPCs {secs / count * 1e6 * clients:N1} us/request per client {usage.TrailingText}")); + var httpRow = BenchResults.Row(Id("ours-http-1", "kestrel-async-http-1"), count / secs, count, usage); + httpRow["usPerRequest"] = Math.Round(secs / count * 1e6 * clients, 1); + resultRows.Add(httpRow); print($" HTTP single done ({count / secs:N0} RPC/s)"); // HTTP, a batch per POST. await HttpRun(httpUrl, new[] { batch }, batchSize, clients, 0.5); (count, secs, usage) = await HttpRun(httpUrl, new[] { batch }, batchSize, clients, seconds); rows.Add(new BenchmarkRunner.ChartRow($"HTTP, batch of {batchSize}/POST", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-http-100", "kestrel-async-http-100"), count / secs, count, usage)); print($" HTTP batch done ({count / secs:N0} RPC/s)"); // TCP, pipelined. TcpRun(tcpPort, inputs, clients, 0.5, pipeline, ResultPrefix); (count, secs, usage) = TcpRun(tcpPort, inputs, clients, seconds, pipeline, ResultPrefix); rows.Add(new BenchmarkRunner.ChartRow($"TCP, {pipeline} pipelined{(asyncMethods ? ", inline methods" : "")}", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row(Id("ours-tcp", "ours-tcp-async-inline"), count / secs, count, usage)); print($" TCP done ({count / secs:N0} RPC/s)"); if (asyncMethods) @@ -104,10 +116,22 @@ internal static async Task RunAsync(Action print, double seconds = 3, in TcpRun(tcpPort, yieldInputs, clients, 0.5, pipeline, ResultPrefix); (count, secs, usage) = TcpRun(tcpPort, yieldInputs, clients, seconds, pipeline, ResultPrefix); rows.Add(new BenchmarkRunner.ChartRow($"TCP, {pipeline} pipelined, yielding methods", count / secs, $"{count,12:N0} RPCs {usage.TrailingText}")); + resultRows.Add(BenchResults.Row("ours-tcp-async-yield", count / secs, count, usage)); print($" TCP yielding done ({count / secs:N0} RPC/s)"); } BenchmarkRunner.PrintBarChart($"Kestrel benchmark - {Config.Serializer.Name} - EnableAsyncMethods = {(asyncMethods ? "true" : "false")} - RPC/s by transport", "Transport", rows); + + if (BenchResults.Enabled) + { + var doc = BenchResults.Document(asyncMethods ? "kestrel-async" : "kestrel", seconds); + doc["clients"] = clients; + doc["pipeline"] = pipeline; + doc["batchSize"] = batchSize; + doc["asyncMethods"] = asyncMethods; + doc["rows"] = resultRows; + BenchResults.Write(asyncMethods ? "kestrel-async" : "kestrel", doc, print); + } } finally { diff --git a/TestServer_Console/Program.cs b/TestServer_Console/Program.cs index 4fec7ce..e638998 100644 --- a/TestServer_Console/Program.cs +++ b/TestServer_Console/Program.cs @@ -33,16 +33,22 @@ static void Main(string[] args) // `dotnet run -- --scale [seconds] [workers] [threshold]` is the release gate for the ProcessAsync path: // the inline rows at 1, 2 and N workers, three paired runs, medians; exit code 1 when N/1 is below the threshold. // After the gate it prints a per-serializer diagnostics table that never changes the exit code; a final - // `--no-diagnostics` argument skips it. + // `--no-diagnostics` argument skips it. The worker count defaults to the core count. if (args.Length > 0 && args[0] == "--scale") { double seconds = args.Length > 1 && double.TryParse(args[1], out var s) ? s : 3; - int workers = args.Length > 2 && int.TryParse(args[2], out var t) ? t : 16; + int workers = args.Length > 2 && int.TryParse(args[2], out var t) ? t : Environment.ProcessorCount; double threshold = args.Length > 3 && double.TryParse(args[3], out var r) ? r : 4.0; bool diagnostics = !(args.Length > 1 && args[args.Length - 1] == "--no-diagnostics"); - bool pass = AsyncBenchmark.ScaleAsync(Console.WriteLine, seconds, workers, threshold).GetAwaiter().GetResult(); + var report = BenchResults.Enabled ? BenchResults.Document("scale", seconds) : null; + bool pass = AsyncBenchmark.ScaleAsync(Console.WriteLine, seconds, workers, threshold, report: report).GetAwaiter().GetResult(); Environment.ExitCode = pass ? 0 : 1; - if (diagnostics) AsyncBenchmark.ScaleDiagnosticsAsync(Console.WriteLine, seconds, workers).GetAwaiter().GetResult(); + if (diagnostics) AsyncBenchmark.ScaleDiagnosticsAsync(Console.WriteLine, seconds, workers, report).GetAwaiter().GetResult(); + if (report != null) + { + report["exitCode"] = Environment.ExitCode; + BenchResults.Write("scale", report, Console.WriteLine); + } return; } @@ -75,7 +81,8 @@ static void Main(string[] args) return; } - // `dotnet run -- --sweep [seconds] [output.json]` measures every library and transport at 1, 2, 4, 8 and 16 connections. + // `dotnet run -- --sweep [seconds] [output.json]` measures every library and transport at 1, 2, 4, ... connections up to + // the core count; with JSONRPC_BENCH_RESULTS set and no output path, the file goes to /sweep-.json. if (args.Length > 0 && args[0] == "--sweep") { double seconds = args.Length > 1 && double.TryParse(args[1], out var s) ? s : 2; diff --git a/benchmarks/Baseline/Program.cs b/benchmarks/Baseline/Program.cs index f0d992e..b7506d3 100644 --- a/benchmarks/Baseline/Program.cs +++ b/benchmarks/Baseline/Program.cs @@ -1,6 +1,13 @@ using System; using System.Collections.Generic; using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Runtime; +using System.Runtime.InteropServices; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; using System.Threading; using System.Threading.Tasks; using AustinHarris.JsonRpc; @@ -47,21 +54,28 @@ public static int Main() while (warm.Elapsed < TimeSpan.FromSeconds(1)) RunBatch(5000); var rows = new List<(int size, double rps)>(); + var resultRows = new JsonArray(); int cnt = 50; for (int iteration = 1; iteration <= 8; iteration++) { cnt *= iteration; int batches = 0; + var cpu = CpuWindow.Start(); var sw = Stopwatch.StartNew(); do { RunBatch(cnt); batches++; } while (sw.Elapsed < TimeSpan.FromMilliseconds(500)); sw.Stop(); + var usage = cpu.Stop(); double rps = (long)cnt * batches / sw.Elapsed.TotalSeconds; rows.Add((cnt, rps)); Console.WriteLine($"#{iteration} batch {cnt,10:N0}: {rps,14:N0} RPC/s ({(long)cnt * batches,12:N0} RPCs in {sw.Elapsed.TotalSeconds:F3} s)"); + var row = new JsonObject { ["id"] = "headline-123-string", ["batch"] = cnt, ["rpcPerSec"] = Math.Round(rps), ["rpcs"] = (long)cnt * batches, ["seconds"] = Math.Round(sw.Elapsed.TotalSeconds, 3) }; + if (usage.Count > 0) row["cpu"] = usage; + resultRows.Add(row); } double peak = 0; int peakSize = 0; foreach (var (size, rps) in rows) if (rps > peak) { peak = rps; peakSize = size; } Console.WriteLine($"peak {peak:N0} RPC/s at batch {peakSize:N0}"); + WriteResults(version, resultRows); GC.KeepAlive(service); return 0; } @@ -79,5 +93,150 @@ private static void RunBatch(int count) done.Wait(); } } + + /// + /// With JSONRPC_BENCH_RESULTS set, writes baseline-<JSONRPC_BENCH_RUN>.json there in the shape TestServer_Console's + /// results use (benchmarks/charts/ingest.py reads it): one row per batch size, id headline-123-string. + /// + private static void WriteResults(Version version, JsonArray rows) + { + var dir = Environment.GetEnvironmentVariable("JSONRPC_BENCH_RESULTS"); + if (string.IsNullOrWhiteSpace(dir)) return; + int run = int.TryParse(Environment.GetEnvironmentVariable("JSONRPC_BENCH_RUN"), out int r) && r > 0 ? r : 1; + var commit = Environment.GetEnvironmentVariable("JSONRPC_BENCH_COMMIT"); + var doc = new JsonObject + { + ["mode"] = "baseline", + ["machine"] = Machine(), + ["runtime"] = $"{RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}", + ["serializer"] = "Json.NET (1.2.3)", + ["date"] = DateTime.UtcNow.ToString("yyyy-MM-dd"), + ["seconds"] = 0.5, + ["commit"] = string.IsNullOrEmpty(commit) ? null : commit, + ["run"] = run, + ["cores"] = Environment.ProcessorCount, + ["version"] = version.ToString(), + ["rows"] = rows, + }; + Directory.CreateDirectory(dir); + var path = Path.Combine(dir, $"baseline-{run}.json"); + File.WriteAllText(path, doc.ToJsonString(new JsonSerializerOptions { WriteIndented = true }) + "\n", new UTF8Encoding(false)); + Console.WriteLine($"wrote {path}"); + } + + /// CPU model, capacity, OS and runtime: the string TestServer_Console's machine line gives on Linux. + private static string Machine() + { + var overridden = Environment.GetEnvironmentVariable("JSONRPC_BENCH_MACHINE"); + if (!string.IsNullOrWhiteSpace(overridden)) return overridden; + string model = Environment.GetEnvironmentVariable("PROCESSOR_IDENTIFIER") ?? RuntimeInformation.ProcessArchitecture.ToString(); + int hostCpus = 0; + bool named = false; + if (OperatingSystem.IsLinux()) + { + try + { + foreach (var line in File.ReadLines("/proc/cpuinfo")) + { + if (!named && line.StartsWith("model name", StringComparison.Ordinal)) { model = line.Split(':', 2)[1].Trim(); named = true; } + if (line.StartsWith("processor", StringComparison.Ordinal) && line.Contains(':')) hostCpus++; + } + } + catch (IOException) { } + } + string capacity = Environment.ProcessorCount + " cores"; + if (OperatingSystem.IsLinux() && CpuWindow.Quota() > 0 && hostCpus > 0) capacity += $" of {hostCpus} host CPUs, cgroup quota"; + return $"{model}, {capacity}, {RuntimeInformation.OSDescription}, {RuntimeInformation.FrameworkDescription}, Release, {(GCSettings.IsServerGC ? "Server GC" : "Workstation GC")}"; + } + } + + /// + /// CPU over one timed window, with TestServer_Console's CpuMeter arithmetic: the process as a share of wall time times + /// the cores, the machine from cgroup cpu.stat under a quota, else /proc/stat, else GetSystemTimes. There are no client + /// threads here, so there is no client figure. + /// + internal sealed class CpuWindow + { + private readonly Stopwatch _wall = Stopwatch.StartNew(); + private readonly double _process = ProcessSeconds(); + private readonly double _quota = Quota(); + private readonly ulong _busy, _total; + private readonly bool _system; + + private CpuWindow() { _system = SystemTimes(out _busy, out _total); } + + internal static CpuWindow Start() => new CpuWindow(); + + internal JsonObject Stop() + { + double wall = _wall.Elapsed.TotalSeconds; + var o = new JsonObject(); + if (_system && SystemTimes(out ulong busy, out ulong total)) + { + // Under a quota the counters are cgroup microseconds of CPU; otherwise busy and total ticks. + double system = _quota > 0 ? Percent((busy - _busy) / 1_000_000.0, wall * _quota) : Percent(busy - _busy, total - _total); + o["system"] = Math.Round(system, 1); + } + o["process"] = Math.Round(Percent(ProcessSeconds() - _process, wall * Environment.ProcessorCount), 1); + return o; + } + + private static double Percent(double amount, double capacity) => capacity > 0 ? Math.Clamp(100 * amount / capacity, 0, 100) : 0; + + private static double ProcessSeconds() + { + using var p = Process.GetCurrentProcess(); + return p.TotalProcessorTime.TotalSeconds; + } + + /// The cgroup v2 CPU quota in cores, or 0 when there is none. + internal static double Quota() + { + try + { + if (!OperatingSystem.IsLinux() || !File.Exists("/sys/fs/cgroup/cpu.max")) return 0; + var parts = File.ReadAllText("/sys/fs/cgroup/cpu.max").Split((char[])null, StringSplitOptions.RemoveEmptyEntries); + if (parts.Length < 2 || parts[0] == "max") return 0; + double quota = double.Parse(parts[0], System.Globalization.CultureInfo.InvariantCulture); + double period = double.Parse(parts[1], System.Globalization.CultureInfo.InvariantCulture); + return quota > 0 && period > 0 ? quota / period : 0; + } + catch (Exception) { return 0; } + } + + private bool SystemTimes(out ulong busy, out ulong total) + { + busy = total = 0; + try + { + if (OperatingSystem.IsLinux() && _quota > 0) + { + var line = File.ReadLines("/sys/fs/cgroup/cpu.stat").FirstOrDefault(s => s.StartsWith("usage_usec ", StringComparison.Ordinal)); + return line != null && ulong.TryParse(line.AsSpan("usage_usec ".Length), out busy); + } + if (OperatingSystem.IsLinux()) + { + var parts = File.ReadLines("/proc/stat").First().Split((char[])null, StringSplitOptions.RemoveEmptyEntries); + if (parts.Length < 6 || parts[0] != "cpu") return false; + for (int i = 1; i < parts.Length; i++) total += ulong.Parse(parts[i]); + busy = total - ulong.Parse(parts[4]) - ulong.Parse(parts[5]); + return true; + } + if (OperatingSystem.IsWindows() && GetSystemTimes(out var idle, out var kernel, out var user)) + { + busy = kernel.Value - idle.Value + user.Value; + total = kernel.Value + user.Value; + return true; + } + } + catch (Exception) { } + return false; + } + + [StructLayout(LayoutKind.Sequential)] + private struct FileTime { internal uint Low; internal uint High; internal readonly ulong Value => ((ulong)High << 32) | Low; } + + [DllImport("kernel32.dll")] + private static extern bool GetSystemTimes(out FileTime idle, out FileTime kernel, out FileTime user); } } diff --git a/benchmarks/charts/benchmarks.json b/benchmarks/charts/benchmarks.json index 59c2ad8..7aa4ef7 100644 --- a/benchmarks/charts/benchmarks.json +++ b/benchmarks/charts/benchmarks.json @@ -1,153 +1,206 @@ { "schema": 1, - "machine": "AMD Ryzen 7 7800X3D (8 cores / 16 threads, 4.2 GHz), 64 GB, Windows 11", - "runtime": ".NET 10, Release, Server GC, built-in serializer", + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC, built-in serializer", + "summary": [ + { "what": "Library alone, {workers} threads", "api": "`Process(bytes)`, dedicated threads", "source": "sync:ours-sync:last", "details": "[Sync](#sync-the-library-alone)" }, + { "what": "Library alone, {workers} workers", "api": "`ProcessAsync(bytes)`, awaited workers, methods that complete inline", "source": "asyncn:asyncn-sync-none,asyncn-task-flow,asyncn-task-none,asyncn-valuetask-flow,asyncn-valuetask-none:range", "details": "[Async](#async-processasync-awaited-workers); the spread is across registrations, not runs" }, + { "what": "Library alone, {workers} workers, one real suspension per request", "api": "`ProcessAsync(bytes)`, `yieldsOnce`", "source": "asyncn:asyncn-yield-none", "details": "" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-tcp", "details": "[Kestrel](#kestrel-through-the-aspnetcore-package)" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = true`, methods that complete inline", "source": "kestrel:ours-tcp-async-inline", "details": "" }, + { "what": "Kestrel TCP, 256 pipelined", "api": "`EnableAsyncMethods = true`, methods that suspend once", "source": "kestrel:ours-tcp-async-yield", "details": "" }, + { "what": "Kestrel HTTP, batch of 100 per POST", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-http-100", "details": "" }, + { "what": "Kestrel HTTP, one request per POST", "api": "`EnableAsyncMethods = false`", "source": "kestrel:ours-http-1", "details": "HTTP/1.1 round trips dominate" }, + { "what": "Legacy string API, thread pool", "api": "`Task Process(string)`, batches of {batch}", "source": "legacy:legacy-string:best", "details": "[Legacy](#legacy-string-api-scheduled-synchronous-execution); the 1.x overloads, not the byte path" } + ], "sets": { "headline": { "title": "JSON-RPC.Net 1.2.3 and 2.0 on one machine", - "date": "2026-09-25", - "source": "README.md, Performance table; branch async-scratch-cache", - "harness": "benchmarks/Baseline (1.2.3 from NuGet), then TestServer_Console menu entry t, --sync 3 16 and --async 3 16, one session", - "workload": "the five benchmark requests; the string rows are the best batch size of the thread-pool loop, the byte rows a loop on 16 dedicated threads or 16 awaited workers", - "conditions": "idle box; one run per row", - "policy": "one run per row, all in one session; the day-to-day ranges are the sync, async, legacy and kestrel sets", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "benchmarks/Baseline (1.2.3 from NuGet), TestServer_Console menu entry t, --sync 3 and --async 3 32", + "workload": "the five benchmark requests; the string rows are the best batch size of the thread-pool loop, the byte rows a loop on {workers} dedicated threads or {workers} awaited workers", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "2 runs of benchmarks/Baseline and 3 runs of the t entry, each at its best batch size by median; 2 runs of --sync and 2 runs of --async at 32; low to high over the runs", + "workers": 32, + "subtitle": ["The same five requests on one machine in one job, {date}: {host}, .NET 10, Server GC.", "String rows: the best batch size of the thread-pool loop. Byte rows: {workers} dedicated threads or {workers} awaited workers in a loop."], + "alt": "JSON-RPC.Net 1.2.3 and 2.0 on one machine, low to high over the runs of one job on {date}: {p:headline-123-string} requests per second through the 1.2.3 string API, {p:headline-20-string} through the same API on 2.0, and {p:headline-20-sync} and {p:headline-20-async} through the 2.0 byte entry points", "groups": [ { "heading": "1.2.3, the last 1.x release on NuGet", "rule": false, "series": ["headline-123-string"] }, { "heading": "2.0, the same string API and the same loop", "rule": false, "series": ["headline-20-string"] }, { "heading": "2.0, the byte entry points", "rule": false, "series": ["headline-20-sync", "headline-20-async"] } ], "series": [ - { "id": "headline-123-string", "label": "Task Process(string), thread pool", "note": "best batch size, 1,200; Json.NET on every request", "family": "scale", "low": 3080000, "high": 3080000, "status": "single", "published": "3.08 M" }, - { "id": "headline-20-string", "label": "Task Process(string), thread pool", "note": "best batch size, 36,000; the 1.x overloads, kept for compatibility", "family": "ours", "low": 13300000, "high": 13300000, "status": "single", "published": "13.3 M" }, - { "id": "headline-20-sync", "label": "Process(bytes), 16 dedicated threads", "note": "the library alone; 0 B per numeric request", "family": "ours", "low": 31700000, "high": 31700000, "status": "single", "published": "31.7 M" }, - { "id": "headline-20-async", "label": "ProcessAsync(bytes), 16 awaited workers", "note": "ValueTask methods, RpcContextFlow.None; 6 B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M" } - ] + { "id": "headline-123-string", "label": "Task Process(string), thread pool", "note": "best batch size, {batch}; Json.NET on every request", "family": "scale", "low": 1289438, "high": 2412083, "status": "range", "published": "1.29 M to 2.41 M", "row": "1.2.3, `Task Process(string)`, thread pool, best batch size", "batch": 6000, "median": 1850760.5, "values": [1289438, 2412083], "n": 2, "cpu": { "system": 68.5, "process": 68.5 } }, + { "id": "headline-20-string", "label": "Task Process(string), thread pool", "note": "best batch size, {batch}; the 1.x overloads, kept for compatibility", "family": "ours", "low": 9825595, "high": 11144579, "status": "range", "published": "9.83 M to 11.1 M", "row": "2.0, the same string API and the same loop", "batch": 6000, "median": 11110511, "values": [11110511, 11144579, 9825595], "n": 3, "cpu": { "system": 87.5, "process": 87.5 }, "against": "6.0×" }, + { "id": "headline-20-sync", "label": "Process(bytes), {workers} dedicated threads", "note": "the library alone; 0 B per numeric request", "family": "ours", "low": 44239898, "high": 45671189, "status": "range", "published": "44.2 M to 45.7 M", "row": "2.0, `Process(bytes)`, {workers} dedicated threads", "median": 44955543.5, "values": [44239898, 45671189], "n": 2, "cpu": { "system": 91.4, "process": 91.4 }, "against": "24.3×" }, + { "id": "headline-20-async", "label": "ProcessAsync(bytes), {workers} awaited workers", "note": "ValueTask methods, RpcContextFlow.None; {bytes} B per request", "family": "ours", "low": 36255749, "high": 40521407, "status": "range", "published": "36.3 M to 40.5 M", "row": "2.0, `ProcessAsync(bytes)`, {workers} awaited workers", "bytes": 6, "median": 38388578, "values": [36255749, 40521407], "n": 2, "cpu": { "system": 96.9, "process": 96.9 }, "against": "20.7×" } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 9 }, "sync": { "title": "JSON-RPC.Net alone, by worker threads", - "date": "2026-09-25", - "source": "README.md, Sync table; branch async-scratch-cache", - "harness": "TestServer_Console --sync 3, and --sync 2 for single rows", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console --sync 3 (1, 2, 4, ... threads up to 32)", "workload": "the five benchmark requests through the byte-level JsonRpcProcessor.Process in a loop; no scheduler, no transport", - "conditions": "idle box; three 3 s runs of --sync 3 in one session", - "policy": "low and high over three runs; ns per request is the harness's reported figure for one of them", - "x": { "name": "worker threads", "values": [1, 2, 4, 8, 16] }, + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 2 runs; ns per request per thread is the median of the runs' reported figures", + "workers": 32, + "subtitle": ["Byte entry point in a loop, no scheduler, no transport. {host}, .NET 10, Server GC, {runs_word} runs."], + "alt": "JSON-RPC.Net alone, by worker threads: aggregate requests per second as a low-to-high band, and the reported ns per request per thread", + "x": { "name": "worker threads", "values": [1, 2, 4, 8, 16, 32], "caption": "worker threads (each step doubles; {cores} cores)" }, "series": [ { "id": "ours-sync", "label": "JSON-RPC.Net, byte entry point", "family": "ours", + "note": "0 bytes for numeric shapes, one string for `StringMe`", "points": [ - { "low": 4570000, "high": 5000000, "status": "range", "published": "4.6 M to 5.0 M", "ns": 214 }, - { "low": 8500000, "high": 9100000, "status": "range", "published": "8.5 M to 9.1 M", "ns": 235 }, - { "low": 16600000, "high": 17100000, "status": "range", "published": "16.6 M to 17.1 M", "ns": 234 }, - { "low": 22800000, "high": 27700000, "status": "range", "published": "22.8 M to 27.7 M", "ns": 292 }, - { "low": 31700000, "high": 36400000, "status": "range", "published": "31.7 M to 36.4 M", "ns": 439 } + { "low": 3174372, "high": 3175966, "median": 3175169, "values": [3175966, 3174372], "n": 2, "status": "range", "published": "3.17 M to 3.18 M", "cpu": { "system": 3.2, "process": 3.2 }, "ns": 315 }, + { "low": 4310687, "high": 4660256, "median": 4485471.5, "values": [4660256, 4310687], "n": 2, "status": "range", "published": "4.31 M to 4.66 M", "cpu": { "system": 6.3, "process": 6.3 }, "ns": 446 }, + { "low": 10665149, "high": 11951987, "median": 11308568, "values": [11951987, 10665149], "n": 2, "status": "range", "published": "10.7 M to 12 M", "cpu": { "system": 12.4, "process": 12.4 }, "ns": 355 }, + { "low": 18372188, "high": 19841401, "median": 19106794.5, "values": [19841401, 18372188], "n": 2, "status": "range", "published": "18.4 M to 19.8 M", "cpu": { "system": 24.6, "process": 24.6 }, "ns": 419 }, + { "low": 34467034, "high": 35823680, "median": 35145357, "values": [34467034, 35823680], "n": 2, "status": "range", "published": "34.5 M to 35.8 M", "cpu": { "system": 47.9, "process": 47.9 }, "ns": 456 }, + { "low": 44239898, "high": 45671189, "median": 44955543.5, "values": [44239898, 45671189], "n": 2, "status": "range", "published": "44.2 M to 45.7 M", "cpu": { "system": 91.4, "process": 91.4 }, "ns": 712 } ] } - ] + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 2 }, - "async16": { - "title": "ProcessAsync, 16 awaited workers, by registration", - "date": "2026-09-25", - "source": "README.md, Async table, 16-worker column; branch async-scratch-cache", - "harness": "TestServer_Console --async 3 16", + "asyncn": { + "title": "ProcessAsync, {workers} awaited workers, by registration", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console --async 3 32", "workload": "the five benchmark requests (yieldsOnce: one request) through the byte-level JsonRpcProcessor.ProcessAsync from Task.Run workers that await each call", - "conditions": "idle box; one 3 s run per row", - "policy": "one run per row", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 2 runs of each row", + "workers": 32, "groups": [ - { "heading": "methods that complete inline, RpcContextFlow.None", "rule": false, "series": ["async16-sync-none", "async16-task-none", "async16-valuetask-none"] }, - { "heading": "methods that complete inline, RpcContextFlow.Flow", "rule": false, "series": ["async16-task-flow", "async16-valuetask-flow"] }, - { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["async16-yield-none", "async16-yield-flow"] } + { "heading": "methods that complete inline, RpcContextFlow.None", "rule": false, "series": ["asyncn-sync-none", "asyncn-task-none", "asyncn-valuetask-none"] }, + { "heading": "methods that complete inline, RpcContextFlow.Flow", "rule": false, "series": ["asyncn-task-flow", "asyncn-valuetask-flow"] }, + { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["asyncn-yield-none", "asyncn-yield-flow"] } ], "series": [ - { "id": "async16-sync-none", "label": "synchronous methods, None", "note": "6 B per request, the StringMe result string", "family": "ours", "low": 24600000, "high": 24600000, "status": "single", "published": "24.6 M" }, - { "id": "async16-task-none", "label": "Task, None", "note": "67 B per request, the service's own Task.FromResult", "family": "ours", "low": 30000000, "high": 30000000, "status": "single", "published": "30.0 M" }, - { "id": "async16-valuetask-none", "label": "ValueTask, None", "note": "6 B per request", "family": "ours", "low": 32100000, "high": 32100000, "status": "single", "published": "32.1 M" }, - { "id": "async16-task-flow", "label": "Task, Flow", "note": "251 B per request", "family": "ours", "low": 22200000, "high": 22200000, "status": "single", "published": "22.2 M" }, - { "id": "async16-valuetask-flow", "label": "ValueTask, Flow", "note": "190 B per request", "family": "ours", "low": 23400000, "high": 23400000, "status": "single", "published": "23.4 M" }, - { "id": "async16-yield-none", "label": "yieldsOnce, None", "note": "559 B per request", "family": "ours", "low": 8960000, "high": 8960000, "status": "single", "published": "8.96 M" }, - { "id": "async16-yield-flow", "label": "yieldsOnce, Flow", "note": "741 B per request", "family": "ours", "low": 6890000, "high": 6890000, "status": "single", "published": "6.89 M" } - ] + { "id": "asyncn-sync-none", "label": "synchronous methods, None", "note": "{bytes} B per request, the StringMe result string", "family": "ours", "low": 26710958, "high": 28656762, "status": "range", "published": "26.7 M to 28.7 M", "bytes": 6, "median": 27683860, "values": [26710958, 28656762], "n": 2, "cpu": { "system": 93.4, "process": 93.4 } }, + { "id": "asyncn-task-flow", "label": "Task, Flow", "note": "{bytes} B per request", "family": "ours", "low": 27004517, "high": 27572270, "status": "range", "published": "27 M to 27.6 M", "row": "`Task`, Flow", "bytes": 251, "median": 27288393.5, "values": [27004517, 27572270], "n": 2, "cpu": { "system": 79.8, "process": 79.8 } }, + { "id": "asyncn-task-none", "label": "Task, None", "note": "{bytes} B per request, the service's own Task.FromResult", "family": "ours", "low": 34390026, "high": 35453995, "status": "range", "published": "34.4 M to 35.5 M", "row": "`Task`, None", "bytes": 67, "median": 34922010.5, "values": [35453995, 34390026], "n": 2, "cpu": { "system": 93, "process": 93 } }, + { "id": "asyncn-valuetask-flow", "label": "ValueTask, Flow", "note": "{bytes} B per request", "family": "ours", "low": 28613257, "high": 31438045, "status": "range", "published": "28.6 M to 31.4 M", "row": "`ValueTask`, Flow", "bytes": 190, "median": 30025651, "values": [28613257, 31438045], "n": 2, "cpu": { "system": 84.7, "process": 84.7 } }, + { "id": "asyncn-valuetask-none", "label": "ValueTask, None", "note": "{bytes} B per request", "family": "ours", "low": 36255749, "high": 40521407, "status": "range", "published": "36.3 M to 40.5 M", "row": "`ValueTask`, None", "bytes": 6, "median": 38388578, "values": [36255749, 40521407], "n": 2, "cpu": { "system": 96.9, "process": 96.9 } }, + { "id": "asyncn-yield-flow", "label": "yieldsOnce, Flow", "note": "{bytes} B per request", "family": "ours", "low": 2739377, "high": 2775513, "status": "range", "published": "2.74 M to 2.78 M", "row": "`yieldsOnce`, Flow", "bytes": 377, "median": 2757445, "values": [2775513, 2739377], "n": 2, "cpu": { "system": 89.8, "process": 89.8 } }, + { "id": "asyncn-yield-none", "label": "yieldsOnce, None", "note": "{bytes} B per request", "family": "ours", "low": 2548190, "high": 2552560, "status": "range", "published": "2.55 M", "row": "`yieldsOnce`, None", "bytes": 802, "median": 2550375, "values": [2552560, 2548190], "n": 2, "cpu": { "system": 91.5, "process": 91.5 } } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 2 }, "async1": { "title": "ProcessAsync, one awaited worker, by registration", - "date": "2026-09-25", - "source": "README.md, Async table, 1-worker column; branch async-scratch-cache", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", "harness": "TestServer_Console --async 3 1", - "workload": "as async16, one worker", - "conditions": "idle box; one 3 s run per row", - "policy": "one run per row", + "workload": "as asyncn, one worker", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 2 runs of each row", "groups": [ { "heading": "methods that complete inline, RpcContextFlow.None", "rule": false, "series": ["async1-sync-none", "async1-task-none", "async1-valuetask-none"] }, { "heading": "methods that complete inline, RpcContextFlow.Flow", "rule": false, "series": ["async1-task-flow", "async1-valuetask-flow"] }, { "heading": "one real suspension per request (yieldsOnce)", "rule": true, "series": ["async1-yield-none", "async1-yield-flow"] } ], "series": [ - { "id": "async1-sync-none", "label": "synchronous methods, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M" }, - { "id": "async1-task-none", "label": "Task, None", "family": "ours", "low": 3470000, "high": 3470000, "status": "single", "published": "3.47 M" }, - { "id": "async1-valuetask-none", "label": "ValueTask, None", "family": "ours", "low": 3760000, "high": 3760000, "status": "single", "published": "3.76 M" }, - { "id": "async1-task-flow", "label": "Task, Flow", "family": "ours", "low": 2940000, "high": 2940000, "status": "single", "published": "2.94 M" }, - { "id": "async1-valuetask-flow", "label": "ValueTask, Flow", "family": "ours", "low": 3110000, "high": 3110000, "status": "single", "published": "3.11 M" }, - { "id": "async1-yield-none", "label": "yieldsOnce, None", "family": "ours", "low": 1140000, "high": 1140000, "status": "single", "published": "1.14 M" }, - { "id": "async1-yield-flow", "label": "yieldsOnce, Flow", "family": "ours", "low": 915000, "high": 915000, "status": "single", "published": "915 k" } - ] + { "id": "async1-sync-none", "label": "synchronous methods, None", "family": "ours", "low": 2513106, "high": 2520838, "status": "range", "published": "2.51 M to 2.52 M", "bytes": 6, "median": 2516972, "values": [2520838, 2513106], "n": 2, "cpu": { "system": 3.3, "process": 3.3 } }, + { "id": "async1-task-flow", "label": "Task, Flow", "family": "ours", "low": 1978020, "high": 1981182, "status": "range", "published": "1.98 M", "row": "`Task`, Flow", "bytes": 251, "median": 1979601, "values": [1981182, 1978020], "n": 2, "cpu": { "system": 3.3, "process": 3.3 } }, + { "id": "async1-task-none", "label": "Task, None", "family": "ours", "low": 2629355, "high": 2655947, "status": "range", "published": "2.63 M to 2.66 M", "row": "`Task`, None", "bytes": 67, "median": 2642651, "values": [2655947, 2629355], "n": 2, "cpu": { "system": 3.4, "process": 3.4 } }, + { "id": "async1-valuetask-flow", "label": "ValueTask, Flow", "family": "ours", "low": 2047241, "high": 2059920, "status": "range", "published": "2.05 M to 2.06 M", "row": "`ValueTask`, Flow", "bytes": 190, "median": 2053580.5, "values": [2059920, 2047241], "n": 2, "cpu": { "system": 3.2, "process": 3.2 } }, + { "id": "async1-valuetask-none", "label": "ValueTask, None", "family": "ours", "low": 2769948, "high": 2818163, "status": "range", "published": "2.77 M to 2.82 M", "row": "`ValueTask`, None", "bytes": 6, "median": 2794055.5, "values": [2818163, 2769948], "n": 2, "cpu": { "system": 3.3, "process": 3.3 } }, + { "id": "async1-yield-flow", "label": "yieldsOnce, Flow", "family": "ours", "low": 563130, "high": 589434, "status": "range", "published": "563 k to 589 k", "row": "`yieldsOnce`, Flow", "bytes": 740, "median": 576282, "values": [563130, 589434], "n": 2, "cpu": { "system": 18.1, "process": 18.1 } }, + { "id": "async1-yield-none", "label": "yieldsOnce, None", "family": "ours", "low": 543546, "high": 565206, "status": "range", "published": "544 k to 565 k", "row": "`yieldsOnce`, None", "bytes": 548, "median": 554376, "values": [543546, 565206], "n": 2, "cpu": { "system": 21.2, "process": 21.2 } } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "workers": 1, + "cores": 32, + "run_count": 2 }, "legacy": { "title": "Legacy string API through the thread pool, by batch size", - "date": "2026-09-25", - "source": "README.md, Legacy table; branch async-scratch-cache", - "harness": "TestServer_Console, menu entry t", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console, menu entry t with stdout redirected (BenchmarkRunner.Benchmark)", "workload": "the five benchmark requests through the 1.x Task Process(string) overload, submitted in batches from every core through Parallel.For", - "conditions": "idle box; three runs of the t entry in one session", - "policy": "low and high over three runs", - "x": { "name": "batch size", "values": [50, 300, 6000, 36000, 252000, 2016000] }, + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 3 runs", + "x": { "name": "batch size", "values": [50, 100, 300, 1200, 6000, 36000, 252000, 2016000] }, "series": [ { "id": "legacy-string", "label": "JSON-RPC.Net, 1.x string overloads", "family": "ours", "points": [ - { "low": 1570000, "high": 2030000, "status": "range", "published": "1.6 M to 2.0 M" }, - { "low": 7990000, "high": 9190000, "status": "range", "published": "8.0 M to 9.2 M" }, - { "low": 12360000, "high": 13260000, "status": "range", "published": "12.4 M to 13.3 M" }, - { "low": 11570000, "high": 13290000, "status": "range", "published": "11.6 M to 13.3 M" }, - { "low": 7470000, "high": 7800000, "status": "range", "published": "7.5 M to 7.8 M" }, - { "low": 7490000, "high": 8030000, "status": "range", "published": "7.5 M to 8.0 M" } + { "low": 1078909, "high": 1435139, "median": 1086828, "values": [1086828, 1435139, 1078909], "n": 3, "status": "range", "published": "1.08 M to 1.44 M", "cpu": { "system": 76.7, "process": 76.7 } }, + { "low": 1434129, "high": 1512488, "median": 1501647, "values": [1512488, 1434129, 1501647], "n": 3, "status": "range", "published": "1.43 M to 1.51 M", "cpu": { "system": 89.7, "process": 89.7 } }, + { "low": 3724002, "high": 4103938, "median": 3922107, "values": [3922107, 4103938, 3724002], "n": 3, "status": "range", "published": "3.72 M to 4.1 M", "cpu": { "system": 85.9, "process": 85.9 } }, + { "low": 7551462, "high": 7803867, "median": 7615069, "values": [7551462, 7803867, 7615069], "n": 3, "status": "range", "published": "7.55 M to 7.8 M", "cpu": { "system": 86.4, "process": 86.2 } }, + { "low": 9825595, "high": 11144579, "median": 11110511, "values": [11110511, 11144579, 9825595], "n": 3, "status": "range", "published": "9.83 M to 11.1 M", "cpu": { "system": 87.5, "process": 87.5 } }, + { "low": 10284576, "high": 10782349, "median": 10416737, "values": [10782349, 10284576, 10416737], "n": 3, "status": "range", "published": "10.3 M to 10.8 M", "cpu": { "system": 81.9, "process": 81.9 } }, + { "low": 5958403, "high": 6778844, "median": 5963409, "values": [5958403, 5963409, 6778844], "n": 3, "status": "range", "published": "5.96 M to 6.78 M", "cpu": { "system": 71, "process": 70.9 } }, + { "low": 3397534, "high": 6145911, "median": 5607426, "values": [6145911, 3397534, 5607426], "n": 3, "status": "range", "published": "3.4 M to 6.15 M", "cpu": { "system": 76, "process": 76 } } ] } - ] + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 3 }, "kestrel": { "title": "JSON-RPC.Net by transport", - "date": "2026-09-25", - "source": "README.md, Kestrel table; branch async-scratch-cache", - "harness": "TestServer_Console --kestrel 3", - "workload": "AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, 16 clients on the server's 8 cores", - "conditions": "idle box; three 3 s runs of --kestrel 3 and two of --kestrel 3 async, one session", - "policy": "low and high over the runs: three for the default rows, two for the EnableAsyncMethods = true rows", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console --kestrel 3 and --kestrel 3 async (32 clients)", + "workload": "AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, {workers} clients on the server's {cores} cores", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over the runs: 2 runs of --kestrel for the EnableAsyncMethods = false rows, 2 runs of --kestrel async for the EnableAsyncMethods = true rows", + "workers": 32, + "subtitle": ["AustinHarris.JsonRpc.AspNetCore on Kestrel, loopback, {workers} clients on the server's {cores} cores; 3 s runs, intervals span them.", "HTTP clients await one POST at a time; TCP keeps 256 requests in flight per connection."], + "alt": "JSON-RPC.Net by transport: HTTP single, HTTP batch of 100 and TCP pipelined, as low-to-high intervals on a log axis, with the in-process figure for scale", "groups": [ { "heading": "EnableAsyncMethods = false (the default)", "rule": false, "series": ["ours-http-1", "ours-http-100", "ours-tcp"] }, { "heading": "EnableAsyncMethods = true: every document through ProcessAsync", "rule": false, "series": ["ours-tcp-async-inline", "ours-tcp-async-yield"] }, - { "heading": "Reference: no transport", "rule": true, "series": ["ours-inproc-16"] } + { "heading": "Reference: no transport", "rule": true, "series": ["ours-inproc-n"] } ], "series": [ - { "id": "ours-http-1", "label": "HTTP, 1 request per POST", "note": "each client awaits one POST at a time; 88 to 130 µs per round trip", "family": "ours", "low": 123000, "high": 182000, "status": "range", "published": "123 k to 182 k" }, - { "id": "ours-http-100", "label": "HTTP, batch of 100 per POST", "note": "one POST outstanding per client, 100 RPCs in it", "family": "ours", "low": 12800000, "high": 14500000, "status": "range", "published": "12.8 M to 14.5 M" }, - { "id": "ours-tcp", "label": "TCP, 256 pipelined", "note": "JsonRpcConnectionHandler, 256 requests in flight per connection", "family": "ours", "low": 14300000, "high": 16500000, "status": "range", "published": "14.3 M to 16.5 M" }, - { "id": "ours-tcp-async-inline", "label": "TCP, 256 pipelined, methods that complete inline", "note": "--kestrel 3 async", "family": "ours", "low": 15000000, "high": 15400000, "status": "range", "published": "15.0 M to 15.4 M" }, - { "id": "ours-tcp-async-yield", "label": "TCP, 256 pipelined, methods that suspend once", "note": "five async Task methods awaiting Task.Yield(); documents are sequential per connection", "family": "ours", "low": 1250000, "high": 1290000, "status": "range", "published": "1.25 M to 1.29 M" }, - { "id": "ours-inproc-16", "label": "in-process, 16 threads", "note": "the same requests through the byte entry point", "family": "scale", "low": 33800000, "high": 37100000, "status": "range", "published": "33.8 M to 37.1 M" } - ] + { "id": "ours-inproc-n", "label": "in-process, {workers} threads", "note": "the same requests through the byte entry point", "family": "scale", "low": 47335159, "high": 49559560, "status": "range", "published": "47.3 M to 49.6 M", "median": 48447359.5, "values": [49559560, 47335159], "n": 2, "cpu": { "system": 92.8, "process": 92.8 } }, + { "id": "ours-http-1", "label": "HTTP, 1 request per POST", "note": "each client awaits one POST at a time; {us} µs per round trip", "family": "ours", "low": 155808, "high": 162103, "status": "range", "published": "156 k to 162 k", "latency_us": [197, 205], "row_note": "{us} µs per round trip per client depending on the run; HTTP/1.1 request-response is the cost, not the server", "median": 158955.5, "values": [162103, 155808], "n": 2, "cpu": { "system": 90.6, "process": 90.5 } }, + { "id": "ours-http-100", "label": "HTTP, batch of 100 per POST", "note": "one POST outstanding per client, 100 RPCs in it", "family": "ours", "low": 14356502, "high": 14757580, "status": "range", "published": "14.4 M to 14.8 M", "median": 14557041, "values": [14757580, 14356502], "n": 2, "cpu": { "system": 91.6, "process": 91.6 } }, + { "id": "ours-tcp", "label": "TCP, 256 pipelined", "note": "JsonRpcConnectionHandler, 256 requests in flight per connection", "family": "ours", "low": 18454461, "high": 18506274, "status": "range", "published": "18.5 M", "row_note": "ring-buffer clients, one thread each, streaming framer", "median": 18480367.5, "values": [18454461, 18506274], "n": 2, "cpu": { "system": 95.3, "process": 95.3, "clients": 19 } }, + { "id": "ours-tcp-async-inline", "label": "TCP, 256 pipelined, methods that complete inline", "note": "--kestrel 3 async", "family": "ours", "low": 17024435, "high": 17080213, "status": "range", "published": "17 M to 17.1 M", "row": "TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that complete inline", "n": 2, "row_note": "`--kestrel 3 async`, {runs_word} runs", "median": 17052324, "values": [17024435, 17080213], "cpu": { "system": 95.4, "process": 95.4, "clients": 17.9 } }, + { "id": "ours-tcp-async-yield", "label": "TCP, 256 pipelined, methods that suspend once", "note": "five async Task methods awaiting Task.Yield(); documents are sequential per connection", "family": "ours", "low": 2863591, "high": 2867023, "status": "range", "published": "2.86 M to 2.87 M", "row": "TCP, 256 pipelined, `EnableAsyncMethods = true`, methods that suspend once", "row_note": "five `async Task` methods awaiting `Task.Yield()`", "median": 2865307, "values": [2867023, 2863591], "n": 2, "cpu": { "system": 94.8, "process": 94.8, "clients": 12.1 } } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 4 }, "compare": { - "title": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections", - "date": "2026-09-23", - "source": "README.md, comparison table; commit 192a97f", - "harness": "TestServer_Console --compare 3", - "workload": "the same five calls on the same Kestrel; 16 connections or channels with 256 requests in flight each; every request carries \"jsonrpc\":\"2.0\"", - "conditions": "WSL VM shut down; two 3 s runs", - "policy": "observed low and high over two runs; a single figure is one value that both runs rounded to", + "title": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, {workers} connections", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console --compare 3 (32 connections)", + "workload": "the same five calls on the same Kestrel; {workers} connections or channels with 256 requests in flight each; every request carries \"jsonrpc\":\"2.0\"", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 2 runs; a single figure is one value that every run rounded to", + "workers": 32, + "subtitle": ["Same five calls, same Kestrel, {workers} connections or channels with 256 requests in flight each, {runs_word} 3 s runs.", "Every request carries \"jsonrpc\":\"2.0\" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value."], + "alt": "JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET at {workers} connections: low-to-high intervals on a log axis, grouped by library, with the in-process paths under a rule", "versions": "StreamJsonRpc 2.25.29, gRPC for .NET 2.84.0", "groups": [ { "heading": "JSON-RPC.Net", "rule": false, "series": ["ours-tcp-jsonrpc"] }, @@ -155,49 +208,191 @@ { "heading": "gRPC for .NET 2.84: its .NET client shares the server's cores", "rule": false, "series": ["grpc-unary", "grpc-stream"] } ], "series": [ - { "id": "ours-tcp-jsonrpc", "label": "over Kestrel TCP", "note": "raw documents", "family": "ours", "low": 13700000, "high": 14600000, "status": "range", "published": "13.7 M to 14.6 M" }, - { "id": "sjr-newline-stj", "label": "over Kestrel TCP", "note": "newline framing, System.Text.Json formatter", "family": "theirs", "low": 1380000, "high": 1440000, "status": "range", "published": "1.38 M to 1.44 M" }, - { "id": "sjr-cl-stj", "label": "over Kestrel TCP", "note": "Content-Length framing, System.Text.Json formatter", "family": "theirs", "low": 1410000, "high": 1450000, "status": "range", "published": "1.41 M to 1.45 M" }, - { "id": "sjr-cl-newtonsoft", "label": "over Kestrel TCP", "note": "Content-Length framing, Json.NET formatter (default)", "family": "theirs", "low": 625000, "high": 625000, "status": "single", "published": "625 k" }, - { "id": "grpc-unary", "label": "unary calls", "note": "HTTP/2 + protobuf, 16 channels × 256 in flight", "family": "grpc", "low": 192000, "high": 198000, "status": "range", "published": "192 k to 198 k" }, - { "id": "grpc-stream", "label": "bidirectional stream", "note": "one stream per channel, 256 in flight, batched writes", "family": "grpc", "low": 200000, "high": 209000, "status": "range", "published": "200 k to 209 k" } - ] + { "id": "ours-tcp-jsonrpc", "label": "over Kestrel TCP", "note": "raw documents", "family": "ours", "low": 16438116, "high": 16778975, "status": "range", "published": "16.4 M to 16.8 M", "row": "JSON-RPC.Net over Kestrel TCP, raw documents", "median": 16608545.5, "values": [16778975, 16438116], "n": 2, "cpu": { "system": 94.7, "process": 94.6, "clients": 17 } }, + { "id": "sjr-newline-stj", "label": "over Kestrel TCP", "note": "newline framing, System.Text.Json formatter", "family": "theirs", "low": 846928, "high": 878656, "status": "range", "published": "847 k to 879 k", "row": "StreamJsonRpc over Kestrel TCP, newline framing, System.Text.Json formatter", "median": 862792, "values": [846928, 878656], "n": 2, "cpu": { "system": 86.8, "process": 86.8, "clients": 10.1 } }, + { "id": "sjr-cl-stj", "label": "over Kestrel TCP", "note": "Content-Length framing, System.Text.Json formatter", "family": "theirs", "low": 1060037, "high": 1151116, "status": "range", "published": "1.06 M to 1.15 M", "row": "StreamJsonRpc over Kestrel TCP, `Content-Length` framing, System.Text.Json formatter", "median": 1105576.5, "values": [1060037, 1151116], "n": 2, "cpu": { "system": 89.4, "process": 89.4, "clients": 10.1 } }, + { "id": "sjr-cl-newtonsoft", "label": "over Kestrel TCP", "note": "Content-Length framing, Json.NET formatter (default)", "family": "theirs", "low": 560352, "high": 583191, "status": "range", "published": "560 k to 583 k", "row": "StreamJsonRpc over Kestrel TCP, `Content-Length` framing, Json.NET formatter (its default)", "median": 571771.5, "values": [560352, 583191], "n": 2, "cpu": { "system": 82, "process": 82, "clients": 7.8 } }, + { "id": "grpc-unary", "label": "unary calls", "note": "HTTP/2 + protobuf, {workers} channels × 256 in flight", "family": "grpc", "low": 157574, "high": 166452, "status": "range", "published": "158 k to 166 k", "row": "gRPC for .NET, unary calls over HTTP/2 (Grpc.Net.Client, {workers} channels × 256 in flight)", "median": 162013, "values": [157574, 166452], "n": 2, "cpu": { "system": 92.7, "process": 92.7 } }, + { "id": "grpc-stream", "label": "bidirectional stream", "note": "one stream per channel, 256 in flight, batched writes", "family": "grpc", "low": 703425, "high": 732602, "status": "range", "published": "703 k to 733 k", "row": "gRPC for .NET, one bidirectional stream per channel, 256 in flight, batched writes", "median": 718013.5, "values": [703425, 732602], "n": 2, "cpu": { "system": 96.5, "process": 96.5 } } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "cores": 32, + "run_count": 2 }, "inprocess": { "title": "In-process paths: different execution boundaries", - "date": "2026-09-23", - "source": "README.md, comparison table; commit 192a97f", - "harness": "TestServer_Console --compare 3", + "date": "2026-09-28", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", + "harness": "TestServer_Console --compare 3 (32 connections)", "workload": "the same five calls with no network: a direct call, a Pipe pair, a typed proxy", - "conditions": "WSL VM shut down; two 3 s runs", - "policy": "observed low and high over two runs", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 2 runs; a single figure is one value that every run rounded to", + "subtitle": ["No network. A direct call, a Pipe pair and a typed proxy cross different boundaries, so these rows are not a", "like-for-like comparison with each other or with the transport rows. Intervals span {runs_word} 3 s runs."], + "alt": "In-process paths: JSON-RPC.Net direct call, StreamJsonRpc Pipe pair and StreamJsonRpc typed proxy, as low-to-high intervals on a log axis", "groups": [ { "heading": "", "rule": false, "series": ["ours-direct-1", "sjr-pipe", "sjr-proxy"] } ], "series": [ - { "id": "ours-direct-1", "label": "JSON-RPC.Net, direct call, 1 thread", "note": "bytes in, bytes out, no transport", "family": "ours", "low": 2600000, "high": 3600000, "status": "range", "published": "2.6 M to 3.6 M" }, - { "id": "sjr-pipe", "label": "StreamJsonRpc, Pipe pair, 1 client", "note": "newline framing, System.Text.Json, 256 pipelined", "family": "theirs", "low": 117000, "high": 142000, "status": "range", "published": "117 k to 142 k" }, - { "id": "sjr-proxy", "label": "StreamJsonRpc, typed proxy", "note": "sequential await per call, 10 µs per round trip", "family": "theirs", "low": 96000, "high": 97000, "status": "range", "published": "96 k to 97 k" } - ] + { "id": "ours-direct-1", "label": "JSON-RPC.Net, direct call, 1 thread", "note": "bytes in, bytes out, no transport", "family": "ours", "low": 2974336, "high": 3008500, "status": "range", "published": "2.97 M to 3.01 M", "row": "JSON-RPC.Net in-process, 1 thread (direct call, bytes in, bytes out)", "median": 2991418, "values": [3008500, 2974336], "n": 2, "cpu": { "system": 3.1, "process": 3.1 } }, + { "id": "sjr-pipe", "label": "StreamJsonRpc, Pipe pair, 1 client", "note": "newline framing, System.Text.Json, 256 pipelined", "family": "theirs", "low": 133143, "high": 153717, "status": "range", "published": "133 k to 154 k", "row": "StreamJsonRpc in-process, 1 client over a `Pipe` pair, newline framing, System.Text.Json formatter, 256 pipelined", "median": 143430, "values": [153717, 133143], "n": 2, "cpu": { "system": 19.8, "process": 19.8 } }, + { "id": "sjr-proxy", "label": "StreamJsonRpc, typed proxy", "note": "sequential await per call, {us} µs per round trip", "family": "theirs", "low": 53734, "high": 56199, "status": "range", "published": "53.7 k to 56.2 k", "latency_us": [18, 19], "row": "StreamJsonRpc typed proxy, sequential `await` per call, in-process pipes", "rpc_note": "{us} µs per round trip", "median": 54966.5, "values": [56199, 53734], "n": 2, "cpu": { "system": 12.8, "process": 12.8 } } + ], + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "runtime": ".NET 10.0.12, Release, Server GC", + "workers": 32, + "cores": 32, + "run_count": 2 }, "sweep": { "title": "Every library and transport, by client connections", - "source": "TestServer_Console --sweep 2 ; one file per run, matched by the pattern below", + "source": "Hugging Face job 6ab9b3936b030d633f69b0fb, commit 6c3c272", "runs": "sweep*.json", - "workload": "the same five calls on the same Kestrel, clients on the server's 8 cores; each cell warmed for 2 s, then timed", - "conditions": "WSL VM running and other sessions active: a separate session from the tables, so its absolute figures are not comparable with them", - "policy": "observed low, high and median over the runs; each run contributes one observation per cell", - "x": { "name": "client connections (gRPC: channels)" }, + "workload": "the same five calls on the same Kestrel, clients on the server's {cores} cores; each cell warmed for 2 s, then timed", + "conditions": "Hugging Face Jobs cpu-performance, the whole 32-core host (cgroup quota 32 of 32 CPUs), one job on 2026-09-28", + "policy": "low, high and median over 5 runs; each run contributes one observation per cell", + "subtitle": ["Same five calls, same Kestrel, clients on the server's {cores} cores, {date}. TCP and gRPC keep {pipeline} requests in flight", "per connection; HTTP clients await one POST at a time (1 or 100 RPCs). {runs_text}."], + "alt": "Every library and transport by client connections, 1 to {x_max}: three panels on a shared log axis, one per library, with a marker shape and dash per setting and whiskers spanning {runs_word} runs", + "x": { "name": "client connections (gRPC: channels)", "values": [1, 2, 4, 8, 16, 32] }, "series": [ - { "id": "sweep-ours-tcp", "name": "JSON-RPC.Net, TCP", "label": "TCP", "family": "ours", "style": 0, "concurrency": "256 requests in flight per connection" }, - { "id": "sweep-ours-http-100", "name": "JSON-RPC.Net, HTTP, batch of 100 per POST", "label": "HTTP, batch of 100 per POST", "family": "ours", "style": 1, "concurrency": "one POST outstanding per connection, 100 RPCs in it" }, - { "id": "sweep-ours-http-1", "name": "JSON-RPC.Net, HTTP, 1 request per POST", "label": "HTTP, 1 request per POST", "family": "ours", "style": 2, "concurrency": "one POST outstanding per connection" }, - { "id": "sweep-sjr-newline-stj", "name": "StreamJsonRpc, TCP, newline + System.Text.Json", "label": "TCP, newline + System.Text.Json", "family": "theirs", "style": 0, "concurrency": "256 requests in flight per connection" }, - { "id": "sweep-sjr-cl-stj", "name": "StreamJsonRpc, TCP, Content-Length + System.Text.Json", "label": "TCP, Content-Length + System.Text.Json", "family": "theirs", "style": 1, "concurrency": "256 requests in flight per connection" }, - { "id": "sweep-sjr-cl-newtonsoft", "name": "StreamJsonRpc, TCP, Content-Length + Json.NET", "label": "TCP, Content-Length + Json.NET", "family": "theirs", "style": 2, "concurrency": "256 requests in flight per connection" }, - { "id": "sweep-grpc-unary", "name": "gRPC for .NET, unary", "label": "unary", "family": "grpc", "style": 0, "concurrency": "256 calls in flight per channel" }, - { "id": "sweep-grpc-stream", "name": "gRPC for .NET, bidirectional stream", "label": "bidirectional stream", "family": "grpc", "style": 1, "concurrency": "one stream per channel, 256 in flight, batched writes" } - ] + { + "id": "sweep-ours-tcp", + "name": "JSON-RPC.Net, TCP", + "label": "TCP", + "family": "ours", + "style": 0, + "concurrency": "256 requests in flight per connection", + "points": [ + { "low": 1504315, "high": 1663416, "median": 1582632, "n": 5, "values": [1504315, 1581792, 1582632, 1663416, 1597854], "status": "range", "cpu": { "system": 10.8, "process": 10.8, "clients": 1.2 }, "published": "1.5 M to 1.66 M" }, + { "low": 2264852, "high": 2906285, "median": 2864827, "n": 5, "values": [2807580, 2891600, 2864827, 2906285, 2264852], "status": "range", "cpu": { "system": 18.1, "process": 18.1, "clients": 2.4 }, "published": "2.26 M to 2.91 M" }, + { "low": 4709242, "high": 5123054, "median": 4982397, "n": 5, "values": [5123054, 4728503, 4709242, 5025529, 4982397], "status": "range", "cpu": { "system": 31.4, "process": 31.4, "clients": 4.9 }, "published": "4.71 M to 5.12 M" }, + { "low": 8355818, "high": 8541684, "median": 8408546, "n": 5, "values": [8436939, 8408546, 8355818, 8382932, 8541684], "status": "range", "cpu": { "system": 53.4, "process": 53.4, "clients": 7.9 }, "published": "8.36 M to 8.54 M" }, + { "low": 13672167, "high": 13923831, "median": 13838572, "n": 5, "values": [13838572, 13792074, 13672167, 13923831, 13909240], "status": "range", "cpu": { "system": 89.8, "process": 89.8, "clients": 15.1 }, "published": "13.7 M to 13.9 M" }, + { "low": 17341543, "high": 17762271, "median": 17518939, "n": 5, "values": [17455330, 17518939, 17341543, 17612181, 17762271], "status": "range", "cpu": { "system": 96.7, "process": 96.7, "clients": 17.7 }, "published": "17.3 M to 17.8 M" } + ] + }, + { + "id": "sweep-ours-http-100", + "name": "JSON-RPC.Net, HTTP, batch of 100 per POST", + "label": "HTTP, batch of 100 per POST", + "family": "ours", + "style": 1, + "concurrency": "one POST outstanding per connection, 100 RPCs in it", + "points": [ + { "low": 711106, "high": 780521, "median": 743574, "n": 5, "values": [747452, 743574, 780521, 730180, 711106], "status": "range", "cpu": { "system": 10.6, "process": 10.6 }, "published": "711 k to 781 k" }, + { "low": 1536182, "high": 1575796, "median": 1544718, "n": 5, "values": [1575796, 1544556, 1546103, 1544718, 1536182], "status": "range", "cpu": { "system": 16.7, "process": 16.7 }, "published": "1.54 M to 1.58 M" }, + { "low": 2998846, "high": 3049007, "median": 3018942, "n": 5, "values": [3018942, 3011600, 3032005, 3049007, 2998846], "status": "range", "cpu": { "system": 28.6, "process": 28.6 }, "published": "3 M to 3.05 M" }, + { "low": 5444182, "high": 5528558, "median": 5486596, "n": 5, "values": [5528558, 5500819, 5480732, 5486596, 5444182], "status": "range", "cpu": { "system": 48.2, "process": 48.2 }, "published": "5.44 M to 5.53 M" }, + { "low": 9282494, "high": 9423705, "median": 9347086, "n": 5, "values": [9282494, 9298140, 9347086, 9423705, 9353620], "status": "range", "cpu": { "system": 80.4, "process": 80.4 }, "published": "9.28 M to 9.42 M" }, + { "low": 14356886, "high": 14604824, "median": 14568739, "n": 5, "values": [14568739, 14559938, 14356886, 14590783, 14604824], "status": "range", "cpu": { "system": 93.3, "process": 93.3 }, "published": "14.4 M to 14.6 M" } + ] + }, + { + "id": "sweep-ours-http-1", + "name": "JSON-RPC.Net, HTTP, 1 request per POST", + "label": "HTTP, 1 request per POST", + "family": "ours", + "style": 2, + "concurrency": "one POST outstanding per connection", + "points": [ + { "low": 14258, "high": 15268, "median": 14732, "n": 5, "values": [15268, 14732, 14971, 14549, 14258], "status": "range", "cpu": { "system": 9.8, "process": 9.8 }, "published": "14.3 k to 15.3 k" }, + { "low": 28591, "high": 29774, "median": 28672, "n": 5, "values": [28672, 28591, 29265, 29774, 28622], "status": "range", "cpu": { "system": 15.4, "process": 15.4 }, "published": "28.6 k to 29.8 k" }, + { "low": 54512, "high": 55763, "median": 54983, "n": 5, "values": [54983, 54848, 54512, 55003, 55763], "status": "range", "cpu": { "system": 26.1, "process": 26.1 }, "published": "54.5 k to 55.8 k" }, + { "low": 97476, "high": 98859, "median": 98738, "n": 5, "values": [98843, 98859, 98738, 98651, 97476], "status": "range", "cpu": { "system": 44.3, "process": 44.3 }, "published": "97.5 k to 98.9 k" }, + { "low": 164467, "high": 168059, "median": 166437, "n": 5, "values": [168059, 166437, 164889, 166932, 164467], "status": "range", "cpu": { "system": 75.1, "process": 75.1 }, "published": "164 k to 168 k" }, + { "low": 287938, "high": 290499, "median": 289276, "n": 5, "values": [287938, 288711, 289588, 290499, 289276], "status": "range", "cpu": { "system": 94.8, "process": 94.8 }, "published": "288 k to 290 k" } + ] + }, + { + "id": "sweep-sjr-newline-stj", + "name": "StreamJsonRpc, TCP, newline + System.Text.Json", + "label": "TCP, newline + System.Text.Json", + "family": "theirs", + "style": 0, + "concurrency": "256 requests in flight per connection", + "points": [ + { "low": 169122, "high": 195042, "median": 181536, "n": 5, "values": [195042, 169122, 181536, 192370, 180901], "status": "range", "cpu": { "system": 20.3, "process": 20.3, "clients": 2.5 }, "published": "169 k to 195 k" }, + { "low": 290819, "high": 310815, "median": 296378, "n": 5, "values": [302641, 310815, 296378, 290819, 292461], "status": "range", "cpu": { "system": 36.7, "process": 36.6, "clients": 5.1 }, "published": "291 k to 311 k" }, + { "low": 462815, "high": 481397, "median": 476216, "n": 5, "values": [481397, 476216, 478417, 466228, 462815], "status": "range", "cpu": { "system": 66.7, "process": 66.7, "clients": 10.3 }, "published": "463 k to 481 k" }, + { "low": 761947, "high": 791998, "median": 768721, "n": 5, "values": [765565, 761947, 791998, 768721, 776653], "status": "range", "cpu": { "system": 85.2, "process": 85.2, "clients": 11.9 }, "published": "762 k to 792 k" }, + { "low": 858189, "high": 908097, "median": 881160, "n": 5, "values": [908097, 858189, 902865, 866567, 881160], "status": "range", "cpu": { "system": 87.1, "process": 87.1, "clients": 10.9 }, "published": "858 k to 908 k" }, + { "low": 924264, "high": 1008400, "median": 971720, "n": 5, "values": [991712, 954696, 1008400, 924264, 971720], "status": "range", "cpu": { "system": 89.5, "process": 89.5, "clients": 10.1 }, "published": "924 k to 1.01 M" } + ] + }, + { + "id": "sweep-sjr-cl-stj", + "name": "StreamJsonRpc, TCP, Content-Length + System.Text.Json", + "label": "TCP, Content-Length + System.Text.Json", + "family": "theirs", + "style": 1, + "concurrency": "256 requests in flight per connection", + "points": [ + { "low": 196223, "high": 210829, "median": 197636, "n": 5, "values": [197627, 196223, 210829, 197636, 203954], "status": "range", "cpu": { "system": 20.5, "process": 20.5, "clients": 2.5 }, "published": "196 k to 211 k" }, + { "low": 337259, "high": 352609, "median": 341014, "n": 5, "values": [341343, 339821, 352609, 341014, 337259], "status": "range", "cpu": { "system": 37, "process": 37, "clients": 4.9 }, "published": "337 k to 353 k" }, + { "low": 552090, "high": 563704, "median": 557259, "n": 5, "values": [552293, 562128, 557259, 563704, 552090], "status": "range", "cpu": { "system": 67.8, "process": 67.8, "clients": 10 }, "published": "552 k to 564 k" }, + { "low": 943109, "high": 968434, "median": 953507, "n": 5, "values": [953507, 968434, 951676, 943109, 959551], "status": "range", "cpu": { "system": 90.4, "process": 90.4, "clients": 11.5 }, "published": "943 k to 968 k" }, + { "low": 1117138, "high": 1265533, "median": 1255931, "n": 5, "values": [1265533, 1260862, 1255931, 1117138, 1242741], "status": "range", "cpu": { "system": 92.9, "process": 92.9, "clients": 10.7 }, "published": "1.12 M to 1.27 M" }, + { "low": 1115974, "high": 1250026, "median": 1233730, "n": 5, "values": [1233730, 1222980, 1249031, 1115974, 1250026], "status": "range", "cpu": { "system": 91.6, "process": 91.6, "clients": 10.1 }, "published": "1.12 M to 1.25 M" } + ] + }, + { + "id": "sweep-sjr-cl-newtonsoft", + "name": "StreamJsonRpc, TCP, Content-Length + Json.NET", + "label": "TCP, Content-Length + Json.NET", + "family": "theirs", + "style": 2, + "concurrency": "256 requests in flight per connection", + "points": [ + { "low": 122182, "high": 139087, "median": 133171, "n": 5, "values": [131421, 136732, 122182, 133171, 139087], "status": "range", "cpu": { "system": 19.2, "process": 19.2, "clients": 2.3 }, "published": "122 k to 139 k" }, + { "low": 204910, "high": 220368, "median": 213573, "n": 5, "values": [210770, 213573, 204910, 217147, 220368], "status": "range", "cpu": { "system": 35, "process": 35, "clients": 4.7 }, "published": "205 k to 220 k" }, + { "low": 356420, "high": 369044, "median": 363840, "n": 5, "values": [364211, 369044, 356420, 363840, 358624], "status": "range", "cpu": { "system": 64.3, "process": 64.3, "clients": 9.6 }, "published": "356 k to 369 k" }, + { "low": 616843, "high": 642518, "median": 634055, "n": 5, "values": [642518, 616843, 632229, 634055, 636066], "status": "range", "cpu": { "system": 86.4, "process": 86.4, "clients": 11.8 }, "published": "617 k to 643 k" }, + { "low": 660979, "high": 733715, "median": 686417, "n": 5, "values": [665977, 725538, 686417, 660979, 733715], "status": "range", "cpu": { "system": 87.2, "process": 87.2, "clients": 10.1 }, "published": "661 k to 734 k" }, + { "low": 618631, "high": 740627, "median": 668408, "n": 5, "values": [655445, 740627, 668408, 618631, 717800], "status": "range", "cpu": { "system": 84.8, "process": 84.8, "clients": 9.1 }, "published": "619 k to 741 k" } + ] + }, + { + "id": "sweep-grpc-unary", + "name": "gRPC for .NET, unary", + "label": "unary", + "family": "grpc", + "style": 0, + "concurrency": "256 calls in flight per channel", + "points": [ + { "low": 136259, "high": 142853, "median": 137954, "n": 5, "values": [137954, 137109, 138886, 142853, 136259], "status": "range", "cpu": { "system": 82.6, "process": 82.5 }, "published": "136 k to 143 k" }, + { "low": 252190, "high": 255854, "median": 254309, "n": 5, "values": [254309, 255854, 252190, 253783, 255434], "status": "range", "cpu": { "system": 94.1, "process": 94.1 }, "published": "252 k to 256 k" }, + { "low": 328837, "high": 339254, "median": 336472, "n": 5, "values": [328837, 339254, 336472, 338634, 335192], "status": "range", "cpu": { "system": 93.3, "process": 93.3 }, "published": "329 k to 339 k" }, + { "low": 337860, "high": 356705, "median": 350513, "n": 5, "values": [350513, 356528, 356705, 346085, 337860], "status": "range", "cpu": { "system": 92.5, "process": 92.5 }, "published": "338 k to 357 k" }, + { "low": 308571, "high": 323245, "median": 314681, "n": 5, "values": [313991, 308571, 314681, 322720, 323245], "status": "range", "cpu": { "system": 87.9, "process": 87.9 }, "published": "309 k to 323 k" }, + { "low": 269442, "high": 289596, "median": 274986, "n": 5, "values": [289596, 269442, 275939, 274986, 272509], "status": "range", "cpu": { "system": 88.4, "process": 88.4 }, "published": "269 k to 290 k" } + ] + }, + { + "id": "sweep-grpc-stream", + "name": "gRPC for .NET, bidirectional stream", + "label": "bidirectional stream", + "family": "grpc", + "style": 1, + "concurrency": "one stream per channel, 256 in flight, batched writes", + "points": [ + { "low": 35607, "high": 36814, "median": 36543, "n": 5, "values": [36814, 36543, 35607, 36219, 36574], "status": "range", "cpu": { "system": 24.7, "process": 24.7 }, "published": "35.6 k to 36.8 k" }, + { "low": 69178, "high": 70267, "median": 69901, "n": 5, "values": [70148, 69822, 69178, 69901, 70267], "status": "range", "cpu": { "system": 41.2, "process": 41.2 }, "published": "69.2 k to 70.3 k" }, + { "low": 125211, "high": 125542, "median": 125414, "n": 5, "values": [125211, 125542, 125256, 125414, 125456], "status": "range", "cpu": { "system": 68.2, "process": 68.2 }, "published": "125 k to 126 k" }, + { "low": 223925, "high": 228992, "median": 228506, "n": 5, "values": [228992, 228889, 228506, 223925, 227039], "status": "range", "cpu": { "system": 89.8, "process": 89.8 }, "published": "224 k to 229 k" }, + { "low": 423644, "high": 437574, "median": 434284, "n": 5, "values": [437574, 434284, 436949, 432957, 423644], "status": "range", "cpu": { "system": 96.5, "process": 96.5 }, "published": "424 k to 438 k" }, + { "low": 893246, "high": 913031, "median": 901600, "n": 5, "values": [893246, 913031, 893936, 901600, 906325], "status": "range", "cpu": { "system": 97.6, "process": 97.6 }, "published": "893 k to 913 k" } + ] + } + ], + "run_files": ["sweep.json", "sweep-2.json", "sweep-3.json", "sweep-4.json", "sweep-5.json"], + "seconds_per_cell": 2, + "pipeline": 256, + "date": "2026-09-28", + "machine": "AMD EPYC 7R13 Processor, 32 cores of 32 host CPUs, cgroup quota, Ubuntu 24.04.5 LTS, .NET 10.0.12, Release, Server GC", + "cores": 32, + "runtime": ".NET 10.0.12, Release, Server GC", + "run_count": 5, + "harness": "TestServer_Console --sweep 2; one file per run, matched by the pattern in runs" }, "wasm": { "title": "In the browser: interpreter vs AOT", @@ -205,8 +400,9 @@ "source": "samples/WasmHost/README.md table; commit 41371ec", "harness": "samples/WasmHost, Run benchmark in Chrome 152", "workload": "add(1, 2) through each interop path, 20,000 RPCs per row", - "conditions": "8-core desktop; interpreter from dotnet run, AOT from dotnet publish -c Release with the wasm-tools workload", + "conditions": "a browser measurement on an 8-core desktop, not the benchmark job's host; interpreter from dotnet run, AOT from dotnet publish -c Release with the wasm-tools workload", "policy": "interpreter: one run; AOT: the better of two runs. The interpreter-to-AOT distance is a deployment difference, not a range", + "subtitle": ["samples/WasmHost in Chrome 152, .NET 10, 20,000 RPCs per row. Interpreter: one run. AOT: the better of two runs.", "Batches are per RPC. The interpreter-to-AOT distance is a deployment difference, not a range."], "modes": [ { "id": "interp", "label": "interpreter", "marker": "circle", "policy": "one run" }, { "id": "aot", "label": "AOT", "marker": "diamond", "policy": "better of two runs" } @@ -231,5 +427,6 @@ { "id": "wasm-loop", "label": "ProcessMany, the string request N times", "family": "scale", "values": { "interp": 17600, "aot": 169800 }, "published": { "interp": "17,600", "aot": "169,800" } } ] } - } + }, + "host": "AMD EPYC 7R13, 32 cores" } diff --git a/benchmarks/charts/compare-connections-dark.svg b/benchmarks/charts/compare-connections-dark.svg index 706d09f..79b9087 100644 --- a/benchmarks/charts/compare-connections-dark.svg +++ b/benchmarks/charts/compare-connections-dark.svg @@ -1,290 +1,331 @@ - -Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision eacac12e8d55. + +Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 32 cores, 2026-09-28. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision 697798ae199d. Every library and transport, by client connections -Same five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight +Same five calls, same Kestrel, clients on the server's 32 cores, 2026-09-28. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. JSON-RPC.Net 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + TCP -at 16: 10.9 M to 13 M - - - - - - - - - - - - - - - - - - - - - +at 32: 17.3 M to 17.8 M + + + + + + + + + + + + + + + + + + + + + + + + + HTTP, batch of 100 per POST -at 16: 8.64 M to 10.2 M - - - - - - - - - - - - - - - - - - - - - +at 32: 14.4 M to 14.6 M + + + + + + + + + + + + + + + + + + + + + + + + + HTTP, 1 request per POST -at 16: 178 k to 210 k +at 32: 288 k to 290 k StreamJsonRpc 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + TCP, newline + System.Text.Json -at 16: 1.06 M to 1.24 M - - - - - - - - - - - - - - - - - - - - - +at 32: 924 k to 1.01 M + + + + + + + + + + + + + + + + + + + + + + + + + TCP, Content-Length + System.Text.Json -at 16: 1.12 M to 1.37 M - - - - - - - - - - - - - - - - - - - - - +at 32: 1.12 M to 1.25 M + + + + + + + + + + + + + + + + + + + + + + + + + TCP, Content-Length + Json.NET -at 16: 544 k to 677 k +at 32: 619 k to 741 k gRPC for .NET 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + unary -at 16: 360 k to 404 k - - - - - - - - - - - - - - - - - - - - - +at 32: 269 k to 290 k + + + + + + + + + + + + + + + + + + + + + + + + + bidirectional stream -at 16: 233 k to 291 k +at 32: 893 k to 913 k client connections (gRPC: channels); each step doubles requests per second, log scale, shared by the panels; marker at the median, whisker low to high diff --git a/benchmarks/charts/compare-connections.svg b/benchmarks/charts/compare-connections.svg index 661419a..a1dc321 100644 --- a/benchmarks/charts/compare-connections.svg +++ b/benchmarks/charts/compare-connections.svg @@ -1,290 +1,331 @@ - -Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision eacac12e8d55. + +Every library and transport, by client connectionsSame five calls, same Kestrel, clients on the server's 32 cores, 2026-09-28. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. Data revision 697798ae199d. Every library and transport, by client connections -Same five calls, same Kestrel, clients on the server's 8 cores, 2026-09-23. TCP and gRPC keep 256 requests in flight +Same five calls, same Kestrel, clients on the server's 32 cores, 2026-09-28. TCP and gRPC keep 256 requests in flight per connection; HTTP clients await one POST at a time (1 or 100 RPCs). 5 runs of 2 s per point; whiskers span the runs. JSON-RPC.Net 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + TCP -at 16: 10.9 M to 13 M - - - - - - - - - - - - - - - - - - - - - +at 32: 17.3 M to 17.8 M + + + + + + + + + + + + + + + + + + + + + + + + + HTTP, batch of 100 per POST -at 16: 8.64 M to 10.2 M - - - - - - - - - - - - - - - - - - - - - +at 32: 14.4 M to 14.6 M + + + + + + + + + + + + + + + + + + + + + + + + + HTTP, 1 request per POST -at 16: 178 k to 210 k +at 32: 288 k to 290 k StreamJsonRpc 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + TCP, newline + System.Text.Json -at 16: 1.06 M to 1.24 M - - - - - - - - - - - - - - - - - - - - - +at 32: 924 k to 1.01 M + + + + + + + + + + + + + + + + + + + + + + + + + TCP, Content-Length + System.Text.Json -at 16: 1.12 M to 1.37 M - - - - - - - - - - - - - - - - - - - - - +at 32: 1.12 M to 1.25 M + + + + + + + + + + + + + + + + + + + + + + + + + TCP, Content-Length + Json.NET -at 16: 544 k to 677 k +at 32: 619 k to 741 k gRPC for .NET 10 k - - - -100 k - - - -1 M - - - -10 M + + + +100 k + + + +1 M + + + +10 M + - - - + + + + 1 -2 -4 -8 -16 - - - - - - - - - - - - - - - - - - - - - +2 +4 +8 +16 +32 + + + + + + + + + + + + + + + + + + + + + + + + + unary -at 16: 360 k to 404 k - - - - - - - - - - - - - - - - - - - - - +at 32: 269 k to 290 k + + + + + + + + + + + + + + + + + + + + + + + + + bidirectional stream -at 16: 233 k to 291 k +at 32: 893 k to 913 k client connections (gRPC: channels); each step doubles requests per second, log scale, shared by the panels; marker at the median, whisker low to high diff --git a/benchmarks/charts/compare-streamjsonrpc-dark.svg b/benchmarks/charts/compare-streamjsonrpc-dark.svg index 1ccb1fd..3a06ca5 100644 --- a/benchmarks/charts/compare-streamjsonrpc-dark.svg +++ b/benchmarks/charts/compare-streamjsonrpc-dark.svg @@ -1,8 +1,8 @@ - -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision eacac12e8d55. + +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 32 connectionsSame five calls, same Kestrel, 32 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision 697798ae199d. -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections -Same five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 32 connections +Same five calls, same Kestrel, 32 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. 100 k @@ -24,39 +24,40 @@ JSON-RPC.Net over Kestrel TCP raw documents - - - -13.7 M to 14.6 M + + + +16.4 M to 16.8 M StreamJsonRpc 2.25 over Kestrel TCP newline framing, System.Text.Json formatter - - - -1.38 M to 1.44 M + + + +847 k to 879 k over Kestrel TCP Content-Length framing, System.Text.Json formatter - - - -1.41 M to 1.45 M + + + +1.06 M to 1.15 M over Kestrel TCP Content-Length framing, Json.NET formatter (default) - - -625 k + + + +560 k to 583 k gRPC for .NET 2.84: its .NET client shares the server's cores unary calls -HTTP/2 + protobuf, 16 channels × 256 in flight - - - -192 k to 198 k +HTTP/2 + protobuf, 32 channels × 256 in flight + + + +158 k to 166 k bidirectional stream one stream per channel, 256 in flight, batched writes - - - -200 k to 209 k + + + +703 k to 733 k diff --git a/benchmarks/charts/compare-streamjsonrpc.svg b/benchmarks/charts/compare-streamjsonrpc.svg index c1cc29b..42f7f40 100644 --- a/benchmarks/charts/compare-streamjsonrpc.svg +++ b/benchmarks/charts/compare-streamjsonrpc.svg @@ -1,8 +1,8 @@ - -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connectionsSame five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision eacac12e8d55. + +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 32 connectionsSame five calls, same Kestrel, 32 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. Data revision 697798ae199d. -JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 16 connections -Same five calls, same Kestrel, 16 connections or channels with 256 requests in flight each, two 3 s runs. +JSON-RPC.Net vs StreamJsonRpc vs gRPC for .NET, 32 connections +Same five calls, same Kestrel, 32 connections or channels with 256 requests in flight each, two 3 s runs. Every request carries "jsonrpc":"2.0" (StreamJsonRpc requires it). Intervals span the runs; a dot is a single value. 100 k @@ -24,39 +24,40 @@ JSON-RPC.Net over Kestrel TCP raw documents - - - -13.7 M to 14.6 M + + + +16.4 M to 16.8 M StreamJsonRpc 2.25 over Kestrel TCP newline framing, System.Text.Json formatter - - - -1.38 M to 1.44 M + + + +847 k to 879 k over Kestrel TCP Content-Length framing, System.Text.Json formatter - - - -1.41 M to 1.45 M + + + +1.06 M to 1.15 M over Kestrel TCP Content-Length framing, Json.NET formatter (default) - - -625 k + + + +560 k to 583 k gRPC for .NET 2.84: its .NET client shares the server's cores unary calls -HTTP/2 + protobuf, 16 channels × 256 in flight - - - -192 k to 198 k +HTTP/2 + protobuf, 32 channels × 256 in flight + + + +158 k to 166 k bidirectional stream one stream per channel, 256 in flight, batched writes - - - -200 k to 209 k + + + +703 k to 733 k diff --git a/benchmarks/charts/explorer.html b/benchmarks/charts/explorer.html index 20af779..714cbbd 100644 --- a/benchmarks/charts/explorer.html +++ b/benchmarks/charts/explorer.html @@ -5,7 +5,7 @@ JSON-RPC.NET benchmark explorer - +