Skip to content

Add telemetry-function experiment results (inline cost, exact baseline, accuracy vs size) - #33

Open
zzylol wants to merge 1 commit into
feat/ci-tiersfrom
feat/experiment-results
Open

zzylol wants to merge 1 commit into
feat/ci-tiersfrom
feat/experiment-results

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Being rerun. The stack below now runs the sketch, exact and raw pipelines side by side (#25), and experiments.py has pipelines instead of inline-cost. The results below are from the earlier setup; this PR will be updated with the pipelines and accuracy-size reruns.

Part 10 of the telemetry-functions stack: the measured results of experiments 1 (inline cost) and 2 (accuracy versus size), generated by report_experiments.py from experiments.py output (#30). Every stage is a pinned asap-collector container on 56-core x86 servers, with 10-second windows. Full tables and CSVs are in docs/results/experiments.md.

Inline cost: branch CPU with and without each function

Branch CPU with and without each function

Function Added branch CPU per point Branch CPU at 200,000 points/s per source
none (forwarding only) (about 0.3 µs in total) 0.06 cores
KLL latency quantiles 2.7–4.1 µs 0.60 cores
Count-Min + heap, top-k 3.7–5.3 µs 0.81 cores
HLL over 4 attribute keys 9–11 µs (about 2.3 µs per key) one core reached at about 100,000 points/s
  • The global stages (merge, detector) receive a few KB to 180 KB per window and use under 0.01 cores.
  • The exact baseline, which receives every raw point, needs a full core at about 95,000 points/s for cardinality and about 400,000 points/s for latency quantiles.
  • Answers arrive 2–3 s after a window ends (these runs used a 2-second merge grace).
  • These per-point costs put a branch below the ~1M points/s per core reference. The next step is a profile of the per-point path; decoding OTAP into observations and the per-attribute split are the likely costs, not yet measured.

Accuracy versus size (one window per setting)

Setting Error Merge bytes per window
KLL k = 50 / 100 / 200 / 400 p99 median error 13.3% / 7.0% / 7.5% / 2.0% (routes ≥ 20,000 points) 39 / 58 / 96 / 181 KB
Count-Min 256 / 512 / 1,024 / 2,048 columns, 100,000 keys recall@10 1.0 at every width; top-10 count error 6.0% / 2.8% / 1.3% / 0.46% 7 / 11 / 12 / 19 KB

Single windows, so neighbouring settings can cross (KLL k = 100 vs 200).

@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from 0965b48 to 17186b1 Compare October 12, 2026 22:09
@zzylol
zzylol force-pushed the feat/experiment-results branch from 67930ea to 4cf35de Compare October 12, 2026 22:09
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from 17186b1 to f263a6c Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the feat/experiment-results branch from 4cf35de to be08a0a Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from f263a6c to f19756b Compare October 19, 2026 02:31
@zzylol
zzylol force-pushed the feat/experiment-results branch from be08a0a to 19c8143 Compare October 19, 2026 02:31
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from f19756b to 330dfc9 Compare October 19, 2026 14:57
@zzylol
zzylol force-pushed the feat/experiment-results branch from 19c8143 to 016efad Compare October 19, 2026 14:57
Inline cost (branch CPU with and without each function, 20k-200k points/s
per source), the exact baseline's own cost, and accuracy versus sketch size
(KLL k, Count-Min width over 100,000 keys), measured on 56-core x86 servers
with every stage a pinned asap-collector container. Generated by
report_experiments.py from experiments.py output.
@zzylol
zzylol changed the base branch from docs/governance-and-rfc to feat/ci-tiers October 19, 2026 17:32
@zzylol
zzylol force-pushed the feat/experiment-results branch from 016efad to 82a6d7e Compare October 19, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant