Skip to content

Report heavy-hitter precision, recall, F1 and ARE in the top-k demo - #35

Open
zzylol wants to merge 1 commit into
perf/observe-without-allocationfrom
feat/hh-accuracy
Open

zzylol wants to merge 1 commit into
perf/observe-without-allocationfrom
feat/hh-accuracy

Conversation

@zzylol

@zzylol zzylol commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Part 12 of the telemetry-functions stack: identification and counting accuracy of the top-k use case's heavy hitters, live in the demo.

Why

The top-k dashboard showed the sketch's and the exact top-10 side by side, and acceptance checked recall@10. Neither said how well the sketch identifies heavy hitters, or how far off its counts are. A top-10 comparison cannot separate precision from recall either: both lists always have 10 entries.

What

  • hh-accuracy (urn:asap:processor:heavy_hitter_accuracy, common/hh-accuracy.yaml): a new stage outside the three pipelines; they still share no processor. In the top-k use case both detectors send it a copy of their answers through the official fanout. Per window and top-k series:
    • Heavy hitters (truth): keys with at least φ = 1% of the window's points, by the exact counts.
    • Reported: keys whose sketch estimate reaches the same 1%.
    • It emits asap_hh_precision (TP / reported), asap_hh_recall (TP / truth), asap_hh_f1, asap_hh_are (mean |estimate − exact| / exact over the true positives; the sketch has no count for a heavy hitter it missed), asap_hh_truth and asap_hh_reported.
  • Top 20 instead of top 10. About 10 routes reach 1% in this scenario (rank 10 has about 1.07% of the points, rank 11 about 0.97%), so a top-10 list could not show whether more heavy hitters exist. The sketch heap and both detectors now report the top 20; novelty therefore also watches the top 20. The top-10 panels and checks filter on rank. A window whose exact top 20 is entirely above the threshold could still hide heavy hitters; it gets only asap_hh_incomplete.
  • Dashboard: a precision/recall/F1 panel and an ARE panel (with the truth and reported counts on the right axis); the pipeline panel describes hh-accuracy.
  • Acceptance adds "heavy-hitter precision and recall ≥ 0.9"; experiments.py accuracy-size records the four metrics per Count-Min width. CI and compose add the 19th stage (CPU 18).

Result (56-core x86 server, 1-minute windows, 2 × 20,000 points/s, Count-Min 4 × 512)

First window on the live demo: 10 heavy hitters by exact, 11 reported by the sketch. Precision 0.91, recall 1.00, F1 0.95, ARE 1.95%. The false positive is the rank-11 route at about 0.97% of the points. Count-Min only overestimates, and its collision error (about 1,000–1,300 points here, 0.05%) lifts it over 1%. That is the precision failure the metric is meant to expose.

Verification

  • Unit tests: heavy_hitter_accuracy on a hand-checked window (a false positive pushed over the threshold, a missed heavy hitter, ARE over the true positives, the truncated-list case); the node only reports a window after both methods answered, ignores lower/upper bounds, and labels the gauges with the series and φ.
  • Live on the demo stack, as above.
  • cargo fmt --check, cargo clippy --all-targets --features otap-engine (no warnings) and cargo test --features otap-engine pass.
  • Not yet run: the full acceptance with the new check. It needs the same server as the running demo, and will run together with the Add telemetry-function experiment results (inline cost, exact baseline, accuracy vs size) #33 experiment rerun.

@zzylol
zzylol force-pushed the feat/hh-accuracy branch 2 times, most recently from f0b5fe4 to bffc387 Compare October 19, 2026 20:15
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from ab6f04b to 74a8594 Compare October 19, 2026 20:15
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 74a8594 to f3d2ad5 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from f3d2ad5 to 971825f Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 971825f to 11092ab Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 11092ab to 21f5802 Compare October 19, 2026 20:17
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 21f5802 to 05ea8f8 Compare October 19, 2026 20:17
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch 2 times, most recently from bfbdb55 to 541925f Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 541925f to 7da0bc7 Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from 7da0bc7 to c03faab Compare October 19, 2026 20:19
A new observer stage, hh-accuracy (urn:asap:processor:heavy_hitter_accuracy),
sits outside the three pipelines. In the top-k use case both detectors send
it a copy of their answers through the official fanout. Per window and top-k
series it takes the heavy hitters as the keys with at least phi (1%) of the
window's points by the exact counts, and the sketch's reported keys as those
whose estimate reaches the same threshold. It then emits asap.hh.precision,
asap.hh.recall, asap.hh.f1, asap.hh.are (mean relative count error over the
true positives), asap.hh.truth and asap.hh.reported.

About 10 routes reach 1% in this scenario, so the sketch heap and both
detectors now report the top 20, and the top-10 panels and checks filter on
rank. A window whose exact top 20 is all above the threshold could hide
heavy hitters and only gets asap.hh.incomplete. The dashboard gains
precision/recall/F1 and ARE panels; acceptance checks precision and recall
>= 0.9, and the accuracy-size experiment records them.
@zzylol
zzylol force-pushed the perf/observe-without-allocation branch from c03faab to 26e7bb2 Compare October 19, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant