Skip to content

Run every stage as a pinned collector container; sketch vs exact answers with bounds - #25

Open
zzylol wants to merge 5 commits into
feat/scenario-generatorfrom
feat/collector-containers
Open

zzylol wants to merge 5 commits into
feat/scenario-generatorfrom
feat/collector-containers

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 3 of the RFC 0001 stack. This is the deployment the use cases run on.

Before → after

  • Before: the production asap_sketches processor only ran in-process or behind the file-passing demo. The container benchmark used a separate model processor.
  • After: every stage is the official OTAP engine (asap-collector: the upstream controller with the ASAP nodes linked in), in its own container pinned to its own CPU, talking OTLP/HTTP over the network. The sketch answers are shown next to exact answers computed from the same traffic.
generator-a -> branch-a --sketch--> merge -> detector --> Prometheus -> Grafana
generator-b -> branch-b --sketch--/
                  \--raw copy (official fanout)--> exact ------/

New pieces

  • asap-collector: the upstream df_engine startup (admin /metrics, /readyz, signals) with the ASAP nodes registered.
  • window_merge: merges sources' sketches by the window they describe, not by arrival time, and emits a window a grace period (5 s) after it ends. Later arrivals are counted as late.
  • sketch_detector: turns merged windows into answers with bounds. HLL ±1.96 standard errors; KLL quantiles at q ± the normalized rank error; Count-Min lower bound = estimate − e/w·N.
  • exact_answers: the same answers from every raw point by event-time window. It is both the ground truth and the exact baseline. Both emit asap_quantile, asap_distinct, asap_topk and asap_points, labelled method="sketch" or "exact", as plain OTLP gauges with labels on each point.
  • aggregate_by is exposed in the processor config.
  • Fix: tumbling windows close by event time. Windows used to close on a timer started at the first message, so up to a window of later points landed in the previous window. Measured before the fix: the first window held 751,060 points in the sketch against 175,630 exact. A batch whose event time passes the window's end now closes it first, and the idle timer runs every second. After the fix, sketch and exact hold identical point counts in every window.
  • deploy/telemetry-functions: Dockerfile, compose.yaml, stage pipelines, Prometheus (OTLP receiver plus a scrape of each engine's CPU, RSS and bytes), Grafana and Loki, and run.sh.

Result (56-core x86 server, 1-minute windows, 20,000 points/s per source)

Latency dashboard

For the busiest route, the sketch p99 was 199.5 ms against 201.7 ms exact, inside the sketch's bound of 166–295 ms. For routes with 20,000–100,000 points per window, the p99 can differ from exact by 10–40%. That is KLL's rank guarantee (about 1.3% at k = 200) in a log-normal tail, where a small rank shift is a large value shift; the bound covers it. The accuracy-vs-k experiment comes in a later PR.

Validation

Run on a 56-core x86 server with deploy/telemetry-functions/run.sh. cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head. The end-to-end acceptance checks come in a later PR of this stack.

asap-collector --config pipeline.yaml runs the upstream controller (startup,
admin /metrics and /readyz, signal handling) with the asap_sketches
processor and the scenario generator registered, so every pipeline stage
can run as a real collector process in its own container.
Sketches with the same series and window are merged and emitted once, a
grace period after the window ends; later arrivals are counted as late.
The ASAPv1 decode used by the carrier check becomes a shared
sketches::decode_snapshot.
Both emit asap.quantile, asap.distinct, asap.topk and asap.points gauges
in one shape, labelled method=sketch or method=exact, so dashboards can
overlay them. Sketch answers carry lower/upper bounds from each sketch's
configuration (HLL 95% standard error, KLL normalized rank error, Count-Min
overestimate). exact_answers groups raw points into event-time windows and
is both the ground truth and the exact baseline.
A tumbling window used to close only on a timer started at the first
message, so up to one window of later points landed in the previous window.
Now a batch whose event time passes the open window's end closes it first,
and the idle timer runs every second. aggregate_by selects the labels that
identify a series (empty keeps every label).
deploy/telemetry-functions: generators, branches (sketch + raw copy via the
official fanout), window merge, detector and the exact baseline each run
asap-collector in their own container on their own CPU, over OTLP/HTTP.
Prometheus receives sketch and exact answers over OTLP and scrapes every
engine's own telemetry (CPU time, RSS, payload bytes per stage); Grafana
overlays the sketch answer and its bound on the exact answer. Answers are
written as plain OTLP gauges with labels on each point.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant