Repository navigation
perf: chrome-headless-shell, faster wait, gif workers, bounded capture queue - #39
Merged
Merged
Conversation
benchmark/: fixtures served with simulated latency, browsers launched by the harness, interleaved variants, CPU and memory of the browser process tree read from /proc. Experiments: wait strategies, browser binaries, launch flags, screenshot and pdf encoders, gif pipeline, concurrency, CDP transport, the real service over HTTP, and the public applications of a data-fair instance. Findings and recommendations in benchmark/FINDINGS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… + 1s Pages that do not call window.triggerCapture paid a 2s floor: Chrome's networkidle0 fires 0.5-1.5s after the last request, then a fixed 1s sleep. Now puppeteer's network idle (500ms) then 200ms without DOM mutation, animation frame, finite animation or font loading, capped at 2s (config.wait). On the public applications of opendata.koumoul.com that do not call triggerCapture: 2093ms -> 1075ms median, identical screenshots. Long chart animations are now waited for instead of captured half drawn. A capture-no-trigger warning names the pages that do not call triggerCapture. Large (config.pngFastEncodingPixels) and full page png screenshots are encoded for speed: the encoding runs on the browser's main thread, shared by all the captures. Fixes: type=jpg returned png bytes labelled image/jpeg, type=gif on a page without animation support returned png bytes labelled image/gif. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… enabled The console listener did 2 CDP round trips per argument of every console message even with the debug output off: node CPU x4 on a page logging 500 messages (benchmark: console). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captures waiting for a browser context were kept whatever happened to their client: after a proxy timeout or a user leaving, they were still rendered for nobody and delayed the next ones (benchmark e2e: 30 abandoned requests made a 0.7s capture wait 18s). The routes now pass an AbortSignal, aborted when the response closes before being sent: the capture is skipped if it is still queued, stopped otherwise (499, not logged as a failure). The queue is bounded (config.queue, QUEUE_MAX_WAITING and QUEUE_MAX_WAIT_TIME): beyond 100 waiting captures, or after 20s of waiting, a 503 that the caller can retry instead of an answer after the proxy gave up. promiseTimeout now clears its timer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The png decoding and the gif encoding (pure JS, NeuQuant quantization) of an animated capture ran on the main thread: ~4s of CPU for 30 frames at 800x450, stalling every other request of the service meanwhile. The frames are now collected during the capture and encoded by a worker thread (same libraries, same output). Benchmark e2e, a probe requesting api-docs every 20ms during a gif capture: p90 83ms -> 1.2ms, max 131ms -> 4.6ms. config.gifThreads (GIF_THREADS, default 2) also bounds the number of gifs encoded at the same time. tmp-promise is not needed anymore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The image installs Debian's chromium-headless-shell (the "old" headless Chromium, without the browser UI layer) instead of google-chrome-stable from Google's repository. Measured: half the memory at idle, 20-45% less browser CPU per capture, pdf x1.6 faster, +30% throughput on paint heavy pages. On the image: 1.12GB -> 904MB, container memory 324 -> 142MiB idle and ~780 -> ~550MiB under 10 concurrent captures. It is packaged for amd64 and arm64 (the google repository was pinned to amd64), updated with the distribution, and still sandboxed with seccomp.json. wget is now purged from the image, it was only kept because google-chrome-stable depended on it. Development, tests and CI use the chrome-headless-shell downloaded by puppeteer (npx puppeteer browsers install chrome-headless-shell), with --no-sandbox: outside of a container AppArmor forbids its sandbox on Ubuntu. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pying, small gifs failed The encoded gif was returned with move() on the output of Buffer.concat, which is a slice of the worker's shared Buffer pool below half of Buffer.poolSize: the transfer threw a DataCloneError and every gif under ~32KB failed. It is now joined in an ArrayBuffer of its own. The frames were cloned (copied) to the worker. They are now transferred (transferList): the large frames puppeteer returns own their ArrayBuffer and move without any copy, the small ones (slices of the Buffer pool) are copied once first. The main thread no longer holds a second copy of all the frames during the encoding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The screenshot was decoded from base64 by puppeteer, copied, then encoded back to base64 for the JSON response: 4-8ms of synchronous work on the main thread for the large full page captures, and 2 extra copies of the image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…f the cpu Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y a bounded semaphore The pool reused its incognito contexts from one capture to the next and only deleted their cookies: localStorage, IndexedDB and Cache Storage written by a capture were visible to the next ones of the same origin, whatever the user they were made for. Each capture now runs in its own context, closed after it. It costs a few milliseconds (benchmark: context). This also removes the pool and its cleanup, destroy and retry paths. The captures wait for a slot of the service's semaphore, which gained what the pool provided and more: a bounded queue, a wait timeout, and the removal of a waiting call when its client leaves. /inspect uses the same mechanism instead of checking its socket. The default wait timeout is 30s: data-fair gives up after 60s without a response and retries once. The metrics keep their names (df_capture_acquire_context_*, df_capture_contexts_cleanups) for the existing dashboards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance work on capture, measured with a new local benchmark (
npm run benchmark, results inbenchmark/FINDINGS.md).chromium-headless-shellin the image) instead of google-chrome-stable: half the memory, 20-45% less CPU per capture, smaller image. Development and tests use puppeteer's build (npx puppeteer browsers install chrome-headless-shell).window.triggerCapture, wait for network idle (500 ms) and then visual quiet (200 ms, capped at 2 s), instead ofnetworkidle0plus a fixed 1 s. On public applications, the wait drops from ~2.1 s to ~1.1 s with identical pixels. These captures log a warning (capture-no-trigger) to show which applications should calltriggerCapture.GIF_THREADS, default 2), with the frames transferred rather than copied. Event loop blocking at p90 goes from 83 ms to 1 ms.QUEUE_MAX_WAITING,QUEUE_MAX_WAIT_TIME). An overloaded service answers 503, and a capture whose client left is dropped (499).type=jpgreturns a JPEG. Agifrequest on a page without animation is labelled png./inspectreturns the browser's base64 as is.Why: under steady load of application captures, memory is the bottleneck and the CPU goes to WebGL maps rendered in software. This change lowers the cost per capture, makes overload fail fast, and closes a state leak between captures.
Heads-up:
triggerCapture. Pages that keep rendering long after network idle are captured after at most 2 s of settling.capture-no-triggeris logged at warn level for every such capture, so expect some log volume.Retry-Afteryet.df_capture_contexts_cleanupsnow counts closed contexts.