refactor(rendering): drive the Media Foundation decoder from one request state machine by cesarfrancots · Pull Request #2545 · CapSoftware/Cap

cesarfrancots · 2026-10-14T03:54:47Z

Stacked on #2541. The first 10 commits belong to that PR. This PR adds the last two: fix(video-decode): read past Media Foundation samples that carry no frame and refactor(rendering): drive the Media Foundation decoder from one request state machine. Once #2541 merges, the diff shrinks to those two.

Summary

The Windows Media Foundation decoder thread used to be one 300-line loop. Batching, a mid-batch readahead slide, deferred requests and four separate fallback chains were all tangled together in it, which made its frame selection hard to reason about and impossible to test. This PR splits it into two parts:

FrameRequests holds the policy and never touches Media Foundation. It owns the cache, the waiting requests and the readahead, and it answers next_step() with Seek, Decode one sample, or Wait.
The thread takes in requests, carries out each step, and sends back the answers.

One answer rule. Once the decoder moves past a requested frame, the request gets the newest sample at or below that frame. VFR samples that round to the same frame resolve the same way no matter when the request arrives. Requests are served lowest first, and each gets its own seek when decoding forward cannot reach it. The cache keeps every sample since the last seek and evicts the oldest, so that newest sample is always in the cache.

Unit tests drive the policy with a fake sample stream. They cover:

VFR collisions and skipped frames
readahead sliding with every request
readahead never evicting the frame being shown
seeking back for a passed frame
long GOPs
a request landing inside a seek's pre-roll
several requests in flight
end of stream
read errors

Each new rule was mutation-checked: reverting it fails its test.

Behaviour changes

Several requests in flight each get their own frame. An export transition between two segments of the same recording sends both requests through one decoder. Previously the highest request was the target, and a lower request far from it got the target's frame.
A request behind the decoder with nothing cached at or below it seeks back. Previously, within half the cache size, it got the next decoded frame, which is a later one.
Every request slides the readahead, cache hits included. A sequential reader no longer drains the cache while the decoder sits idle.
Readahead never evicts the sample that answers the latest request. Frames too large for the 128 MiB cache budget used to get a later frame. The per-batch time budget is gone; readahead is bounded by its frame count.
A failed seek is no longer treated as a successful one. Waiting requests are answered from the cache instead of from wherever the old read position happens to be.
read_sample no longer returns Ok(None) for a sample that carries no frame. Stream ticks used to look like the end of the stream; it now reads past them.

Verification

cargo test -p cap-rendering --lib media_foundation: 13 pass. cargo test -p cap-video-decode --lib passes.
cargo clippy -p cap-rendering -p cap-video-decode --all-targets -- -D warnings is clean.
Export output is byte-identical. The 1080p60 export of the reference recording has the same sha256 before and after (a88dca3b…, SSIM 1.0). It matched on every run: three builds, seven exports.
No performance regression. These are interleaved A/B runs with scripts/capbench.py (from perf(windows): fence-waited readbacks, overlapped MF export decode, cheaper screen capture #2541) against the perf(windows): fence-waited readbacks, overlapped MF export decode, cheaper screen capture #2541 build. Each row is the median, with a bootstrap 95% CI.

Metric	#2541	This PR	Change
export-1080p60 wall (n=5)	7.65 s	7.52 s	−1.8% [−6.2%, +13.5%] same
export-1080p60 CPU (n=5)	15.2 s	14.5 s	−4.4% [−9.0%, +11.5%] same
decode random access avg (n=10)	66.7 ms	67.1 ms	+0.6% [−3.3%, +2.1%] same
decode sequential (n=10)	305.6 fps	308.1 fps	+0.8% [−0.5%, +2.0%] same
playback pipeline (n=5)	129.6 fps	137.3 fps	+5.9% [−7.8%, +22.7%] same
playback scrub p50 (n=5)	46.5 ms	45.7 ms	−1.7% [−6.1%, +10.8%] same

One 5-rep run flagged random access as +25% WORSE. A 10-rep, three-way rerun showed that was session noise: the #2541 build itself ranged from 68 to 86 ms across sessions, and the rerun shows +0.6% with a CI of [−3.3%, +2.1%]. Media Foundation returned no frameless samples on this fixture, so the reader change is not a factor here.

The win here is correctness and structure, not speed. The trends lean the right way but stay within noise.

Notes for reviewers

ffmpeg.rs keeps its own readahead and eviction helpers. They are deliberately not shared. The MF cache evicts strictly oldest first, because that is what keeps "newest sample at or below a frame" exact. The FFmpeg helper evicts by direction.
Today the MF zero-copy path is never taken, because the plane texture handles are always null. Cached MF frames are therefore CPU NV12 copies, so a 4096×2304 frame still leaves room for the readahead window plus the frame being shown.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

The PR separates Media Foundation frame selection into FrameRequests and leaves the decoder thread to carry out seeks and reads.

Media Foundation requests now get frames through one seek-and-decode policy.
Rendered frames now wait on the GPU work that produced them.
Windows capture reuses Direct3D frame buffers.
Cursor capture reuses the last image while the cursor shape stays the same.
Windows media benchmarks can run with timed playback and scripted measurements.
Exports can log how long rendering and frame sending take.

_{Reviews (12) · Last reviewed commit: "Merge branch 'perf/windows-media-pipelin..." · Reviewed by Greptile}

Examples live in target/<profile>/examples, next to deps/, but the DXC lookup only stepped up from deps/. Examples fell back to FXC, which took ~13s to compile the shaders on every editor open and skewed every benchmark built as an example.

…marks scripts/capbench.py drives the existing measurement binaries (cap export, cap record + cap-test validate, and the cap-editor playback/decode/startup examples) against the CI performance fixture and records wall time, whole-process-tree CPU time and peak commit (Windows job object), plus per-benchmark metrics parsed from their output. Variants (builds and/or env overrides) are interleaved per repetition with alternating order so drift and thermal state hit both sides equally; `compare` reports the median change with a bootstrap 95% interval and SSIM of the exported files. Single-shot runs on the reference machine varied by ~25%, which is larger than most individual changes. The playback example gains --timer-1ms to measure under the 1ms timer resolution the editor holds while playing.

Backport of gfx-rs/wgpu#8273 (wgpu 27). Device::wait signalled the device's single auto-reset idler event, so when several threads waited on one device a wakeup could reach the wrong waiter and the other one slept until the 60s CLEANUP_WAIT_MS timeout while wgpu-core held the fence lock, blocking every queue.submit on that device. The desktop app shares one device between editor, preparing-preview and screenshot renderers, and readbacks are about to block on the fence, so concurrent waits become routine.

…-polling Preview and export readbacks polled map_async with yield, then sleep(100us), then sleep(1ms). Outside timeBeginPeriod every sleep rounds up to the ~15.6ms Windows timer tick, which pinned render+readback at ~24ms regardless of resolution. Block on the submission's fence in spawn_blocking (device.poll WaitForSubmissionIndex) so the wait ends when the GPU signals. capbench, 5 interleaved reps (Ryzen 9 9950X3D, RTX 4070 Ti, fixture 3024x1964 + camera): playback pipeline 31.3 -> 92.2 fps, render p50 24.2 -> 4.1 ms, p95 46.0 -> 17.0 ms scrub p50/p95 79.6/123.5 -> 63.5/90.2 ms editor first frame 38.7 -> 13.9 ms exports unchanged

After answering a request the MF decoder reads ahead up to 8-30 frames before it looks at its channel again. A sequential reader (export, playback prefetch) therefore waited out the rest of the batch for its next frame, then found the decoder idle while it consumed the cached ones, so decode and render alternated instead of overlapping. Poll for requests between samples, answer cache hits immediately and slide the readahead window with them; misses are handed back to the normal request path. Answering from the cache mid-batch exposed a timing dependence that already existed: the decoders number samples at the recorded fps (59/57/30 on the fixture), so a VFR stream regularly has two samples rounding to the same frame and the later one replaces the first in the cache. Whether a request got the first or the second depended on whether it arrived before the second was decoded, so repeated exports differed by a source frame at a few dozen positions. A frame is now answered only once the decoder has moved past it, which always yields the last sample for that frame; repeated exports are byte-identical. capbench, interleaved, same build with the window slide toggled (n=4): export 1080p60 9.79 -> 8.45 s wall (-14%), 15.7 -> 18.0 s CPU export 720p30 8.11 -> 7.72 s wall (-5%), 13.7 -> 15.2 s CPU and against the previous build (n=5): MF random access 87.7 -> 69.1 ms avg, 119 -> 97 ms p95. The extra CPU is mostly NVIDIA driver threads now running concurrently with rendering; 4K exports are NVENC-bound and unchanged.

…OP_STATS The renderer already accumulates per-stage time (decode wait, render, next-frame decode, join, send) behind CAP_RENDER_LOOP_STATS=1, but only the render farm read it. MP4 exports now log it at completion, together with the time the forwarder spent blocked on the encoder, which is what showed the 1080p export to be decode-bound and 4K to be NVENC-bound.

Every captured frame was copied into a freshly allocated FFmpeg frame (8 MB at 1080p, 33 MB at 4K). Allocations that large come straight from new OS pages on Windows, so each frame also paid the page faults and zeroing, measured at ~2 ms per 1080p frame (~9 ms at 4K) against ~0.45 ms for the copy itself. Take the buffers from an AVBufferPool instead; they return to the pool when the encoder and the duplicate-frame holder drop them. capbench, 10 s recordings of a 60 fps test pattern, 5 interleaved reps (with the cursor change that follows; instant mode has no cursor thread, so its number is this change alone): studio 60 fps 0.309 -> 0.248 cores (-20%), delivered fps unchanged instant 30 fps 0.205 -> 0.167 cores (-18%) peak commit +7 MB (the pooled buffers)

…nchanged The cursor thread rendered the cursor into a DIB, trimmed it, PNG-encoded it and hashed the PNG on every tick (~32 Hz) only to find the same id. That costs ~250 us per tick (~0.8% of a core for the whole recording) against ~0.5 us for returning the previous result when GetCursorInfo reports the same HCURSOR.

Windows exports held a 1 ms timer resolution because the NV12 readback wait slept in 1 ms steps. Readbacks now block on the GPU fence and nothing on the export's per-frame path sleeps: CLI exports (1080p60, 5 interleaved reps) take 8.62 s without the hold and 8.70 s with it, so the process no longer raises its timer resolution for the length of an export.

…ailure logs `cap-test validate` exits 0 for an invalid recording, so read its `valid` and `errors` and count such runs as failures instead of comparing them. Failure logs were written under the work directory that is deleted after the run; copy them to <out>/failures/ and record that path.

…rame ReadSample hands back a null sample for stream ticks (gaps in the stream), and read_sample reported those as Ok(None), the same as the end of the stream. The decoder then stopped a batch early and answered with an older frame. Read again until a sample, the end-of-stream flag or an error.

…est state machine The decoder thread mixed batching, a mid-batch readahead slide, deferred requests and four fallback chains in one loop. The policy now lives in FrameRequests, which never touches Media Foundation: the thread takes in requests, asks it for the next step (seek, decode one sample, or wait) and does it. Tests drive it with a fake sample stream. Once the decoder moves past a requested frame, the request gets the newest sample at or below that frame. Requests are served lowest first, each with its own seek when decoding forward cannot reach it, so the "first sample after" fallback only applies to a frame before the stream's first sample. The cache keeps every sample since the last seek and evicts the oldest, so the newest sample at or below a frame is always cached. Behaviour changes: - Several requests in flight (export transitions between two segments of one recording decode both through the same decoder) each get their own frame. The highest used to be the target, and a lower one far from it got the target's frame. - A request behind the decoder with nothing cached at or below it seeks back. Within half the cache size it used to get the next decoded, later frame. - Every request slides the readahead, cache hits included, so a sequential reader no longer drains the cache while the decoder idles. - Readahead stops at its frame count, or before the next sample would evict the one answering the latest request (frames too big for the cache budget used to get a later frame); the per-batch time budget is gone. - Samples outside the old cache window are cached too, so a frame after a VFR gap past the window gets its own sample, not the one before the gap. - A failed seek answers what waits from the cache instead of reading on from the old position as if it had landed. Export output is byte-identical to the previous decoder on the reference recording.

vercel · 2026-10-14T03:54:55Z

@cesarfrancots is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

…-machine

richiemcilroy · 2026-10-14T06:44:26Z

Thanks for the PR. Greptile currently has this at 4/5, and we only merge PRs once Greptile scores the current head 5/5. Please address the remaining Greptile feedback, then comment @greptileai to request a re-review, and repeat until it reaches 5/5.

cesarfrancots · 2026-10-14T14:28:55Z

@greptileai

cesarfrancots · 2026-10-14T14:45:44Z

@richiemcilroy Thanks. The Greptile feedback is addressed and all review threads are resolved. Greptile now scores the current head (9d0ef5a) 5/5. This PR is still stacked on #2541, so once that merges the diff shrinks to the two decoder commits.

…-machine

cesarfrancots · 2026-10-16T04:53:44Z

@greptileai

…peline

… colour matrix

…-machine

cesarfrancots · 2026-10-16T05:06:26Z

@greptileai

A benchmark that exits 0 without its metrics (decode-benchmark does when it cannot create a decoder) was kept with a warning, and its short wall time could make a broken build look faster. Such runs are now invalid runs, like failed ffprobe or recording validation. The job's CPU and memory figures stay optional because they exist only on Windows, and instant recordings no longer list delivered_fps and captured_ratio, which cap-test validate never reports for them.

…-machine

cesarfrancots · 2026-10-16T05:14:53Z

@greptileai

…peline

…-machine

cesarfrancots · 2026-10-16T13:53:03Z

@greptileai

… with failed reads A missing cap-test or ffprobe raised out of the run loop after earlier benchmarks had finished, so nothing was saved; each bench now names the helpers it needs and the startup check finds them before any run. decode-benchmark exits 0 and times only the reads that succeeded, so failed sequential reads, seeks or random reads now make the run invalid.

…-machine

cesarfrancots · 2026-10-16T14:01:44Z

@greptileai

…-machine

…peline

cesarfrancots · 2026-10-16T15:21:33Z

@greptileai

…tory cmd_run deletes <out>/work before copying the fixture into it, so a fixture reused from an earlier --keep-work run under the same --out was deleted before it was copied. Overlapping fixture and work paths now stop the run before anything is deleted.

…-machine

cesarfrancots · 2026-10-16T15:29:13Z

@greptileai

run_once writes each log inside the work directory, which is deleted at the end unless --keep-work, so --keep-logs alone kept nothing. Kept logs are now copied to <out>/logs, like failure logs to <out>/failures.

…-machine

cesarfrancots · 2026-10-16T15:30:30Z

@greptileai

…peline The readahead path this branch adds to the Media Foundation decoder now maps request times with request_frame, like main's other decoder paths.

…-machine The request state machine now maps request times with request_frame, like main's other decoder paths.

…-machine

cesarfrancots · 2026-10-18T02:02:17Z

@greptileai

…tion test after_submit takes the submission index on this branch; the macOS-only test main added still called it with the queue alone, so cap-rendering's tests did not compile on macOS.

…runs with failed frames A --bin-dir inside <out>/work passed the binary checks and was then deleted with the work directory. playback-pipeline-benchmark exits 0 and times only the frames that decoded and rendered, so runs reporting decode or render failures are now invalid, like decode-benchmark runs with failed reads.

…-machine

cesarfrancots · 2026-10-18T02:12:47Z

@greptileai

cesarfrancots · 2026-10-18T02:13:47Z

The macOS test finding is fixed here too (merged up from #2541 in 1e10ac5), and the branch includes the latest main.

@greptileai review

cesarfrancots added 12 commits October 13, 2026 19:18

greptile-apps Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread crates/export/src/mp4.rs

cesarfrancots added 2 commits October 13, 2026 22:06

docs(export): note that the loop stats are process-wide

610ba39

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

9d0ef5a

…-machine

cesarfrancots mentioned this pull request Oct 14, 2026

perf(video-decode): pipeline the Media Foundation GPU readback #2583

Open

This was referenced Oct 15, 2026

perf(recording): take the cursor position from the recorder's mouse sample #2621

Merged

perf(recording): reuse one X11 connection and unchanged cursor PNGs #2622

Open

Merge remote-tracking branch 'origin/main' into perf/mf-decoder-state…

67cdf7b

…-machine

cesarfrancots added 3 commits October 15, 2026 22:55

Merge remote-tracking branch 'origin/main' into perf/windows-media-pi…

5fd3a90

…peline

fix(rendering): tag readahead Media Foundation frames with the source…

470f90d

… colour matrix

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

246c17d

…-machine

greptile-apps Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread scripts/capbench.py Outdated

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

ec11a9b

…-machine

cesarfrancots added 2 commits October 16, 2026 07:49

Merge remote-tracking branch 'origin/main' into perf/windows-media-pi…

f7aa44c

…peline

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

a7ac091

…-machine

cesarfrancots added 2 commits October 16, 2026 08:01

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

2f22c03

…-machine

cesarfrancots added 2 commits October 16, 2026 09:18

Merge remote-tracking branch 'origin/main' into perf/mf-decoder-state…

e908f42

…-machine

Merge remote-tracking branch 'origin/main' into perf/windows-media-pi…

b95bfa6

…peline

cesarfrancots added 2 commits October 16, 2026 09:28

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

0d5f379

…-machine

cesarfrancots added 2 commits October 16, 2026 09:30

fix(scripts): keep successful-run logs when --keep-logs is set

03c72d9

run_once writes each log inside the work directory, which is deleted at the end unless --keep-work, so --keep-logs alone kept nothing. Kept logs are now copied to <out>/logs, like failure logs to <out>/failures.

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

65a4b7b

…-machine

cesarfrancots added 3 commits October 17, 2026 19:56

Merge remote-tracking branch 'origin/main' into perf/windows-media-pi…

9afd214

…peline The readahead path this branch adds to the Media Foundation decoder now maps request times with request_frame, like main's other decoder paths.

Merge remote-tracking branch 'origin/main' into perf/mf-decoder-state…

ad4041d

…-machine The request state machine now maps request times with request_frame, like main's other decoder paths.

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

3074903

…-machine

greptile-apps Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread crates/rendering/src/frame_pipeline.rs

cesarfrancots added 3 commits October 17, 2026 20:11

fix(rendering): pass the submission index in the macOS surface comple…

e26f981

…tion test after_submit takes the submission index on this branch; the macOS-only test main added still called it with the queue alone, so cap-rendering's tests did not compile on macOS.

Merge branch 'perf/windows-media-pipeline' into perf/mf-decoder-state…

1e10ac5

…-machine

Conversation

cesarfrancots commented Oct 14, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour changes

Verification

Notes for reviewers

Confidence Score: 5/5

Summary

Uh oh!

vercel Bot commented Oct 14, 2026

Uh oh!

Uh oh!

richiemcilroy commented Oct 14, 2026

Uh oh!

cesarfrancots commented Oct 14, 2026

Uh oh!

cesarfrancots commented Oct 14, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 16, 2026

Uh oh!

cesarfrancots commented Oct 18, 2026

Uh oh!

Uh oh!

cesarfrancots commented Oct 18, 2026

Uh oh!

cesarfrancots commented Oct 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cesarfrancots commented Oct 14, 2026 •

edited by greptile-apps Bot

Loading