perf(audio): reduce caption mixing time and contributor-count memory

[dennisimoo]

Caption preparation allocates a u32 contributor count for every 48 kHz frame while decoded mic/system audio and the mix buffer are also live. A source contributes over one continuous interval, so only its start and end can change that count.

Replace the recording-sized count buffer with two boundary events per source and average each span in place. Clip the source iterator once instead of checking the destination for every frame. Keep main’s direct accumulation into the appended output slice, then divide that slice over the boundary intervals. Source/channel accumulation order, offsets, silence and previous takes stay unchanged.

This saves 329.6 MiB per continuous 30-minute take. Rebased onto main after #2607: its removal of the separate f32 mix buffer is preserved, so both memory savings stack.

Measurements

Linux x86_64, Xeon Platinum 8175M, Rust 1.88.0, opt-level 3/thin LTO/one codegen unit. These refreshed measurements compile the included example and each revision’s transcription module directly with rustc, excluding unrelated crate dependencies. Three interleaved fresh processes per version and fixture, pinned to one CPU, with no concurrent builds. Baseline: 2b2bf844.

Synthetic decoded 48 kHz audio	Assembly median, before → after	Process peak RSS, before → after
1-minute microphone	36.00 → 15.01 ms	34.88 → 23.83 MiB
1-minute stereo system + microphone	53.28 → 21.74 ms	56.83 → 45.89 MiB
30-minute microphone	1075.76 → 451.64 ms	990.71 → 661.08 MiB
30-minute stereo system + microphone	1605.59 → 660.31 ms	1649.89 → 1320.26 MiB

Assembly takes 58–59% less time in the long cases. The timer excludes fixture generation and output checksumming; RSS includes decoded inputs. This is audio assembly, not decoding, model inference, or total caption-generation time. All output-bit checksums match.

Validation

• cargo +1.88.0 check -p cap-audio --locked
• cargo +1.88.0 fmt --all --check
• cargo +1.88.0 test -p cap-audio --locked --lib: 83 passed, one ignored.
• cargo +1.88.0 clippy -p cap-audio --all-targets --locked -- -D warnings
• The official reference recording produces 1,476,298 bit-identical mixed samples. A separate harness using Cap's FFmpeg libraries and the GPUI normalization/WAV-writing steps also produces byte-identical 16 kHz WAVs. The native GPUI app and speech model were not run.

Added regression test	Coverage
interval_counts_match_per_sample_counts_and_preserve_previous_takes	1,152 append comparisons (exact bits except arithmetic NaN payload/sign) against the old algorithm: offsets, clipping, silence, multichannel input and non-finite samples
adjacent_overlapping_and_disjoint_source_boundaries_keep_exact_averages	Tied boundaries, overlap averages, empty/clipped sources and gaps
invalid_append_leaves_existing_samples_unchanged	Invalid timing/channel data leaves prior audio untouched

Reproduce the benchmark

With the repository's native dependencies configured, build the included example to exercise the same fixture through the crate:

CARGO_PROFILE_RELEASE_OPT_LEVEL=3 CARGO_PROFILE_RELEASE_DEBUG=0 \
  cargo +1.88.0 build --release --locked -p cap-audio \
  --example transcription-assembly-benchmark

/usr/bin/time -v target/release/examples/transcription-assembly-benchmark 1800 mic-system

Repeat with 60 seconds and mic. For a crate-linked baseline, copy the same example into a worktree at 2b2bf844, build with identical settings, and alternate the two binaries in fresh processes. The stereo fixture delays the microphone by 125 ms. On Linux, taskset -c 0 can pin each process as in the table.

AI-assisted with Codex.

Confidence Score: 5/5

The PR appears safe to merge; no blocking issues were found.

Summary

Replaces per-frame contributor counts with two boundary events per source and clips each source before mixing.

• Caption assembly mixes each source span without a recording-sized count array.

_{Reviews (2) · Last reviewed commit: "perf(audio): replace per-sample transcri..." · Reviewed by Greptile}

[vercel]

@dennisimoo is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[richiemcilroy]

Thanks @dennisimoo. #2607 just merged and changed the same function, so this now conflicts with main in crates/audio/src/transcription_timing.rs. #2607 accumulates straight into output (output.resize(start + frames, 0.0), then sums = &mut output[start..]) with a per-sample u32 count. This PR replaces the per-sample counts with source boundaries.

Both savings can stack: sum into the output slice as main now does, and divide over your boundary intervals instead of the counts vector. Could you rebase onto main that way? Once it is 5/5 again, we will run the full suite and merge it.

[dennisimoo]

@greptile review

[dennisimoo]

Rebased onto main in 7606d1e. The mix still accumulates directly into output[start..] as in #2607, and overlap averaging now uses the source boundaries instead of a per-frame count buffer. Both savings stack.

Validation: 83 cap-audio tests passed, one ignored; cargo check, Clippy with -D warnings, and formatting passed. The reference recording still produces 1,476,298 bit-identical mixed samples and byte-identical 16 kHz WAVs in the separate harness. Refreshed measurements against current main are in the PR description.

A fresh Greptile review is running. The GitHub Actions workflows are waiting for maintainer approval.