perf(audio): save 330 MiB while preparing 30-minute captions

[AksharP5]

Generating captions for a long Studio recording keeps a full-length scratch mix alongside the final audio while decoded mic/system tracks are still live. Mix into the new output tail and average it in place, preserving prior takes, source offsets, silence, and the existing arithmetic.

This removes one f32 buffer: 329.6 MiB for a 30-minute take at the GPUI caption path’s 48 kHz rate. Fallible reservations still finish before the output length changes.

Measured additional live heap inside append_transcription_audio, excluding input samples and an existing output prefix:

Current take	Before	After	Saved
1 minute	32.96 MiB	21.97 MiB	10.99 MiB
5 minutes	164.79 MiB	109.86 MiB	54.93 MiB
30 minutes	988.77 MiB	659.18 MiB	329.59 MiB
5 minutes appended after 25 minutes	164.79 MiB	109.86 MiB	54.93 MiB

The std-only harness includes the actual source module. All 56 paired fixture processes and eight additional checks with GPUI release optimization flags produced matching output-bit digests and confirmed the table’s before/after heap reductions. Each fixture keeps a stereo system source and mono mic source live. Timing is excluded because unrelated host compilation resumed during the run; decoding, normalization, WAV encoding, and speech recognition are outside this measurement.

Validation: cargo fmt --all, cargo check -p cap-audio, cargo test -p cap-audio --lib transcription_timing (7 tests), and cargo clippy -p cap-audio --all-targets -- -D warnings pass with Rust 1.88. The harness also passes 22 bitwise edge cases, invalid-input checks, and injected allocation failures. The two added regression tests cover appending without changing earlier takes and preserving output when validation/reservation fails.

Checked the changed files in all 264 public open PRs before publishing; none overlaps this function. Related #2539, #2582, and #1786 affect other stages.

Reproduce the memory and output checks without FFmpeg or a model download

Save the Rust block below as harness.rs in an empty directory and create its sources subdirectory. From this checkout, write the actual baseline and candidate modules there:

git show 5705ee8515b2f862f4136f2e380ca23df4985e95:crates/audio/src/transcription_timing.rs > /path/to/bench/sources/baseline.rs
git show 74c211078290d9bf940e37c6c9354521ee217389:crates/audio/src/transcription_timing.rs > /path/to/bench/sources/candidate.rs

From that benchmark directory, using Rust 1.88.0:

for version in baseline candidate; do
  CAP_TRANSCRIPTION_SOURCE="$PWD/sources/$version.rs" rustc --edition=2024 \
    -C opt-level=3 -C debuginfo=0 -C lto=thin -C codegen-units=1 \
    harness.rs -o "$version"
  "./$version" validate
  for case in studio-1m studio-5m studio-30m studio-25m-prefix-5m; do
    "./$version" case "$case"
  done
done

peak_additional_bytes is the measured live-heap increase. fnv1a64_sample_bits_le covers every output sample bit. The large cases need about 2 GiB available memory. The initial 56-process run used -C opt-level=3 -C debuginfo=0; the eight profile-confirmation processes used the full flags above and agreed exactly on allocations and output. All runs were retained.

mod subject {
    include!(env!("CAP_TRANSCRIPTION_SOURCE"));
}
mod baseline {
    include!("sources/baseline.rs");
}

use std::alloc::{GlobalAlloc, Layout, System};
use std::hint::black_box;
use std::sync::atomic::{AtomicBool, AtomicIsize, AtomicUsize, Ordering::Relaxed};
use subject::{TranscriptionAudioSource as Source, TranscriptionAudioTake as Take};

struct Counted;
static LIVE: AtomicUsize = AtomicUsize::new(0);
static PEAK: AtomicUsize = AtomicUsize::new(0);
static ALLOCATIONS: AtomicUsize = AtomicUsize::new(0);
static REQUESTED: AtomicUsize = AtomicUsize::new(0);
static TRACK: AtomicBool = AtomicBool::new(false);
static FAIL_AFTER: AtomicIsize = AtomicIsize::new(-1);

fn fail_allocation() -> bool {
    let remaining = FAIL_AFTER.load(Relaxed);
    if remaining < 0 {
        return false;
    }
    FAIL_AFTER.store(remaining - 1, Relaxed);
    remaining == 0
}

fn allocated(old_size: usize, new_size: usize) {
    let live = if new_size >= old_size {
        LIVE.fetch_add(new_size - old_size, Relaxed) + new_size - old_size
    } else {
        LIVE.fetch_sub(old_size - new_size, Relaxed) - old_size + new_size
    };
    if TRACK.load(Relaxed) {
        ALLOCATIONS.fetch_add(1, Relaxed);
        REQUESTED.fetch_add(new_size, Relaxed);
        PEAK.fetch_max(live, Relaxed);
    }
}

unsafe impl GlobalAlloc for Counted {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        if fail_allocation() {
            return std::ptr::null_mut();
        }
        let pointer = unsafe { System.alloc(layout) };
        if !pointer.is_null() {
            allocated(0, layout.size());
        }
        pointer
    }

    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
        if fail_allocation() {
            return std::ptr::null_mut();
        }
        let pointer = unsafe { System.alloc_zeroed(layout) };
        if !pointer.is_null() {
            allocated(0, layout.size());
        }
        pointer
    }

    unsafe fn realloc(&self, pointer: *mut u8, layout: Layout, size: usize) -> *mut u8 {
        if fail_allocation() {
            return std::ptr::null_mut();
        }
        let result = unsafe { System.realloc(pointer, layout, size) };
        if !result.is_null() {
            allocated(layout.size(), size);
        }
        result
    }

    unsafe fn dealloc(&self, pointer: *mut u8, layout: Layout) {
        LIVE.fetch_sub(layout.size(), Relaxed);
        unsafe { System.dealloc(pointer, layout) };
    }
}

#[global_allocator]
static ALLOCATOR: Counted = Counted;

fn source(frames: usize, channels: usize, offset_secs: f64, seed: u64) -> Source {
    let samples = (0..frames * channels)
        .map(|index| {
            let value = (index as u64)
                .wrapping_mul(1_103_515_245)
                .wrapping_add(seed);
            ((value >> 8) % 65_537) as f32 / 65_536.0 - 0.5
        })
        .collect();
    Source {
        samples,
        channels,
        offset_secs,
    }
}

fn fixture(name: &str) -> Vec<Take> {
    let seconds = match name {
        "studio-1m" => 60,
        "studio-5m" | "studio-25m-prefix-5m" => 300,
        "studio-30m" => 1_800,
        _ => panic!("Unknown case: {name}"),
    };
    vec![Take {
        display_duration_secs: seconds as f64,
        sources: vec![
            source(seconds * 48_000 - 3_600, 2, 0.173, 11),
            source(seconds * 48_000 - 9_600, 1, -0.093, 71),
        ],
    }]
}

fn oracle(takes: &[Take], rate: u32, prefix: &[f32]) -> Result<Vec<f32>, String> {
    let mut output = prefix.to_vec();
    for take in takes {
        let take = baseline::TranscriptionAudioTake {
            display_duration_secs: take.display_duration_secs,
            sources: take
                .sources
                .iter()
                .map(|source| baseline::TranscriptionAudioSource {
                    samples: source.samples.clone(),
                    channels: source.channels,
                    offset_secs: source.offset_secs,
                })
                .collect(),
        };
        baseline::append_transcription_audio(&mut output, &take, rate)?;
    }
    Ok(output)
}

fn bits(samples: &[f32]) -> Vec<u32> {
    samples.iter().map(|sample| sample.to_bits()).collect()
}

fn validate() {
    let prefix = [1.25, -0.0, f32::from_bits(0x7fc0_1234)];
    let mut cases = Vec::new();
    for frames in [0, 1, 2, 7, 31, 257] {
        for offset in [-0.5, 0.0, 0.375] {
            cases.push(vec![Take {
                display_duration_secs: frames as f64 / 8.0,
                sources: vec![
                    source(frames + 4, 2, offset, 11),
                    source(frames / 2, 1, -offset, 71),
                ],
            }]);
        }
    }
    cases.push(vec![Take {
        display_duration_secs: 1.0,
        sources: Vec::new(),
    }]);
    cases.push(vec![Take {
        display_duration_secs: 1.0,
        sources: vec![source(8, 1, 0.0, 1)],
    }]);
    cases.push(vec![
        Take {
            display_duration_secs: 2.0,
            sources: vec![source(24, 3, 0.125, 91)],
        },
        Take {
            display_duration_secs: 1.0,
            sources: Vec::new(),
        },
        Take {
            display_duration_secs: 0.5,
            sources: vec![source(4, 1, 0.0, 1)],
        },
    ]);
    cases.push(vec![Take {
        display_duration_secs: 1.0,
        sources: vec![Source {
            samples: vec![
                -0.0,
                0.0,
                f32::INFINITY,
                f32::NEG_INFINITY,
                f32::from_bits(0x7fc0_1234),
                0.25,
                -0.5,
                0.0,
            ],
            channels: 1,
            offset_secs: 0.0,
        }],
    }]);
    for takes in &cases {
        let expected = oracle(takes, 8, &prefix).unwrap();
        let mut actual = prefix.to_vec();
        for take in takes {
            subject::append_transcription_audio(&mut actual, take, 8).unwrap();
        }
        assert_eq!(bits(&actual), bits(&expected));
        let assembled = subject::assemble_transcription_audio(takes, 8).unwrap();
        assert_eq!(bits(&assembled), bits(&expected[prefix.len()..]));
    }
    for (take, rate) in [
        (
            Take {
                display_duration_secs: f64::NAN,
                sources: Vec::new(),
            },
            8,
        ),
        (
            Take {
                display_duration_secs: -1.0,
                sources: Vec::new(),
            },
            8,
        ),
        (
            Take {
                display_duration_secs: 1.0,
                sources: Vec::new(),
            },
            0,
        ),
        (
            Take {
                display_duration_secs: 1.0,
                sources: vec![Source {
                    samples: vec![0.0],
                    channels: 0,
                    offset_secs: 0.0,
                }],
            },
            8,
        ),
        (
            Take {
                display_duration_secs: 1.0,
                sources: vec![Source {
                    samples: vec![0.0],
                    channels: 2,
                    offset_secs: 0.0,
                }],
            },
            8,
        ),
        (
            Take {
                display_duration_secs: 1.0,
                sources: vec![source(8, 1, f64::INFINITY, 1)],
            },
            8,
        ),
    ] {
        let expected = oracle(std::slice::from_ref(&take), rate, &prefix).unwrap_err();
        let mut actual = prefix.to_vec();
        assert_eq!(
            subject::append_transcription_audio(&mut actual, &take, rate).unwrap_err(),
            expected
        );
        assert_eq!(bits(&actual), bits(&prefix));
    }
    let take = Take {
        display_duration_secs: 8.0,
        sources: vec![source(64, 1, 0.0, 7)],
    };
    for successful_allocations in [0, 1] {
        let mut output = prefix.to_vec();
        FAIL_AFTER.store(successful_allocations, Relaxed);
        let result = subject::append_transcription_audio(&mut output, &take, 8);
        FAIL_AFTER.store(-1, Relaxed);
        assert_eq!(result.unwrap_err(), "Transcription audio is too large");
        assert_eq!(bits(&output), bits(&prefix));
    }
    let expected = baseline::assemble_transcription_audio(&[], 48_000).unwrap();
    assert!(expected.is_empty());
    println!(
        "{{\"validation\":\"passed\",\"bitwise_cases\":{},\"invalid_input_cases\":6,\"allocation_failure_cases\":2}}",
        cases.len()
    );
}

fn digest(samples: &[f32]) -> u64 {
    let mut hash = 0xcbf2_9ce4_8422_2325_u64;
    for sample in samples {
        for byte in sample.to_bits().to_le_bytes() {
            hash = (hash ^ u64::from(byte)).wrapping_mul(0x100_0000_01b3);
        }
    }
    hash
}

fn peak_rss_kib() -> usize {
    std::fs::read_to_string("/proc/self/status")
        .unwrap()
        .lines()
        .find_map(|line| {
            line.strip_prefix("VmHWM:")?
                .split_whitespace()
                .next()?
                .parse()
                .ok()
        })
        .unwrap()
}

fn run_case(name: &str, measure: bool) {
    let takes = fixture(name);
    let input_bytes: usize = takes
        .iter()
        .flat_map(|take| &take.sources)
        .map(|source| source.samples.len() * std::mem::size_of::<f32>())
        .sum();
    let mut output = if name == "studio-25m-prefix-5m" {
        source(25 * 60 * 48_000, 1, 0.0, 97).samples
    } else {
        Vec::new()
    };
    let prefix_samples = output.len();
    let before = LIVE.load(Relaxed);
    PEAK.store(before, Relaxed);
    ALLOCATIONS.store(0, Relaxed);
    REQUESTED.store(0, Relaxed);
    let start = measure.then(std::time::Instant::now);
    TRACK.store(true, Relaxed);
    for take in &takes {
        subject::append_transcription_audio(&mut output, take, 48_000).unwrap();
    }
    TRACK.store(false, Relaxed);
    let elapsed = start.map(|start| start.elapsed().as_nanos());
    let peak = PEAK.load(Relaxed);
    let after = LIVE.load(Relaxed);
    let allocations = ALLOCATIONS.load(Relaxed);
    let requested = REQUESTED.load(Relaxed);
    let rss = peak_rss_kib();
    let output_hash = digest(&output);
    black_box(&takes);
    black_box(&output);
    println!(
        "{{\"case\":\"{name}\",\"sample_rate\":48000,\"takes\":{},\"input_sample_bytes\":{input_bytes},\"prefix_samples\":{prefix_samples},\"output_samples\":{},\"output_capacity_bytes\":{},\"input_live_bytes\":{before},\"peak_live_bytes\":{peak},\"peak_additional_bytes\":{},\"retained_additional_bytes\":{},\"allocation_calls\":{allocations},\"requested_bytes\":{requested},\"peak_rss_kib\":{rss},\"fnv1a64_sample_bits_le\":\"{output_hash:016x}\",\"elapsed_ns\":{}}}",
        takes.len(),
        output.len(),
        output.capacity() * 4,
        peak - before,
        after - before,
        elapsed.map_or_else(|| "null".to_string(), |elapsed| elapsed.to_string())
    );
}

fn main() {
    let args: Vec<String> = std::env::args().collect();
    match args.get(1).map(String::as_str) {
        Some("validate") => validate(),
        Some("case") => run_case(
            args.get(2).expect("case name required"),
            args.get(3).is_some_and(|arg| arg == "--measure"),
        ),
        _ => panic!(
            "Usage: harness validate | case studio-1m|studio-5m|studio-30m|studio-25m-prefix-5m [--measure]"
        ),
    }
}

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

append_transcription_audio now mixes directly into the output’s new tail, removing one full-length f32 buffer.

• Caption mixing reuses its output tail to save memory.

Review checks were source-based; no tests or benchmarks were run.

_{Reviews (1) · Last reviewed commit: "perf(audio): reduce memory while prepari..." · Reviewed by Greptile}

[vercel]

@AksharP5 is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.