perf(recording): encode macOS studio audio with AudioToolbox AAC

[hridaya423]

Referred from this post by @cap.

Summary

On macOS, studio recordings encode the microphone and system audio tracks with FFmpeg's built-in aac encoder. In a profile of the GPUI app during a studio recording, that encoder was the largest unclaimed CPU cost in the recording engine. AudioToolbox's AAC encoder (aac_at, already in the bundled FFmpeg) does the same job for about 7.5x less CPU at the same output bitrate.

This PR makes FragmentedAudioFile use aac_at on macOS. The decoded audio timeline stays sample-identical to the current output, so A/V sync does not change.

Benchmarks

Encoder cost

cargo run --release -p cap-enc-ffmpeg --example recording-audio-benchmark -- <input> encodes the same input through FragmentedAudioFile with each encoder. Rounds alternate between the encoders. The table shows the median of 5 rounds. CPU is process user+sys time from getrusage. M4 Pro, 48 kHz input.

75 s of speech:

Channels	FFmpeg aac	AudioToolbox aac_at	CPU ratio	Bitrate
Mono	29.2 ms CPU per second of audio	3.7 ms	0.13	216 → 192 kbps
Stereo	48.5 ms	6.5 ms	0.13	273 → 253 kbps

60 s synthetic tone and noise (the default input without an argument):

Channels	FFmpeg aac	AudioToolbox aac_at	CPU ratio	Bitrate
Mono	24.5 ms	3.3 ms	0.13	173 → 197 kbps
Stereo	46.7 ms	6.3 ms	0.14	245 → 253 kbps

In the GPUI app

I ran the release cap-gpui binary with an isolated profile and CAP_GPUI_AUTO_RECORD=studio:…, recording with the microphone and system audio on. One build covers both cases. CAP_RECORDING_FFMPEG_AAC=1 selects the old encoder.

• Profile. /usr/bin/sample ran for 45 s in the middle of a 120 s recording. Samples inside FragmentedAudioMuxer::send_audio_frame fell from 5,684 to 403. In the AudioToolbox run, the encoder itself (AudioConverterFillComplexBuffer) took 364 samples. In the FFmpeg run, avcodec_send_frame took 3,229.
• Whole-process CPU. Over 25 s windows of alternating 45 s recordings, the median was 23.2% of a core with FFmpeg (8 runs) and 22.8% with AudioToolbox (7 runs). The run-to-run spread was about ±4 points because the machine was busy. This number therefore shows no regression. It does not measure the saving precisely; the encoder benchmark above does that.

On this macOS build, system audio loopback captured only silence in both variants. Silence costs almost nothing with either encoder. The in-app saving therefore comes from the microphone track. With non-silent system audio, the stereo row above also applies.

What changed

All changes are in crates/enc-ffmpeg.

• AACEncoder::init_for_recording. On macOS, this opens aac_at with aac_at_mode=abr, at 192 kbps for mono and 256 kbps for stereo. These targets match what the current encoder actually writes. ABR keeps silence small; CBR would fill silence up to the full bitrate. If aac_at fails to open, the function logs a warning and uses AACEncoder::init. CAP_RECORDING_FFMPEG_AAC=1 forces the old path. On other platforms, the function calls AACEncoder::init.
• AACEncoder::init is unchanged for every other caller (camera, DASH, segmented, Windows, export). It now delegates to a shared init_with_codec. That function selects the sample format and sample rate from what the codec supports, because aac_at reports no supported-rate list.
• Leading trim. Fragmented MP4 output keeps encoder priming in the decoded timeline. FFmpeg's aac primes 1024 samples and aac_at primes 2112. A direct switch would move all recorded audio about 22.7 ms later. AudioEncoderBase now drops initial_padding - 1024 samples (1088 for aac_at) before they reach the encoder. The value comes from the opened encoder, not a constant.
• End trim. aac_at accepts only full frames, so libavcodec pads the last frame and the packet durations include the padding. An EndTrimmedSink drops or shortens packets after the last input sample. This applies only to codecs without SMALL_LAST_FRAME or VARIABLE_FRAME_SIZE, so FFmpeg's aac output does not change.
• FragmentedAudioFile::init uses init_for_recording. A #[doc(hidden)] init_with_encoder_preference lets tests and the benchmark select an encoder without environment variables.

Correctness

New tests in mux/fragmented_audio.rs encode through both encoders and decode the result:

• A 1-sample click decodes at exactly sample 24_000 + 1024 with both encoders, in mono and in stereo.
• The recording pipeline fills capture gaps with synthesized silence and gives the muxer sample-count timestamps (output_pipeline/core.rs). A test copies that input shape: two clicks around a 1.5 s silence fill, with irregular frame sizes and the same nanosecond timestamp rounding. Both clicks decode at the same sample with both encoders. The decoded length and track end also match.
• For an input of 47,123 samples (not a multiple of the frame size), the track ends at the same sample with both encoders.
• On macOS, FragmentedAudioFile::init selects aac_at.
• 10 s of stereo silence stays under 64 KiB.

The input that this PR does not cover is a raw timestamp jump that reaches the encoder without a silence fill. With such input, audio after the jump can decode up to one frame away from the FFmpeg position, because the leading trim moves the frame grid. The presentation timestamps still match. The recording pipeline does not produce this input today.

Trade-offs

• Segment start. The first 1088 input samples (22.7 ms) of each audio file become encoder priming. This is the cost of keeping every later sample in its current position.
• Microphone file size. Quiet room audio now encodes at about 197 kbps instead of about 160 kbps, roughly 0.3 MB more per minute. Louder speech encodes about 10% smaller. Silence stays smaller than before.
• Overlap with perf(export): encode MP4 export audio with AudioToolbox on macOS #2623. That PR moves export audio to AudioToolbox and also touches audio/aac.rs and audio/base.rs. I used the same init_with_codec and EndTrimmedSink names so that the second PR to merge needs only a small rebase.

Verification

• cargo test -p cap-enc-ffmpeg: 157 passed.
• cargo clippy -p cap-enc-ffmpeg --all-targets -- -D warnings: clean.
• cargo check -p cap-recording: OK.
• I made real studio recordings in the GPUI app with both encoders. Both produced valid audio-input.m4a and system_audio.m4a files with matching durations, and the editor opened them.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings.

Summary

The PR switches macOS studio audio to AudioToolbox AAC, with FFmpeg AAC as a fallback.

• macOS studio recordings use AudioToolbox AAC when it can open.
• Recorded audio keeps its established start and end with either encoder.

_{Reviews (3) · Last reviewed commit: "chore(enc-ffmpeg): drop descriptive head..." · Reviewed by Greptile}

[vercel]

@hridaya423 is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[hridaya423]

fixed

[hridaya423]

@greptileai review

[hridaya423]

@greptileai review