Skip to content

feat(studio)!: a two-size program that follows the bandwidth grant, and local source tracks - #21

Merged
kixelated merged 30 commits into
mainfrom
quest/m0/encode-control
Nov 4, 2026
Merged

kixelated merged 30 commits into
mainfrom
quest/m0/encode-control

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Live encode control: the studio published one H.264 program rendition at a fixed bitrate, plus every captured source's own track, always encoding. The tracks claimed bitrate on the connection's bandwidth::Allocator through the catalog, but nothing followed the grant, a viewer on a slow link had no smaller rendition, and every source held a hardware encode session whether or not anything used it.

Decisions

Settled by the maintainer (PR comments, 2026-11-03 and 2026-11-04):

  1. Ladder. ✅ The program gets 2 sizes, and the half size encodes only while something watches it. Each source gets 1 rendition. Consumer GPUs cap encode sessions (GeForce NVENC at 8, a base M4 has one engine).
  2. Keyframes on join. ✅ Use moq main's moq_video::encode::Producer::demand() and capture's cut-on-join, available since chore(moq): pin moq fd07f5b for the audio capture split and mix router #27 pinned moq fd07f5b. No moq change.
  3. Sources. ✅ Not published to the relay by default. A source's own track exists locally and encodes only while something in the studio uses it. "Published independently" is its own quest (publish-independently), not built here.
  4. Replay arming (2026-11-04). ✅ "On program" counts as used: every source the program scene shows encodes its local track while shown, so a replay has its past. Off program a source encodes only while something reads its track (a recording, #18), and replaying a source with no history is refused with the reason. Audio sources have no own track yet (audio-source-tracks applies the "feeds the program mix" half), so the rule here covers picture sources.
  5. Recording (2026-11-04, coordinator relaying, after #18 landed). ✅ Per-source recordings read the local source broadcast; a recorded source counts as used; the program recording records the canvas size; replay keeps working.
  6. Recording encoder (2026-11-04, maintainer). ✅ Option (c): while a recording runs, the program recording gets its own full-quality canvas-size encoder, independent of the bandwidth grant, existing only while recording, as OBS's advanced output mode records with its own encoder. The relay ladder keeps following the grant. It counts in the Output panel's session count. Options not picked: (a) one encoder shared with the relay, whose file follows the grant and has a gap while the canvas size is disabled; (b) never disabling the canvas size while recorded.

This supersedes the earlier draft's 3-rung ladders for the program and every source, and its rewrite of the quest to "Keyframes on request".

Approach

  • Program ladder (rs/moq-studio/src/ladder.rs, engine.rs): the canvas size, plus half of it when that is at least 180 lines. Each rendition is a Gate on the program clock (its own Reservation at 0.07 bits per pixel, a moq_mux::rate::Control fed the grant every frame, and its track's demand) and an encode thread. The half size ranks one priority above the canvas size, so the allocator funds it first; the canvas size uses Policy::lower, so when the grant cannot fund it over the half size it is advertised enabled: false and resumes on a keyframe. The canvas size keeps today's rule (always, or watched with --on-demand) and holds its session while skipping; the half size closes its encoder, giving the session back, while unwatched. A failed half size drops alone; the canvas size keeps the video.
  • GPU scale: moq_scene::Scaler draws the program at half size before export (a 4-tap box filter), so zero-copy export still holds. It binds the program texture per frame, since the compositor replaces it on a canvas resize.
  • Live resize (feat(studio): rename and reorder scenes, set the fade time, and resize the canvas live #16) and drain (fix(studio): drain the program encoder before a pause, a resize, and at exit #23) carry over per rendition: a new canvas ends every rendition (each finishes its track), and one probe thread probes both sizes, the half size on the encoder the canvas size picked.
  • Keyframes on join: each rendition's track advertises refresh = 2 (capture's ratio) and cuts when Demand::wanted() passes the track's largest group, once per asked-for group (Joins). An encoder that refuses cut() at open (V4L2 without the control) logs and leaves groups to its GOP.
  • Grant following: each encode thread retunes with set_bitrate; an encoder that cannot retune stays at its ceiling and logs. The catalog no longer claims bandwidth (with_bandwidth), since each track reserves its own ceiling; the audio reserves its 128 kb/s explicitly.
  • Local source tracks (moq_scene::publish): Publish owns a broadcast and catalog of its own, on the program's clock, that is never announced. Video encodes only while its source is on program (Publish::show, fed from the on-air scenes plus the top layer, clones resolved to their capture) or its track is read, and drains its encoder, marking the gap, otherwise. Replay of a source reads that broadcast.
  • Sessions: moq_scene::publish::Sessions counts open encoders (program and sources); the Output panel shows it, and the encode work per rendition.
  • Replay: Take refuses a span that holds nothing ("nothing of the last N s is held"); the studio adds "name keeps a past only while it is on program or recorded". This also covers an --on-demand program nobody watched. A gap marker (one empty media frame, read with the rendition's container) is not a picture. A program replay takes the size whose pictures cover the most time inside the span (Take::covered, from each picture to the next, broken by a gap marker or a missing group), the larger on a tie, and a take marks what it holds enabled, so a rendition disabled since still exports.
  • Recording (merged from feat(studio): record the program, and optionally each source, to MP4 #18): Recorder::new takes the recording broadcast and the local source broadcast. A source's file reads its local track, and that read keeps the source encoding whether or not it is on program, so feat(studio): record the program, and optionally each source, to MP4 #18's Publish::on_demand is gone.
  • Recording encoder (decision 6): the ladder has a third rendition, Role::Record, at the canvas size in a recording broadcast announced only on a private origin no relay connection carries. Its gate reserves on an unlimited allocator, so it encodes at its ceiling whatever the grant, and only while a recording reads it, giving its session back otherwise. It has its own exporter, so a program frame is copied once more while recording. The mix on program (#32) is encoded a second time into that broadcast, on an audio-record thread beside audio-encode, following the same pick, only while a recording reads it, marking the gap when the last one stops so the next starts on the program clock. A failure of that copy shows in the Output panel as "recording audio off", leaving the published audio running. The program's file holds that canvas size alone. Without --connect, nobody can watch the published sizes, so the published canvas size opens no session; only the recording's encoder encodes the program.
  • Half size fails alone: a half size the encoder refuses at probe, or whose encoder fails, leaves the canvas size running; the reason shows in the Output panel (Load::error), not as the video's error, and the canvas size left alone stops disabling (Gate::alone). The Output panel shows only the sizes the program has (Stats::sizes).
  • Use without a picture: a source's capture loop races the next frame against Video::settle, which wakes on the shown set (a watch) or its track's demand whenever use and the encoder's state disagree. A source leaving use gives its session back at once, even while a still screen delivers no frame. A display or window going into use with its encoder closed reopens its capture, which starts with the picture as it is, and maps the new stream's timeline onto the program clock afresh, so a still screen put on program or recorded encodes its current picture without waiting for its screen to change. A camera keeps delivering, so it waits for its next frame.
  • Recording audio errors: a failure of the recording's audio copy goes to its own Stats::recording_audio_error, shown in the Output panel, so Health does not report the published audio stopped.
  • Health: a Granted column (each program track's share, or "off" for a disabled rendition); source tracks graded from the local catalog.

Measurements

Encoder sessions, by construction and checked in tests:

Encoder Sessions
The program's canvas size, live 1, or none without --connect
Its half size 1 while someone watches it
The recording's own canvas size 1 while recording
Each captured source 1 while on program or recorded

A live scene showing a camera, a display, and a window holds 4 sessions, 5 with the half size watched, and 6 while recording too, within GeForce's 8. main held 1 + one per captured source always (4 for the same three, whatever is shown); the earlier draft's 3-rung ladders would have held up to 12. the_program_reaches_a_relay asserts 1 session before the half size is watched, 2 once it is, 2 while recording (the canvas size and the recording's), and 1 after a resize under 360 lines; the_program_records_without_a_relay asserts 0 sessions before recording and 1 while recording.

Encode wall time per frame on this Apple Silicon Mac (release build, 150 frames of a moving I420 pattern, 1080p30 sources, five encoders open at once on one thread; the host was under load from other agents, so read the ratios):

Encoder Program 1920x1080 Program 960x540 Each 1080p source on program All five
VideoToolbox (includes the I420 upload) 7.82 ms 3.01 ms 7.85 ms 34.4 ms
openh264 2.54 ms 0.67 ms 2.52 ms 10.8 ms

The half size adds about 38% (VideoToolbox) or 26% (openh264) to the program's encode while watched, nothing while not. Three visible 1080p30 sources plus the watched program sum to about one frame period of VideoToolbox time, so on a single-engine Mac, sources on program are what approaches the encoder's throughput, not the ladder.

Impact

  • Breaking for viewers composing scenes: the program catalog no longer lists sources under dev.moq.studio, and no source track reaches the relay. Migrate by reading the program, or wait for Published independently.
  • The program catalog lists two video renditions (one for a canvas under 360 lines), each with refresh; the canvas size may be enabled: false while the grant cannot fund it.
  • Video bitrate follows the first relay connection's estimate.
  • --on-demand applies to the canvas size; the half size is always on demand. Its help says so.
  • A source replays only from while it was on program or recorded, and encodes while recorded.
  • Recording: the program's file holds the canvas size from the recording's own encoder, at full quality whatever the uplink, costing one more encode session and one more program copy while recording; a recorded source encodes while recorded.
  • Output panel: encoder sessions, and per-rendition work. Health panel: a Granted column.
  • Public API (moq-scene): Publish::new(clock, kind, sessions) replaces new(broadcast, catalog, bandwidth, kind), and Publish::on_demand (from feat(studio): record the program, and optionally each source, to MP4 #18) is gone, every source being on demand; new Publish::{consume, catalog, show}, Sessions, Scaler, Take::covered; Encoding::new takes Sessions, and gains watched, demand, cut, set_bitrate, set_enabled, and a public discontinuity; Encode gains set_bitrate. No wire change beyond the catalog contents above.
  • README: Output, Source tracks, Replay, and Health sections; quest encode-control.md is finished and deleted.

Choices made without asking

  • The half-size floor of 180 lines, so a tiny canvas never asks a hardware encoder for a size under its minimum.
  • Guided setup still prices and times the canvas size alone: the half size costs a quarter of the bits and runs only while watched, and the grant funds it before the canvas size.
  • The refresh ratio is advertised on every program track, since whether the encoder cuts is known only once it opens; an encoder that cannot cut leaves joiners on the cached group, as with no ratio.
  • Source tracks advertise no refresh: their readers are local.
  • "Read" counts any subscriber to a source's local track, so a replay's own momentary copy does not start the encoder for long.

Validation

  • Ported to moq ffcb7cd (#35): no catalog opts out of HLS any more, a nested source catalog clears its media timing, and a reservation names its join interval (with_interval). The ladder tests pass under the new zero-max-age track default.
  • just ci on c95e8e4 (lints with default and all features: capture, openh264, cef; 187 tests): passed. An earlier run failed bun run check on the join page from a stale node_modules (an npm @moq/net 0.4.7 left from before the merge of feat(studio): renew a joined guest's token while they stay in the room #28); a clean bun install fixed it, and main passes the same check.
  • GPU tests on this Mac with openh264, on 693188e (and 4 more runs each, all passing): engine::tests::the_program_reaches_a_relay (both sizes through a relay, sessions 1 → 2 → 1, a recording beside it on its own encoder, a clip) and engine::tests::the_program_records_without_a_relay (no session until recording, then only the recording's; a resize going on in a second file; quitting mid-recording): passed. The latter failed 4 of 6 runs while the recording broadcast's catalog went away with the audio thread at quit; held in Program, it passed 14 of 14.
  • New tests: ladder::tests (sizes, priorities, gate follows grant and viewers, a gate left alone never disables, one cut per join), publish::tests::a_video_encodes_while_used (sessions only while on program or read), publish::tests::a_video_leaving_program_closes_its_encoder_at_once, replay::tests::{a_take_of_nothing_is_refused, a_take_of_a_gap_holds_no_pictures, coverage_counts_time_not_pictures}, record::tests::the_program_records_its_canvas_size, the Granted and enabled columns in health::tests::source_tracks_are_graded_by_source.

Follow-ups

  • A longer program GOP now that joins cut keyframes: measure the bitrate saved before picking one.

Review

Reviewer: Codex, gpt-6.1-sol, medium effort (quest review --base origin/main --new), a new session after the hand-resolved merge of #32 (mixes). Round 1 reviewed head c95e8e476b22cf62b9b19e6f680d730d3c1d106b and reported no findings. The merge kept #32's audio and audio-encode threads, gave the recording's copy its own audio-record thread following the same mix pick, and moved quest/m0/record-mix.md off the deleted encode-control.md onto the recording broadcast.

Previous session, after the hand-resolved merge of #29, #31 and #34: Codex, gpt-6.1-sol, medium effort (quest review --base origin/main --new), a new session after the hand-resolved merge of #29, #31 and #34. Round 5 reviewed head bb666eabb5b42578f64927fa2de9eb789ecdcfd3 and reported no findings; e103f9d since is a conflict-free merge of main (#36).

Previous session, after the merge of #18 (design change): round 5 reviewed b874149 with no findings.

  • Round 1 (22baf8a, the merge of feat(studio): record the program, and optionally each source, to MP4 #18 and the online review fixes): medium, replay.rs: a size disabled in the middle of the span counted as covering it. Fixed in 6a994eb: a take counts the pictures inside the span.
  • Round 2 (6a994eb): no findings.
  • Round 3 (2f71465, the recording encoder in 8cfa730 and the moq ffcb7cd port): medium, engine.rs: the recording's audio copy skipped writes while no recording read it but kept its anchor, so a later recording's audio would run behind its video by the idle time. Fixed in 71b3248: the copy marks a discontinuity as the last recording stops, re-anchoring on the next frame.
  • Round 4 (4c91769, also merging feat(scene): browser sources share one persistent profile, and reload #29): no findings. Design checkpoint at round 4: the approach holds (the ladder, local source tracks, and the recording's own encoder as decided); nothing remains.
  • Round 5 (b874149, the online review's replay ranking moved to time covered): no findings.
  • Online review on 6a994eb (review), answered here: replay ranked by picture count, fixed in b874149; static-source arming fixed in d571fd0 and 07c8dbf.
  • Online reviews on 7065cb9, answered in this comment and inline: the half-size probe, the threshold after the half size fails, marker-only replays, the half size's failure reported as the video's, and the Output panel's columns are fixed in 22baf8a; a source leaving program without a frame is fixed, and putting a still source on program without a new frame waits on moq (follow-up above).

Earlier session, before the merge of #18:

  • Round 1 (0dc4b60):
    • medium, replay.rs: a program replay took the first catalog rendition, so with the canvas size disabled it saved audio only. Fixed in ce95160: one target per size, largest first, taking the first take that holds video.
    • medium, health.rs: the local broadcast's track names can repeat the program's, so a source row picked up the program's grant and viewer feedback. Fixed in ce95160: grants and feedback apply to program rows only; the test now shares a name.
  • Round 2 (e3b5b8c, also merging feat(studio): renew a joined guest's token while they stay in the room #28): medium, replay.rs: a take copied enabled: false, which the MP4 export filters out. Fixed in 7065cb9: a take marks its renditions enabled.
  • Round 3 (7065cb9): no findings.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 4 commits November 2, 2026 20:16
Quest-Claim: fd8ff08aeb866861
The program and each captured source encode a ladder (the picture's size,
then 2/3 and 1/3 of it); each rung reserves its ceiling, follows its grant
through moq_mux::rate, disables when it cannot be funded, and encodes the
smaller sizes only while watched. Catalogs no longer claim bandwidth, so no
track is counted twice; the Health panel shows each track's grant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	rs/moq-studio/src/engine.rs
@kixelated kixelated changed the title Live encode control feat(studio)!: simulcast ladders that follow the bandwidth grant Nov 3, 2026
@kixelated

Copy link
Copy Markdown
Contributor Author

Maintainer decisions (relayed 2026-11-03):

  1. Fewer encodings. The program publishes two sizes: full, plus half size, which encodes only while watched. Sources publish only their full-size track, with no ladder. Encoder sessions are scarce: GeForce drivers cap concurrent NVENC sessions at 8, and a base M4 has one encode engine. Camera + display + program should stay at no more than 4 sessions.
  2. Keyframes on join come from moq main. moq_video::encode::Producer::demand() already exposes a joiner's wanted group on main, and capture there cuts on join with a refresh ratio. The studio moves to moq main (just upgrade-moq, its own PR) and then uses that, instead of a new moq change. This PR iterates once that bump lands.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Contributor Author

Maintainer decision 3 (relayed 2026-11-03): publishing a source's own track is optional and off by default. Each source gets a setting, off by default and saved in the scene file. A source publishes its full-size track only when that setting is on, so by default the program is the only encoding. This replaces #3's always-on per-source tracks and fits the session budget above. Browser sources keep their own rule: an opt-in URL in the layout, never a video track.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Contributor Author

Maintainer decision 3, revised (relayed 2026-11-03), replacing the earlier comment: sources are not published to the relay by default. A source's own track still exists locally. It encodes only while something in the studio uses it (an isolated source recording, a replay of that source) and is never announced to the relay. Publishing a source independently, so that a service can move a webcam or overlay itself, becomes a "Published independently" checkbox that also leaves the source out of the composited program. That goes in the layout quest (quest/m0/compose/layout.md) at low priority, since no such service exists yet. This PR only stops publishing sources by default; the program ladder stays at two sizes as in decision 1.

(Written by Claude Opus 5.5)

kixelated and others added 6 commits November 3, 2026 22:39
Takes main's tree whole: the ladder is reworked onto the current code in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd local source tracks

The program publishes the canvas size and half of it, the half size encoding
only while watched; each follows its share of the connection's bandwidth and
cuts a keyframe for a joining viewer. Sources keep their own tracks in a
broadcast the studio never announces, encoding only while on program or read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n program

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title feat(studio)!: simulcast ladders that follow the bandwidth grant feat(studio)!: a two-size program that follows the bandwidth grant, and local source tracks Nov 4, 2026
kixelated and others added 3 commits November 3, 2026 23:36
…p source rows off relay feedback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ables them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 4, 2026 07:48
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 55 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 48c08a26-b755-4e32-8a62-09e1de6f4c1f
Commits

Reviewing files that changed from the base of the PR and between acd4cd7 and c95e8e4.

Files selected for processing: 17
  • README.md
  • quest/m0/README.md
  • quest/m0/encode-control.md
  • quest/m0/publish-independently.md
  • quest/m0/record-mix.md
  • rs/moq-scene/src/lib.rs
  • rs/moq-scene/src/publish.rs
  • rs/moq-scene/src/replay.rs
  • rs/moq-scene/src/scale.rs
  • rs/moq-scene/src/source.rs
  • rs/moq-studio/src/engine.rs
  • rs/moq-studio/src/health.rs
  • rs/moq-studio/src/ladder.rs
  • rs/moq-studio/src/main.rs
  • rs/moq-studio/src/record.rs
  • rs/moq-studio/src/replay.rs
  • rs/moq-studio/src/ui.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 7065cb9be8e7130b8ec33bcbaf5fab3c0e69f0a7.

  • [P2] Keep the canvas rung if the optional half-size probe fails (rs/moq-studio/src/engine.rs:1163–1170). A backend can open the canvas size but reject the half size. The ? discards the successful full-size probe, so startup fails or a resize leaves all video stopped until the next canvas change, rather than dropping only the optional rung. Preserve the full-size probe when the half-size one fails and test that case.
  • [P2] Clear the fallback threshold after the half-size rung fails (rs/moq-studio/src/engine.rs:1096–1101, 1225–1228). Removing the failed half-size rung leaves the canvas gate's Policy::lower set to its ceiling. On a constrained probed connection the sole remaining rendition can stay disabled below that threshold, leaving no video. Reconfigure the surviving gate as the only rung and test half-size encoder failure under a low grant.
  • [P2] Do not select marker-only video for a program replay (rs/moq-scene/src/replay.rs:270–311; rs/moq-studio/src/replay.rs:92–102). A disabled canvas rendition emits an empty-media discontinuity frame. copy_held counts that group as video, so the largest target wins even when it holds no picture and the half-size target has the requested recent media. Require actual video media when choosing a target, and test a replay wholly inside the disabled interval. More generally, prefer coverage of the requested interval over a larger rung with only older partial history.
  • [P2] React to source-use changes without waiting for a capture frame (rs/moq-scene/src/publish.rs:514–519; rs/moq-scene/src/source.rs:504). Use is checked only inside Video::encode, which runs after the next capture frame. ScreenCaptureKit can stop delivering frames when a display is unchanged. A static source taken off program can retain its encoder indefinitely; one captured off program and then shown can fail to start replay history until its pixels change. Drive the use transition independently of incoming frames and make the current picture available when arming it; test both transitions with a paused capture.

Verification: GitHub-only static review of the ladder, source/replay paths, pinned moq behavior and tests. These scenarios are source-traced, not reproduced; I did not run GPU, capture, replay or relay tests.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The half-size ladder promises to "drop alone" — the failure handling doesn't fully deliver that. Two paths still let a half-size problem take down the whole output or misreport it as a stopped program.

Reviewed changes

Reviewed all 15 files and 13 commits over 18f2520..7065cb9 (base main at 18f2520). I read the raw diff plus surrounding code in engine.rs, publish.rs, source.rs, collection.rs, replay.rs, health.rs, ui.rs, and scale.rs, and checked the Exporter/rate::Control contracts against the pinned moq SHA fd07f5b7. I could not build or run the tests: the moq/ submodule is empty in this checkout, so the API expectations are verified by reading the pinned source rather than by compiling.

The design is a real improvement — the ladder now follows the grant, the canvas rung keeps its session while skipping, and run_program correctly isolates a runtime half failure by dropping only that rung. The comments below are the places where the half-size story leaks out of that isolation, plus one display issue for canvases that have no half size at all.

Startup abort on half-size probe

publish() does Probe::new(gpu, canvas).await?, so a canvas whose encoder accepts the full size but rejects the half size fails the engine start entirely, even though the body promises the half drops alone. Depending on your intent here (start anyway with one rung, or surface the error but keep the canvas publishing), the fix differs; I flagged it inline rather than prescribing one.

fail_video for a non-program rendition

The fail closure in encode routes both rungs through Stats::fail_video, whose contract is "why the output stopped". For a half-size rung the canvas keeps publishing, yet Health reports "The output stopped" and problems() skips the encoder-behind/drop analysis because it checks local.failed.is_none(). That's the same misreport the isolation was meant to avoid, just in the health surface instead of the ladder.

Half-size column with no half size

ladder::sizes returns a single size when the half's shorter side is under MIN_SIDE, leaving stats.renditions[1] at all zeros, but the Output panel renders a fixed [Load; 2] with a "Half size" heading. Small canvases therefore show a rendition that doesn't exist.

Nothing else in the diff needs a change: the hold: canvas behavior, the Encoder::skip fallback without cuts, and the session/discontinuity accounting all match what the PR body claims.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using big-pickle (free) | 𝕏

Comment thread rs/moq-studio/src/engine.rs Resolved
Comment thread rs/moq-studio/src/engine.rs Resolved
Comment thread rs/moq-studio/src/ui.rs Resolved
kixelated and others added 2 commits November 4, 2026 03:17
Recordings read each source's track from the local broadcast, where reading it
keeps the source encoding, and the program's file holds its canvas size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers, and a source leaving program closes its encoder at once

- A half size the encoder refuses at probe leaves the canvas size running, and
  its failure is reported apart from the video's; the canvas size left alone
  stops disabling below it.
- A replay takes the size whose pictures cover the most of the span, counting
  no gap marker as a picture.
- A source's encoder closes as it leaves program, without waiting on a capture
  frame a still screen never delivers.
- The Output panel shows only the sizes the program has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head b874149 against base 3a4171a.

  • [P2] Count the final held picture when ranking coverage (rs/moq-scene/src/replay.rs:327–365). The new score sums only intervals between pictures and never the last picture's duration to the requested end, although replay keeps displaying that picture until the span ends unless a gap/end marker clears it. For two gap-free ten-second rungs starting together, a 15 fps canvas scores about 9.933 s while a 30 fps half-size rung scores 9.966 s. The half-size rung therefore wins even though both cover the full replay, defeating the intended largest-rung tie-break whenever the canvas drops more frames. Account for the held picture up to the appropriate end or gap boundary and test two complete unequal-frame-rate rungs as well as the partial-history case.

The change fixes the earlier picture-count example where the higher-rate rendition stops partway, but coverage ranking still has this boundary bias. The prior private-recording replay finding remains unchanged.

Verification limits: GitHub-only static review of scoring and replay display paths; I did not run GPU or replay tests.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ One minor issue in the recording-only audio path, not covered by the existing reviews.

Reviewed changes

Reviewed PR #21 at b874149, focusing on the replay coverage rewrite and the recording-only encoder split. The replay boundary issue at this head is already reported by OpenAI review 5772439939, and the private-recording replay candidate issue by 5772339781; I left both there rather than duplicate them.

  • Rank a take by covered time — Take::covered sums the time between kept pictures, resetting at gap markers and missing group sequences, and take() compares coverage before the larger size.
  • Separate the recording audio copy — run_audio keeps the program and recording producers independent, restarts the recording copy on the program clock after a gap, and aborts only the copy that failed.
  • Private recording broadcast — the program file reads a canvas-size encoder in a broadcast no relay sees; the pinned moq_mux catalog releases a rendition's entry on drop.

The inline note is the only new finding: a recording-copy audio failure currently reports the live program output as stopped.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using big-pickle (free) | 𝕏

Comment thread rs/moq-studio/src/engine.rs Resolved
kixelated and others added 2 commits November 4, 2026 09:24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	README.md
…'s to report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head d0be9dd3b02659cfebfa1327cf77592d8b262661 against base 501fbb585c3873cf0da768327084ce71d39a178c. Recording-audio finish, discontinuity and write failures now report separately from live program audio, and the Output panel displays the recording warning. No new actionable finding in this update or the overlay integration.

The previously reported final-held-picture coverage bias and the omission of private recording history from program replay remain unchanged. The still-source arming limitation also remains tracked as a follow-up.

Verification: GitHub-only static review; I did not run tests, record media, or exercise GPU/audio devices.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed b874149..d0be9dd: the recording audio copy's failures now report separately from the published program audio, plus the merge of origin/main.

  • Separated the recording audio copy's failure — run_audio's finish, discontinuity, and write paths call the new Stats::fail_recording_audio, so a recording-only audio failure no longer marks the live program output stopped; the Output panel shows "recording audio off". The published producer still uses fail_audio.
  • Merged origin/main — the overlay module, README, and quest updates came in with the merge; the reordering around Stats, Ladder::count, and the encode fail closure is behaviorally identical.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

…ture

A display or window delivers nothing while its screen stands still, so one put
on program or recorded with its encoder closed would encode nothing until the
screen changed. Going into use now reopens its capture, which starts with the
picture as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head d571fd096befa823e6094b7bacccd69535a03585 against base 501fbb585c3873cf0da768327084ce71d39a178c.

[P2] Re-anchor the source clock after reopening capture (rs/moq-scene/src/source.rs:571-577; rs/moq-scene/src/publish.rs:515-522). Each capture open creates a fresh private timestamp epoch, but the existing Video keeps the offset from the old stream. After a display has encoded for a while, goes off program and is later reopened, its new near-zero timestamp maps back near the original start. The container can reject the resumed keyframe as a timestamp rewind, aborting the local track; even a first late use can stamp its history in the distant past. Reset the capture-to-program mapping when reopening, and test a use/off/use cycle with elapsed time and a fresh capture timeline.

The earlier final-picture coverage and private-recording replay findings remain unchanged. This update addresses the separate still-source arming limitation by reopening capture, subject to the clock issue above.

Verification: GitHub-only static review of the changed paths and pinned capture/container code; no device, replay or recording test was run.

(Written by review (OpenAI))

…ck afresh

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 07c8dbf5088649d9649fa1a226fcbe0bad5f7446 against base 501fbb585c3873cf0da768327084ce71d39a178c. The reopened-capture timestamp finding is addressed: a successful reopen clears the old mapping before the next frame, and the new test advances the program clock and starts the capture timeline again. No new actionable finding in this update.

The previously reported final-picture coverage bias and omission of private recording history from program replay remain unchanged.

Verification: GitHub-only static review; the regression test simulates reopening rather than exercising a capture device. I did not run tests or record media.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed d0be9dd..07c8dbf, the still-source reopen and its timeline re-anchor.

  • Reopened a still display or window capture on use — Video::settle now wakes on either direction of a used-vs-open disagreement, and source.rs::device drops and reopens the capture for Display/Window sources, so a static screen's local track starts with the current picture instead of waiting for a change. Cameras still arm on their next frame, and the settling flag keeps the loop from spinning.
  • Re-anchored a reopened capture's timeline — Video::reopened clears epoch, so the fresh capture timeline maps onto the program clock now rather than the old start (the timestamp rewind OpenAI reported for d571fd0). The test asserts the first picture after a reopen is stamped at or after the current clock.

The prior recording-audio finding is confirmed resolved by d0be9dd (recording-copy failures report on recording_audio_error, not the live program's fail_audio); the earlier replay-coverage and private-recording-replay items stay tracked in the OpenAI reviews and are unchanged by this delta.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

kixelated and others added 2 commits November 4, 2026 10:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	rs/moq-scene/src/lib.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed 07c8dbf..bb666ea, the merge of origin/main's slide/wipe/move transition refactor (#33) into the branch.

  • Merged origin/main's transitions — the sole conflict, rs/moq-scene/src/lib.rs's module docs, resolves to describe both the new Transition and the branch's Scaler; ui.rs picks up the Transition/Direction picker, and engine.rs takes only comment-wording changes ("fading out" → "leaving in a transition").
  • Confirmed the half-size path still matches the compositor — Compositor::program() still returns a COPY_SRC | TEXTURE_BINDING Rgba8Unorm texture and Scaler::scale re-binds it per call, so the merge's new incoming/outgoing targets do not affect the scaled rendition. Compositor::on_air() now also covers a scene leaving a transition, which correctly keeps its sources encoding while still visible.

The earlier opens remain tracked in the OpenAI reviews and are unchanged by this merge: final-held-picture coverage bias (5772439939), private recording history omitted from program replay (5772339781), and static-source arming as a moq follow-up.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

kixelated and others added 2 commits November 4, 2026 10:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	README.md
#	rs/moq-studio/src/engine.rs
#	rs/moq-studio/src/main.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pullfrog

pullfrog Bot commented Nov 4, 2026

Copy link
Copy Markdown

Your OpenCode Zen account is out of credit. Top it up, then re-trigger Pullfrog.

OpenCode Zen's response
provider error: Free usage exceeded, subscribe to Go

Top up OpenCode Zen →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using big-pickle (free)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: c95e8e4, against base acd4cd7.

No new actionable finding in the hand-merged audio-mix integration or recording-mix plan. Published and recording audio now consume the selected mix independently; recording demand, gap handling and separate failure reporting are preserved. The capture-clock re-anchor fix is unchanged.

Two prior findings remain open:

  • Final-picture coverage bias: rs/moq-scene/src/replay.rs:330–370 still omits the last held picture's interval, so a complete higher-frame-rate half-size rendition can outrank the complete canvas-size rendition. Include that final hold when comparing coverage.
  • Private recording history omitted from replay: rs/moq-studio/src/engine.rs:465–505 still builds program replay targets only from the published broadcast. Include retained recording history so an on-demand program with no viewers can replay what its recording captured.

Overall direction: the merge preserves the intended audio split; resolve the two replay findings before calling the replay behavior complete.

Verification limits: GitHub-only static review and base/delta comparison. I did not run GPU, audio, recording or replay tests, or execute quest context; I read its pinned source instead.

(Written by review (OpenAI))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant