Skip to content

feat(studio)!: unlink canvases, record and grade each canvas, and preview only what is seen - #54

Closed
kixelated wants to merge 7 commits into
mainfrom
quest/m0/multi-canvas
Closed

kixelated wants to merge 7 commits into
mainfrom
quest/m0/multi-canvas

Conversation

@kixelated

@kixelated kixelated commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

moq-dev/studio#42 drew further canvases, but every canvas switched scenes together, only the main canvas recorded, the Health panel graded only the main broadcast, and every canvas drew its preview each frame whether anyone looked or not. Multiple canvases asks for the rest.

Approach

  • Unlinking (moq-scene): preview, program, keyframe clocks, and the transition under way move from Compositor into a Switcher. The main canvas has one, which every linked canvas shares; a canvas whose Extra::transition is set is unlinked and gets its own, cloned from the main one's so it keeps showing what it showed. Compositor::switcher(canvas) / switcher_mut(canvas) return the one that drives a canvas; rename, retain, and on_air cover every switcher, so a scene on air on any canvas cannot be removed. Relinking cuts the canvas to the main canvas's program. Draft::set_extra_transition unlinks, relinks, or sets the unlinked canvas's own transition; Collection::takes(canvas) says which transition a canvas uses.
  • Preview cost: Compositor::set_previewed(&[canvas]) names the canvases that draw a preview; every other draws its program alone, and the overlay painter and keyframe clocks skip what no preview shows. Output::previewed() says whether the last render drew it. The UI previews only the canvas it shows, and switches the picker's canvas once that canvas's preview has been drawn, so it never flashes a stale one.
  • Recording per canvas: the recording's encoder is no longer a rung of the main canvas's ladder. It is a Line of its own (Encoders::recording) fed whichever canvas Extra::recorded names, or the main one; a canvas of another size reprobes, as any line does, so the recorder goes on in a new file. record.rs is untouched. Draft::set_recorded moves the pick; removing the recorded canvas sends recordings back to the main one. Stats::recording reports its encoder.
  • Health per canvas: each further canvas's catalog solicits feedback beneath its own path, and health::Canvases grades each one beside the main health::Broadcast, with its viewers, the relays' watch count, and its tracks. The uplink is graded on every canvas's tracks together (each further canvas's under its name/ prefix, as its grants already are). Encoder and viewer problems on a further canvas name it ("On canvas tall, ..."). Readers run on the main runtime, not the program clock's, and a canvas removed and added back keeps its grading.
  • UI: the canvas picker beside Preview now picks what Scenes switches too, and marks unlinked canvases. The Program panel gains Linked for a further canvas, and its transition picker edits that canvas's own transition while unlinked. The Output panel gains a Canvas picker for the recording and a row for the recording's encoder. The Health panel shows the shared relays and uplink, then each canvas's watching, tracks, and viewers, then the sources' tracks.

Measurements

compositor::bench::sources_against_canvases, release, Apple M-series GPU (Metal); 1080p main canvas plus 1080x1920 further canvases, each item a quarter canvas at half opacity, GPU waited on per frame. "all" draws every canvas's preview (as before this PR), "main" only the main canvas's, as when the studio shows the main canvas:

sources canvases dense, every preview dense, main preview sparse, every preview sparse, main preview
4 1 1.63 ms 0.64 ms 0.64 ms 0.64 ms
4 2 0.95 ms 0.79 ms 0.79 ms 0.72 ms
4 4 1.54 ms 1.11 ms 1.20 ms 0.92 ms
16 1 1.25 ms 1.27 ms 1.24 ms 1.24 ms
16 2 2.13 ms 1.69 ms 1.42 ms 1.31 ms
16 4 4.15 ms 2.61 ms 1.80 ms 1.51 ms
64 1 3.99 ms 3.93 ms 3.96 ms 3.99 ms
64 2 7.54 ms 5.78 ms 4.12 ms 4.05 ms
64 4 14.63 ms 9.30 ms 4.50 ms 4.22 ms

Skipping the previews nobody sees saves a third of the frame time with four dense canvases (14.6 ms to 9.3 ms at 64 sources), and a sparse canvas still costs about the same however many sources the collection has. One canvas is unchanged within noise (the 4-source, 1-canvas "every preview" cell is a warm-up outlier). A second run while other agents loaded the GPU was 2 to 5 times slower throughout and is not shown.

GPU copies and encoder load: unchanged per canvas. The recording still costs one encode session and one copy of the recorded canvas's program per frame while recording, and none otherwise; it no longer needs to be the main canvas.

Impact

  • Scene file: [canvases.<name>] transition = {...} unlinks a canvas, which then switches scenes on its own by that transition; recorded = true has recordings hold that canvas, on one canvas at most. Additive: existing files load unchanged.
  • Panels: Linked in the Program panel for a further canvas; Scenes and the transition picker act on the canvas Preview shows; the canvas picker marks unlinked canvases; a Canvas picker and the recording's encoder row in the Output panel; Health per canvas.
  • Outputs: each further canvas's catalog asks viewers for feedback under <broadcast>/<name>/viewers/. A recording can hold any canvas.
  • API (moq-scene): Switcher and Output exported; Compositor::{preview_scene, program_scene, preview_since, set_preview, cut, transition, transitioning} move to Switcher, reached by Compositor::switcher{,_mut}; new Compositor::set_previewed, Output::previewed, Extra::{transition, recorded}, Collection::{recorded, takes}, Draft::{set_extra_transition, set_recorded}. A compositor previews only its main canvas until told otherwise. No wire changes beyond the new feedback solicitation.
  • README.md updated (Panels, Canvases, scene file, encoder sessions, Recording, Health).

Choices made without asking

  • Linking is a scene-file property of the canvas, like the main canvas's transition, not a session toggle, so a reload keeps it.
  • Unlinking starts the canvas from the main canvas's scenes and transition, so nothing visibly changes until the creator switches it; relinking cuts it to the main canvas's program.
  • The recorded canvas can change while recording, as the recorded mix can; a canvas of another size goes on in a new file.
  • Destinations do not exist yet, so they pick their canvas in Studio destinations, whose Plan now says so.

Decisions

Settled by the maintainer on 2026-11-05:

  • ✅ Scene-file shape: keep as built. [canvases.<name>] transition = {...} unlinks a canvas, and recorded = true picks it for recordings, on one canvas at most. (Rejected: linked = false beside a transition ignored while linked; a top-level recording = "<canvas>" or a [recording] table holding canvas and mix.)
  • ✅ UI: keep the picker. The canvas picker beside Preview chooses what Preview, Program, Scenes, and Items act on, Linked sits beside Cut for a further canvas, and the picker marks unlinked canvases. (Rejected: a Scenes and Program pair per unlinked canvas.)

Alternatives

  • A recording rung on every canvas's ladder, each in its own recording broadcast: costs a probe per canvas and changes Recorder's API; one recording line fed the picked canvas does neither.
  • A top-level recording = "<canvas>" beside [mixes] recording, or a [recording] table holding canvas and mix: see Waiting.
  • Previewing every canvas and hiding the unseen ones in the UI: keeps paying for draws no one sees.

Follow-ups

None.

Validation

On macOS (Apple GPU, Metal):

  • just ci passes at 5ce216b (lint, both clippy feature sets, every test).
  • GPU tests, --run-ignored all with openh264: every compositor::tests::*, including the new an_unlinked_canvas_switches_on_its_own and only_a_previewed_canvas_draws_its_preview; engine::tests::the_program_records_without_a_relay, now recording a further canvas of another size into a new file; engine::tests::the_program_of_a_further_canvas_has_its_own_broadcast, now checking its catalog solicits feedback and Health grades it, then forgets it once removed; engine::tests::the_program_reaches_a_relay; setup::tests::times_every_canvas.
  • Unit tests: collection::tests::a_further_canvas_unlinks_and_records, edit::tests::the_recorded_canvas_moves_and_falls_back, health::tests::{the_encoder_falling_behind_is_named, tracks_are_graded_by_canvas_and_source, viewer_feedback_times_each_track}.

Review

Codex (quest review), gpt-6.1-sol, medium effort.

  • Round 0 (a stale session the branch name kept from moq-dev/studio#42), head ef68db4: 1 finding. A hidden Preview tab still drew its canvas's preview. Fixed in f26a7f3: only a Preview panel drawn last frame asks for one, and shown again it waits a frame rather than showing a stale picture.
  • Round 1 (new session), head f26a7f3: 1 finding. A further canvas's audio, listed by reference to the main broadcast, showed no Sent or Granted. Fixed in 5ce216b: a rendition listed by reference is graded under the main broadcast's track.
  • Round 2, head 5ce216b: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 7 commits November 5, 2026 08:31
Quest-Claim: 903e184462767cb6
…rade each canvas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	rs/moq-studio/src/ui.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…adcast writes it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Multiple canvases feat(studio)!: unlink canvases, record and grade each canvas, and preview only what is seen Nov 5, 2026
@kixelated
kixelated marked this pull request as ready for review November 5, 2026 17:50
@coderabbitai

coderabbitai Bot commented Nov 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 56 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0b8fcd66-1c10-4f25-9a7f-885f373800be
Commits

Reviewing files that changed from the base of the PR and between 8d8c5ea and 5ce216b.

Files selected for processing: 12

  • README.md
  • quest/m0/README.md
  • quest/m0/destinations.md
  • quest/m0/multi-canvas.md
  • rs/moq-scene/src/collection.rs
  • rs/moq-scene/src/compositor.rs
  • rs/moq-scene/src/edit.rs
  • rs/moq-scene/src/lib.rs
  • rs/moq-studio/src/engine.rs
  • rs/moq-studio/src/health.rs
  • rs/moq-studio/src/ui.rs
  • rs/moq-studio/tests/browser.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit 5ce216b. No actionable findings. Shared/independent switchers fit linked canvases, a single movable recording line avoids redundant encoder ladders, and preview gating appropriately reduces unseen GPU work.

Verification: GitHub-only static review of implementation, tests and relevant rendering/recording lifecycle paths. I did not run builds, tests, UI, GPU, relay, or performance execution.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed changes

  • Unlinked canvases (moq-scene): preview, program, keyframe clocks, and the in-flight transition move into a Switcher; linked canvases share the main one, and a canvas with Extra::transition gets its own switcher cloned from the main one. rename, retain, and on_air cover every switcher, and relinking cuts the canvas to the main program.
  • Preview gating: Compositor::set_previewed names the canvases that draw a preview; every other output draws its program alone, and the overlay painter and keyframe clocks skip unseen previews. The UI previews only the canvas it shows and advances the picker once that preview has been drawn.
  • Per-canvas recording: recording becomes its own Line (Encoders::recording) fed whichever canvas Extra::recorded names, falling back to the main canvas, and reprobing when the canvas size changes. Stats::recording reports its encoder, and Draft::set_recorded moves the pick and falls back on removal.
  • Per-canvas health: each further canvas solicits viewer feedback beneath its own path and is graded by health::Canvases; the uplink is graded across every canvas's tracks, encoder/viewer problems name the canvas, and readers run on the main runtime.
  • UI, docs, and tests: the canvas picker drives Scenes/Program/Items and marks unlinked canvases, the Program panel gains Linked, the Output panel gains a recording Canvas picker and encoder row, and Health is per canvas; README, scene-file docs, and tests are updated, and the multi-canvas quest is marked complete.

✅ No new issues found.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

@kixelated

Copy link
Copy Markdown
Contributor Author

Closing: the maintainer is replacing canvases with published scenes (each scene has a size and a Publish checkbox, a scene source draws another scene or the program, and mixes become audio scenes). The parts that survive (Health per published broadcast, a recording picking its scene, skipping an unseen preview) carry into the new quest plan.

(Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant