Skip to content

feat(droid-control): point callouts at on-screen terminal text - #56

Merged
factory-ain3sh merged 5 commits into
masterfrom
ain3sh/droid-control-annotations
Oct 11, 2026
Merged

factory-ain3sh merged 5 commits into
masterfrom
ain3sh/droid-control-annotations

Conversation

@factory-droid-dev

@factory-droid-dev factory-droid-dev Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Description

What

Showcase callouts used to sit at hand-measured coordinates. They can now point at text on a terminal recording's screen. "at": {"text": "Auto (High)"} draws a box on exactly those terminal cells, keeps it on the text while the screen scrolls or redraws, and joins it by a straight line to a card placed where it covers the least text. A region target ({x, y, w, h}) points the same way on any clip type, and a point ({x, y}) still centres a card on it. "key": "shift+tab" adds Shift and Tab keycaps that press once as the card settles.

Every callout now draws as the droids-view pointer element (dimmed frame, accent box, solid card with an accent top bar). Existing at: {x, y} props therefore render differently; nothing else in a showcase changes (see Verification).

Why

Hand-placed callouts drift. In the pointer-widget screenshot that prompted this, the box sits on "a task for a" instead of its target, and the line crosses the text it annotates. A callout that misses or covers its subject reads as a rendering bug.

How

The renderer replays the .cast through avt, the emulator agg renders with, so the box lands on the cells agg drew. Callout::new resolves each text target at t on the shown cast clips, then tracks it frame by frame; the box glides over 8 frames when the text moves. Placement tries four sides at increasing distances and scores each against the text seen near the target over the callout's life (covered text ×4, text crossed by the line ×10, extra line length ×8). While a pointing callout shows, the frame dims to 25% through an even-odd path with a hole at each target. Above or below a narrow target, the card shifts left so the line drops straight.

Visual Evidence

Captured on a 100x24 Droid-style cast rendered by render-showcase.sh (video and stills at f4a0dfd).

Demo. Watch the first callout land on the Auto (High) footer and press its keycaps, then the second box follow RELEASE_NOTES.md +12 -0 up three rows as output streams in.

d-callouts.mp4

Frame 252. The box sits on the Auto (High) cells, the card sits below on a straight line, and the Shift/Tab keycaps are lit mid-press.

Frame 252: callout box on Auto (High) with Shift and Tab keycaps lit

Frame 400. The second box has followed its text up three rows; the card stays to the right without covering text.

Frame 400: callout box on RELEASE_NOTES.md +12 -0 with the card to its right

Motion sheet. The first callout's entrance, line growth, keycap press, and release, six frames apart.

Motion sheet of the first callout, six frames apart

Repro Recipe

{ "effects": [
  { "fx": "callout", "t": 3.7, "dur": 2.4, "text": "Cycle the autonomy level", "key": "shift+tab", "at": { "text": "Auto (High)" } }
] }

scripts/render-showcase.sh --props props.json --output out.mp4 session.cast, with any cast that shows the target text at t.

Related Issue

None. Follow-up to #55 and the pointer-widget discussion in Slack.

Reviewer Guide

Diff shape: most new code is in two new files, callout.rs (483 lines) and cast.rs (169). The rest is wiring, a layout refactor in content.rs, and docs.
Review depth: Standard. Placement scoring and tracking in callout.rs deserve the closest read.

Commits:

  1. 7e854a6 adds text and region targets, tracking, placement, and keycaps.
  2. dcd2e0c clamps resample crops that overran the source by an f32 rounding error (a fractionally scaled card's shadow hit it and panicked).
  3. 59967ff restyles every callout as the droids-view pointer; placement and tracking are unchanged.
  4. c0663f8 is cleanup with no output change: Stage::source_frame, named constants, frame_of in timing.rs.
  5. f4a0dfd lowers the callout dim from 50% to 25% so the rest of the screen stays readable.

Read order:

  1. props.rs: Target, the untagged at union (region, text, point), and key.
  2. cast.rs: v2/v3 cast parsing, Replay::find (cell span of a string, wide characters included) and Replay::ink (lines of text).
  3. callout.rs: Callout::new (resolve and track), placement, pointer, then Overlays::callouts (dim layer, box, line, card, keycaps).
  4. content.rs: window layout moves into windows(), shared by compose and the new clip_rects (clip frame to screen, entrances and zooms included). Stage::source_frame is the one clip-frame mapping that both decoding and text tracking use.
  5. showcase.rs, clips.rs, stage.rs, main.rs: wiring. Clips carry their probed size and source cast.
  6. Tests: cast parsing and cell lookup through wide characters, the three target shapes, placement avoiding text, straight vs slanted lines, gliding, agg's cell geometry.

Open for pushback:

  • The callout dim is a fixed 25% (DIM in callout.rs), not a prop. The box and line already single out the target, so a per-showcase knob would mostly let droids pick contrast they can't judge. The spotlight effect keeps its own dim prop.
  • Point callouts now use the pointer card (30px Geist Light, left-aligned, accent top bar) instead of the 24px centred pill (Overlays::card).

Risk & Impact

  • Existing point callouts look different. For the feat(droid-control): render showcases with fframes instead of Remotion #55 demo, only frames inside its callout window change.
  • Malformed at gives a vaguer error. Because at is an untagged union, a bad value reports "data did not match any variant of untagged enum Target" instead of naming the bad percentage.
  • Text targets fail loudly. The render stops when no shown cast clip has the text on screen at t, so a typo can't produce a misplaced box.
  • v3 casts now get their real size. stage.rs used to read only v2 width/height and passed agg 120x36 for v3 casts. Both versions now read through cast.rs (term.cols/term.rows for v3).

Verification

Behavior verified @ f4a0dfd: on the 100x24 cast above, the first box lands on the Auto (High) cells with the card below the footer, and its keycaps light and press at frame 18. The second box follows its text up three rows with the card on a straight line. In a side-by-side render with two casts (at 7e854a6), each text target resolves on its own clip.
Regression: rendered the #55 demo with master's binary and this branch's. Lossless stills at frames 140, 240, 340, 420 (mid-zoom) and 600 are pixel-identical @ c0663f8. Frame 300 differs because of the point-callout restyle. A full-encode diff at 7e854a6 showed only the callout window changing; it was not rerun after the restyle.
Not tested: text targets under objectFit: cover or fill (both go through the same clip_rects mapping as the default contain, but neither was rendered).
Standard validators: cargo fmt, cargo clippy --all-targets --release -D warnings, cargo test --release (45 passed) @ f4a0dfd; plugin Python suite (7 run, 3 skipped) @ 7e854a6 (no Python changed since).

A callout's `at` can now name text on a cast clip's screen. The
composition replays the cast through avt (the emulator agg renders
with), boxes the text's cells, and follows the text as the screen
changes. A region target boxes part of any clip, and a point keeps the
card alone. The card goes where it covers the least text over the
callout's life, on a straight leader, or one with a single jog near a
screen edge. An optional `key` adds keycaps. Callouts stage in (box,
leader, card) and leave in reverse.

Clips now carry their probed size and source cast. Window layout is
computed in one place for both drawing and the clip-to-screen mapping.

Co-authored-by: factory-ain3sh <ainesh@factory.ai>
factory-droid Bot and others added 3 commits October 11, 2026 05:55
A crop worked out in f32 can overrun the source image by a rounding
error, which made fast_image_resize panic. A fractionally scaled card's
shadow hit it.

Co-authored-by: factory-ain3sh <ainesh@factory.ai>
Port the pointer element from the droids-view video: the frame dims to
50% around an accent box on the target, a straight accent line grows
from the card to the box with a dot at each end, and the card is a
solid panel under a 2px accent bar with a SHORTCUT tag over Geist Light
text. Keycaps sit on a slab and press once, lit in the accent, as the
card settles. Placement and text tracking are unchanged. The leader is
one straight line, slanting only where the screen edge pushes the card
off line, and cards above or below a narrow target shift so it drops
straight.

Co-authored-by: factory-ain3sh <ainesh@factory.ai>
Callout tracking and clip decoding share Stage::source_frame, so the
two can't drift apart. frame_of moves to timing.rs on FPS. The key-name
helper becomes cap_label (keycap draws), clip_rects is worked out once
per callout, and the card's spacing and key release get named constants.

Co-authored-by: factory-ain3sh <ainesh@factory.ai>
@factory-droid-dev
factory-droid-dev Bot force-pushed the ain3sh/droid-control-annotations branch from 15214a7 to c0663f8 Compare October 11, 2026 05:56
At 50% the rest of the screen was hard to read while a callout showed.

Co-authored-by: factory-ain3sh <ainesh@factory.ai>
@factory-ain3sh
factory-ain3sh merged commit aa8be50 into master Oct 11, 2026
2 checks passed
@factory-ain3sh
factory-ain3sh deleted the ain3sh/droid-control-annotations branch October 11, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant