Repository navigation
feat(droid-control): point callouts at on-screen terminal text - #56
Merged
Merged
Conversation
A callout's `at` can now name text on a cast clip's screen. The composition replays the cast through avt (the emulator agg renders with), boxes the text's cells, and follows the text as the screen changes. A region target boxes part of any clip, and a point keeps the card alone. The card goes where it covers the least text over the callout's life, on a straight leader, or one with a single jog near a screen edge. An optional `key` adds keycaps. Callouts stage in (box, leader, card) and leave in reverse. Clips now carry their probed size and source cast. Window layout is computed in one place for both drawing and the clip-to-screen mapping. Co-authored-by: factory-ain3sh <ainesh@factory.ai>
A crop worked out in f32 can overrun the source image by a rounding error, which made fast_image_resize panic. A fractionally scaled card's shadow hit it. Co-authored-by: factory-ain3sh <ainesh@factory.ai>
Port the pointer element from the droids-view video: the frame dims to 50% around an accent box on the target, a straight accent line grows from the card to the box with a dot at each end, and the card is a solid panel under a 2px accent bar with a SHORTCUT tag over Geist Light text. Keycaps sit on a slab and press once, lit in the accent, as the card settles. Placement and text tracking are unchanged. The leader is one straight line, slanting only where the screen edge pushes the card off line, and cards above or below a narrow target shift so it drops straight. Co-authored-by: factory-ain3sh <ainesh@factory.ai>
Callout tracking and clip decoding share Stage::source_frame, so the two can't drift apart. frame_of moves to timing.rs on FPS. The key-name helper becomes cap_label (keycap draws), clip_rects is worked out once per callout, and the card's spacing and key release get named constants. Co-authored-by: factory-ain3sh <ainesh@factory.ai>
factory-droid-dev
Bot
force-pushed
the
ain3sh/droid-control-annotations
branch
from
October 11, 2026 05:56
15214a7 to
c0663f8
Compare
At 50% the rest of the screen was hard to read while a callout showed. Co-authored-by: factory-ain3sh <ainesh@factory.ai>
factory-ain3sh
approved these changes
Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What
Showcase callouts used to sit at hand-measured coordinates. They can now point at text on a terminal recording's screen.
"at": {"text": "Auto (High)"}draws a box on exactly those terminal cells, keeps it on the text while the screen scrolls or redraws, and joins it by a straight line to a card placed where it covers the least text. A region target ({x, y, w, h}) points the same way on any clip type, and a point ({x, y}) still centres a card on it."key": "shift+tab"addsShiftandTabkeycaps that press once as the card settles.Every callout now draws as the droids-view pointer element (dimmed frame, accent box, solid card with an accent top bar). Existing
at: {x, y}props therefore render differently; nothing else in a showcase changes (see Verification).Why
Hand-placed callouts drift. In the pointer-widget screenshot that prompted this, the box sits on "a task for a" instead of its target, and the line crosses the text it annotates. A callout that misses or covers its subject reads as a rendering bug.
How
The renderer replays the
.castthrough avt, the emulator agg renders with, so the box lands on the cells agg drew.Callout::newresolves each text target atton the shown cast clips, then tracks it frame by frame; the box glides over 8 frames when the text moves. Placement tries four sides at increasing distances and scores each against the text seen near the target over the callout's life (covered text ×4, text crossed by the line ×10, extra line length ×8). While a pointing callout shows, the frame dims to 25% through an even-odd path with a hole at each target. Above or below a narrow target, the card shifts left so the line drops straight.Visual Evidence
Captured on a 100x24 Droid-style cast rendered by
render-showcase.sh(video and stills atf4a0dfd).Demo. Watch the first callout land on the
Auto (High)footer and press its keycaps, then the second box followRELEASE_NOTES.md +12 -0up three rows as output streams in.d-callouts.mp4
Frame 252. The box sits on the
Auto (High)cells, the card sits below on a straight line, and theShift/Tabkeycaps are lit mid-press.Frame 400. The second box has followed its text up three rows; the card stays to the right without covering text.
Motion sheet. The first callout's entrance, line growth, keycap press, and release, six frames apart.
Repro Recipe
{ "effects": [ { "fx": "callout", "t": 3.7, "dur": 2.4, "text": "Cycle the autonomy level", "key": "shift+tab", "at": { "text": "Auto (High)" } } ] }scripts/render-showcase.sh --props props.json --output out.mp4 session.cast, with any cast that shows the target text att.Related Issue
None. Follow-up to #55 and the pointer-widget discussion in Slack.
Reviewer Guide
Diff shape: most new code is in two new files,
callout.rs(483 lines) andcast.rs(169). The rest is wiring, a layout refactor incontent.rs, and docs.Review depth: Standard. Placement scoring and tracking in
callout.rsdeserve the closest read.Commits:
7e854a6adds text and region targets, tracking, placement, and keycaps.dcd2e0cclampsresamplecrops that overran the source by an f32 rounding error (a fractionally scaled card's shadow hit it and panicked).59967ffrestyles every callout as the droids-view pointer; placement and tracking are unchanged.c0663f8is cleanup with no output change:Stage::source_frame, named constants,frame_ofintiming.rs.f4a0dfdlowers the callout dim from 50% to 25% so the rest of the screen stays readable.Read order:
props.rs:Target, the untaggedatunion (region, text, point), andkey.cast.rs: v2/v3 cast parsing,Replay::find(cell span of a string, wide characters included) andReplay::ink(lines of text).callout.rs:Callout::new(resolve and track),placement,pointer, thenOverlays::callouts(dim layer, box, line, card, keycaps).content.rs: window layout moves intowindows(), shared bycomposeand the newclip_rects(clip frame to screen, entrances and zooms included).Stage::source_frameis the one clip-frame mapping that both decoding and text tracking use.showcase.rs,clips.rs,stage.rs,main.rs: wiring. Clips carry their probed size and source cast.Open for pushback:
DIMincallout.rs), not a prop. The box and line already single out the target, so a per-showcase knob would mostly let droids pick contrast they can't judge. Thespotlighteffect keeps its owndimprop.Overlays::card).Risk & Impact
atgives a vaguer error. Becauseatis an untagged union, a bad value reports "data did not match any variant of untagged enum Target" instead of naming the bad percentage.t, so a typo can't produce a misplaced box.stage.rsused to read only v2width/heightand passed agg 120x36 for v3 casts. Both versions now read throughcast.rs(term.cols/term.rowsfor v3).Verification
Behavior verified @
f4a0dfd: on the 100x24 cast above, the first box lands on theAuto (High)cells with the card below the footer, and its keycaps light and press at frame 18. The second box follows its text up three rows with the card on a straight line. In a side-by-side render with two casts (at7e854a6), each text target resolves on its own clip.Regression: rendered the #55 demo with master's binary and this branch's. Lossless stills at frames 140, 240, 340, 420 (mid-zoom) and 600 are pixel-identical @
c0663f8. Frame 300 differs because of the point-callout restyle. A full-encode diff at7e854a6showed only the callout window changing; it was not rerun after the restyle.Not tested: text targets under
objectFit: coverorfill(both go through the sameclip_rectsmapping as the defaultcontain, but neither was rendered).Standard validators:
cargo fmt,cargo clippy --all-targets --release -D warnings,cargo test --release(45 passed) @f4a0dfd; plugin Python suite (7 run, 3 skipped) @7e854a6(no Python changed since).