[pkajaba]

Software-host parity work from #702, piece two: picture in picture. SoftwarePiPSource already published the AVSampleBufferDisplayLayer and transport answers, but every host had to write its own AVPictureInPictureController and playback delegate. This moves that into the engine.

What it does

• makePictureInPictureController() on SoftwarePiPSource returns an AVPictureInPictureController around the display layer, or nil when PiP is unsupported on the device. SoftwarePiPSource.isSupported (new, public) answers the same question without building one.
• SoftwarePiPPlaybackDelegate (source.playbackDelegate) maps play, pause, skip and the time range onto the engine's transport. Skip is clamped to the VOD duration and its completion handler fires once the seek has settled. With no content loaded the range is invalid, so the window shows no scrubber.
• Invalidation. The engine calls invalidatePlaybackState() on the controller it made when state or duration changes, so the play/pause button and the timeline follow the engine.
• No main-thread blocking. The delegate callbacks have no thread contract, so they never hop synchronously to main. The source keeps a lock-guarded snapshot of its transport answers (refreshed at init, on every main-thread read, and on state or duration changes). Off main the getters return the snapshot, and setPlaying hops asynchronously.
• The picture fits the PiP window (all platforms: the layer shape; macOS: the window size). The system window mirrors the display layer's geometry, so while pictureInPictureActive is set and a bound AetherPlayerView shows a software session, the layer is laid out to the picture instead of to the view. On iOS (and on tvOS if a host sets the flag) that is the picture's aspect-fit rectangle, or the aspect-fill rectangle clipped by the view under resizeAspectFill; the view's layer is reshaped and clipsToBounds is toggled while PiP is active, and both are restored when it stops. On macOS the layer is laid out at the size of the system PiP window, so its own videoGravity fits the picture. That window is found with public AppKit only (a new visible panel whose aspect matches the picture, when exactly one does), and its size is followed through the public frame-change notification, with the render-size callback as a second trigger. The search stops watching window updates at the first update after five seconds, and it starts again when the picture's aspect becomes known or changes (a panel taken before the size was known is dropped if it does not match). Observers are removed and the layer is restored when PiP stops. Nothing is hidden, and if no window is found the shaped layout is kept.
• A session that ends clears the flag without asking for the background decision. stopInternal still resets pictureInPictureActive (a host that drops its controller with the source never sees a did-stop), but on tvOS that reset no longer asks for the background decision from inside the teardown; only a window that closes does.

API notes

• Additive. The host still owns the controller, sets pictureInPictureActive from its callbacks, and rebuilds the controller whenever $softwarePiPSource changes (a reload replaces the source, and the content source holds its delegate weakly).
• The engine invalidates only the latest controller it made.
• iOS in practice. tvOS AVKit does not evaluate sample-buffer content sources (FB9751461), so this is not expected to show a window there.
• docs/api.md, README.md and the CHANGELOG (Unreleased > Added) are updated.

Tested on Mac

• swift test, full run: 5107 swift-testing tests in 699 suites plus 815 XCTest (1 skipped), 0 failures.
• LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 swift test --skip-build --disable-xctest: 5107 tests pass.
• The PiP suites (source and delegate, window tracker, in-place item handover, background decision): 71 tests pass. The getter test that failed on the CI runner now runs its background half on a dedicated Thread; it passes alone three times.
• Generic tvOS, iOS and visionOS Simulator builds succeed (build-for-testing for tvOS and iOS, build for visionOS; visionOS is build only, PiP is not exercised there).
• macOS PiP window fit: unit tests for the fit maths and for the window tracker (window filters, resize reports, stop and release cleanup, the five second search budget, the aspect arriving after start), plus engine-level tests (the area drives the layer; the picture size arriving during PiP re-arms the match).
• tvOS reset: a test pins that the flag going false is a window close only when stopInternal is not the one clearing it. The settle itself runs on tvOS only, so that platform is compile-checked, not run.
• Checked by hand in a Mac test app with a software-host session at 16:9, 2.4:1 and 4:3: the layer matches the PiP window at the default size, follows programmatic window resizes, and returns to normal after PiP stops. (Run before the review changes; the tracker's search budget and re-arm were not part of that run.)
• The controller tests run where PiP is supported and are gated on that, rather than passing silently. They cover getters answering from a background thread while main is busy, the invalidation path, and that a released source is not retained.

Known issue: system bar (OS defect FB22411168)

On macOS the system draws an opaque bar over the top of the PiP window for sample-buffer sources whenever the window is taller than a height that depends on the video's aspect (roughly 350 pt for 16:9, 467 pt for 2.4:1, 200 pt for 4:3 at the default scale). The bar overlays the top of the picture; there is no public API to remove it or to read its height, so the picture is fitted to the whole window and the bar is left alone. At the default window size, 16:9 and 2.4:1 show no bar; 4:3 loses about 114 pt at the top. The player-layer route has no such bar. Filed as FB22411168.

Known limitations

• Under videoGravity = .resize the main view shows the picture letterboxed (undistorted) while PiP is active, because the window mirrors the layer's shape; it returns to stretched when PiP ends. resizeAspect and resizeAspectFill look the same in the main view.
• Any load on the software path ends PiP (next episode with auto-advance, and reloads such as an audio switch), because each load builds a new display layer. On iOS that means PiP plus auto-advance closes the window and continues audio-only in the background. Keeping the display layer across loads, as the native path hands the item over, is a possible follow-up.

Still needs a real window (macOS)

• Drag a corner of the real PiP window and confirm the picture re-fits live (only programmatic resizes were tried).
• A second PiP session in one run, and PiP on the next item.
• The return-to-app animation, and pause and play from the real PiP controls.
• A stream whose aspect changes mid-PiP: the adopted window is briefly dropped and found again, and the picture re-fits.

Still needs Apple TV / devices

Not yet tried on hardware, so this stays a draft until these pass with a host that uses the helper:

• Start PiP during software-host playback, including a Dolby Vision FEL title.
• Play and pause from the PiP window.
• Scrubber and duration are correct.
• Return to full screen on iOS, watching the return-to-app animation: the bound view's layer jumps from the fitted rectangle back to the full bounds when the flag flips, and clipsToBounds is restored.
• App to background and back.
• Stop playback while in PiP.
• Aspect-fill does not draw outside the player view during PiP.

[superuser404notfound]

Thanks for this, Pavel. Having the delegate and the controller in the engine is the right place for it, and the off-main snapshot design reads well: nothing in the delegate path waits on main.

Local run

Merged onto current main (2394ab2e). The only conflict is the README PiP / Subtitles rows, trivial: your PiP row plus main's VobSub addition in the Subtitles row. On that merge:

• swift test: 779 XCTest (1 skipped) + 4676 swift-testing, 0 failures.
• LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 swift test --skip-build --disable-xctest: 4676, 0 failures.
• SoftwarePiP|PictureInPicture filter alone: 34/34, three runs.

The red CI test

delegate getters answer from a background queue while the main thread is busy failed on the macOS runner (SoftwarePiPSourceTests.swift:163, the 5 s semaphore wait timed out, test took 27.5 s). It passes locally, alone and in the full run. The production path cannot block there (the getters only read the snapshot under its lock), so my reading is that the test's own block did not get scheduled: DispatchQueue.global() is width-limited, and on the 3-core runner the rest of the suite can hold that pool for longer than 5 s. Could you run the background half on a dedicated Thread instead (e.g. Thread { ... }.start())? That removes the dependency on pool capacity. Holding main for up to 5 s also stalls every other @MainActor test running in parallel, so a dedicated thread keeps that window short in practice too.

Questions

1. tvOS re-entrancy in stopInternal. The new pictureInPictureActive = false runs in the middle of stopInternal, and on tvOS the didSet calls settleBackgroundDecisionAfterPictureInPictureClosed() from there, while the teardown is half done. Since sample-buffer PiP does not present on tvOS (FB9751461) this is probably unreachable in practice, but a host can still set the flag during a software session. Could you either gate the reset to non-tvOS, or confirm that the background settle is safe to run re-entrantly at that point?
2. setPictureFit is not macOS-only. The PR text says iOS and tvOS are unchanged, which is true for the window tracker, but updatePictureFit() runs on every platform, so on iOS the bound view's layer is reshaped and clipsToBounds toggled while PiP is active. That is probably what you want, but please include it explicitly in the iOS device pass, especially the return-to-app animation (the layer jumps from the fitted rect back to full bounds when the flag flips).

Smaller points

• Tracker cost while unmatched. Until a window is adopted, the tracker observes NSWindow.didUpdateNotification app-wide. If no panel ever matches (out-of-process window on a future macOS, aspect outside tolerance), that observer stays for the whole PiP session and filters NSApp.windows on every update. Worth a cap (stop watching after N updates or a few seconds, keep the fit layout).
• Unknown aspect. With softwareDisplaySize still nil at start, aspectMatches accepts any size, so any single new visible NSPanel is adopted. Low risk, but re-arming the match once the size is known would be tighter.
• SoftwarePiPSource.isSupported is new public API and is not named in docs/ yet.
• The CHANGELOG entries do not mention the macOS window fit or FB22411168 (the README does).

Known limitation, fine for this PR

Any load on the software path ends PiP (next episode with auto-advance, and reloads such as an audio switch). On iOS that means PiP plus auto-advance closes the window and continues audio-only in the background. It is documented, so no change needed here; keeping the display layer across loads for a hand-over like the native path would be a good follow-up.

Happy to merge once the test is hardened and your device list has been run.

[pkajaba]

Thanks for the careful review and for running it on current main. Everything is in four new commits on top of the PR head, no history rewrite (e22c98b..ecfead5).

Red test (3808096). The getter test now runs its background half on a dedicated Thread, as you suggested, so it no longer depends on the capacity of the global pool. The main thread is still held for the wait, but the test only waits for the thread. The other PiP tests do not use a global queue while holding main. It passes alone three times and in the full run.

tvOS re-entrancy (f1c696e). I did not gate the reset to non-tvOS, because the reason for it holds on every platform: a host that drops its controller with the source never sees a did-stop, so a gone window must not count as live for the keepalive policy. Instead the reset inside stopInternal is marked as the teardown's own, and the tvOS didSet only asks for the background decision when a window really closes. Today the settle was only kept from running there by its softwareHost == nil guard, so this makes it explicit and not dependent on the order of the teardown. There is a test for the rule; the settle itself only runs on tvOS, so that platform is compile-checked.

Q2, iOS fit. Yes, intended: updatePictureFit() runs on every platform, so on iOS the bound view's layer is reshaped and clipsToBounds toggled while PiP is active, and both go back when the flag flips. I have fixed the PR text (it said iOS and tvOS were unchanged, which is only true for the window tracker) and added it to the device list, with the return-to-app animation spelled out.

Tracker cost (1c02b1c). The search for the window now has a budget of five seconds (PictureInPictureAreaTracker.searchBudget): the window is in place within about a second of the host reporting PiP active, so that leaves room for a slow start, and after it the window-update observer is removed and the fit layout stays. A new search starts when the picture's aspect changes or an adopted window goes away. Tested with the injectable windows and clock.

Unknown aspect (same commit). When the engine learns the picture size during PiP, the tracker gets the aspect: a window adopted without it that does not match is dropped (no area reported, the view's setting restored) and the match starts again, and a tracker that found nothing gets to look with the aspect. Tests cover the tracker and the engine wiring.

Docs and CHANGELOG (ecfead5). SoftwarePiPSource.isSupported is named in docs/api.md, and the macOS window fit and FB22411168 are in the CHANGELOG, README and docs/api.md (the README row did not have them on this branch, so I added them there too).

Known limitation. Agreed, it stays as it is for this PR, and I have put it into the PR text as a possible follow-up (keeping the display layer across loads).

Tests on my side after the changes: full swift test 5107 swift-testing tests plus 815 XCTest (1 skipped) pass, the strict pool run passes, and the tvOS, iOS and visionOS builds succeed. The device list is still to run; I will report back here when it has, and keep this as a draft until then.