diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bfed7755..9da0a2564 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,36 @@ the public-API contract. ## [Unreleased] +### Added + +- **The software host's PiP source now builds its own controller.** + `SoftwarePiPSource.makePictureInPictureController()` returns an `AVPictureInPictureController` + around the display layer (nil when PiP is unsupported), with a `SoftwarePiPPlaybackDelegate` + serving play, pause, skip and the time range from the engine's transport, as the AVPlayer path + gets from AVKit. Skip is clamped to the VOD duration, and the window's playback state is + invalidated when the engine's state or duration changes. A host no longer re-implements the + delegate; it still owns the controller, rebuilds it when `softwarePiPSource` changes, and sets + `pictureInPictureActive`. +- **Ending a software session closes its PiP window.** The engine stops the PiP of the controller it + made, before the host stops, and resets `pictureInPictureActive` to false. A host with its own + controller closes PiP when `softwarePiPSource` becomes nil. The software path builds a new display + layer per load, so any load ends PiP there (no hand-over as on the native path). +- **A bound `AetherPlayerView` shapes the software layer like the picture while PiP is active.** A PiP + window mirrors the layer's own geometry, so the layer takes the picture's aspect-fit rectangle + (`resizeAspect`, `resize`) or its aspect-fill rectangle clipped by the view (`resizeAspectFill`), and + returns to the full view afterwards. A `videoGravity` change during PiP re-applies it at once. A host + that mounts `softwarePiPSource.layer` in its own view sizes the layer itself. +- **macOS: the software layer fits the system PiP window.** While PiP is active the layer is laid out at the + window's size (found through public AppKit and followed on resize; the picture-shaped layout stays when no window is + found), so the layer's `videoGravity` fits the picture. The system draws an opaque bar over the top of that + window for sample-buffer sources, which no public API removes (Apple FB22411168). A picture of another + aspect during PiP (16:9 to 4:3 and back) keeps the same window: AVKit resizes it to the new picture, and the + layer follows the window instead of falling back to the view's fit. +- **`SoftwarePiPSource.isSupported`** says whether the system offers PiP at all; the controller helper returns + nil when it is false. +- **The PiP source keeps its off-main snapshot fresh without the controller helper**, so a host using + `playbackDelegate` with its own content source gets current state and duration. + ### Fixed - **A 24.000 fps source is programmed as 24 Hz, not 23.976 Hz (AE#884, thanks @akhanwav).** `FrameRateSnap` sent diff --git a/README.md b/README.md index 7f4f27687..9f94f933e 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ A scannable summary; the depth for each row lives in **[docs/formats.md](docs/fo | Surround | 5.1 / 7.1 with correct `AudioChannelLayout` | | Audio-only | `LoadOptions.audioOnly`: lean pipeline, no video machinery, system Now-Playing on tvOS / iOS | | Background audio | Audio keeps playing when the app backgrounds on iOS: native AVPlayer stays alive, the software path drops video and keeps decoding audio (`backgroundPlaybackEnabled` / `pictureInPictureActive`); a paused session survives quick app switches for a grace window (`backgroundTeardownGraceSeconds`, default 15 s) before the wedge-safe teardown runs. tvOS tears down immediately (wedge-safe) unless a PiP window is active, which keeps the pipeline and loopback server alive; with PiP the software path keeps decoding video too (the window needs frames). Hosts gate corrective actions on the published `isSessionReady` | -| Picture in Picture | Native path: hosts build `AVPictureInPictureController` around `currentAVPlayer`, or around the session's own `nativePlayerLayer` when they render through `bind(view:)` rather than AVKit; while `pictureInPictureActive` a native->native load hands the AVPlayerItem over in place so the window survives next-episode transitions. Software path: published `softwarePiPSource` carries the `AVSampleBufferDisplayLayer` plus transport answers on the enqueued frames' PTS axis for sample-buffer PiP. While PiP is active the software path also composites active subtitle cues (text and PGS/DVB bitmaps) into the decoded frames, so the window shows subtitles the host overlay cannot reach. On the native path, a selected bitmap track's compositions are OCR-recognized into its WebVTT rendition, so PGS subtitles render in the PiP window there as well. iOS renders sample-buffer PiP; tvOS AVKit does not evaluate sample-buffer content sources (Apple FB9751461, verified through tvOS 26) | +| Picture in Picture | Native path: hosts build `AVPictureInPictureController` around `currentAVPlayer`, or around the session's own `nativePlayerLayer` when they render through `bind(view:)` rather than AVKit; while `pictureInPictureActive` a native->native load hands the AVPlayerItem over in place so the window survives next-episode transitions. Software path: published `softwarePiPSource` carries the `AVSampleBufferDisplayLayer` plus transport answers on the enqueued frames' PTS axis for sample-buffer PiP; `makePictureInPictureController()` builds the controller with a `SoftwarePiPPlaybackDelegate` over the engine's play, pause, skip and time range. While PiP is active the software path also composites active subtitle cues (text and PGS/DVB bitmaps) into the decoded frames, so the window shows subtitles the host overlay cannot reach. While PiP is active a bound `AetherPlayerView` shapes the software layer like the picture (aspect-fit, or aspect-fill clipped by the view) so the window mirrors it without offset or crop; a host mounting `softwarePiPSource.layer` itself sizes it the same way. On macOS the layer is laid out at the size of the system PiP window (found through public AppKit, followed on resize; the picture-shaped layout is kept when no window is found), and the system draws an opaque bar over the top of that window that no public API removes (Apple FB22411168). Ending the session closes the helper controller's window, and a load on the software path ends PiP (no hand-over as on the native path). On the native path, a selected bitmap track's compositions are OCR-recognized into its WebVTT rendition, so PGS subtitles render in the PiP window there as well. iOS renders sample-buffer PiP; tvOS AVKit does not evaluate sample-buffer content sources (Apple FB9751461, verified through tvOS 26) | | Subtitles | Text (SRT / ASS / SSA / VTT / mov_text) inline, bitmap (PGS / DVB / DVD, plus local VobSub `.idx` + `.sub` sidecars) as `CGImage`, in-band CEA-608 closed captions (field-1, from an `eia_608`/`c608` demuxable track or extracted from A53 `cc_data` embedded in the video bitstream: H.264/HEVC SEI on the native path, decoded-frame side data such as MPEG-2 on the software path, with the caption track surfacing lazily on first real caption data), DVB teletext decoded to text cues with broadcaster colour preserved and a selectable caption page (libzvbi, `LoadOptions.teletextPage` at load, `setTeletextPage(_:)` while the channel plays), external files as first-class tracks (registered, listed, selected like embedded streams), a live channel's own HLS `SUBTITLES` renditions surfaced as tracks and fetched only once one is selected, opt-in raw ASS markup + fonts; embedded-text cues harvested from the producer's own read (instant enable, no side-channel bandwidth); opt-in native WebVTT renditions (one per text track incl. load-declared external files, language-tagged) so subtitles survive PiP / AirPlay / external display (`LoadOptions.prepareNativeSubtitles`); bitmap tracks (PGS / DVB / DVD, embedded and external .sup) join as OCR-fed renditions: on-device Vision text recognition runs while the track is selected and fills a language-tagged text rendition, so bitmap subtitles survive PiP / AirPlay / external display on the native path too (lossy by design; fullscreen keeps the pixel-accurate overlay) | | Frames | Off-playback `FrameExtractor`: `thumbnail` (scrub preview) + `snapshot` (frame-accurate) | | Audio tap | Opt-in `installAudioTap()`: decoded playback audio as mono Float32 48 kHz PCM with source-PTS timestamps, off the render path (live transcription, ShazamKit); delivers on the loopback, remote-HLS (VOD + live), and software paths. The stream is bound to its session, so a session-preserving reload (audio / subtitle / disc-title switch, `reloadAtCurrentPosition`) finishes it and the host re-installs on stream end | diff --git a/Sources/AetherEngine/AetherEngine.swift b/Sources/AetherEngine/AetherEngine.swift index 792564d47..d018d4c5a 100644 --- a/Sources/AetherEngine/AetherEngine.swift +++ b/Sources/AetherEngine/AetherEngine.swift @@ -940,6 +940,10 @@ public final class AetherEngine: ObservableObject { /// the keepalive policy + pause-safety read it. public var pictureInPictureActive = false { didSet { + #if os(macOS) + updatePictureInPictureTracking() + #endif + updatePictureFit() nativeHost?.pictureInPictureActive = pictureInPictureActive // SW-PiP Phase C: flip the frame compositor with the PiP state so subtitles appear in the // window and never double-draw under the fullscreen host overlay. @@ -947,12 +951,20 @@ public final class AetherEngine: ObservableObject { #if os(tvOS) // PiP window closed while backgrounded: nothing keeps the app running anymore, so the // wedge-safe teardown is due before idle suspension (mirrors the iOS pause-while-backgrounded path). - if oldValue && !pictureInPictureActive { + if closingPictureInPictureOwesBackgroundDecision(wasActive: oldValue) { settleBackgroundDecisionAfterPictureInPictureClosed() } #endif } } + /// True while `stopInternal` clears `pictureInPictureActive` itself. + var isResettingPictureInPictureForStop = false + /// Whether the PiP flag going false is a window closing that owes the background decision. Not when + /// `stopInternal` is the one clearing it: that is the teardown the decision would schedule, and it + /// is half done at that point. + func closingPictureInPictureOwesBackgroundDecision(wasActive: Bool) -> Bool { + wasActive && !pictureInPictureActive && !isResettingPictureInPictureForStop + } /// #127: seconds a PAUSED session survives backgrounding (iOS) before the wedge-safe teardown runs, /// held under a background-task assertion so a quick app switch resumes without a pipeline rebuild. /// 0 restores the immediate teardown. Ignored on tvOS. Keep well under the ~30 s system allowance, @@ -1104,7 +1116,22 @@ public final class AetherEngine: ObservableObject { /// against it, and it is cleared with the session so the next source cannot be laid out against /// this one's rectangle. The native and bypass paths mount an `AVPlayerLayer`, which measures its /// own `videoRect` and carries `AVPlayerItem.presentationSize`; this stays nil there. - @Published public internal(set) var softwareDisplaySize: CGSize? + @Published public internal(set) var softwareDisplaySize: CGSize? { + didSet { + updatePictureFit() + #if os(macOS) + updatePictureInPictureAspect() + #endif + } + } + #if os(macOS) + /// macOS: the content size of the system PiP window while a software session is in it, nil + /// otherwise. The display layer is laid out to it (AE#810). + var pictureInPictureArea: CGSize? { + didSet { if oldValue != pictureInPictureArea { updatePictureFit() } } + } + var pictureInPictureAreaTracker: PictureInPictureAreaTracker? + #endif /// #288: the native-path counterpart of `softwarePiPSource.layer`. `AVPictureInPictureController` /// wants the layer, not the player, so a host presenting its own PiP on the native path (tvOS has @@ -1722,6 +1749,8 @@ public final class AetherEngine: ObservableObject { // Still correct for hosts that use the engine's layer directly and for the SW displayLayer. nativeHost?.playerLayer.videoGravity = newValue softwareHost?.displayLayer.videoGravity = newValue + // The PiP fit depends on the gravity and no layout pass follows a change. + if softwareHost != nil { boundView?.refreshHostedLayerFrame() } } } var _videoGravity: AVLayerVideoGravity = .resizeAspect @@ -1989,9 +2018,43 @@ public final class AetherEngine: ObservableObject { presentCurrentLayer() } + /// While PiP is active the bound view shapes the software layer to the picture so the window does + /// not inherit the view's aspect; otherwise it fills the view. + func updatePictureFit() { + boundView?.setPictureFit(pictureInPictureActive && softwareHost != nil ? softwareDisplaySize : nil) + #if os(macOS) + boundView?.setPictureArea(pictureInPictureActive && softwareHost != nil ? pictureInPictureArea : nil) + #endif + } + + #if os(macOS) + /// The picture's size arrived or changed: let the window tracker match against its aspect. + func updatePictureInPictureAspect() { + guard pictureInPictureActive, softwareHost != nil, + let size = softwareDisplaySize, size.width > 0, size.height > 0 else { return } + pictureInPictureAreaTracker?.updateAspect(size.width / size.height) + } + + /// macOS: follow the system PiP window while a software session is in it, see + /// `PictureInPictureAreaTracker`. Stopped (and the area cleared) when PiP ends. + func updatePictureInPictureTracking() { + if pictureInPictureActive && softwareHost != nil { + if pictureInPictureAreaTracker == nil { + pictureInPictureAreaTracker = PictureInPictureAreaTracker { [weak self] in self?.pictureInPictureArea = $0 } + } + let aspect = softwareDisplaySize.flatMap { $0.width > 0 && $0.height > 0 ? $0.width / $0.height : nil } + pictureInPictureAreaTracker?.start(known: softwarePiPSource?.windowsBeforeController ?? .weakObjects(), + host: boundView?.window, aspect: aspect) + } else { + pictureInPictureAreaTracker?.stop() + } + } + #endif + /// Attaches nativeHost.playerLayer or softwareHost.displayLayer to the bound view. No-op when no host. func presentCurrentLayer() { guard let view = boundView else { return } + defer { updatePictureFit() } if let host = nativeHost { view.attach(host.playerLayer) } else if let host = softwareHost { @@ -8117,6 +8180,15 @@ public final class AetherEngine: ObservableObject { decodedVideoFormat = nil // #314: same detach on the software path, where the outgoing renderer's decode thread is what // can still hand a frame over while the next host comes up. + // The window shows this session's layer: close it while the host still feeds that layer, and + // drop the host-set flag so the keepalive policy does not treat a gone window as live (a host + // that drops its controller with the source never sees a did-stop). + if let pipSource = softwarePiPSource { + pipSource.endPictureInPicture() + isResettingPictureInPictureForStop = true + pictureInPictureActive = false + isResettingPictureInPictureForStop = false + } softwareHost?.setVideoFrameTimeObserver(nil) softwareHost?.stop() softwarePiPSource = nil diff --git a/Sources/AetherEngine/Native/SoftwarePiPSource.swift b/Sources/AetherEngine/Native/SoftwarePiPSource.swift index 5807cb06e..533ab141f 100644 --- a/Sources/AetherEngine/Native/SoftwarePiPSource.swift +++ b/Sources/AetherEngine/Native/SoftwarePiPSource.swift @@ -1,46 +1,216 @@ import AVFoundation +import Combine import CoreMedia +import os +#if os(macOS) +import AppKit +#endif +#if canImport(AVKit) +import AVKit +#endif /// SW-PiP bridge (Sodalite SW-PiP Phase A): everything a host needs to build an -/// AVPictureInPictureController ContentSource around the software path, WITHOUT AVKit entering the -/// engine. The analog of `currentAVPlayer` for sample-buffer PiP: the layer plus the four transport -/// answers backing AVPictureInPictureSampleBufferPlaybackDelegate. Time answers live here because -/// the enqueued frames' PTS axis (source axis, synchronizer clock) is engine knowledge. +/// AVPictureInPictureController ContentSource around the software path. The layer and the transport +/// answers are plain engine state; AVKit is touched only by the optional controller helpers below. +/// The analog of `currentAVPlayer` for sample-buffer PiP: the layer plus the four transport answers +/// backing AVPictureInPictureSampleBufferPlaybackDelegate. Time answers live here because the +/// enqueued frames' PTS axis (source axis, synchronizer clock) is engine knowledge. @MainActor public final class SoftwarePiPSource { + /// The two transport answers the PiP window polls, copied out of the main actor so a delegate + /// callback on another thread can answer without waiting for main. + struct Snapshot: Sendable { + var isPaused: Bool + var range: CMTimeRange + } + public let layer: AVSampleBufferDisplayLayer let isLive: Bool + /// Last values computed on the main actor. Read off main by the playback delegate. + nonisolated let snapshot = OSAllocatedUnfairLock(initialState: Snapshot(isPaused: true, range: .invalid)) private weak var engine: AetherEngine? + #if canImport(AVKit) && !os(watchOS) + private weak var controller: AVPictureInPictureController? + #endif + #if os(macOS) + /// The windows that existed when the controller was made, so the PiP window can be told apart later. + private(set) var windowsBeforeController: NSHashTable? + #endif + private var stateCancellables: Set = [] init(layer: AVSampleBufferDisplayLayer, isLive: Bool, engine: AetherEngine) { self.layer = layer self.isLive = isLive self.engine = engine + refreshSnapshot() + // One subscription per source, independent of any controller: a host that uses + // `playbackDelegate` with its own ContentSource still gets a fresh off-main snapshot. + let refresh: @Sendable () -> Void = { [weak self] in + Task { @MainActor in + guard let self else { return } + self.refreshSnapshot() + #if canImport(AVKit) && !os(watchOS) + self.controller?.invalidatePlaybackState() + #endif + } + } + engine.$state.dropFirst().sink { _ in refresh() }.store(in: &stateCancellables) + engine.$duration.dropFirst().sink { _ in refresh() }.store(in: &stateCancellables) } - /// Playable range on the enqueued frames' PTS axis (source axis). Live sources report an - /// indefinite range so the window shows live UI. - public func timeRange() -> CMTimeRange { - guard let engine else { - return CMTimeRange(start: .negativeInfinity, duration: .positiveInfinity) + /// Recompute the transport answers from the engine and publish them to `snapshot`. + @discardableResult + func refreshSnapshot() -> Snapshot { + let fresh: Snapshot + if let engine { + fresh = Snapshot( + isPaused: engine.state != .playing, + range: AetherEngine.softwarePiPTimeRange( + isLive: isLive, + sourceTime: engine.sourceTime, + currentTime: engine.currentTime, + duration: engine.duration + ) + ) + } else { + // No engine means no content to describe, which is not the same as a live source. + fresh = Snapshot(isPaused: true, range: .invalid) } - return AetherEngine.softwarePiPTimeRange( - isLive: isLive, - sourceTime: engine.sourceTime, - currentTime: engine.currentTime, - duration: engine.duration - ) + snapshot.withLock { $0 = fresh } + return fresh + } + + /// Playable range on the enqueued frames' PTS axis (source axis). Live sources report an + /// indefinite range so the window shows live UI. `.invalid` when the engine is gone: there is + /// no content to describe, which is not the same as a live source. + /// The read also refreshes the off-main snapshot on purpose. + public func timeRange() -> CMTimeRange { refreshSnapshot().range } + + /// The read also refreshes the off-main snapshot on purpose. + public var isPaused: Bool { refreshSnapshot().isPaused } + + /// Close the PiP window of the controller this source made, if any. The engine calls this when + /// the session ends; a host that built its own controller closes it on `softwarePiPSource == nil`. + func endPictureInPicture() { + #if canImport(AVKit) && !os(watchOS) + controller?.stopPictureInPicture() + #endif } - public var isPaused: Bool { engine?.state != .playing } + /// The PiP window changed size (the render-size delegate callback): macOS looks again for the + /// window's picture area. + func pictureInPictureRenderSizeChanged() { + #if os(macOS) + engine?.pictureInPictureAreaTracker?.refresh() + #endif + } public func setPlaying(_ playing: Bool) { if playing { engine?.play() } else { engine?.pause() } } public func skip(by seconds: Double) { + Task { await skipAndWait(by: seconds) } + } + + /// The seek target for a PiP skip button: relative to the playhead, never before 0, and on a + /// finite VOD never past the end. Live sources have no end to clamp to. + nonisolated static func skipTarget(isLive: Bool, currentTime: Double, duration: Double, by seconds: Double) -> Double { + let target = max(0, currentTime + seconds) + guard !isLive, duration.isFinite, duration > 0 else { return target } + return min(target, duration) + } + + /// `skip(by:)` that returns once the seek has settled, for the PiP skip completion handler. + func skipAndWait(by seconds: Double) async { guard let engine else { return } - let target = max(0, engine.currentTime + seconds) - Task { await engine.seek(to: target) } + let target = Self.skipTarget(isLive: isLive, currentTime: engine.currentTime, duration: engine.duration, by: seconds) + await engine.seek(to: target) + } + + #if canImport(AVKit) && !os(watchOS) + /// The playback delegate that maps the PiP window's play, pause, skip and time-range queries onto + /// this source. Held here because `ContentSource` does not keep its delegate alive. + public private(set) lazy var playbackDelegate = SoftwarePiPPlaybackDelegate(source: self) + + /// True when the system offers PiP at all. The layer exists for the life of the source; the check + /// is system-wide. + public static var isSupported: Bool { AVPictureInPictureController.isPictureInPictureSupported() } + + /// A ready `AVPictureInPictureController` for this session's display layer, or nil when PiP is + /// not supported on this device. A reload replaces `AetherEngine.softwarePiPSource` and the + /// content source holds its delegate weakly, so rebuild the controller on every source change. + /// When `state` or `duration` changes, the source refreshes its snapshot and invalidates the + /// playback state of the controller it made (the latest one). The host keeps the controller, sets its delegate and + /// calls `startPictureInPicture()`; it must also set `AetherEngine.pictureInPictureActive` from + /// the controller's will-start / did-stop callbacks. tvOS AVKit accepts the source but does not + /// present sample-buffer content (README, Picture in Picture). + public func makePictureInPictureController() -> AVPictureInPictureController? { + guard Self.isSupported else { return nil } + let content = AVPictureInPictureController.ContentSource( + sampleBufferDisplayLayer: layer, playbackDelegate: playbackDelegate) + #if os(macOS) + // Weak, so a window freed before PiP starts cannot be mistaken for the PiP window by address. + let before = NSHashTable.weakObjects() + (NSApp?.windows ?? []).forEach(before.add) + windowsBeforeController = before + #endif + let made = AVPictureInPictureController(contentSource: content) + controller = made + return made + } + #endif +} + +#if canImport(AVKit) && !os(watchOS) +/// `AVPictureInPictureSampleBufferPlaybackDelegate` over a `SoftwarePiPSource`. AVKit gives these +/// callbacks no thread contract, and the main thread may be waiting on AVKit itself, so nothing here +/// blocks on main: on main the live values are read, elsewhere the last published snapshot. +public final class SoftwarePiPPlaybackDelegate: NSObject, AVPictureInPictureSampleBufferPlaybackDelegate, @unchecked Sendable { + private weak var source: SoftwarePiPSource? + + init(source: SoftwarePiPSource) { + self.source = source + } + + public func pictureInPictureController(_ controller: AVPictureInPictureController, setPlaying playing: Bool) { + if Thread.isMainThread { + MainActor.assumeIsolated { source?.setPlaying(playing) } + } else { + Task { @MainActor [weak source] in source?.setPlaying(playing) } + } + } + + public func pictureInPictureControllerTimeRangeForPlayback(_ controller: AVPictureInPictureController) -> CMTimeRange { + guard let source else { return .invalid } + if Thread.isMainThread { return MainActor.assumeIsolated { source.timeRange() } } + return source.snapshot.withLock { $0.range } + } + + public func pictureInPictureControllerIsPlaybackPaused(_ controller: AVPictureInPictureController) -> Bool { + guard let source else { return true } + if Thread.isMainThread { return MainActor.assumeIsolated { source.isPaused } } + return source.snapshot.withLock { $0.isPaused } + } + + public func pictureInPictureController(_ controller: AVPictureInPictureController, didTransitionToRenderSize newRenderSize: CMVideoDimensions) { + if Thread.isMainThread { + MainActor.assumeIsolated { source?.pictureInPictureRenderSizeChanged() } + } else { + Task { @MainActor [weak source] in source?.pictureInPictureRenderSizeChanged() } + } + } + + public func pictureInPictureController(_ controller: AVPictureInPictureController, skipByInterval skipInterval: CMTime, completion completionHandler: @escaping @Sendable () -> Void) { + let seconds = skipInterval.seconds + guard skipInterval.isNumeric, seconds.isFinite else { + completionHandler() + return + } + Task { @MainActor [weak source] in + await source?.skipAndWait(by: seconds) + completionHandler() + } } } +#endif diff --git a/Sources/AetherEngine/View/AetherPlayerView.swift b/Sources/AetherEngine/View/AetherPlayerView.swift index 6b857a388..47bf81933 100644 --- a/Sources/AetherEngine/View/AetherPlayerView.swift +++ b/Sources/AetherEngine/View/AetherPlayerView.swift @@ -83,11 +83,128 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { } #endif + /// The picture's size while picture in picture is active (nil otherwise). A system PiP window + /// mirrors the display layer's own geometry, so a layer sized to this view carries the view's aspect + /// into the window as offset or clipped video. While set, the software display layer is sized to + /// the picture's rectangle for the active gravity instead (aspect-fit, or the larger aspect-fill + /// rectangle with this view clipping it), so the window gets a layer of the picture's shape. Under + /// `resizeAspect` and `resizeAspectFill` this view looks the same; under `resize` the picture is + /// shown undistorted while PiP is up. + private var pictureFit: CGSize? + /// The size of the macOS system PiP window's picture area while it is known (nil otherwise). It takes + /// precedence over `pictureFit`: the layer is laid out at the origin with exactly this size and its + /// own gravity fits the picture. The system draws an opaque bar over the top of that window, covering + /// part of the picture; its height is not public, so we fit to the whole window and leave the bar + /// alone (FB22411168). + private var pictureArea: CGSize? + /// `clipsToBounds` / `masksToBounds` as the host had it, restored when the fit ends. + private var clipsBeforeFit: Bool? + + /// Engine-internal. Shape the hosted software layer to the picture (`size` non-nil, PiP active) or + /// back to the full view (nil). No-op for a layer that is not a sample buffer layer. + func setPictureFit(_ size: CGSize?) { + guard pictureFit != size else { return } + pictureFit = size + #if canImport(AppKit) + hostedLayer?.autoresizingMask = layerAutoresizing + #endif + updateClipping() + applyLayerFrame() + } + + /// Engine-internal. Lay the hosted software layer out to the PiP window's picture area (`size` + /// non-nil) or back to the `pictureFit` / full view layout (nil). Called only on macOS. + func setPictureArea(_ size: CGSize?) { + let size = size.flatMap { $0.width > 0 && $0.height > 0 ? $0 : nil } + guard pictureArea != size else { return } + pictureArea = size + #if canImport(AppKit) + hostedLayer?.autoresizingMask = layerAutoresizing + #endif + updateClipping() + applyLayerFrame() + } + + /// Engine-internal. Re-applies the hosted layer's frame now, for a state it depends on that changed + /// without a layout pass (the display layer's gravity). Not a layout pass on macOS during PiP, + /// where the layer's autoresizing is off. + func refreshHostedLayerFrame() { + updateClipping() + applyLayerFrame() + } + + #if canImport(AppKit) + /// Full-bounds sizing normally, so the layer follows a live window drag. During PiP the layer is + /// the picture's rectangle rather than the bounds: it keeps its size and stays centred during a + /// drag, and the next layout pass fits it. + private var layerAutoresizing: CAAutoresizingMask { + if pictureArea != nil { return [] } + return pictureFit == nil + ? [.layerWidthSizable, .layerHeightSizable] + : [.layerMinXMargin, .layerMaxXMargin, .layerMinYMargin, .layerMaxYMargin] + } + #endif + + /// The aspect-fill rectangle is larger than the view, so the view clips it while the fit is on. + private var wantsClipping: Bool { + guard let sample = hostedLayer as? AVSampleBufferDisplayLayer else { return false } + // The area frame can exceed this view, so the view clips it. + if pictureArea != nil { return true } + return pictureFit != nil && sample.videoGravity == .resizeAspectFill + } + + private func updateClipping() { + #if canImport(UIKit) + let current = clipsToBounds + #elseif canImport(AppKit) + let current = layer?.masksToBounds ?? false + #endif + var target = current + if wantsClipping { + if clipsBeforeFit == nil { clipsBeforeFit = current } + target = true + } else if let saved = clipsBeforeFit { + clipsBeforeFit = nil + target = saved + } + guard target != current else { return } + #if canImport(UIKit) + clipsToBounds = target + #elseif canImport(AppKit) + layer?.masksToBounds = target + #endif + } + + /// Frame for the hosted layer inside `bounds`: the rectangle `content` occupies under `gravity` + /// (aspect-fit for `resizeAspect` and `resize`, the larger aspect-fill rectangle for + /// `resizeAspectFill`), centred; the full bounds when `content` is nil. + nonisolated static func hostedFrame(in bounds: CGRect, fitting content: CGSize?, + gravity: AVLayerVideoGravity = .resizeAspect) -> CGRect { + guard let content, content.width > 0, content.height > 0, bounds.width > 0, bounds.height > 0 else { return bounds } + let ratios = (bounds.width / content.width, bounds.height / content.height) + let scale = gravity == .resizeAspectFill ? max(ratios.0, ratios.1) : min(ratios.0, ratios.1) + let size = CGSize(width: content.width * scale, height: content.height * scale) + return CGRect(x: bounds.midX - size.width / 2, y: bounds.midY - size.height / 2, + width: size.width, height: size.height) + } + + /// Frame for the layer while the PiP window's picture area is known: origin at zero, the area's size. + /// The layer's own gravity fits the picture inside it. + nonisolated static func pictureAreaFrame(_ area: CGSize) -> CGRect { + CGRect(origin: .zero, size: area) + } + + private var hostedFrame: CGRect { + guard let sample = hostedLayer as? AVSampleBufferDisplayLayer else { return bounds } + if let pictureArea { return Self.pictureAreaFrame(pictureArea) } + return Self.hostedFrame(in: bounds, fitting: pictureFit, gravity: sample.videoGravity) + } + private func applyLayerFrame() { guard hostedLayer != nil || stillLayer != nil else { return } CATransaction.begin() CATransaction.setDisableActions(true) - hostedLayer?.frame = bounds + hostedLayer?.frame = hostedFrame presenter.layout(bounds) CATransaction.commit() } @@ -118,8 +235,12 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { // bounds with both axes flexible keeps it full-bounds throughout. layer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable] #endif - layer.frame = bounds hostedLayer = layer + #if canImport(AppKit) + layer.autoresizingMask = layerAutoresizing + #endif + updateClipping() + layer.frame = hostedFrame CATransaction.commit() } @@ -128,6 +249,9 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { /// `attach`: a layer that has moved to another surface stays there. func detach() { clearStill() + // The PiP layout belongs to the engine that set it; a view bound again later starts clean. + pictureFit = nil + pictureArea = nil guard let hosted = hostedLayer else { return } CATransaction.begin() CATransaction.setDisableActions(true) @@ -135,6 +259,7 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { hosted.removeFromSuperlayer() } hostedLayer = nil + updateClipping() CATransaction.commit() } diff --git a/Sources/AetherEngine/View/PictureInPictureAreaTracker.swift b/Sources/AetherEngine/View/PictureInPictureAreaTracker.swift new file mode 100644 index 000000000..1ee84f0e9 --- /dev/null +++ b/Sources/AetherEngine/View/PictureInPictureAreaTracker.swift @@ -0,0 +1,238 @@ +#if os(macOS) +import AppKit + +/// Follows the size of the system picture in picture window on macOS while a software session is in +/// it, so the display layer can be laid out to that window (AE#810). +/// +/// AVKit gives no public size for that window, so it is found with public AppKit only: the one visible +/// panel in `NSApp.windows` that did not exist when the PiP controller was made (it is already there by +/// the time a host reports PiP active) and whose content has the picture's aspect. Its content view +/// posts the public frame change notification, which fires when the window first settles, on every +/// resize and when PiP stops. `refresh()` is a second trigger, for the render-size delegate callback. +/// +/// This relies on the system PiP window being an in-process panel. If that ever stops being true, or +/// the match is not unique, nothing is adopted, no area is reported and the layer keeps the +/// `AetherPlayerView.setPictureFit` layout, which is safe. The search for the window is bounded by +/// `searchBudget`, so an app-wide window observer does not outlive a window that never turns up. +@MainActor +final class PictureInPictureAreaTracker { + private let onChange: (CGSize?) -> Void + private let windows: () -> [NSWindow] + private let isVisible: (NSWindow) -> Bool + private let now: () -> TimeInterval + private var known = NSHashTable.weakObjects() + private weak var hostWindow: NSWindow? + private var aspect: CGFloat? + private var isTracking = false + private var searchDeadline: TimeInterval = 0 + private var searchExpired = false + /// The adopted window was matched without a known aspect, so it still has to pass the first one. + private var adoptionUnchecked = false + private let registrations = Registrations() + + /// Aspect difference up to which a panel counts as showing the picture. + static let aspectTolerance: CGFloat = 0.02 + + /// How long the search for the window may keep observing window updates, in seconds. The system + /// window is in place within about a second of the host reporting PiP active, so five seconds + /// leaves room for a slow start; past it, no panel is going to match (an out-of-process window, + /// an aspect outside the tolerance) and the app-wide observer would only cost every update for + /// the rest of the session. The fit layout stays. A new search starts when an adopted window goes + /// away, or when the picture's aspect changes while no window is adopted. + static let searchBudget: TimeInterval = 5 + + /// `windows`, `isVisible` and `now` are injectable for tests; the defaults are the app's windows + /// and the system uptime. + init(windows: @escaping () -> [NSWindow] = { NSApp?.windows ?? [] }, + isVisible: @escaping (NSWindow) -> Bool = { $0.isVisible }, + now: @escaping () -> TimeInterval = { ProcessInfo.processInfo.systemUptime }, + onChange: @escaping (CGSize?) -> Void) { + self.windows = windows + self.isVisible = isVisible + self.now = now + self.onChange = onChange + } + + /// Begin looking for the PiP window. `known` are the windows that existed before the controller was + /// made (empty when unknown: any matching visible panel then qualifies); `host` is the window of the + /// view showing the video and is never taken for it; `aspect` is the picture's width over height, or + /// nil when unknown. Idempotent. + func start(known: NSHashTable, host: NSWindow?, aspect: CGFloat?) { + guard !isTracking else { return } + isTracking = true + self.known = known + hostWindow = host + self.aspect = aspect + beginSearch() + refresh() + } + + /// The picture's aspect arrived or changed. AVKit resizes the PiP window to each new picture shape, + /// and the new aspect can arrive before that resize has happened. So a window matched under a known + /// aspect stays the PiP window until PiP ends, through any number of aspect changes, and its size is + /// followed as it changes. A window adopted before the aspect was known has not been matched against + /// the picture yet: the first aspect checks it, and it is dropped if it does not match. With no window + /// adopted, the search starts again under the new aspect. + func updateAspect(_ newAspect: CGFloat) { + guard isTracking, newAspect > 0 else { return } + if let aspect, aspect > 0, abs(newAspect / aspect - 1) <= Self.aspectTolerance { return } + aspect = newAspect + if adoptionUnchecked { + adoptionUnchecked = false + if let view = registrations.view, !aspectMatches(view.bounds.size) { + registrations.release() + onChange(nil) + } + } + if registrations.view == nil { + beginSearch() + refresh() + } + } + + /// Stop following, give the content view back its notification setting and report no area. + func stop() { + guard isTracking else { return } + isTracking = false + registrations.teardown() + // Drop the reference only: the table belongs to the source that made it and is reused. + known = .weakObjects() + hostWindow = nil + aspect = nil + searchExpired = false + adoptionUnchecked = false + onChange(nil) + } + + /// Find the window if it is not found yet, then report its content size. + func refresh() { + guard isTracking else { return } + if registrations.view == nil { + // The window we followed is gone: look for the next one with a fresh budget. + if registrations.dropFrameObserver() { beginSearch() } + if !searchExpired { adoptWindow() } + } + if let view = registrations.view { report(view.bounds.size) } + } + + private func adoptWindow() { + let matches = windows().filter { window in + !known.contains(window) && window !== hostWindow && window is NSPanel && isVisible(window) + && window.contentView.map { aspectMatches($0.bounds.size) } == true + } + guard matches.count == 1, let view = matches[0].contentView else { + if now() >= searchDeadline { + searchExpired = true + registrations.dropUpdateObserver() + } else { + watchWindowUpdates() + } + return + } + registrations.dropUpdateObserver() + adoptionUnchecked = aspect == nil + registrations.adopt(view, frameObserver: NotificationCenter.default.addObserver( + forName: NSView.frameDidChangeNotification, object: view, queue: .main + ) { [weak self] _ in + MainActor.assumeIsolated { self?.refresh() } + }) + } + + private func beginSearch() { + searchDeadline = now() + Self.searchBudget + searchExpired = false + } + + /// Until a window is adopted, every window update is a chance to find it. Cheap and public. + private func watchWindowUpdates() { + guard !registrations.isWatchingUpdates else { return } + registrations.watchUpdates(NotificationCenter.default.addObserver( + forName: NSWindow.didUpdateNotification, object: nil, queue: .main + ) { [weak self] _ in + MainActor.assumeIsolated { self?.refresh() } + }) + } + + private func aspectMatches(_ size: CGSize) -> Bool { + guard size.width > 0, size.height > 0 else { return false } + guard let aspect, aspect > 0 else { return true } + return abs(size.width / size.height / aspect - 1) <= Self.aspectTolerance + } + + private func report(_ size: CGSize) { + guard size.width > 0, size.height > 0 else { return } + onChange(size) + } + + /// The notification tokens and the adopted view, kept apart from the main-actor state so a tracker + /// that is released while tracking still removes its observers and gives the view its setting back + /// (a MainActor deinit can't touch non-Sendable stored state under Swift 6). + private final class Registrations: @unchecked Sendable { + private let lock = NSLock() + private var updateObserver: NSObjectProtocol? + private var frameObserver: NSObjectProtocol? + private weak var adopted: NSView? + private var postedBefore = false + + var view: NSView? { lock.withLock { adopted } } + var isWatchingUpdates: Bool { lock.withLock { updateObserver != nil } } + + func watchUpdates(_ token: NSObjectProtocol) { + lock.withLock { updateObserver = token } + } + + func dropUpdateObserver() { + let token = lock.withLock { () -> NSObjectProtocol? in + defer { updateObserver = nil } + return updateObserver + } + token.map(NotificationCenter.default.removeObserver) + } + + /// Returns whether there was a frame observer to drop. + @discardableResult + func dropFrameObserver() -> Bool { + let token = lock.withLock { () -> NSObjectProtocol? in + defer { frameObserver = nil } + return frameObserver + } + token.map(NotificationCenter.default.removeObserver) + return token != nil + } + + @MainActor + func adopt(_ view: NSView, frameObserver token: NSObjectProtocol) { + lock.withLock { + adopted = view + postedBefore = view.postsFrameChangedNotifications + frameObserver = token + } + view.postsFrameChangedNotifications = true + } + + /// Remove every observer and restore the adopted view's setting. + func teardown() { + dropUpdateObserver() + release() + } + + /// Drop the frame observer and give the adopted view its setting back. + func release() { + dropFrameObserver() + let (view, before) = lock.withLock { () -> (NSView?, Bool) in + defer { adopted = nil } + return (adopted, postedBefore) + } + guard let view else { return } + if Thread.isMainThread { + MainActor.assumeIsolated { view.postsFrameChangedNotifications = before } + } else { + nonisolated(unsafe) let unsafeView = view + DispatchQueue.main.async { unsafeView.postsFrameChangedNotifications = before } + } + } + + deinit { teardown() } + } +} +#endif diff --git a/Tests/AetherEngineTests/PictureInPictureAreaTrackerTests.swift b/Tests/AetherEngineTests/PictureInPictureAreaTrackerTests.swift new file mode 100644 index 000000000..51f61a81a --- /dev/null +++ b/Tests/AetherEngineTests/PictureInPictureAreaTrackerTests.swift @@ -0,0 +1,409 @@ +#if os(macOS) +import AppKit +import AVFoundation +import Testing +@testable import AetherEngine + +/// AE#810: the tracker that follows the system PiP window's size. Plain panels stand in for that window +/// and are never ordered on screen; visibility is injected. +@Suite("PiP area tracker", .serialized) +@MainActor +struct PictureInPictureAreaTrackerTests { + private func panel(_ size: CGSize) -> NSPanel { + _ = NSApplication.shared + let panel = NSPanel(contentRect: CGRect(origin: .zero, size: size), styleMask: [], backing: .buffered, defer: true) + let content = NSView(frame: CGRect(origin: .zero, size: size)) + // A plain view posts frame changes by default; start from off so the restore is observable. + content.postsFrameChangedNotifications = false + panel.contentView = content + return panel + } + + private func pump() { RunLoop.main.run(until: Date().addingTimeInterval(0.05)) } + + private final class Reports { + var sizes: [CGSize?] = [] + var last: CGSize? { sizes.last ?? nil } + } + + private func tracker(_ windows: [NSWindow], _ reports: Reports) -> PictureInPictureAreaTracker { + PictureInPictureAreaTracker(windows: { windows }, isVisible: { _ in true }, onChange: { reports.sizes.append($0) }) + } + + private func known(_ windows: [NSWindow]) -> NSHashTable { + let table = NSHashTable.weakObjects() + windows.forEach(table.add) + return table + } + + @Test("only the new panel with the picture's aspect is adopted") + func filters() { + let old = panel(CGSize(width: 556, height: 312)) + let host = panel(CGSize(width: 556, height: 312)) + let notPanel = NSWindow(contentRect: CGRect(x: 0, y: 0, width: 556, height: 312), styleMask: [], backing: .buffered, defer: true) + notPanel.contentView = NSView(frame: CGRect(x: 0, y: 0, width: 556, height: 312)) + let wrongAspect = panel(CGSize(width: 400, height: 400)) + let target = panel(CGSize(width: 556, height: 312)) + let reports = Reports() + let tracker = tracker([old, host, notPanel, wrongAspect, target], reports) + tracker.start(known: known([old]), host: host, aspect: 16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312)) + target.contentView?.setFrameSize(CGSize(width: 700, height: 394)) + pump() + #expect(reports.last == CGSize(width: 700, height: 394)) + tracker.stop() + } + + @Test("two matching panels adopt nothing") + func ambiguousAdoptsNothing() { + let reports = Reports() + let tracker = tracker([panel(CGSize(width: 556, height: 312)), panel(CGSize(width: 320, height: 180))], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + #expect(reports.sizes.isEmpty) + tracker.stop() + } + + @Test("a panel that settles to the picture's aspect later is found on a window update") + func foundOnUpdate() { + let target = panel(CGSize(width: 100, height: 100)) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + #expect(reports.sizes.isEmpty) + target.contentView?.setFrameSize(CGSize(width: 556, height: 312.75)) + NotificationCenter.default.post(name: NSWindow.didUpdateNotification, object: target) + pump() + #expect(reports.last == CGSize(width: 556, height: 312.75)) + tracker.stop() + } + + @Test("without a picture aspect any single new panel qualifies") + func noAspect() { + let reports = Reports() + let tracker = tracker([panel(CGSize(width: 300, height: 200))], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + #expect(reports.last == CGSize(width: 300, height: 200)) + tracker.stop() + } + + @Test("stop reports no area, restores the view's setting and removes the observers") + func stopCleansUp() { + let target = panel(CGSize(width: 556, height: 312)) + let view = try! #require(target.contentView) + #expect(!view.postsFrameChangedNotifications) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + #expect(view.postsFrameChangedNotifications) + tracker.stop() + #expect(reports.sizes.last as CGSize?? == .some(nil)) + #expect(!view.postsFrameChangedNotifications) + let count = reports.sizes.count + // Turn posting back on so a frame observer left behind would hear the change. + view.postsFrameChangedNotifications = true + view.setFrameSize(CGSize(width: 800, height: 450)) + NotificationCenter.default.post(name: NSWindow.didUpdateNotification, object: target) + pump() + #expect(reports.sizes.count == count) + } + + @Test("a second start adds no second observer, and the window-update observer goes once a panel is adopted") + func singleRegistration() { + let target = panel(CGSize(width: 556, height: 312)) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + let afterStart = reports.sizes.count + #expect(afterStart == 1) + NotificationCenter.default.post(name: NSWindow.didUpdateNotification, object: target) + pump() + #expect(reports.sizes.count == afterStart) + target.contentView?.setFrameSize(CGSize(width: 600, height: 337)) + pump() + #expect(reports.sizes.count == afterStart + 1) + tracker.stop() + } + + @Test("a tracker released while tracking gives the view its setting back") + func releaseRestores() { + let target = panel(CGSize(width: 556, height: 312)) + let view = try! #require(target.contentView) + do { + let tracker = tracker([target], Reports()) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + #expect(view.postsFrameChangedNotifications) + } + #expect(!view.postsFrameChangedNotifications) + } + + @Test("stop leaves the source's snapshot alone, so a second start on it still filters") + func stopKeepsSnapshot() { + let old = panel(CGSize(width: 556, height: 312)) + let target = panel(CGSize(width: 556, height: 312)) + let snapshot = known([old]) + let reports = Reports() + let tracker = tracker([old, target], reports) + tracker.start(known: snapshot, host: nil, aspect: 16.0 / 9.0) + tracker.stop() + #expect(snapshot.count == 1 && snapshot.contains(old)) + tracker.start(known: snapshot, host: nil, aspect: 16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312)) + // `old` was still filtered: with it counted, two panels would have matched and nothing been adopted. + tracker.stop() + } + + private final class Clock { + var value: TimeInterval = 100 + } + + private func tracker(_ windows: [NSWindow], _ reports: Reports, clock: Clock) -> PictureInPictureAreaTracker { + PictureInPictureAreaTracker(windows: { windows }, isVisible: { _ in true }, now: { clock.value }, + onChange: { reports.sizes.append($0) }) + } + + private func postUpdate(_ window: NSWindow) { + NotificationCenter.default.post(name: NSWindow.didUpdateNotification, object: window) + pump() + } + + @Test("the search for the window stops observing updates after its budget, and keeps no area") + func searchExpires() { + let target = panel(CGSize(width: 100, height: 100)) + let reports = Reports() + let clock = Clock() + let tracker = tracker([target], reports, clock: clock) + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + + // Inside the budget a late match is still found (see foundOnUpdate); here it just keeps looking. + clock.value += PictureInPictureAreaTracker.searchBudget - 1 + postUpdate(target) + #expect(reports.sizes.isEmpty) + + // The first update past the budget ends the observation. + clock.value += 2 + postUpdate(target) + #expect(reports.sizes.isEmpty) + target.contentView?.setFrameSize(CGSize(width: 556, height: 312.75)) + postUpdate(target) + #expect(reports.sizes.isEmpty) + + // Stopping and starting again is a new search. + tracker.stop() + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312.75)) + tracker.stop() + } + + @Test("a new aspect after the search expired starts a new search") + func aspectRestartsExpiredSearch() { + let target = panel(CGSize(width: 400, height: 300)) + let reports = Reports() + let clock = Clock() + let tracker = tracker([target], reports, clock: clock) + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + clock.value += PictureInPictureAreaTracker.searchBudget + 1 + postUpdate(target) + #expect(reports.sizes.isEmpty) + tracker.updateAspect(4.0 / 3.0) + #expect(reports.last == CGSize(width: 400, height: 300)) + tracker.stop() + } + + @Test("an aspect that arrives after start finds a window the unknown aspect could not tell apart") + func aspectArrivesLate() { + let a = panel(CGSize(width: 556, height: 312)) + let b = panel(CGSize(width: 400, height: 400)) + let reports = Reports() + let tracker = tracker([a, b], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + #expect(reports.sizes.isEmpty) + tracker.updateAspect(16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312)) + tracker.stop() + } + + @Test("a window adopted under an unknown aspect is dropped when the known aspect does not match it") + func wrongAdoptionIsDropped() { + let wrong = panel(CGSize(width: 400, height: 400)) + let right = panel(CGSize(width: 556, height: 312)) + let wrongView = try! #require(wrong.contentView) + var windows: [NSWindow] = [wrong] + let reports = Reports() + let tracker = PictureInPictureAreaTracker(windows: { windows }, isVisible: { _ in true }, + onChange: { reports.sizes.append($0) }) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + #expect(reports.last == CGSize(width: 400, height: 400)) + #expect(wrongView.postsFrameChangedNotifications) + + tracker.updateAspect(16.0 / 9.0) + #expect(reports.sizes.last as CGSize?? == .some(nil)) + #expect(!wrongView.postsFrameChangedNotifications) + + // The real window turns up afterwards and is adopted; the wrong one is still ignored. + windows.append(right) + postUpdate(right) + #expect(reports.last == CGSize(width: 556, height: 312)) + wrongView.setFrameSize(CGSize(width: 500, height: 500)) + pump() + #expect(reports.last == CGSize(width: 556, height: 312)) + tracker.stop() + } + + @Test("an adopted window that fits the arriving aspect is kept, and a repeated aspect changes nothing") + func matchingAdoptionIsKept() { + let target = panel(CGSize(width: 556, height: 312)) + let view = try! #require(target.contentView) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + let count = reports.sizes.count + tracker.updateAspect(16.0 / 9.0) + tracker.updateAspect(16.0 / 9.0) + #expect(reports.sizes.count == count) + #expect(view.postsFrameChangedNotifications) + tracker.stop() + } + + @Test("a window matched under a known aspect is kept through aspect changes, in either direction") + func adoptionSurvivesAspectChanges() { + let target = panel(CGSize(width: 556, height: 312)) + let view = try! #require(target.contentView) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: 16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312)) + let count = reports.sizes.count + tracker.updateAspect(4.0 / 3.0) + tracker.updateAspect(16.0 / 9.0) + tracker.updateAspect(4.0 / 3.0) + #expect(reports.sizes.count == count) + #expect(view.postsFrameChangedNotifications) + // AVKit resizes the window to the new picture shape, a little after the aspect arrives. The kept + // window reports each of those sizes. + view.setFrameSize(CGSize(width: 480, height: 360)) + pump() + #expect(reports.last == CGSize(width: 480, height: 360)) + view.setFrameSize(CGSize(width: 500, height: 281)) + pump() + #expect(reports.last == CGSize(width: 500, height: 281)) + tracker.stop() + #expect(reports.last == nil) + } + + @Test("a window matched under the first known aspect is kept when the aspect changes again") + func firstAspectCheckedOnce() { + let target = panel(CGSize(width: 556, height: 312)) + let reports = Reports() + let tracker = tracker([target], reports) + tracker.start(known: .weakObjects(), host: nil, aspect: nil) + tracker.updateAspect(16.0 / 9.0) + #expect(reports.last == CGSize(width: 556, height: 312)) + let count = reports.sizes.count + tracker.updateAspect(4.0 / 3.0) + #expect(reports.sizes.count == count) + #expect(reports.last == CGSize(width: 556, height: 312)) + tracker.stop() + } + + @Test("engine wiring: a picture aspect change during PiP keeps the window's area and the layer on it") + func engineKeepsAreaThroughAspectChange() throws { + let right = panel(CGSize(width: 556, height: 312)) + let windows: [NSWindow] = [right] + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let engine = try AetherEngine() + let host = SoftwarePlaybackHost() + defer { host.stop() } + engine.softwareHost = host + engine.bind(view: view) + let layer = host.displayLayer + engine.pictureInPictureAreaTracker = PictureInPictureAreaTracker( + windows: { windows }, isVisible: { _ in true }, + onChange: { [weak engine] in engine?.pictureInPictureArea = $0 }) + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + + engine.pictureInPictureActive = true + #expect(engine.pictureInPictureArea == CGSize(width: 556, height: 312)) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 556, height: 312)) + + engine.softwareDisplaySize = CGSize(width: 640, height: 480) + #expect(engine.pictureInPictureArea == CGSize(width: 556, height: 312)) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 556, height: 312)) + + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + #expect(engine.pictureInPictureArea == CGSize(width: 556, height: 312)) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 556, height: 312)) + + engine.pictureInPictureActive = false + #expect(engine.pictureInPictureArea == nil) + #expect(layer.frame == full) + } + + @Test("engine wiring: the picture size arriving during PiP re-arms the window match") + func engineRearmsOnDisplaySize() throws { + let wrong = panel(CGSize(width: 400, height: 400)) + let right = panel(CGSize(width: 556, height: 312)) + var windows: [NSWindow] = [wrong] + let engine = try AetherEngine() + let host = SoftwarePlaybackHost() + defer { host.stop() } + engine.softwareHost = host + engine.pictureInPictureAreaTracker = PictureInPictureAreaTracker( + windows: { windows }, isVisible: { _ in true }, + onChange: { [weak engine] in engine?.pictureInPictureArea = $0 }) + + // PiP starts before the first frame told the engine the picture's size: the lone panel is taken. + engine.pictureInPictureActive = true + #expect(engine.pictureInPictureArea == CGSize(width: 400, height: 400)) + + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + #expect(engine.pictureInPictureArea == nil) + windows.append(right) + postUpdate(right) + #expect(engine.pictureInPictureArea == CGSize(width: 556, height: 312)) + engine.pictureInPictureActive = false + #expect(engine.pictureInPictureArea == nil) + } + + @Test("engine wiring: the area lays the layer out only while PiP is active, and clearing it restores") + func engineDrivesPictureArea() throws { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let engine = try AetherEngine() + let host = SoftwarePlaybackHost() + defer { host.stop() } + engine.softwareHost = host + engine.bind(view: view) + let layer = host.displayLayer + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + + // Not in PiP: an area changes nothing. + engine.pictureInPictureArea = CGSize(width: 556, height: 312) + #expect(layer.frame == full) + + engine.pictureInPictureActive = true + engine.pictureInPictureArea = CGSize(width: 417, height: 313) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 417, height: 313)) + engine.pictureInPictureArea = CGSize(width: 720, height: 405) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 720, height: 405)) + + engine.pictureInPictureActive = false + #expect(engine.pictureInPictureArea == nil) + #expect(layer.frame == full) + } + + @Test("a view detached during PiP comes back with its normal layout") + func detachResetsPictureLayout() { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let layer = AVSampleBufferDisplayLayer() + view.attach(layer) + view.setPictureFit(CGSize(width: 1280, height: 720)) + view.setPictureArea(CGSize(width: 417, height: 313)) + view.detach() + view.attach(layer) + #expect(layer.frame == full) + } +} +#endif diff --git a/Tests/AetherEngineTests/SoftwarePiPSourceTests.swift b/Tests/AetherEngineTests/SoftwarePiPSourceTests.swift index 5018d4b30..bb5d6d420 100644 --- a/Tests/AetherEngineTests/SoftwarePiPSourceTests.swift +++ b/Tests/AetherEngineTests/SoftwarePiPSourceTests.swift @@ -1,3 +1,7 @@ +import AVFoundation +#if canImport(AVKit) && os(macOS) +@preconcurrency import AVKit +#endif import Testing import CoreMedia @testable import AetherEngine @@ -33,3 +37,398 @@ struct SoftwarePiPSourceTests { #expect(r.start == .negativeInfinity) } } + +/// The transport answers the PiP window's delegate callbacks map onto. +@Suite("Software PiP transport mapping") +@MainActor +struct SoftwarePiPTransportTests { + @Test("no software session: no PiP source to offer") + func noSessionNoSource() throws { + let engine = try AetherEngine() + #expect(engine.softwarePiPSource == nil) + } + + @Test("skip target is relative, floored at 0, capped at the VOD duration") + func skipTargets() { + #expect(SoftwarePiPSource.skipTarget(isLive: false, currentTime: 100, duration: 600, by: 15) == 115) + #expect(SoftwarePiPSource.skipTarget(isLive: false, currentTime: 100, duration: 600, by: -15) == 85) + #expect(SoftwarePiPSource.skipTarget(isLive: false, currentTime: 5, duration: 600, by: -15) == 0) + #expect(SoftwarePiPSource.skipTarget(isLive: false, currentTime: 595, duration: 600, by: 15) == 600) + } + + @Test("live and unknown-duration sources are not capped") + func skipTargetsUncapped() { + #expect(SoftwarePiPSource.skipTarget(isLive: true, currentTime: 100, duration: 50, by: 15) == 115) + #expect(SoftwarePiPSource.skipTarget(isLive: false, currentTime: 100, duration: 0, by: 15) == 115) + } + + @Test("an idle engine reports paused with an indefinite range, and pause keeps it paused") + func idleEngine() throws { + let engine = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + #expect(source.isPaused) + #expect(source.timeRange().start == .negativeInfinity) + source.setPlaying(false) + #expect(source.isPaused) + #expect(engine.state != .playing) + } + + @Test("a released engine has no content: paused and an invalid range") + func releasedEngine() async throws { + var engine: AetherEngine? = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine!) + engine = nil + #expect(source.isPaused) + #expect(!source.timeRange().isValid) + source.setPlaying(true) + await source.skipAndWait(by: 10) + } + + @Test("skipAndWait on an idle engine returns") + func skipAwaitsIdle() async throws { + let engine = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + await source.skipAndWait(by: 10) + #expect(source.isPaused) + } + + #if canImport(AVKit) && !os(watchOS) && os(macOS) + private final class Counter: @unchecked Sendable { + private let lock = NSLock() + private var n = 0 + func bump() { lock.lock(); n += 1; lock.unlock() } + var value: Int { lock.lock(); defer { lock.unlock() }; return n } + } + + /// Evaluated off the main actor by the trait, so it asks AVKit directly. + nonisolated static let pipSupported = AVPictureInPictureController.isPictureInPictureSupported() + + @Test("delegate callbacks through a real controller: skip completes once, no content is invalid", + .enabled(if: SoftwarePiPTransportTests.pipSupported, "PiP is not supported on this host")) + func delegateThroughController() async throws { + let engine = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + let controller = try #require(source.makePictureInPictureController()) + let delegate = source.playbackDelegate + #expect(delegate.pictureInPictureControllerIsPlaybackPaused(controller)) + #expect(delegate.pictureInPictureControllerTimeRangeForPlayback(controller).start == .negativeInfinity) + + let done = Counter() + await withCheckedContinuation { (c: CheckedContinuation) in + delegate.pictureInPictureController(controller, skipByInterval: CMTime(seconds: 10, preferredTimescale: 600)) { + done.bump() + c.resume() + } + } + // NaN interval: completes immediately, still exactly once. + delegate.pictureInPictureController(controller, skipByInterval: .invalid) { done.bump() } + try await Task.sleep(nanoseconds: 200_000_000) + #expect(done.value == 2) + + let orphan = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: try AetherEngine()) + let orphanController = try #require(orphan.makePictureInPictureController()) + let orphanDelegate = orphan.playbackDelegate + // Engine of `orphan` is released once the temporary above is gone. + #expect(orphanDelegate.pictureInPictureControllerIsPlaybackPaused(orphanController)) + #expect(!orphanDelegate.pictureInPictureControllerTimeRangeForPlayback(orphanController).isValid) + } + + private final class Answers: @unchecked Sendable { + private let lock = NSLock() + private var stored: (paused: Bool, range: CMTimeRange)? + func set(paused: Bool, range: CMTimeRange) { lock.lock(); stored = (paused, range); lock.unlock() } + var value: (paused: Bool, range: CMTimeRange)? { lock.lock(); defer { lock.unlock() }; return stored } + } + + @Test("delegate getters answer from a background thread while the main thread is busy", + .enabled(if: SoftwarePiPTransportTests.pipSupported, "PiP is not supported on this host")) + func gettersOffMainDoNotBlock() throws { + let engine = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + let controller = try #require(source.makePictureInPictureController()) + let delegate = source.playbackDelegate + + let answers = Answers() + let finished = DispatchSemaphore(value: 0) + // A dedicated thread rather than a global queue: its start does not depend on the capacity of + // the shared pool, which the rest of the suite can hold for longer than the wait below. + Thread { + delegate.pictureInPictureController(controller, setPlaying: false) + answers.set( + paused: delegate.pictureInPictureControllerIsPlaybackPaused(controller), + range: delegate.pictureInPictureControllerTimeRangeForPlayback(controller)) + finished.signal() + }.start() + // This test runs on the main thread and holds it here: a getter that waited on main would + // never finish inside the budget. + let outcome = finished.wait(timeout: .now() + 5) + #expect(outcome == .success) + let result = try #require(answers.value) + #expect(result.paused) + #expect(result.range.start == .negativeInfinity) + } + + @Test("the snapshot refreshes from state and duration changes without the controller helper") + func snapshotRefreshesWithoutController() async throws { + let engine = try AetherEngine() + let source = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + // The source starts out paused (the engine is idle), so only a refresh can make it "playing". + engine.state = .playing + await Task.yield() + await Task.yield() + #expect(!source.snapshot.withLock { $0 }.isPaused) + engine.duration = 600 + await Task.yield() + await Task.yield() + let snap = source.snapshot.withLock { $0 } + #expect(!snap.isPaused) + #expect(abs(snap.range.duration.seconds - 600) < 0.001) + } + + @Test("the PiP fit shapes the display layer to the picture under every gravity and restores on exit", + arguments: [AVLayerVideoGravity.resizeAspect, .resizeAspectFill, .resize]) + @MainActor + func pictureFitFollowsPictureInPicture(gravity: AVLayerVideoGravity) throws { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let layer = AVSampleBufferDisplayLayer() + layer.videoGravity = gravity + view.attach(layer) + #expect(layer.frame == full) + view.setPictureFit(CGSize(width: 1280, height: 720)) + // The window mirrors the layer, so the layer must have the picture's aspect whatever the gravity. + #expect(abs(layer.frame.width / layer.frame.height - 16.0 / 9.0) < 0.001) + if gravity == .resizeAspectFill { + #expect(layer.frame.width >= 639.999 && layer.frame.height >= 391.999) + #expect(layer.frame.width > 640.001 || layer.frame.height > 392.001) + #expect(clips(view)) + } else { + #expect(layer.frame.width <= 640.001 && layer.frame.height <= 392.001) + #expect(!clips(view)) + } + #expect(abs(layer.frame.midX - full.midX) < 0.001 && abs(layer.frame.midY - full.midY) < 0.001) + view.setPictureFit(nil) + #expect(layer.frame == full) + #expect(!clips(view)) + } + + @MainActor + private func clips(_ view: AetherPlayerView) -> Bool { + #if canImport(UIKit) + view.clipsToBounds + #else + view.layer?.masksToBounds ?? false + #endif + } + + @Test("the PiP window's picture area takes the layer to the origin at that size and restores on exit", + arguments: [AVLayerVideoGravity.resizeAspect, .resizeAspectFill, .resize]) + @MainActor + func pictureAreaFitsAndRestores(gravity: AVLayerVideoGravity) { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let layer = AVSampleBufferDisplayLayer() + layer.videoGravity = gravity + view.attach(layer) + view.setPictureFit(CGSize(width: 1280, height: 720)) + view.setPictureArea(CGSize(width: 558, height: 313)) + #expect(layer.frame == CGRect(x: 0, y: 0, width: 558, height: 313)) + #expect(layer.videoGravity == gravity) + #expect(clips(view)) + // A window resize re-fits; a view layout pass does not undo it. + view.setPictureArea(CGSize(width: 720, height: 405)) + #if canImport(AppKit) + view.setFrameSize(CGSize(width: 800, height: 500)) + view.layoutSubtreeIfNeeded() + #endif + #expect(layer.frame == CGRect(x: 0, y: 0, width: 720, height: 405)) + // No size (or an empty one) hands the layout back to the picture fit, then the full view. + view.setPictureArea(.zero) + #expect(abs(layer.frame.width / layer.frame.height - 16.0 / 9.0) < 0.001) + view.setPictureFit(nil) + #expect(layer.frame == CGRect(origin: .zero, size: view.bounds.size)) + #expect(!clips(view)) + } + + @Test("the picture area frame is the area at the origin") + func pictureAreaRect() { + #expect(AetherPlayerView.pictureAreaFrame(CGSize(width: 417, height: 313)) == CGRect(x: 0, y: 0, width: 417, height: 313)) + } + + @Test("a host's own clipping setting survives the PiP fit") + @MainActor + func hostClippingRestored() { + let view = AetherPlayerView(frame: CGRect(x: 0, y: 0, width: 640, height: 392)) + #if canImport(UIKit) + view.clipsToBounds = true + #else + view.layer?.masksToBounds = true + #endif + let layer = AVSampleBufferDisplayLayer() + layer.videoGravity = .resizeAspectFill + view.attach(layer) + view.setPictureFit(CGSize(width: 1280, height: 720)) + view.setPictureFit(nil) + #expect(clips(view)) + } + + @Test("the aspect-fit rectangle is centred and never exceeds the bounds") + func fittedRect() { + let r = AetherPlayerView.hostedFrame(in: CGRect(x: 0, y: 0, width: 1200, height: 700), fitting: CGSize(width: 1920, height: 1080)) + #expect(abs(r.width - 1200) < 0.001 && abs(r.height - 675) < 0.001) + #expect(abs(r.midX - 600) < 0.001 && abs(r.midY - 350) < 0.001) + #expect(AetherPlayerView.hostedFrame(in: CGRect(x: 0, y: 0, width: 10, height: 10), fitting: nil) == CGRect(x: 0, y: 0, width: 10, height: 10)) + } + + @Test("the aspect-fill rectangle is centred and covers the bounds") + func filledRect() { + let r = AetherPlayerView.hostedFrame(in: CGRect(x: 0, y: 0, width: 1200, height: 700), + fitting: CGSize(width: 1920, height: 1080), gravity: .resizeAspectFill) + // 1200/1920 = 0.625, 700/1080 = 0.648: fill takes the larger scale. + #expect(abs(r.height - 700) < 0.001 && abs(r.width - 700 * 1920 / 1080) < 0.001) + #expect(abs(r.midX - 600) < 0.001 && abs(r.midY - 350) < 0.001) + #expect(r.minX <= 0 && r.maxX >= 1200) + } + + /// An engine with a software host (never loaded) whose display layer is presented on `view`. + @MainActor + private func engineWithSoftwareLayer(on view: AetherPlayerView) throws -> (AetherEngine, SoftwarePlaybackHost) { + let engine = try AetherEngine() + let host = SoftwarePlaybackHost() + engine.softwareHost = host + engine.bind(view: view) + return (engine, host) + } + + @Test("engine wiring: the PiP flag, the picture size and the gravity re-shape the bound layer") + @MainActor + func engineDrivesPictureFit() throws { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let (engine, host) = try engineWithSoftwareLayer(on: view) + defer { host.stop() } + let layer = host.displayLayer + #expect(layer.frame == full) + + // The size alone changes nothing; the flag alone has no picture to fit. + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + #expect(layer.frame == full) + engine.pictureInPictureActive = true + #expect(abs(layer.frame.width / layer.frame.height - 16.0 / 9.0) < 0.001) + + // A new picture size while in PiP is picked up. + engine.softwareDisplaySize = CGSize(width: 1000, height: 1000) + #expect(abs(layer.frame.width / layer.frame.height - 1) < 0.001) + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + + // A gravity change while in PiP re-lays the layer without a layout pass. + engine.videoGravity = .resizeAspectFill + #expect(layer.frame.width > 640.001 || layer.frame.height > 392.001) + #expect(clips(view)) + engine.videoGravity = .resizeAspect + #expect(layer.frame.width <= 640.001 && layer.frame.height <= 392.001) + #expect(!clips(view)) + + engine.pictureInPictureActive = false + #expect(layer.frame == full) + } + + @Test("engine wiring: presenting the layer on a view applies or clears the fit") + @MainActor + func presentingAppliesFit() throws { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let first = AetherPlayerView(frame: full) + let (engine, host) = try engineWithSoftwareLayer(on: first) + defer { host.stop() } + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + engine.pictureInPictureActive = true + + // A view bound during PiP gets the shaped layer at once. + let second = AetherPlayerView(frame: full) + engine.bind(view: second) + #expect(abs(host.displayLayer.frame.width / host.displayLayer.frame.height - 16.0 / 9.0) < 0.001) + + // A view carrying a stale fit from an earlier PiP is reset when PiP is not active. + engine.pictureInPictureActive = false + let third = AetherPlayerView(frame: full) + third.setPictureFit(CGSize(width: 1280, height: 720)) + engine.bind(view: third) + #expect(host.displayLayer.frame == full) + } + + @Test("engine wiring: ending the session drops the fit") + @MainActor + func stopDropsPictureFit() throws { + let full = CGRect(x: 0, y: 0, width: 640, height: 392) + let view = AetherPlayerView(frame: full) + let (engine, host) = try engineWithSoftwareLayer(on: view) + defer { host.stop() } + engine.softwarePiPSource = SoftwarePiPSource(layer: host.displayLayer, isLive: false, engine: engine) + engine.softwareDisplaySize = CGSize(width: 1280, height: 720) + engine.pictureInPictureActive = true + #expect(host.displayLayer.frame != full) + engine.stopInternal() + #expect(engine.softwareDisplaySize == nil) + #expect(!engine.pictureInPictureActive) + // A view the next session binds does not inherit this one's rectangle. + #expect(host.displayLayer.frame == full) + } + + @Test("ending the session resets pictureInPictureActive and clears the source") + @MainActor + func stopResetsPictureInPictureFlag() throws { + let engine = try AetherEngine() + engine.softwarePiPSource = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + engine.pictureInPictureActive = true + engine.stopInternal() + #expect(engine.softwarePiPSource == nil) + #expect(!engine.pictureInPictureActive) + #expect(!engine.isResettingPictureInPictureForStop) + } + + @Test("a window closing owes the background decision, the reset inside stopInternal does not") + @MainActor + func stopResetOwesNoBackgroundDecision() throws { + let engine = try AetherEngine() + // A window that closes (the flag went true to false) owes it. + #expect(engine.closingPictureInPictureOwesBackgroundDecision(wasActive: true)) + #expect(!engine.closingPictureInPictureOwesBackgroundDecision(wasActive: false)) + // The flag is not a close while the session teardown clears it, on every platform. + engine.isResettingPictureInPictureForStop = true + #expect(!engine.closingPictureInPictureOwesBackgroundDecision(wasActive: true)) + engine.isResettingPictureInPictureForStop = false + // Only a live flag is not a close. + engine.pictureInPictureActive = true + #expect(!engine.closingPictureInPictureOwesBackgroundDecision(wasActive: true)) + } + + @Test("state and duration changes invalidate quietly, and a released source is not retained", + .enabled(if: SoftwarePiPTransportTests.pipSupported, "PiP is not supported on this host")) + func invalidationPathReleases() async throws { + let engine = try AetherEngine() + weak var weakSource: SoftwarePiPSource? + do { + var source: SoftwarePiPSource? = SoftwarePiPSource(layer: AVSampleBufferDisplayLayer(), isLive: false, engine: engine) + weakSource = source + var controller = try #require(source?.makePictureInPictureController()) + engine.state = .paused + engine.duration = 600 + await Task.yield() + await Task.yield() + // Read the sink-refreshed snapshot directly: the main getters would refresh it live. + let snap = try #require(source).snapshot.withLock { $0 } + #expect(snap.isPaused) + #expect(abs(snap.range.duration.seconds - 600) < 0.001) + _ = controller + controller = try #require(source?.makePictureInPictureController()) + _ = controller + source = nil + } + engine.state = .playing + engine.duration = 700 + await Task.yield() + await Task.yield() + #expect(weakSource == nil) + } + #endif +} diff --git a/docs/api.md b/docs/api.md index bb443d941..0699b8a46 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1184,7 +1184,7 @@ say so on the tracker rather than working around it. | `videoGravity` | Fill mode of whichever layer is mounted (`AVPlayerLayer` or the software display layer). Settable; this is the aspect-fit / aspect-fill control. | | `AetherStillView`, `bindStillView(_:)`, `unbindStillView(_:)` | For a host that renders the native path through AVKit rather than `AetherPlayerView`. An audio-track switch swaps the item in place, and `replaceCurrentItem` blanks the player layer until the new item's first frame; the engine holds the outgoing picture over that gap on the bound still view (or, without one, on the bound `AetherPlayerView`). Transparent and input-free; place it above the video and below the host's chrome, e.g. at the top of `AVPlayerViewController.contentOverlayView`, and set its `videoGravity` to the host player's fill. The software path holds its renderer's last picture across the rebuild the same way. Dolby Vision without a displayable base layer (Profile 5, AV1 Profile 10) cannot be held as the output vends it. Without the `DolbyVisionStillReshaping` trait, the engine remuxes the resident segment's video into a standalone file and a player of the surface's own seeks it frame-exactly to the outgoing position, so the system compositor colours the held frame as it colours playback (pixel-identical to the paused outgoing picture on the clips measured; up in 165 to 220 ms on an M1). With the trait, the frame is a 960-wide still decoded through the scrub stills' RPU reshaping. Either way the switch waits for it (up to 3 s, the outgoing picture paused meanwhile); other profiles hold their base layer. Nothing is held in PiP or on external playback. | | `nativePlayerLayer` | The engine's own `AVPlayerLayer`, for a host building `AVPictureInPictureController` around a layer rather than around `currentAVPlayer`. | -| `$softwarePiPSource` | `SoftwarePiPSource` for sample-buffer PiP on the software path: the display layer plus transport answers on the enqueued frames' axis. iOS only in practice; tvOS AVKit does not evaluate sample-buffer content sources (FB9751461). | +| `$softwarePiPSource` | `SoftwarePiPSource` for sample-buffer PiP on the software path: the display layer plus transport answers on the enqueued frames' axis. iOS only in practice; tvOS AVKit does not evaluate sample-buffer content sources (FB9751461). `SoftwarePiPSource.isSupported` (static) says whether the system offers PiP at all; `makePictureInPictureController()` returns a ready `AVPictureInPictureController` (nil when `isSupported` is false) whose content source uses `playbackDelegate`, a `SoftwarePiPPlaybackDelegate` mapping play, pause, skip (clamped to the VOD duration) and the time range onto the engine's transport; the host keeps the controller and sets `pictureInPictureActive` from its callbacks. A reload replaces `softwarePiPSource` and the content source holds its delegate weakly, so rebuild the controller on every `$softwarePiPSource` change. When `state` or `duration` changes, the source invalidates the playback state of the controller it made (the latest one). When `softwarePiPSource` becomes nil (the session ended), the engine stops the controller it made and resets `pictureInPictureActive` to false; a host with its own controller closes PiP at that point and treats `pictureInPictureActive` as false. Any load on the software path ends PiP the same way, because each load builds a new display layer (the native path hands the item over instead). While `pictureInPictureActive`, a bound `AetherPlayerView` sizes the display layer to the picture's rectangle for the active `videoGravity` (aspect-fit for `resizeAspect` and `resize`, the larger aspect-fill rectangle clipped by the view for `resizeAspectFill`), because the window mirrors the layer's own geometry; it restores the full-view frame afterwards, and a gravity change during PiP is applied at once. A host that mounts `softwarePiPSource.layer` in its own view must size the layer to the picture itself. On macOS the bound view also follows the system PiP window: it is found through public AppKit (a new visible panel with the picture's aspect, when exactly one matches; the search stops after a few seconds and re-runs when the window goes away, or when the picture's aspect changes while no window is found) and the layer is laid out at its size, so the layer's own `videoGravity` fits the picture; a window matched under a known aspect is kept through later aspect changes until PiP ends, and its size is followed as the window resizes to the new picture; without a match the picture-shaped layout stays. On macOS the system draws an opaque bar over the top of that window for sample-buffer sources, covering part of the picture; there is no public API to remove it or read its height (FB22411168). | | `$softwareDisplaySize` | The rectangle the software path's picture presents at: coded size under the decoder's pixel aspect. Mirrored, not latched, so a mid-stream resolution change re-shapes it. nil on every other path. | | `sourceVideoWidth`, `sourceVideoHeight`, `sourceVideoPixelAspectRatio` | The source's CODED size and the multiplier that turns it into the presented one (`width * ratio`), read once from the probe (on `.remoteBypass`, from AVPlayer's item video track once it resolves: the delivered size). The ratio is 1 on square pixels and on a declared ratio the engine refuses (#290), never a guess; on the paths that draw, prefer what is on screen (`softwareDisplaySize`, `AVPlayerLayer.videoRect`) over recomputing it here. | | `pictureInPictureActive` | Host-set. Keeps the pipeline and the loopback server alive across a background transition, and keeps the software path decoding video for the window. |