Repository navigation
Metal: a failed GPU command buffer still returns an image as success #19
Description
Seen in leehack/llamadart#1033. On iPhone 16 Pro (A18 Pro), a GPU watchdog restart during the VAE decode makes ggml-metal report command buffer 0 failed with status 5 ... kIOGPUCommandBufferCallbackErrorInnocentVictim. sd.cpp still returns the decoded image as success, and that image is corrupt (in our run, 88% of pixels non-zero against 99.9% in clean runs). The next generate then fails with "backend is in error state".
Expected: report the failed decode as an error so callers can fail loudly. This belongs in the wrapper or upstream, not llamadart (zero-patch rule).
Activity
leehack commentedon Oct 15, 2026
Triage: P1 / owner correctness. This is an existing confirmed false-success report, not a newly proven regression from a recent PR. A failed Metal command buffer must not produce a successful image result. Consumer #1048 mitigates iOS direct-VAE use and later error-state handling, but does not replace the owner contract repair.
Acceptance should include controlled backend-failure coverage that discards invalid image results and reports the failure; correct reload/disposal behavior; ordinary-success and CPU negative controls; and affected physical Metal evidence when available. Do not count software rendering as device-GPU proof. Keep this fix separate from the post-v0.2 upstream qualification in #20 unless that exact candidate demonstrably fixes it. No publication or paid/device run is authorized by this triage.
leehack commentedon Oct 19, 2026
Delivery evidence — 2026-10-19: #21, merged as
ea609150a6a55746a2b98ddc7955b60e07867cce, is published in v0.2.0-4 after successful release run 39150529681. This owner guard detects recorded Metal failures even when generation returns image buffers, discards invalid outputs and prevents reuse until reload. Controlled failure/cleanup/success tests and healthy SD Turbo CPU/actual macOS Metal smokes are recorded in the PR; this is not physical-iPhone failure reproduction or watchdog prevention (core #1047).The owner upstream remains
3f8527a46c54ecf4cb4ed6003da8e8982283c73c. Public core 0.15.1 still pins v0.2.0-3; merged core #1205 adopts v0.2.0-4 and #1208 supplies patch-compatible qualified companion handling / older-runtime error backstop. Core and required SD companion 0.15.2 publication/consumer verification remain pending. Owner source correction and public-package delivery are distinct; #20's raw-source/ABI qualification remains separate. Leaving the existing closed state unchanged.leehack commentedon Oct 20, 2026
Consumer delivery verified: core #1213 / 0.15.2 published the Metal failure guard adoption and patch-compatible companion handling. Published core 0.15.4 and SD companion 0.15.2 select owner v0.2.0-4 manifest, still upstream
3f8527a46c54ecf4cb4ed6003da8e8982283c73c. Qualified locked older companions keep their documented old-runtime/backstop route. This satisfies the previously pending public-consumer delivery boundary, not induced physical-iPhone watchdog/OOM recovery or newer raw source; core #1047 remains separate. Leaving the closed state unchanged.