Repository navigation
feat(teapot): Name the faulting shader of a GPU crash from uploaded DXIL binaries - #129684
Draft
tustanivsky wants to merge 18 commits into
Draft
tustanivsky wants to merge 18 commits into
tustanivsky wants to merge 18 commits into
Conversation
This was referenced Nov 3, 2026
tustanivsky
force-pushed
the
feat/gpu-shader-symbols
branch
from
November 3, 2026 15:48
06e5fcc to
8e206d6
Compare
Register two debug file kinds for GPU shader symbolication: dxil (DXC shader binary, DXBC magic, code_id = 16-hex Aftermath hash) and dxilpdb (DXC shader PDB, MSF magic with an ILDN part). Both share the DXC hash as debug_id and arrive through the existing chunk-upload and DIF assemble API, which now accepts an optional code_id. When teapot answers "partial" with dxil/dxil_pdb entries in missing_difs, look the files up by code_id / debug_id, presign the objectstore-backed ones and call teapot once more with shader_binaries / shader_pdbs and a "-2" idempotency key. Any failure in that round falls back to the first response. Add SENTRY_TEAPOT_INLINE_ATTACHMENT_FALLBACK (default off) so a dev server whose Relay keeps attachments out of objectstore can still hand teapot presigned URLs.
Resolve shader DIFs with one presign helper and emit the teapot request entries directly, instead of a dataclass and four lookup helpers. Inline the dxil code_id and shader_group validation like the other DIF kinds and reduce the shader group backfill to its condition.
The native frame renderer shows package, function and filename only, so the group teapot puts into frame.module never reached the page. Write it to the gpu.shader_group, gpu.material and gpu.shader_class tags, the gpu_crash context, the culprit and the frame's filename when UE's sentinel or nothing fills that slot.
Relay streams every attachment to objectstore before the GPU crash task runs, so an attachment without a stored_id is an error, not a case to paper over.
The assemble endpoint answers from the existing row for a known checksum and never runs the task, so a group that a later cook supplies for an already uploaded dxil binary was dropped.
The Linux decoder never reads a shader PDB, so only the DXIL binary is stored, resolved and sent to teapot. The dxilpdb kind, the sibling PDB lookup and the shader_pdbs request field return with line numbers.
tustanivsky
force-pushed
the
feat/gpu-shader-symbols
branch
from
November 3, 2026 16:00
8e206d6 to
57d7064
Compare
tustanivsky
marked this pull request as ready for review
November 4, 2026 08:17
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3e1c53a. Configure here.
tustanivsky
marked this pull request as draft
November 4, 2026 08:54
BYK
pushed a commit
to getsentry/toolkit
that referenced
this pull request
Nov 4, 2026
## Summary Sentry can now name the shader that caused a GPU crash, but only if it has the game's compiled shaders. Unreal Engine builds export them as `.dxil` files (one per shader, over ten thousand per game). This PR lets `sentry debug-files upload` upload them as `dxil` debug files and makes the upload path fast enough for sets of that size. This is the toolkit version of getsentry/sentry-cli#3489, which was implemented earlier to test with the Unreal SDK which has not migrated to the Sentry toolkit yet (getsentry/sentry-unreal#1680). ## Changes - The scanner recognises a DXIL container by its header, before the files reach the object parser, which does not know shaders. The debug id is the 32-hex compiler hash the engine uses as the file name. `dxil` joins the `--type` list; `--id` accepts the bare file name or the hyphenated id. Shaders have none of the symbol/unwind/source features the other formats are filtered on, so those filters do not apply to them. - The GPU crash decoder identifies a shader by NVIDIA's own hash of it, which only NVIDIA's Windows library can compute. The Sentry Unreal plugin computes it at build time and writes `sentry-shader-manifest.tsv` next to the shaders. The CLI reads that manifest once per directory and sends the hash and the shader group as the new `code_id` and `shader_group` assemble fields. Without a manifest the shader is uploaded without them. - Assemble requests are sent in batches of a thousand files. - Missing chunks are uploaded in batched requests, packed up to the server's `chunksPerRequest` and `maxRequestSize` limits, with a chunk shared by several files sent once. They were sent one request per chunk, so ten thousand small files took a lot of time. - The assemble request is repeated right after the chunks are delivered, and the processing wait starts from that point. Before, the wait started before the upload; when the upload alone outlasted it, the command exited without the request that starts processing, and reported the files as not found or timed out although every chunk was on the server. The last three points apply to every debug file type. Files that are not shaders otherwise take exactly the old path. `debug-files find` and `check` do not learn about shaders here. ## Possible Follow-ups Shader PDBs would add source file and line context to GPU crashes. They are left out for now because the server cannot use them yet, but the manifest already lists the PDB that belongs to each binary. ## Related - getsentry/sentry#129684 adds the `dxil` debug file format and the assemble fields - getsentry/teapot#22 names the shader from the binary - getsentry/sentry-unreal#1678 exports and hashes the shaders after the cook - getsentry/sentry-cli#3489 is the legacy CLI version of this change
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

When a game's GPU hangs or faults, the NVIDIA Aftermath SDK writes a crash dump and Sentry sends it to Teapot (Sentry's GPU crash decoder) to turn it into an event. The dump identifies the shader that was running only by a 64-bit hash. Teapot has no name to go with that hash, so today the event shows a placeholder such as
compute_01and the developer has to guess which shader it was.This PR closes that gap with debug files, the same way native crashes get symbolicated. A game can upload its compiled shaders (DXIL binaries) as debug files, keyed by that Aftermath hash. When Teapot reports which shaders it could not name, Sentry looks them up, hands Teapot presigned links to the uploaded binaries and asks it to decode again. The event then shows the shader's real entry point and, for material shaders, the material and pass it belongs to.
Key Changes
dxildebug file format. The debug ID is the compiler's hash from the file name, the code ID is the Aftermath hash computed by the uploader, and an optional shader group from the engine's shader symbol export is stored with the file. Both arrive as new optional fields of the assemble request; re-uploading a known binary with a group fills it in.gpu.shader_group,gpu.materialandgpu.shader_classtags, part of the GPU crash context, and the frame's filename when Teapot provides none, so it is visible in the issue list and on the stack trace.No migration. Events without uploaded shaders are processed exactly as before.
Known Limitations
Source file and line numbers are out of scope. Mapping a DXIL shader back to source needs the shader PDBs and the Windows build of the Aftermath library, which Teapot does not run on.
Testing
Unit tests cover the debug file format, the lookup and presigning, the second round and its fallbacks, and the event fields. Verified end to end against a local stack with a UE 5.8 game, for a global compute shader and a material pixel shader: both events completed with the shader name and group and no missing debug files.
Before:
After:
Related Items
--type dxil(getsentry/sentry-cliDuring artifact lookup, also consider full paths w/o host #3489 is the legacy CLI version)