Skip to content

feat(teapot): Name the faulting shader of a GPU crash from uploaded DXIL binaries - #129684

Draft
tustanivsky wants to merge 18 commits into
getsentry:masterfrom
tustanivsky:feat/gpu-shader-symbols
Draft

tustanivsky wants to merge 18 commits into
getsentry:masterfrom
tustanivsky:feat/gpu-shader-symbols

Conversation

@tustanivsky

@tustanivsky tustanivsky commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

When a game's GPU hangs or faults, the NVIDIA Aftermath SDK writes a crash dump and Sentry sends it to Teapot (Sentry's GPU crash decoder) to turn it into an event. The dump identifies the shader that was running only by a 64-bit hash. Teapot has no name to go with that hash, so today the event shows a placeholder such as compute_01 and the developer has to guess which shader it was.

This PR closes that gap with debug files, the same way native crashes get symbolicated. A game can upload its compiled shaders (DXIL binaries) as debug files, keyed by that Aftermath hash. When Teapot reports which shaders it could not name, Sentry looks them up, hands Teapot presigned links to the uploaded binaries and asks it to decode again. The event then shows the shader's real entry point and, for material shaders, the material and pass it belongs to.

Key Changes

  • New dxil debug file format. The debug ID is the compiler's hash from the file name, the code ID is the Aftermath hash computed by the uploader, and an optional shader group from the engine's shader symbol export is stored with the file. Both arrive as new optional fields of the assemble request; re-uploading a known binary with a group fills it in.
  • Second Teapot round. When Teapot's response is partial, Sentry collects the shader binaries it reported missing, presigns the ones the project has uploaded and calls Teapot again with a distinct idempotency key. Any failure in this round keeps the first response and does not touch the circuit breaker.
  • Shader group on the event. It becomes the culprit, the gpu.shader_group, gpu.material and gpu.shader_class tags, part of the GPU crash context, and the frame's filename when Teapot provides none, so it is visible in the issue list and on the stack trace.
  • Housekeeping: one dxil file is kept per debug ID when redundant debug files are cleaned up.

No migration. Events without uploaded shaders are processed exactly as before.

Known Limitations

Source file and line numbers are out of scope. Mapping a DXIL shader back to source needs the shader PDBs and the Windows build of the Aftermath library, which Teapot does not run on.

Testing

Unit tests cover the debug file format, the lookup and presigning, the second round and its fallbacks, and the event fields. Verified end to end against a local stack with a UE 5.8 game, for a global compute shader and a material pixel shader: both events completed with the shader name and group and no missing debug files.

Before:

image

After:

image

Related Items

Register two debug file kinds for GPU shader symbolication: dxil
(DXC shader binary, DXBC magic, code_id = 16-hex Aftermath hash) and
dxilpdb (DXC shader PDB, MSF magic with an ILDN part). Both share the
DXC hash as debug_id and arrive through the existing chunk-upload and
DIF assemble API, which now accepts an optional code_id.

When teapot answers "partial" with dxil/dxil_pdb entries in
missing_difs, look the files up by code_id / debug_id, presign the
objectstore-backed ones and call teapot once more with shader_binaries
/ shader_pdbs and a "-2" idempotency key. Any failure in that round
falls back to the first response.

Add SENTRY_TEAPOT_INLINE_ATTACHMENT_FALLBACK (default off) so a dev
server whose Relay keeps attachments out of objectstore can still hand
teapot presigned URLs.
Resolve shader DIFs with one presign helper and emit the teapot request
entries directly, instead of a dataclass and four lookup helpers. Inline the
dxil code_id and shader_group validation like the other DIF kinds and reduce
the shader group backfill to its condition.
The native frame renderer shows package, function and filename only, so
the group teapot puts into frame.module never reached the page. Write it
to the gpu.shader_group, gpu.material and gpu.shader_class tags, the
gpu_crash context, the culprit and the frame's filename when UE's
sentinel or nothing fills that slot.
Relay streams every attachment to objectstore before the GPU crash task
runs, so an attachment without a stored_id is an error, not a case to
paper over.
The assemble endpoint answers from the existing row for a known checksum
and never runs the task, so a group that a later cook supplies for an
already uploaded dxil binary was dropped.
The Linux decoder never reads a shader PDB, so only the DXIL binary is
stored, resolved and sent to teapot. The dxilpdb kind, the sibling PDB
lookup and the shader_pdbs request field return with line numbers.
@tustanivsky
tustanivsky force-pushed the feat/gpu-shader-symbols branch from 8e206d6 to 57d7064 Compare November 3, 2026 16:00
@tustanivsky
tustanivsky marked this pull request as ready for review November 4, 2026 08:17
@tustanivsky
tustanivsky requested review from a team as code owners November 4, 2026 08:18

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3e1c53a. Configure here.

Comment thread src/sentry/lang/native/gpu.py Resolved
Comment thread src/sentry/api/endpoints/debug_files.py Resolved
@tustanivsky
tustanivsky marked this pull request as draft November 4, 2026 08:54
BYK pushed a commit to getsentry/toolkit that referenced this pull request Nov 4, 2026
## Summary

Sentry can now name the shader that caused a GPU crash, but only if it
has the game's compiled shaders. Unreal Engine builds export them as
`.dxil` files (one per shader, over ten thousand per game). This PR lets
`sentry debug-files upload` upload them as `dxil` debug files and makes
the upload path fast enough for sets of that size.

This is the toolkit version of getsentry/sentry-cli#3489, which was
implemented earlier to test with the Unreal SDK which has not migrated
to the Sentry toolkit yet (getsentry/sentry-unreal#1680).

## Changes

- The scanner recognises a DXIL container by its header, before the
files reach the object parser, which does not know shaders. The debug id
is the 32-hex compiler hash the engine uses as the file name. `dxil`
joins the `--type` list; `--id` accepts the bare file name or the
hyphenated id. Shaders have none of the symbol/unwind/source features
the other formats are filtered on, so those filters do not apply to
them.
- The GPU crash decoder identifies a shader by NVIDIA's own hash of it,
which only NVIDIA's Windows library can compute. The Sentry Unreal
plugin computes it at build time and writes `sentry-shader-manifest.tsv`
next to the shaders. The CLI reads that manifest once per directory and
sends the hash and the shader group as the new `code_id` and
`shader_group` assemble fields. Without a manifest the shader is
uploaded without them.
- Assemble requests are sent in batches of a thousand files.
- Missing chunks are uploaded in batched requests, packed up to the
server's `chunksPerRequest` and `maxRequestSize` limits, with a chunk
shared by several files sent once. They were sent one request per chunk,
so ten thousand small files took a lot of time.
- The assemble request is repeated right after the chunks are delivered,
and the processing wait starts from that point. Before, the wait started
before the upload; when the upload alone outlasted it, the command
exited without the request that starts processing, and reported the
files as not found or timed out although every chunk was on the server.

The last three points apply to every debug file type. Files that are not
shaders otherwise take exactly the old path. `debug-files find` and
`check` do not learn about shaders here.

## Possible Follow-ups

Shader PDBs would add source file and line context to GPU crashes. They
are left out for now because the server cannot use them yet, but the
manifest already lists the PDB that belongs to each binary.

## Related

- getsentry/sentry#129684 adds the `dxil` debug file format and the
assemble fields
- getsentry/teapot#22 names the shader from the binary
- getsentry/sentry-unreal#1678 exports and hashes the shaders after the
cook
- getsentry/sentry-cli#3489 is the legacy CLI version of this change

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant