Debug training differences with your AI agent.

Website · OSDI ’26 Paper · Live Demo · Agent Harness

Give your coding agent two training runs. OpGuard helps it find where their computations diverge and inspect the evidence.

Find the root cause

Standard logs · Expand ↗	OpGuard output · Expand ↗
The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. …	First observed difference · rank 0 get_grad_norm_direct · None.append#2246 run1: [0, 1077945344, 0, 0] run2: [0, 1077804032, 0, 0] Unaligned before boundary: 0 / 33 …
~19k input tokens · 8 calls Budget ≤ $0.010	~1.5k input tokens · 1 call Budget ≤ $0.0012
Get rootcause	Get rootcause

Actual root cause

A gradient buffer is overwritten on the default stream before its reduction on another stream finishes. Details ↗

Groq · GPT-OSS 20B · Same model and diagnostic prompt; same-launch evidence. Full logs read in eight parts (a few minutes); OpGuard uses first observed differences. Fixed inference budget caps at Groq list prices, including prompts, reading notes and maximum output; free within quota.

Get started

Install from PyPI with Python 3.10+. The training environment needs CUDA-enabled PyTorch and CUDA/CUPTI development tools.

# Training environment (Linux)
python -m pip install "setuptools>=77.0.3" "packaging>=24.2" wheel ninja
python -m pip install --no-build-isolation opguard

# Agent environment
python -m pip install opguard-harness

Installation details →

Then ask your coding agent:

Use OpGuard to investigate why these two training runs differ: ./run-a/train.py and ./run-b/train.py. My training Python is /path/to/train-env/bin/python. Read opguard-harness tools for the available tools, and explain your findings with evidence.

Works with coding agents that can run shell commands. Agent setup and tool calls →

Impact

Deployed at ByteDance, OpGuard has helped diagnose 20+ production issues, including kernel races and silent data corruption. The paper reports investigations shortened from days to minutes. Read the results →

OSDI ’26 · Best Paper Award nominee. Project overview →

Installation · CLI guide · Examples · Apache-2.0

Paper

OSDI ’26 Paper (PDF) · OpGuard was presented at OSDI ’26 (Best Paper Award nominee):

Ziming Zhou, Yinjie Zhao, Hang Zhu, Wenxiao Wang, Zhihao Bai, Yun Zhang, Shuguang Wang, Haibin Lin, Peng Huang.
OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training.
In Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’26), Seattle, WA, USA, July 2026.

BibTeX

@inproceedings{OpGuard2026OSDI,
  author = {Zhou, Ziming and Zhao, Yinjie and Zhu, Hang and Wang, Wenxiao and Bai, Zhihao and Zhang, Yun and Wang, Shuguang and Lin, Haibin and Huang, Peng},
  title = {{OpGuard}: Bitwise Alignment for Precise and General Debugging of Production {LLM} Training},
  booktitle = {Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation},
  series = {OSDI '26},
  month = {July},
  year = {2026},
  address = {Seattle, WA, USA},
  publisher = {USENIX Association},
}

For SDC machine detection—identifying GPUs that silently corrupt training computations—see SDCs in the Wild: Characterizing and Diagnosing SDC-Defective GPUs in Production LLM Training (OSDI ’26). The paper introduces SDCHunter, which replays the triggering training workload and input data to detect defective GPUs in production clusters.

Contributing

See CONTRIBUTING.md. The website and live demo remain in this repository alongside the Python packages.

About ByteDance Seed Team

Founded in 2023, the ByteDance Seed Team develops advanced AI foundation models. It aims to advance science and benefit society through world-class research.