TraCE: Trajectory-Aware Concepts for Explainable Video Understanding

NeurIPS 2026

Wooil Lee^1†, Junpyo Hong^1†, Jongseo Lee¹, Tuomas Oikarinen², Tsui-Wei Weng², Jinwoo Choi^1*

¹Kyung Hee University, ²University of California San Diego  ·  ^†Equal contribution, ^*Corresponding author

---

✨ Highlights

• 🔍 Proposes TraCE, a concept-based framework that makes which motion source supports a video prediction explicit.
• 🌀 Turns clusters of optical-flow trajectories into concepts for people, objects and interactions, alongside object and scene concepts.
• 🎬 One typed concept interface explains both action recognition and video question answering.
• 🔄 Supports exact per-concept contributions, concept interventions and concept removal tests.

---

🛠️ Installation

conda create -n trace python=3.10 -y
conda activate trace

git clone https://github.com/KHU-VLL/TraCE.git && cd TraCE
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

---

📁 Project Structure

TraCE/
│── src/            # discovery, features, ar, vqa, explain, third_party
│── tools/          # command-line entry points (discovery, extract, train, eval, explain)
│── scripts/        # one pipeline script per dataset
│── configs/        # per-dataset settings (ar/, vqa/, discovery/)
│── annotations/    # splits, class lists, QA files
│── concepts/       # object / scene vocabularies
│── notebooks/      # evaluation and intervention demos
│── requirements.txt
│── README.md

---

📂 Dataset

We used the following datasets in our experiments:

• NTU RGB+D (Mutual)
• Something-Something V2 (Chiral)
• UCF101
• KTH Action
• TOMATO
• STAR

Splits and QA files are in annotations/<dataset>/. Backbone weights (VideoMAE, InternVideo2) go in pretrained/.

---

🔎 Trajectory Concept Discovery

VIDEO_ROOT=data/ntu_mutual/videos bash scripts/ntu_mutual.sh trajectory   # trajectory concepts
VIDEO_ROOT=data/ntu_mutual/videos bash scripts/ntu_mutual.sh text         # object / scene concepts

---

🚀 Training

VIDEO_ROOT=data/ntu_mutual/videos bash scripts/ntu_mutual.sh ar    # action recognition
VIDEO_ROOT=data/ntu_mutual/videos bash scripts/ntu_mutual.sh vqa   # video question answering

---

📊 Evaluation & Visualization

python tools/eval_ar.py --bundle runs/ar/ntu_mutual --features work/ntu_mutual/features_ar/val_videomae.pt \
    --annotation annotations/ntu_mutual/val.csv --removal
python tools/explain_ar.py --bundle runs/ar/ntu_mutual --video clip.mp4 --overlay 3 --out_dir outputs/explain_ar

Evaluation and concept intervention demos are included in:

• notebooks/demo_action_recognition.ipynb
• notebooks/demo_video_qa.ipynb

📎 Citation

@inproceedings{lee2026trace,
  title={TraCE: Trajectory-Aware Concepts for Explainable Video Understanding},
  author={Lee, Wooil and Hong, Junpyo and Lee, Jongseo and Oikarinen, Tuomas and Weng, Tsui-Wei and Choi, Jinwoo},
  booktitle={NeurIPS},
  year={2026}
}

---

📜 License

This project is licensed under the Apache 2.0 License.

---

🙏 Acknowledgements

This work builds upon: DANCE, Label-free-CBM, Multi-Modal-CBM