English | 简体中文

EvaOmni : An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Xiaoxing Hu^†, Wenxi Yao^*, Chaoyuan Shi^*, Ziqi Ye^*, Xiaorong Guo, Peixuan Li, Zhuoran Sun, Ziyang Gong, Peiyuan Zhang, Mingxin Liu, Shuran Ma, Hongbo Liu, Xiaomin Yu, Yu Tian, Wentao Wang, Zhenjie Yang, Jian Hu, Yunhang Shen, Haoyu Cao, and Xue Yang^✉

^† Project Lead · ^* Core Contributions · ^✉ Corresponding Author

If EvaOmni is useful for your research, please consider starring this repository and citing our work! ⭐⭐⭐

📖 Motivation & Overview

EvaOmni is an evidence-grounded benchmark for evaluating audio-visual reasoning in omni-modal large language models. Existing multiple-choice benchmarks often score only whether a model selects the correct option. A model can therefore obtain credit from textual, logical, or single-modality shortcuts without identifying the evidence that supports its answer. EvaOmni requires a model to return both an answer and timestamped audio and visual evidence. The benchmark reports answer accuracy together with evidence coverage and temporal grounding, making it possible to measure whether a correct answer is supported by the relevant moments in both modalities.

Shortcut evaluation across benchmarks, with examples of text-only, audio-only, and visual-only shortcuts.

✨ Highlights

• Evidence-grounded evaluation: models predict an answer together with audio and visual evidence intervals.
• Audio-visual reasoning: questions are designed to require complementary information from the two modalities.
• Shortcut screening: construction includes text-only, audio-only, and visual-only checks.
• Fine-grained taxonomy: 1,027 questions from 392 videos cover 15 capabilities across 3 reasoning levels.
• Grounded metrics: Coverage Grounded Score (C.G.) measures evidence coverage, while Bottleneck Grounded Score (B.G.) measures the weaker of the two modality-localization scores.
• Transparent evaluation: this release includes a runnable metric evaluator, Gemini and MiMo adapters, a prediction schema, and a worked example.

Annotation pipeline and a grounded question example with timestamped audio and visual evidence.

📊 Benchmark Statistics

Item	Value
Videos	392
Question-answer pairs	1,027
Fine-grained capabilities	15
Reasoning levels	3
Video duration	1–12 minutes
Evaluation frame rate	1.0 FPS
Evaluation resolution	480 pixels on the longer side

Capability taxonomy and dataset statistics, including video duration, token length, evidence, and video categories.

Capability levels

• Level 1 — Retrieval and Grounding: temporal alignment, event binding, and entity binding.
• Level 2 — Basic Understanding: causal reasoning, emotional intelligence, intent understanding, atmosphere understanding, implicit event reasoning, semantic disambiguation, temporal understanding, spatial understanding, and conditional counting.
• Level 3 — Complex Reasoning: multi-hop reasoning, long-range reasoning, and multi-evidence chains.

🔥 Benchmark Performance

The current manuscript evaluates 7 open-weight and 17 proprietary Omni LLMs, with a human expert reference baseline. The table below reports Acc., C.G., and B.G. for all three capability levels and overall. All values are percentages and higher is better. Within each model group, bold indicates the best results and underlining indicates the second-best results. These are manuscript results, not outputs generated by this starter release.

Model	Level 1 R.&G.			Level 2 B.U.			Level 3 C.R.			Overall
	Acc.	C.G.	B.G.	Acc.	C.G.	B.G.	Acc.	C.G.	B.G.	Acc.	C.G.	B.G.
Open-Weight Omni LLMs
Qwen2.5-Omni-7B	16.6	1.2	0.6	17.4	0.5	0.0	13.7	0.3	0.0	16.4	0.8	0.3
Qwen3-Omni-Instruct-30B-A3B	28.6	5.1	0.6	22.4	3.4	0.4	22.2	2.4	0.1	25.2	4.0	0.4
Qwen3-Omni-Thinking-30B-A3B	26.3	11.4	4.8	24.4	6.2	1.2	21.6	4.1	0.6	24.8	8.3	2.8
Nemotron3-Nano-Omni-30B-A3B	30.7	15.0	5.3	22.1	13.1	3.9	10.5	4.7	0.7	24.2	12.6	4.0
MiMo-V2.5-310B-A15B	65.4	61.3	30.8	52.3	42.3	15.1	45.1	29.6	6.7	56.8	48.4	20.5
MiMo-V2.6-Flash-310B-A15B	61.3	55.3	29.0	48.6	41.4	14.5	45.8	28.6	5.9	53.6	45.3	19.3
MiMo-V2.6-Pro-1T-A42B	60.1	51.3	28.8	44.3	35.2	12.0	49.7	26.6	5.0	51.8	40.7	18.1
Proprietary Omni LLMs
Doubao-Seed-2.0-Lite	54.1	36.4	10.8	54.1	36.1	11.7	62.7	32.2	5.9	55.4	35.6	10.4
Doubao-Seed-2.1-Lite	78.8	74.8	38.0	67.0	61.5	22.6	59.5	42.7	9.4	70.9	64.3	27.2
Qwen3.5-Omni-Flash	47.5	41.9	15.5	30.9	24.0	7.2	29.4	15.7	2.7	37.7	30.4	10.0
Qwen3.5-Omni-Plus	58.1	52.0	22.1	42.3	26.0	8.6	42.5	19.8	3.3	49.0	36.0	13.5
Qwen3.8-Omni-Flash	65.0	53.6	19.3	52.3	41.4	11.8	56.2	35.9	7.8	58.2	45.7	14.4
Gemini2.5-Flash	59.2	55.2	26.7	39.3	31.4	11.8	41.8	24.6	4.0	48.1	40.4	16.9
Gemini2.5-Flash-Lite	53.0	47.2	22.1	30.9	22.4	8.0	27.5	14.8	2.2	39.7	31.8	13.1
Gemini2.5-Pro	76.5	73.8	42.3	65.0	55.0	22.5	72.5	45.8	9.4	71.0	61.6	28.9
Gemini3.1-Flash-Lite	59.4	52.8	24.2	36.6	30.7	10.5	32.0	19.0	3.0	45.6	38.3	15.2
Gemini3.1-Flash-Lite-Preview	59.7	53.6	25.2	39.3	31.1	10.3	38.6	23.2	3.3	47.8	39.4	15.6
Gemini3.1-Pro-Preview	74.7	72.8	41.5	68.6	59.8	24.8	76.5	48.7	11.2	72.3	63.7	29.8
Gemini3.5-Flash	64.1	59.4	28.8	69.5	61.4	27.4	77.8	47.8	11.3	68.5	58.6	25.6
Gemini3.5-Flash-Lite	60.6	55.6	26.5	38.2	27.9	10.7	42.5	24.2	3.5	48.3	39.1	16.3
Gemini3.6-Flash	80.9	78.0	44.7	68.4	61.3	26.1	73.2	44.2	10.6	74.4	65.8	31.7
Gemini3.7-Flash	85.3	84.0	49.1	73.6	65.3	29.2	79.1	49.1	12.3	79.4	70.8	35.1
Gemini3.8-Flash	83.4	81.9	48.4	73.9	63.3	27.4	74.5	43.1	9.5	78.0	68.2	33.6
Gemini4-Argon	67.1	55.9	29.3	52.0	33.7	13.0	72.5	33.5	7.5	61.4	43.0	19.1
Reference baseline
Human Expert	97.8	97.8	95.8	96.5	94.8	93.2	94.4	93.8	93.0	96.8	96.0	94.3

Gemini3.7-Flash achieves 79.4% accuracy but 35.1% B.G. overall. This gap illustrates why answer correctness and evidence localization should be assessed together. Humans may pause and replay videos in the reported baseline; viewing conditions differ from model inference.

📦 Dataset

Our dataset is hosted on Hugging Face and ModelScope respectively. Use either ModelScope or Hugging Face:

• ModelScope: VisionXLab/EvaOmni-Dataset
• Hugging Face: VisionXLab/EvaOmni-Dataset

Download command:

pip install modelscope
modelscope download --dataset VisionXLab/EvaOmni-Dataset --local_dir ./data

or

pip install huggingface_hub
huggingface-cli download --repo-type dataset --resume-download VisionXLab/EvaOmni-Dataset --local-dir ./data

Note

Use resized_video to reproduce the paper's results, or rawvideo (original videos) for custom preprocessing.

🚀 Quick Start

Clone the repository.

git clone https://github.com/VisionXLab/EvaOmni.git
cd EvaOmni

1. Download the dataset

Download the dataset using either command above. The resulting directory structure is:

data/
├── test_file.json
├── rawvideo/
│   ├── 001.mp4
│   ├── 002.mp4
│   └── ...
└── resized_video/
    ├── 001.mp4
    ├── 002.mp4
    └── ...

2. Choose an adapter and run the evaluation

Each provider has its own adapter under scripts/. All of them read the same three variables from the environment; nothing is hardcoded and API_KEY is never written to the output files. Copy .env.example to .env (gitignored) and fill it in, or export the variables in your shell.

Adapter	Protocol	Supported models
scripts/eval_gemini.py	Gemini generateContent REST	Gemini family, e.g. gemini-3.8-flash
scripts/eval_mimo.py	OpenAI-compatible chat completions	MiMo family, e.g. mimo-v2.6-pro, mimo-v2.6-flash

Common variables:

Variable	Meaning
BASE_URL	Endpoint URL. See the per-adapter notes below for the accepted forms.
API_KEY	Provider API key. Gemini sends it as Authorization: Bearer, MiMo via the OpenAI SDK.
MODEL_NAME	Model identifier passed to the provider.

Note

MiMo-v2.5 is no longer served by the API. Use mimo-v2.6-pro or mimo-v2.6-flash.

Gemini family

BASE_URL accepts either a prefix, to which the script appends /${MODEL_NAME}:generateContent, or a full template containing the {model} placeholder, which is used verbatim:

# 1. Prefix form — the script appends /<model>:generateContent
export BASE_URL="https://<your-host>/v1beta/models"

# 2. Template form — used as-is, for relays whose path differs
export BASE_URL="https://<your-host>/v1beta/models/{model}:generateContent"

export BASE_URL="https://<your-host>/v1beta/models"
export API_KEY="<your-api-key>"
export MODEL_NAME="gemini-3.8-flash"

python scripts/eval_gemini.py \
  --annotation data/test_file.json \
  --video-root data/resized_video \
  --results-dir outputs/gemini \
  --workers 4 \
  --retries 3

The script prints the resolved request URL at startup, so a misconfigured relay path is visible before any video is uploaded. The wrapper scripts/eval.sh runs the same command and accepts ANNOTATION_FILE, VIDEO_ROOT, RESULTS_DIR, WORKERS, RETRIES, and LIMIT overrides:

MODEL_NAME="gemini-3.8-flash" bash scripts/eval.sh

MiMo family

BASE_URL is the OpenAI-compatible base, typically ending in /v1. Videos are sent inline as base64 data URLs, so no upload step is required.

export BASE_URL="https://<your-host>/v1"
export API_KEY="<your-api-key>"
export MODEL_NAME="mimo-v2.6-flash"

python scripts/eval_mimo.py \
  --annotation data/test_file.json \
  --video-root data/resized_video \
  --results-dir outputs/mimo \
  --workers 4 \
  --retries 3

Swap MODEL_NAME for mimo-v2.6-pro to evaluate the Pro variant. Use a separate --results-dir per model so checkpoints do not mix, or pass --tag mimo_pro to keep both runs in one directory. --fps controls the sampling rate sent to the provider (default 1.0, matching the benchmark protocol).

Smoke test

Both adapters accept --limit N to evaluate only the first N questions. Point the results at a scratch directory so the partial run is not mistaken for a completed checkpoint:

MODEL_NAME="mimo-v2.6-flash" python scripts/eval_mimo.py --limit 5 --results-dir outputs/smoke
MODEL_NAME="gemini-3.8-flash" python scripts/eval_gemini.py --limit 5 --results-dir outputs/smoke

--qa-ids 1,2,3 and --levels 1,2 select specific questions or reasoning levels instead.

Resuming and outputs

Both scripts display a live progress view and save each completed result. Rerun the same command to resume: successful questions are skipped and failed questions are retried. Each question is counted once in the summary even if it was attempted several times. Raw responses and human-readable logs are saved in the results directory.

TODO: Adapters for the remaining model families evaluated in the paper (Qwen-Omni, Nemotron-Omni, Doubao-Seed, and the local vLLM pipelines) will be released incrementally.

For provider-specific adapters or an API-free orchestration check, see scripts/eval_example.py. The example supports --mock --limit 8 and writes grounded metric snapshots without making network requests.

📚 Citation

@article{hu2026evaomni,
  title={EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning},
  author={Hu, Xiaoxing and Yao, Wenxi and Shi, Chaoyuan and Ye, Ziqi and Guo, Xiaorong and Li, Peixuan and Sun, Zhuoran and Gong, Ziyang and Zhang, Peiyuan and Liu, Mingxin and Ma, Shuran and Liu, Hongbo and Yu, Xiaomin and Tian, Yu and Wang, Wentao and Yang, Zhenjie and Hu, Jian and Shen, Yunhang and Cao, Haoyu and Yang, Xue},
  journal={arXiv preprint arXiv:2610.28444},
  year={2026}
}

📄 License

Repository code is released under the MIT License. For dataset annotation and media terms, refer to the dataset hosting pages.

🙏 Acknowledgements

We thank all the annotators, reviewers, and open-source communities whose work made this benchmark possible. Please open an issue if you find a broken link, an annotation problem, or an evaluation mismatch.