AuscuTSLM: Time-Series Language Models for Auscultation Audio

AuscuTSLM aligns multi-site cardiac and respiratory auscultation recordings with a frozen Large Language Model via Flamingo-style gated cross-attention, moving beyond isolated classification toward open-ended, patient-level clinical question answering. It is an audio-focused extension of OpenTSLM.

On the CaReSound benchmark, AuscuQA reaches 0.865 F1-macro and 0.952 BERTScore, improving on the fine-tuned CaReAQA baseline across every metric, and shows that a lightweight, scratch-trained raw-waveform encoder rivals large pretrained audio-language models for this task.

Architecture

Audio waveform(s) (up to 30s, one per anatomical site)
    → Audio Encoder (RawAudioTokenizer / Mel / Wav2Vec2 / Whisper / CLAP)
    → Perceiver Resampler (compresses multi-site, multi-clip tokens → K fixed latents)
    → Gated Cross-Attention, inserted between frozen LLM blocks
    → Frozen LLM (Llama-3.2-1B)
    → Free-text answer

Only the encoder (if trainable), the MLP projector, the Perceiver Resampler, and the gated cross-attention adapters are trained (~1.4B trainable params); the LLM backbone stays frozen. This is AudioFlamingo in src/opentslm/model/llm/AudioFlamingo.py . The accompanying paper describes the trained checkpoints and their associated results.

Encoder options

Encoder	--encoder	Trainable	Notes
RawAudioTokenizer	tokenizer	✅	Scratch-trained, 40ms raw-waveform patches. Recommended / paper's main result.
MelSpectrogramTokenizer	mel	✅	Scratch-trained, log-mel + 2D CNN.
Wav2Vec2Encoder	wav2vec2	❄️ frozen	Pretrained on LibriSpeech (speech).
WhisperEncoder	whisper	❄️ frozen	Pretrained multilingual speech recognition.
CLAPEncoder	clap	❄️ frozen	LAION CLAP, pretrained on audio-text pairs.

The paper found the lightweight tokenizer (raw waveform) encoder rivals or beats the large pretrained encoders on this task — see Table 2 in the paper for the full ablation.

Installation

Requires Python ≥3.12.

git clone https://github.com/Fan-loewe/AudioTSLM.git
cd AudioTSLM

# uv (recommended)
command -v uv > /dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --all-groups
source .venv/bin/activate

# or pip
pip install -r requirements.txt -r audio_requirements.txt

LLM access

The default backbone is meta-llama/Llama-3.2-1B. Request access on Hugging Face and authenticate:

huggingface-cli login

Data preparation

CaReSound QA metadata is loaded automatically from tsnngw/CaReSound on the Hugging Face Hub; the underlying audio comes from the public source datasets it's built from (ICBHI, KAUH, CirCor, SPRSound, ZCHSound).

# 1. Download and organize source audio
python scripts/data_prep/audio_data_preparation.py --polybox_url "<your audio archive URL>"

# 2. Merge SPRSound's per-segment clips into one file per patient/position
python scripts/data_prep/audio_data_preparation.py --merge_sprsound

# 3. Verify filenames match CaReSound's patient IDs (writes audio_manifest.json)
python scripts/data_prep/audio_data_preparation.py --verify --audio_dir ./data/caresound_audio/audio_merged

The patient-disjoint train/val/test split used for every result in the paper (2,064 / 440 / 447 patients) is checked in at data/caresound_patient_splits.json, generated by scripts/data_prep/create_patient_splits.py.

Training

python scripts/training/audio_curriculum_learning.py \
    --encoder tokenizer \
    --audio_dir ./data/caresound_audio/audio_merged \
    --llm_id meta-llama/Llama-3.2-1B \
    --batch_size 2 \
    --num_epochs 10 \
    --device cuda

Key arguments (python scripts/training/audio_curriculum_learning.py --help for the full list):

Argument	Default	Description
--encoder	tokenizer	One of tokenizer, mel, wav2vec2, whisper, clap
--llm_id	meta-llama/Llama-3.2-1B	Frozen LLM backbone
--max_audio_length	480000	Samples at 16kHz — 160000/320000/480000 = 10s/20s/30s
--num_latents	64	Perceiver Resampler latent count
--cross_attn_every_n	1	Insert gated cross-attention every N LLM layers
--batch_size / --grad_accum_steps	2 / —	Effective batch size = product of the two

GPU memory guidance

VRAM	Suggested flags
8–12 GB (RTX 3060/3070)	--batch_size 1 --grad_accum_steps 8, consider a smaller --llm_id
16–24 GB (RTX 3090/4090)	--batch_size 2 --grad_accum_steps 4
40+ GB (A100/H100)	--batch_size 8 --grad_accum_steps 2

Cross-attention cost scales with n_text × n_audio, so --max_audio_length and --num_latents are the two biggest levers on memory if you hit OOM.

Evaluation

python scripts/evaluation/evaluate_audio_model.py \
    --checkpoint audio_checkpoints_v5/tokenizer_lat64/tokenizer_stage1_caresound_qa/best_model.pt \
    --encoder tokenizer \
    --llm_id meta-llama/Llama-3.2-1B \
    --audio_dir ./data/caresound_audio/audio_merged

Reports open-ended metrics (Contains-Match, ROUGE-L, METEOR, BERTScore) and binary yes/no classification metrics (accuracy, F1, sensitivity, specificity), overall and per source dataset.

Baseline comparison

scripts/comparison/ holds the code behind the zero-shot/fine-tuned baselines in the paper's Table 2 (Audio-Flamingo3, Qwen2-Audio, Qwen2.5-Omni, CaReAQA); results live in comparison/ at the repo root. Keep this code and its outputs in sync if you add another baseline.

Pretrained weights

The checkpoint behind the paper's headline result is published on Hugging Face: fanwufan/AuscuQA.

Repository layout

scripts/
├── training/     audio_curriculum_learning.py
├── evaluation/   evaluate_audio_{model,cot}.py, recompute_*, test_audio_*
├── data_prep/    audio_data_preparation.py, construct_circor_qa.py, create_patient_splits.py, generate_audio_cot.py
├── comparison/   af3_benchmark_eval.py, qwen_audio_benchmark_eval.py, fix_qwen_*.py
└── utils/        check_openai_balance.py

See scripts/README.md for what each script does. The accompanying paper describes the trained checkpoints and their associated results.

Model code lives under src/opentslm/: model/encoder/ (the 5 encoders above), model/llm/AudioFlamingo.py (the Flamingo model), model/projector/MLPProjector.py, and time_series_datasets/audio/ (CareSoundDataset).

Citation

Wu, F., Wang, T.N., Zumarraga, N., Wang, N., Kreft, M., O'Sullivan, K., Fleisch, E., Aalami, O., Schmiedmayer, P., Jakob, R. and Langer, P., 2026. Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings. arXiv preprint arXiv:2603.13362.

Copyright and License

MIT — see LICENSE.md.

© 2026 ETH Zurich Created by: Fan Wu (ETH Zurich)

This repository includes modified portions of OpenTSLM. Their original copyright and MIT licensing notices are retained in the relevant source files.