Skip to content

cuda: exclude sm_75 from the short-query in-place quantized-K/V MMA FA - #361

Merged
bri-prism merged 1 commit into
PrismML-Eng:prismfrom
umbertov:fix/348-fattn-mma-f16
Oct 24, 2026
Merged

bri-prism merged 1 commit into
PrismML-Eng:prismfrom
umbertov:fix/348-fattn-mma-f16

Conversation

@umbertov

Copy link
Copy Markdown

Overview

Fixes the Turing GPU failure reported in #348.

In ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1, skip the F16 MMA path on Turing GPUs (sm_75). That kernel does not support these GPUs, and using it can make prompt processing fail with the following message:

ERROR: CUDA kernel flash_attn_ext_f16 has no device code compatible with CUDA arch 750. ggml-cuda.cu was compiled for: 750,800,860,890,900,1200,1210

On sm_75, this keeps dispatch on the existing compatible path for this case.

Additional information

This change does not add a new Flash Attention kernel or enable the MMA path on Turing.

I have tested this change on my machine with RTX 2080 ti with the following command line invocation:

./build/bin/llama-server \
  -hf prism-ml/Ternary-Bonsai-2-27B-gguf:TQ1_0 \
  --jinja \
  -ngl 99 \
  -c 131072 \
  --ubatch-size 512 \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  --port 8080 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00

Requirements

  • I have read and agree with the contributing guidelines: yes
  • AI usage disclosure: yes. AI supported the codebase exploration which led to the code change. I reviewed the change and am responsible for it. This PR was not automatically generated.

@bri-prism bri-prism left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent review: posted by the maintainer's coding agent at their request.

Looks correct on reading. The new predicate only drops the exact-Turing case: turing_mma_available(cc) becomes IS_NVIDIA && highest_compiled_arch > TURING. That matches the existing Turing guard a few lines below in the same function. When the branch is skipped, dispatch falls through to the 8-column MMA case for the same type_KV, which your 2080 Ti run exercises. Ampere and newer are unchanged. I can't run sm_75 here.

@bri-prism

Copy link
Copy Markdown
Collaborator

Agent review: posted by the maintainer's coding agent at their request.

Follow-up with runtime evidence on the Turing path, from a second Hyper session. The test used an sm_75-only build (CMAKE_CUDA_ARCHITECTURES=75) on an H200, so ggml_cuda_highest_compiled_arch returns 750 and the Turing branch is taken. Results for test-backend-ops FLASH_ATTN_EXT, prism e3de715 vs. this change:

K/V prism this PR
q8_0 aborts (rc 134) 511/511
q4_0 aborts (rc 134) 605/605

On sm_90 the change is a no-op:

  • FA f16, bf16, q8_0 and q4_0 all pass.
  • KLD is identical with a q8_0 KV cache.
  • tg128 is 271.7 vs 271.9 tok/s on 9B PQ2_0.

This conflicts with #337 on the same line, but #337 is superseded by #355, so this one is the one to keep.

@bri-prism
bri-prism merged commit 2b11c2f into PrismML-Eng:prism Oct 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants