Repository navigation
cuda: exclude sm_75 from the short-query in-place quantized-K/V MMA FA - #361
Conversation
bri-prism
left a comment
There was a problem hiding this comment.
Agent review: posted by the maintainer's coding agent at their request.
Looks correct on reading. The new predicate only drops the exact-Turing case: turing_mma_available(cc) becomes IS_NVIDIA && highest_compiled_arch > TURING. That matches the existing Turing guard a few lines below in the same function. When the branch is skipped, dispatch falls through to the 8-column MMA case for the same type_KV, which your 2080 Ti run exercises. Ampere and newer are unchanged. I can't run sm_75 here.
Follow-up with runtime evidence on the Turing path, from a second Hyper session. The test used an sm_75-only build (
On sm_90 the change is a no-op:
This conflicts with #337 on the same line, but #337 is superseded by #355, so this one is the one to keep. |
Overview
Fixes the Turing GPU failure reported in #348.
In
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1, skip the F16 MMA path on Turing GPUs (sm_75). That kernel does not support these GPUs, and using it can make prompt processing fail with the following message:On
sm_75, this keeps dispatch on the existing compatible path for this case.Additional information
This change does not add a new Flash Attention kernel or enable the MMA path on Turing.
I have tested this change on my machine with RTX 2080 ti with the following command line invocation:
Requirements