Skip to content

ggml-cuda: Add GCN HIP sparse fattn - #30370

Draft
thelittlefireman wants to merge 4 commits into
ggml-org:masterfrom
thelittlefireman:feature_GCN_SPARSE_FATTN_GFX906
Draft

thelittlefireman wants to merge 4 commits into
ggml-org:masterfrom
thelittlefireman:feature_GCN_SPARSE_FATTN_GFX906

Conversation

@thelittlefireman

@thelittlefireman thelittlefireman commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Enable sparse at fattn on GCN. This could be extended to other arch without wmma.

Additional information

qwen QSA grid (256/256, GQA 12, F16 K/V, n_kv_max=2048)

KV nq master us/run branch us/run speedup
16384 1 140.4 66.6 +111%
16384 2 172.9 96.1 +80%
16384 4 256.7 129.4 +98%
16384 8 411.3 200.4 +105%
16384 16 767.8 366.1 +110%
16384 32 1463.4 656.7 +123%
16384 64 3006.3 1263.8 +138%
32768 1 258.9 102.3 +153%
32768 2 338.3 133.6 +153%
32768 4 496.7 167.1 +197%
32768 8 784.9 244.1 +222%
32768 16 1470.0 418.3 +251%
32768 32 2880.1 725.2 +297%
32768 64 5807.5 1384.4 +319%
65536 1 505.6 180.4 +180%
65536 2 644.2 211.1 +205%
65536 4 965.1 243.0 +297%
65536 8 1488.4 324.4 +359%
65536 16 2901.5 502.7 +477%
65536 32 5935.5 834.2 +611%
65536 64 11594.3 1527.6 +659%

256/256 GQA 12, KV=32768, n_kv_max=2048

nq master us/run branch us/run speedup
128 11617.9 2118.9 +448%
256 22616.5 4107.8 +451%
512 44465.6 8062.1 +452%
1024 89488.2 16042.5 +458%
2048 178023.6 31982.1 +457%
4096 354443.4 63874.3 +455%

(decode nq=1, ~25% budget)

geometry min_kv (table) gain at min_kv peak gain (kv)
72/72 8192 +43% +78% (16k)
80/80 16384 +248% +248% (16k)
96/96 16384 +228% +283% (24k)
112/112 16384 +243% +243% (16k)
128/128 8192 +53% +64% (24k)
192/128 16384 +93% +109% (12k)
256/256 4096 n/a (gate dense) +111% (nq=1@16k)
320/256 65536 +74% +163% (128k)
512/512 16384 +61% +346% (64k)
576/512 32768 +33% +107% (64k)

End-to-end model benchmarks (llama-bench, t/s, higher is better)

model (arch) test master t/s branch t/s speedup
micro-qwen4exp Q4_K_M (qwen4exp QSA 256/256 gqa12) pp32768 10696.5 14837.2 +38.7%
micro-qwen4exp Q4_K_M tg1024 250.9 252.9 +0.8%
RalphSeek V4 Flash IQ4_NL pp65536 2812.8 2922.3 +3.9%
RalphSeek V4 Flash IQ4_NL tg512 425.9 426.3 +0.1%

Requirements

@github-actions github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Oct 12, 2026
@thelittlefireman
thelittlefireman force-pushed the feature_GCN_SPARSE_FATTN_GFX906 branch from a9d72b5 to e835868 Compare October 12, 2026 17:05
@thelittlefireman thelittlefireman changed the title ggml-cuda: Add HIP sparse fattn ggml-cuda: Add GCN HIP sparse fattn Nov 15, 2026
@thelittlefireman
thelittlefireman force-pushed the feature_GCN_SPARSE_FATTN_GFX906 branch from 1ad19e5 to 2dcd206 Compare November 15, 2026 17:42
@thelittlefireman

Copy link
Copy Markdown
Contributor Author

/bot review

@ggml-gh-bot

ggml-gh-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Automated code review

Review of PR #30370 (ggml-cuda: Add GCN HIP sparse fattn). Checklists run: General, ggml/backend, Security, Tests (touches ggml/src/ggml-cuda/, tests/test-backend-ops.cpp). No prior inline review context exists on the PR.

Summary of the change

The PR wires the existing sparse flash-attention infrastructure (ggml_cuda_flash_attn_ext_compact_mask + n_kv_max hint) into the generic tile kernel for pre-CDNA AMD (GCN) GPUs, where no mma kernel is available. Gather is done in flash_attn_tile_load_tile via a per-query index list, the mask is applied at the gathered position, and a per-geometry host-side table decides when sparse beats dense.

Correctness

I verified the tricky parts and they check out:

  • Index-list layout matches launch_fattn/compact_mask exactly: the kernel receives n_kv_max (not K->ne[1]) as ne11, so KV_max + list*ne11 and the count at n_kv_max*n_lists + list are both in-bounds, and gridDim.x equals ntiles_x because the sparse launch forces ncols1 == 1 and stream_k == false.
  • The - k_VKQ_0 compensation in load_tile correctly cancels the pre-added base offset for both the K and V gathers (net row = sparse_indices[k]).
  • -1 padding entries and counts are never read: full batches guarantee k_VKQ_0 + i < count, the partial batch is bounded by i_sup == count - k_VKQ_0, and && short-circuits avoid loading indices in the zero-fill branch.
  • Gathered columns are exactly the finite-mask columns, so additive mask semantics are preserved; the count comes from the compact kernel, never from the raw hint, so an over-large hint cannot read OOB.
  • Sparse is only reachable via ncols2 > 1 paths, which all require use_gqa_opt (so max_bias == 0, mask->ne[2] == 1), and use_logit_softcap is excluded by may_use_sparse - see (point 3) below on making that less implicit.
  • Requiring both config_amd and config_amd_rdna to be nonzero is needed so the unconditionally-instantiated flash_attn_tile<..., true> variant passes the device-side static_assert in mixed gfx9/gfx11 builds. Good.

No correctness bug found.

Will slow the review

(point 1) Missing perf data. The PR description says "TODO add benchmark". Per project convention, new kernel paths are expected to come with concrete numbers (throughput on realistic shapes). This matters doubly here because ggml_cuda_fattn_tile_get_sparse_config_gcn (ggml/src/ggml-cuda/fattn.cu:157-172) is a table of unexplained magic numbers: per-geometry min_kv thresholds, min_gqa values, and a special gqa_ratio == 2 ? 32768 : 4096 case for 256/256. Reviewers will ask where each row comes from; please add the benchmark data and a short rationale (e.g. "break-even at X% sparsity measured on gfx906") before requesting review.

(point 2) PR description is essentially empty. No overview of why GCN needs this (no mma available pre-CDNA, dense tile reads the whole cache), no linked issue or related prior work (the NVIDIA/CDNA mma sparse path). Also the template requires describing how AI was used when disclosing "YES" - please fill that in.

(point 3) shall_use_sparse relies on an implicit invariant for alibi. ggml_cuda_flash_attn_ext_mma_f16_shall_use_sparse explicitly checks max_bias == 0.0f; the tile version (fattn.cu:161-182) does not, and is only safe because every ncols2 > 1 call path happens to go through use_gqa_opt. One-line check mirroring the mma version would make it robust against future call paths and easier to review.

(point 4) No Q->ne[1]-dependent guard on the sparse path. The index buffer is n_kv_max * Q->ne[1] * ne33 ints and compact_mask scans the entire mask (Q->ne[1] x K->ne[1]). For a large prefill this can be a very large allocation (e.g. 16k queries with a 2k hint is ~128 MB) plus a full mask scan, where the mma version's K->ne[1] >= 2*n_gather rule implicitly declines for large batches. Your break-even perf tests stop at nq=64. Consider bounding Q->ne[1] (equivalent of the mma n_gather rule) or provide data showing large-batch sparse is a win, not a regression.

(point 5) Test duplication across backends. The ~26 new eval cases (tests/test-backend-ops.cpp:13152+) run on every backend in CI, but only exercise the new sparse path on GCN hardware, which CI does not have; on other backends several shapes (256/256, 512/512, 576/512 with n_kv_max) overlap already-covered mma-sparse tests. The genuinely new coverage (nq>1 with independent per-query masks, 72/80/96/112 geometries) is valuable - consider trimming cases that only duplicate existing coverage to keep CI time in check.

Nits

(point 6) In flash_attn_tile (fattn-tile.cuh:971-974), (sequence % ne33)*gridDim.x + blockIdx.x mixes unsigned gridDim.x into an int list index. Values are small so it is fine, but an int64_t cast on the first factor (as the count lookup already does) would be more obviously safe.

(point 7) may_use_sparse's parameter is named ncols2 but is passed to the config helpers as the total ncols (valid only because the sparse launch pins ncols1 == 1). A short comment or renaming would prevent confusion.

Notes

  • ggml/ changes normally require two maintainer approvals - expect that.
  • Style is clean: ASCII only, concise comments, no trailing whitespace on added lines, naming and structure follow the mma-f16 sparse conventions closely.

This review was generated automatically by pi coding agent using zai-org/GLM-5.3. It may contain mistakes. Maintainers make the final call.

@thelittlefireman
thelittlefireman force-pushed the feature_GCN_SPARSE_FATTN_GFX906 branch from 89cd77e to 2c22414 Compare November 17, 2026 16:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant