Skip to content

sycl: faster concat, get_rows and few-row f16/bf16 mat-vec for short rows - #326

Open
kiljoy001 wants to merge 3 commits into
PrismML-Eng:prismfrom
kiljoy001:sycl-short-row-kernels
Open

kiljoy001 wants to merge 3 commits into
PrismML-Eng:prismfrom
kiljoy001:sycl-short-row-kernels

Conversation

@kiljoy001

Copy link
Copy Markdown

Overview

Three small SYCL fixes for ops that were slow because they left most of the GPU idle. All three show up in the decode of Qwen3.5 gated delta net models (Bonsai 2 27B), where they cost about 3.7 ms per token on an Arc Pro B50:

  • concat: the kernels gave every row its own work-group of up to 256 work-items. The conv-state concat of each gated delta net layer ([3,10240] + [1,10240]) launched 10240 work-groups with 4 active lanes each. The strided kernel now maps one work-item to each dst element, and contiguous concats with rows shorter than a work-group use it too. 28.4 us -> 12.4 us per call.
  • f16/bf16 mat-vec with few rows (dmmv): one sub-group per row leaves the GPU idle and bound by load latency. The 48-row bf16 ssm_alpha/ssm_beta projections (K = 5120) took ~30 us each. Up to 512 rows, each row now gets a 256-wide work-group that splits K and reduces over the group. 32.7 us -> 14.2 us.
  • get_rows: 4-byte copies reached ~110 GB/s on the 3 MB recurrent state rows. When rows and strides are 16-byte aligned, f32/i32 rows are copied as 4-element vectors. 54.5 us -> 24.3 us.

Per-op times are from a per-node profile (eval callback with a sync after each node, so they include a few us of sync overhead), Bonsai 2 27B PQ2_0, one token. Normal decode goes from 51.6 to 47.9 ms per token on that profile.

Results

Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero 1.14.37020, Linux 7.0 xe driver, -ngl 99, Ternary Bonsai models. "prism" is the current prism head (6bfcd79). Both were measured in the same session.

llama-bench, t/s:

model test prism this PR
1.7B PQ2_0 pp512 8198 8121
tg64 185.6 183.7
4B PQ2_0 pp512 3508 3464
tg64 106.0 104.5
27B PQ2_0 pp512 490.8 481.6
tg64 19.27 20.55 (+6.6%)
27B PTQ1_0 pp512 489.7 479.8
tg64 19.02 20.52 (+7.9%)

The Qwen3 models (1.7B, 4B) have no gated delta net layers and are within 2% (tg64 of the 4B is 1.4% lower). pp512 of the 27B is 1-2% lower; I have not tracked down why.

Parallel decode (llama-batched-bench -npp 128 -ntg 64), aggregate decode t/s, 27B:

sequences PQ2_0 prism PQ2_0 this PR PTQ1_0 prism PTQ1_0 this PR
1 18.8 19.8 18.4 19.8
2 34.5 37.1 34.3 37.1
4 55.6 62.9 55.8 62.9
8 80.5 94.5 80.1 94.8
16 98.2 121.3 98.1 121.3
32 111.0 141.0 110.8 141.1

The state get_rows and the conv-state concat run once per sequence, so their share grows with the number of sequences.

Correctness

  • test-backend-ops -b SYCL0, run per op over all 130 ops: CONCAT 192/192, GET_ROWS 119/119, MUL_MAT 1401/1401 (includes the f16/bf16 mat-vec cases). The ops that do not pass fully (CONV_2D 2010/2026, FLASH_ATTN_EXT 4070/4073, LIGHTNING_INDEXER, ROLL 1/2; CPY and SET_ROWS abort) fail identically on prism without this PR.
  • Perplexity (-c 512, 5 chunks of the repository's docs/*.md): identical to prism to all printed digits for 1.7B b1/b512, 4B b1/b512, 1.7B -ngl 0, 4B -ngl 20, 27B PQ2_0 and PTQ1_0 b512.
  • KL divergence against a CPU-only reference (1.7B PQ2_0, -dev none --no-op-offload): identical to prism (b1 0.00127, b512 0.00127; same top token 98.4% / 97.6%).

Additional information

Independent of the XMX kernel PRs; it helps any model with gated delta net layers or small f16/bf16 projections.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - the code and this description were written by Claude (Anthropic) under my direction; I tested it on my own hardware (Intel Arc Pro B50) and reviewed the changes.

@bri-prism bri-prism left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent review: posted by the maintainer's coding agent at their request. Measurements are from a second agent session on an Arc B390 (Xe3, Windows, oneAPI 2025.3.2).

Tested on an Arc B390, merged onto the current tip 73bfe6cc4.

  • test-backend-ops: MUL_MAT 1445/1445, CONCAT 192/192, GET_ROWS 123/123.
  • Greedy output: byte-identical to tip on 6/6 prompts (2B and 9B).
  • 27B MTP: output and draft acceptance identical to tip.
  • Speed on Xe3: roughly neutral. 27B tg +1.3% (MTP +1.3%), 9B tg −0.7%. pp512 is 1.3–2.7% lower on all three models, with the same sign in both rounds (ABBA order).

The gain is probably smaller than on your B50 because tip already fuses the GDN conv chain ("fused DeltaNet conv chain active"), so the concat case this targets may not occur there. I haven't verified that per node. Could you check pp on the B50 against the current tip? If it holds there, this looks fine to merge.

The concat kernels gave every row its own work-group of up to 256 work-items, so a
concat of 4-element rows (the conv state of gated delta net layers, [3,10240] + [1,10240])
launched 10240 work-groups with 4 active lanes each and took ~27 us on an Arc Pro B50.
The strided kernel now maps one work-item to each dst element, and contiguous concats with
rows shorter than a work-group use it too (~12 us for that case).
With one sub-group per row, a mat-vec with few rows leaves most of the GPU idle and is
bound by load latency: the 48-row bf16 ssm_alpha/ssm_beta projections of Qwen3.5 (K = 5120)
took 24-35 us each. Up to 512 rows, each row now gets a 256-wide work-group that splits K
and reduces over the group (~12 us).
Single 4-byte copies reached ~110 GB/s on the 3 MB recurrent state rows of gated delta net
layers (54 us per layer on an Arc Pro B50, timed per node). When the rows and strides are
16-byte aligned, f32/i32 rows are now copied as 4-element vectors (24 us).
@kiljoy001
kiljoy001 force-pushed the sycl-short-row-kernels branch from 5de20c9 to cab0ab2 Compare October 26, 2026 18:14
@kiljoy001

Copy link
Copy Markdown
Author

Rebased onto the current tip (c96baf4ab, which includes #356); the three commits are unchanged.

On the Arc Pro B50 (Xe2, Linux), against the same tip:

A sequential run (all of tip first, then this PR) showed this PR 1-3% lower on pp, the same sign you saw, but the gap went away when the runs were interleaved, so I read it as drift over the length of the run, not the kernels. tg128 is neutral (8B 71.4 vs 71.2, 27B 22.5 vs 22.6).

model tip pp512 #326 pp512 tip tg128 #326 tg128
1.7B PQ2_0 8961 8765 198.5 197.6
8B PQ2_0 2307 2262 71.4 71.2
27B PQ2_0 561 547 22.5 22.6
27B PTQ1_0 555 538 22.3 22.5

(sequential run; the interleaved 8B numbers above are the ones to compare)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants