Repository navigation
Conversation
bri-prism
left a comment
There was a problem hiding this comment.
Agent review: posted by the maintainer's coding agent at their request. Measurements are from a second agent session on an Arc B390 (Xe3, Windows, oneAPI 2025.3.2).
Tested on an Arc B390, merged onto the current tip 73bfe6cc4.
- test-backend-ops: MUL_MAT 1445/1445, CONCAT 192/192, GET_ROWS 123/123.
- Greedy output: byte-identical to tip on 6/6 prompts (2B and 9B).
- 27B MTP: output and draft acceptance identical to tip.
- Speed on Xe3: roughly neutral. 27B tg +1.3% (MTP +1.3%), 9B tg −0.7%. pp512 is 1.3–2.7% lower on all three models, with the same sign in both rounds (ABBA order).
The gain is probably smaller than on your B50 because tip already fuses the GDN conv chain ("fused DeltaNet conv chain active"), so the concat case this targets may not occur there. I haven't verified that per node. Could you check pp on the B50 against the current tip? If it holds there, this looks fine to merge.
The concat kernels gave every row its own work-group of up to 256 work-items, so a concat of 4-element rows (the conv state of gated delta net layers, [3,10240] + [1,10240]) launched 10240 work-groups with 4 active lanes each and took ~27 us on an Arc Pro B50. The strided kernel now maps one work-item to each dst element, and contiguous concats with rows shorter than a work-group use it too (~12 us for that case).
With one sub-group per row, a mat-vec with few rows leaves most of the GPU idle and is bound by load latency: the 48-row bf16 ssm_alpha/ssm_beta projections of Qwen3.5 (K = 5120) took 24-35 us each. Up to 512 rows, each row now gets a 256-wide work-group that splits K and reduces over the group (~12 us).
Single 4-byte copies reached ~110 GB/s on the 3 MB recurrent state rows of gated delta net layers (54 us per layer on an Arc Pro B50, timed per node). When the rows and strides are 16-byte aligned, f32/i32 rows are now copied as 4-element vectors (24 us).
5de20c9 to
cab0ab2
Compare
|
Rebased onto the current tip ( On the Arc Pro B50 (Xe2, Linux), against the same tip:
A sequential run (all of tip first, then this PR) showed this PR 1-3% lower on pp, the same sign you saw, but the gap went away when the runs were interleaved, so I read it as drift over the length of the run, not the kernels. tg128 is neutral (8B 71.4 vs 71.2, 27B 22.5 vs 22.6).
(sequential run; the interleaved 8B numbers above are the ones to compare) |
Overview
Three small SYCL fixes for ops that were slow because they left most of the GPU idle. All three show up in the decode of Qwen3.5 gated delta net models (Bonsai 2 27B), where they cost about 3.7 ms per token on an Arc Pro B50:
[3,10240] + [1,10240]) launched 10240 work-groups with 4 active lanes each. The strided kernel now maps one work-item to each dst element, and contiguous concats with rows shorter than a work-group use it too. 28.4 us -> 12.4 us per call.ssm_alpha/ssm_betaprojections (K = 5120) took ~30 us each. Up to 512 rows, each row now gets a 256-wide work-group that splits K and reduces over the group. 32.7 us -> 14.2 us.Per-op times are from a per-node profile (eval callback with a sync after each node, so they include a few us of sync overhead), Bonsai 2 27B PQ2_0, one token. Normal decode goes from 51.6 to 47.9 ms per token on that profile.
Results
Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero 1.14.37020, Linux 7.0
xedriver,-ngl 99, Ternary Bonsai models. "prism" is the currentprismhead (6bfcd79). Both were measured in the same session.llama-bench, t/s:The Qwen3 models (1.7B, 4B) have no gated delta net layers and are within 2% (tg64 of the 4B is 1.4% lower). pp512 of the 27B is 1-2% lower; I have not tracked down why.
Parallel decode (
llama-batched-bench -npp 128 -ntg 64), aggregate decode t/s, 27B:The state get_rows and the conv-state concat run once per sequence, so their share grows with the number of sequences.
Correctness
test-backend-ops -b SYCL0, run per op over all 130 ops: CONCAT 192/192, GET_ROWS 119/119, MUL_MAT 1401/1401 (includes the f16/bf16 mat-vec cases). The ops that do not pass fully (CONV_2D 2010/2026, FLASH_ATTN_EXT 4070/4073, LIGHTNING_INDEXER, ROLL 1/2; CPY and SET_ROWS abort) fail identically on prism without this PR.-c 512, 5 chunks of the repository'sdocs/*.md): identical to prism to all printed digits for 1.7B b1/b512, 4B b1/b512, 1.7B-ngl 0, 4B-ngl 20, 27B PQ2_0 and PTQ1_0 b512.-dev none --no-op-offload): identical to prism (b1 0.00127, b512 0.00127; same top token 98.4% / 97.6%).Additional information
Independent of the XMX kernel PRs; it helps any model with gated delta net layers or small f16/bf16 projections.
Requirements