Repository navigation
ggml-cpu: repack kernels for targets without dotprod (ARMv8.0 64-bit) for Q4_0/Q8_0 - #32142
rjtokenring wants to merge 2 commits into
Conversation
dfd5157 to
1fcbf43
Compare
|
@ServeurpersoCom please verify. Thanks. |
ServeurpersoCom
left a comment
There was a problem hiding this comment.
Tested on a Cortex-A53 emulated with qemu in my rootless container. This uses qemu user mode, so it needs no KVM, no binfmt and no privileges: qemu translates the aarch64 binary process by process, and with -cpu cortex-a53 any dotprod instruction would crash with SIGILL.
cmake -B build -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a
qemu-aarch64 -L /usr/aarch64-linux-gnu -cpu cortex-a53 build/bin/llama-perplexity -m Qwen3-0.6B-Q4_0.gguf ...
Correctness looks good: the new kernels give the same results as the generic ones, and the int16 sums don't overflow even with every value at its max. A full mul_mat through the repack buffer also matches the plain CPU path, row tails included.
With this PR, Qwen3-0.6B Q4_0 on the emulated A53 gets the same perplexity (9.8626) as the existing dotprod repack on an emulated A76.
Emulation says nothing about speed, so for the performance numbers I trust your measurements on the real board.
@rjtokenring Nit: the ggml_hsum_4x_s16 comment says lanes must stay below 8192, but the pairwise add only needs below 16384.
11e1b5e to
2d677c3
Compare
|
@ServeurpersoCom for completeness, here is a bech performed on a Raspberry Pi 4. Raspberry Pi 4 Model B (4x Cortex-A72 @ 1.5 GHz, no dotprod), gcc 14.2 4 threads
1 thread
|
@ServeurpersoCom comment fixed, thanks! |
|
@max-krasnyansky please review |
|
If you build this, #32142, without dotprod the repack might be making prompt processing slower for cpu inference because repacked weights can't use Accelerate BLAS. The BLAS backend can only accept host buffers and repacked weights are not host buffers. Tested on my macmini m4 with Cmake command: Model tested: Q4_0:
build: 2d677c3 (20) Q8_0:
build: 2d677c3 (20) |
The CPU buffer order is ACCEL, GPU host, CPU extra (repack), CPU, and BLAS uses the plain CPU buffer type, so it is skipped. |
max-krasnyansky
left a comment
There was a problem hiding this comment.
I didn't get a chance to try it on my setups but the repack/ukernel changes look good to me.
|
@rjtokenring take a look at the CI failures. EditorConfig, etc. |
2d677c3 to
28f6159
Compare
|
Thanks @max-krasnyansky! Failures (EditorConfig) were in master and has been fixed after my rebase. |
Overview
ggml-cpu: repack kernels for targets without dotprod (ARMv8.0 64bit) for Q4_0/Q8_0
Tested on Arduino UNOQ with Qwen3.5 0.8B Q4_0 pure and Q8_0
Measured a max throughput increase by x2.6 on prefill, x1.6 in token generation.
llama-bench, tokens/s
llama-server, chat request with a 584-token prompt, tokens/s