Skip to content

ggml-cpu: repack kernels for targets without dotprod (ARMv8.0 64-bit) for Q4_0/Q8_0 - #32142

Open
rjtokenring wants to merge 2 commits into
ggml-org:masterfrom
rjtokenring:cpu-arm-repack-no-dotprod
Open

rjtokenring wants to merge 2 commits into
ggml-org:masterfrom
rjtokenring:cpu-arm-repack-no-dotprod

Conversation

@rjtokenring

Copy link
Copy Markdown
Contributor

Overview

ggml-cpu: repack kernels for targets without dotprod (ARMv8.0 64bit) for Q4_0/Q8_0
Tested on Arduino UNOQ with Qwen3.5 0.8B Q4_0 pure and Q8_0

Measured a max throughput increase by x2.6 on prefill, x1.6 in token generation.

llama-bench, tokens/s

model test master PR
Gemma-3-1b-it Q4_0 (Unsloth) pp128 10.30 25.89
pp512 10.17 24.84
tg128 5.85 8.93
Qwen3.5-0.8B Q4_0 pp128 12.96 31.79
pp512 12.84 31.05
tg128 6.56 9.45

llama-server, chat request with a 584-token prompt, tokens/s

model phase master PR
Qwen3.5-0.8B Q8_0 (Unsloth) prompt 11.84 23.98
generation 5.38 6.85

@rjtokenring
rjtokenring marked this pull request as ready for review November 12, 2026 14:31
@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Nov 12, 2026
@rjtokenring
rjtokenring force-pushed the cpu-arm-repack-no-dotprod branch from dfd5157 to 1fcbf43 Compare November 13, 2026 10:36
@rjtokenring

Copy link
Copy Markdown
Contributor Author

@ServeurpersoCom please verify. Thanks.

@ServeurpersoCom ServeurpersoCom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on a Cortex-A53 emulated with qemu in my rootless container. This uses qemu user mode, so it needs no KVM, no binfmt and no privileges: qemu translates the aarch64 binary process by process, and with -cpu cortex-a53 any dotprod instruction would crash with SIGILL.

cmake -B build -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a
qemu-aarch64 -L /usr/aarch64-linux-gnu -cpu cortex-a53 build/bin/llama-perplexity -m Qwen3-0.6B-Q4_0.gguf ...

Correctness looks good: the new kernels give the same results as the generic ones, and the int16 sums don't overflow even with every value at its max. A full mul_mat through the repack buffer also matches the plain CPU path, row tails included.

With this PR, Qwen3-0.6B Q4_0 on the emulated A53 gets the same perplexity (9.8626) as the existing dotprod repack on an emulated A76.

Emulation says nothing about speed, so for the performance numbers I trust your measurements on the real board.

@rjtokenring Nit: the ggml_hsum_4x_s16 comment says lanes must stay below 8192, but the pairwise add only needs below 16384.

@rjtokenring
rjtokenring force-pushed the cpu-arm-repack-no-dotprod branch from 11e1b5e to 2d677c3 Compare November 16, 2026 11:42
@rjtokenring

Copy link
Copy Markdown
Contributor Author

@ServeurpersoCom for completeness, here is a bech performed on a Raspberry Pi 4.
Performance increase is evident and reproducible also on this platform.

Raspberry Pi 4 Model B (4x Cortex-A72 @ 1.5 GHz, no dotprod), gcc 14.2 -mcpu=cortex-a72, llama-bench -fa 1 -ub 512, master 2d8a32502 vs this PR, mean of 2 interleaved runs.

4 threads

model test master t/s PR t/s speedup
Qwen3.5-0.8B Q4_0 (all tensors Q4_0) pp128 16.57 34.02 2.05x
Qwen3.5-0.8B Q4_0 (all tensors Q4_0) pp512 16.29* 33.79 2.07x*
Qwen3.5-0.8B Q4_0 (all tensors Q4_0) tg128 5.29 5.36 1.01x
Qwen3.5-0.8B Q4_0 (unsloth) pp128 16.32 30.38 1.86x
Qwen3.5-0.8B Q4_0 (unsloth) pp512 16.10* 30.29 1.88x*
Qwen3.5-0.8B Q4_0 (unsloth) tg128 4.82 4.95 1.03x
Qwen3.5-0.8B Q8_0 (unsloth) pp128 17.41 27.62 1.59x
Qwen3.5-0.8B Q8_0 (unsloth) pp512 17.26* 27.92 1.62x*
Qwen3.5-0.8B Q8_0 (unsloth) tg128 3.29 3.54 1.08x
gemma-3-1b-it Q4_0 (unsloth) pp128 13.53 30.27 2.24x
gemma-3-1b-it Q4_0 (unsloth) pp512 12.96* 29.70 2.29x*
gemma-3-1b-it Q4_0 (unsloth) tg128 4.55 4.62 1.01x

1 thread

model test master t/s PR t/s speedup
Qwen3.5-0.8B Q4_0 (all tensors Q4_0) pp128 4.54 10.51 2.32x
Qwen3.5-0.8B Q4_0 (all tensors Q4_0) tg128 2.65 3.98 1.50x

@rjtokenring

Copy link
Copy Markdown
Contributor Author

Tested on a Cortex-A53 emulated with qemu in my rootless container. This uses qemu user mode, so it needs no KVM, no binfmt and no privileges: qemu translates the aarch64 binary process by process, and with -cpu cortex-a53 any dotprod instruction would crash with SIGILL.

cmake -B build -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a
qemu-aarch64 -L /usr/aarch64-linux-gnu -cpu cortex-a53 build/bin/llama-perplexity -m Qwen3-0.6B-Q4_0.gguf ...

Correctness looks good: the new kernels give the same results as the generic ones, and the int16 sums don't overflow even with every value at its max. A full mul_mat through the repack buffer also matches the plain CPU path, row tails included.

With this PR, Qwen3-0.6B Q4_0 on the emulated A53 gets the same perplexity (9.8626) as the existing dotprod repack on an emulated A76.

Emulation says nothing about speed, so for the performance numbers I trust your measurements on the real board.

@rjtokenring Nit: the ggml_hsum_4x_s16 comment says lanes must stay below 8192, but the pairwise add only needs below 16384.

@ServeurpersoCom comment fixed, thanks!

@rjtokenring

Copy link
Copy Markdown
Contributor Author

@max-krasnyansky please review

@gatewaytofredom

Copy link
Copy Markdown

If you build this, #32142, without dotprod the repack might be making prompt processing slower for cpu inference because repacked weights can't use Accelerate BLAS. The BLAS backend can only accept host buffers and repacked weights are not host buffers.

Tested on my macmini m4 with -mcpu=apple-a7 to simulate a build for iOS and visionOS builds which don't have dotproduct. Prompt processing slowed down, but token generation speed is up.

Cmake command:
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_METAL=OFF -DCMAKE_C_FLAGS=-mcpu=apple-a7 -DCMAKE_CXX_FLAGS=-mcpu=apple-a7

Model tested: SmolLM2-360M-Instruct

Q4_0:

model size params backend threads rpk test t/s
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 pp128 1015.60 ± 11.06
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 pp512 984.45 ± 3.92
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 tg128 79.63 ± 0.12
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 pp128 1129.34 ± 4.94
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 pp512 1451.15 ± 9.78
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 tg128 219.27 ± 0.44
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 pp128 217.10 ± 0.29
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 pp512 204.70 ± 0.15
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 tg128 114.52 ± 0.17
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 pp128 785.95 ± 16.95
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 pp512 750.36 ± 1.43
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 tg128 253.78 ± 0.94

build: 2d677c3 (20)

Q8_0:

model size params backend threads rpk test t/s
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 pp128 1184.81 ± 4.42
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 pp512 1015.73 ± 5.52
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 tg128 98.24 ± 0.13
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 pp128 1209.68 ± 25.02
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 pp512 1469.27 ± 15.83
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 tg128 169.30 ± 0.19
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 pp128 192.51 ± 0.15
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 pp512 182.27 ± 0.17
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 tg128 128.98 ± 0.35
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 pp128 717.36 ± 0.31
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 pp512 676.76 ± 0.95
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 tg128 171.65 ± 0.17

build: 2d677c3 (20)

@rjtokenring

Copy link
Copy Markdown
Contributor Author

If you build this, #32142, without dotprod the repack might be making prompt processing slower for cpu inference because repacked weights can't use Accelerate BLAS. The BLAS backend can only accept host buffers and repacked weights are not host buffers.

Tested on my macmini m4 with -mcpu=apple-a7 to simulate a build for iOS and visionOS builds which don't have dotproduct. Prompt processing slowed down, but token generation speed is up.

Cmake command: cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DGGML_METAL=OFF -DCMAKE_C_FLAGS=-mcpu=apple-a7 -DCMAKE_CXX_FLAGS=-mcpu=apple-a7

Model tested: SmolLM2-360M-Instruct

Q4_0:

model size params backend threads rpk test t/s
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 pp128 1015.60 ± 11.06
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 pp512 984.45 ± 3.92
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 0 tg128 79.63 ± 0.12
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 pp128 1129.34 ± 4.94
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 pp512 1451.15 ± 9.78
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 0 tg128 219.27 ± 0.44
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 pp128 217.10 ± 0.29
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 pp512 204.70 ± 0.15
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 1 1 tg128 114.52 ± 0.17
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 pp128 785.95 ± 16.95
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 pp512 750.36 ± 1.43
llama 3B Q4_0 217.39 MiB 361.82 M BLAS 4 1 tg128 253.78 ± 0.94
build: 2d677c3 (20)

Q8_0:

model size params backend threads rpk test t/s
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 pp128 1184.81 ± 4.42
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 pp512 1015.73 ± 5.52
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 0 tg128 98.24 ± 0.13
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 pp128 1209.68 ± 25.02
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 pp512 1469.27 ± 15.83
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 0 tg128 169.30 ± 0.19
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 pp128 192.51 ± 0.15
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 pp512 182.27 ± 0.17
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 1 1 tg128 128.98 ± 0.35
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 pp128 717.36 ± 0.31
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 pp512 676.76 ± 0.95
llama 3B Q8_0 366.80 MiB 361.82 M BLAS 4 1 tg128 171.65 ± 0.17
build: 2d677c3 (20)

The CPU buffer order is ACCEL, GPU host, CPU extra (repack), CPU, and BLAS uses the plain CPU buffer type, so it is skipped.
Master already applies the same precedence for Q4_0/Q8_0 with dotprod and the default macOS target has dotprod, so standard macOS builds already skip Accelerate for these types.
Also -mcpu=apple-a7 does not limit Accelerate, which still uses the M4 matrix unit at runtime.
On Cortex-A53, OpenBLAS gave +3% pp vs about 2x from repack.
The "--no-repack" restores the old behavior.

@max-krasnyansky max-krasnyansky left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't get a chance to try it on my setups but the repack/ukernel changes look good to me.

@max-krasnyansky

Copy link
Copy Markdown
Member

@rjtokenring take a look at the CI failures. EditorConfig, etc.
I'll kick it again one you push the updates.

@rjtokenring
rjtokenring force-pushed the cpu-arm-repack-no-dotprod branch from 2d677c3 to 28f6159 Compare November 18, 2026 21:01
@rjtokenring

Copy link
Copy Markdown
Contributor Author

Thanks @max-krasnyansky! Failures (EditorConfig) were in master and has been fixed after my rebase.
I rebased now with master again: please kick CI jobs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants