Skip to content

Keep g and beta in float32 in gated_delta_update - #4760

Open
jvmenen wants to merge 1 commit into
ml-explore:mainfrom
jvmenen:fix/gdn-float32-gates
Open

jvmenen wants to merge 1 commit into
ml-explore:mainfrom
jvmenen:fix/gdn-float32-gates

Conversation

@jvmenen

@jvmenen jvmenen commented Oct 24, 2026

Copy link
Copy Markdown

Fixes #4725.

gated_delta_update cast g and beta to the dtype of q before the kernel. With bf16 activations a decay just below 1 rounds to 1.0 (0.999, 0.9995 and 0.9999 all do), so the state stops decaying at that token. The kernels already convert both to float32 right after loading; only the round trip before the kernel lost the precision.

g and beta are now cast to float32, and the forward and VJP kernels read them as const device float*. The kernel template type stays that of q, k and v, so there are no new instantiations. Float32 inputs give the same result as before; dg and dbeta are cast back to the caller's dtype by the astype in front of the primitive.

Error compared to a float64 result, for a Qwen3.5-sized layer on an M5 Pro:

inputs output error final state error time
bf16 q/k/v, main 4.8e-2 5.0e-2 4.91 ms
all float32 (current workaround) 1.8e-3 1.3e-3 6.45 ms
bf16 q/k/v, this PR 2.4e-3 1.3e-3 5.20 ms

(4096 tokens, 16 key and 48 value heads, head dim 128, decay values as in a trained model.)

test_gated_delta_half_precision_decay_near_one runs bf16 and fp16 q/k/v with decays between 0.99 and 0.9999 against the same call on float32 inputs, on CPU and on the three GPU routes. It fails on v0.32.4 (state error 2.3e-2) and passes with this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep g and beta in float32 in gated_delta_update when the activations are bf16

1 participant