Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4725.
gated_delta_updatecastgandbetato the dtype ofqbefore the kernel. With bf16 activations a decay just below 1 rounds to 1.0 (0.999, 0.9995 and 0.9999 all do), so the state stops decaying at that token. The kernels already convert both to float32 right after loading; only the round trip before the kernel lost the precision.gandbetaare now cast to float32, and the forward and VJP kernels read them asconst device float*. The kernel template type stays that ofq,kandv, so there are no new instantiations. Float32 inputs give the same result as before;dganddbetaare cast back to the caller's dtype by theastypein front of the primitive.Error compared to a float64 result, for a Qwen3.5-sized layer on an M5 Pro:
(4096 tokens, 16 key and 48 value heads, head dim 128, decay values as in a trained model.)
test_gated_delta_half_precision_decay_near_oneruns bf16 and fp16 q/k/v with decays between 0.99 and 0.9999 against the same call on float32 inputs, on CPU and on the three GPU routes. It fails on v0.32.4 (state error 2.3e-2) and passes with this change.