Repository navigation
fix: guard non-causal embeddings against oversized batches - #678
Open
AvrovaDonz2026 wants to merge 1 commit into
Open
AvrovaDonz2026 wants to merge 1 commit into
AvrovaDonz2026 wants to merge 1 commit into
Conversation
Reject inputs longer than the context batch size before evaluation when the model uses non-causal attention. This avoids returning divergent embeddings for inputs that fit in the context but cannot be evaluated in one batch. Add a Nomic embedding regression test covering the default batch size. Closes: withcatai#674
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
For non-causal embedding models,
getEmbeddingFor()now rejects inputs whose tokenizedlength exceeds the context
batchSizebefore evaluation. Non-causal attention cannotproduce a whole-input embedding when the input is split across decode batches; the
previous behavior silently returned divergent vectors. Causal models keep their existing
behavior.
Fixes #674
Pull-Request Checklist
masterbranchnpm run formatto apply eslint formattingnpm run testpasses with this change (the full model-dependent suite was not run)Fixes #674Verification
npm run buildnpm run test:typescriptnpm run lint:eslintnpx vitest run ./test/modelDependent/nomicEmbedText/embedding.test.ts(5/5passed)npx vitest run ./test/standalone/gguf/ggufStandaloneParser.test.ts --testTimeout=60000(3/3passed)