Skip to content

ggml-openvino: initial MTMD support - #433

Draft
mostafafaheem wants to merge 45 commits into
ravi9:dev_backend_openvinofrom
mostafafaheem:unsqueeze_before_view
Draft

mostafafaheem wants to merge 45 commits into
ravi9:dev_backend_openvinofrom
mostafafaheem:unsqueeze_before_view

Conversation

@mostafafaheem

Copy link
Copy Markdown
Collaborator

Overview

  • Enables recognition of mtmd attention pattern
  • Fixes input tensor views (used by llama-mtmd-debug)
  • Op translator exception now specifies which op is causing the error

Additional information

Requirements

ngxson and others added 30 commits November 16, 2026 19:26
…ml-org#32110)

* opencl: release the WMM flash-attention mask fallback image

The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.

* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel

With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.

* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits

The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
)

A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
…g#32436)

* fix(conversion): Respect max_position_embeddings for granite hybrid

Branch: GraniteHybridContextLengthConversion
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: shorter comment w/ HF reference

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
* opencl: support non-contig silu

Assisted-by: Claude Opus 5.5

* opencl: support f16 silu

Assisted-by: Claude Opus 5.5
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
…-org#31536)

* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
…32392)

This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde0 ("CUDA: refactor MMQ
kernel configuration (ggml-org#24127)").

This code was then refactored in Commit
c49edf0 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (ggml-org#29948)") which
added the has_fusion template parameter but it kept both macro calls.

launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.

The motivation for this is to simplify the code and improve readabilty.
…gml-org#32463)

* ui: replace deprecated packages and fix npm audit findings

Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker

Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.

@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: use SvelteKit 3's update detection and Svelte 5's current patterns

A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.

Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
  checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
  attachments ({@Attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
  setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move to ESLint 10

ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.

ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: override minimatch to clear npm audit's last finding

ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.

Assisted-by: Claude (Opus 5.5 Medium)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: support Minimax-Music3

* fix and clean up
* ui: bring @ggml-org/llama-ui into ui/lib/ui

The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI design system: tokens, surfaces and brand

The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts

Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.

ui/lib/ui
- the 17 base components the app had and the library did not (alert,
  alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
  drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
  skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
  app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
  the consumer imports before the theme, as it does Tailwind
- 0.3.0

ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
  @ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
  scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
  imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
  are removed; components.json moves to the library

Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
  rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
  as its source

The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: AGENTS.md places Llama UI in ui/lib/ui

Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code

The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open submenus outside the menu they come from

A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw submenus like the menu they open from

Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.

The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a late current-node write off a closed or switched conversation

updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.

Assisted-by: Claude (Fable 5.2)

* ui: have Llama Chat use the library's Model UI and app components

Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.

ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
  sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
  and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
  (actions, content, forms, misc) and @ggml-org/llama-ui/models

ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
  ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
  come from the library; Chat's copies and the two constants only they
  used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
  id and applies the user's display settings, ModelOrgAvatar hides the
  avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
  svelte/reactivity has no file of its own, so a library source importing
  it got its types from ui/lib/ui's own install when there was one

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI takes the brand's colours and logo from llama-brand

@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.

- primitives.css drops the 25 tokens it held copies of (the neutral
  greys, sand, the skies and cinder) and builds on llama-brand's
  css/colors.css instead; the skies' gradients point at llama-brand's.
  Apps import that file before theme.css, as they do Tailwind and
  tw-animate-css: the library's stylesheets import no package, so an app
  that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
  node_modules, like the library's other dependencies

The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat take the library's constants, enums and helpers

Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.

- the library gets entry points for them: @ggml-org/llama-ui/constants,
  @ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
  @ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
  the byte units, the memory-fit constants and minMemoryTierGb, the bit
  depth labels, the capability and modality icons, labels and order,
  ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
  formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
  ModelDownloadProgress are supersets of the library's (the latter extends
  it now), and formatFileSize counts in 1024s

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the chat form's add key and model selector a tighter shadow

In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.

- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
  the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
  selector trigger
- DESIGN.md names it next to the other elevations

Assisted-by: Claude (Opus 5.5 Medium)
…led FP8) (ggml-org#32476)

* convert : fix FP8 n-gram table export for qwen4exp

* convert : dequantize block-scaled FP8 weight_scale_inv in ModelOpt checkpoints
Assisted-by: Claude Code (Claude Opus 5.5)
* llama : keep the k-pool layout prefix on sequence edits

Assisted-by: Claude Code

* llama : address review of the k-pool layout update

* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.

Assisted-by: Claude Code

* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY

* llama : remove the k-pool layout verification

* llama : inline the k-pool scan again

* remove dangling new line

* do not search twice

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
am17an and others added 13 commits November 17, 2026 22:40
* CUDA: use block reductions for whole-row pool_1d

Assisted-by: Claude Code

* CUDA: template pool1d row kernel by ggml_op_pool
As discussed in ggml-org#31955, the live preview is now opt-in. With it on, the editor renders blocks as widgets, and moving the cursor down from the first line of a prompt with a few blocks (a heading, a list, a code block) jumps to the last line; with it off, the cursor moves one line at a time. Users with a saved config keep their current value, since the full config is stored in localStorage.

The message input also gets spellcheck, autocorrect and autocapitalize back. CodeMirror turns them off by default, so they were off with the preview on or off; the former textarea and contenteditable inputs had the browser's defaults.
* CUDA: few-row MMQ for MUL_MAT_ID

* CUDA: few-row MMQ for MUL_MAT_ID from 2 tokens on DGX Spark

* CUDA: few-row MUL_MAT_ID: one launcher, tighter pair count, more tests

* CUDA: few-row MUL_MAT_ID: dispatch order MMVQ, few-row, MMQ and review fixes
* sampling : fix crash when both ignore_eos and grammar are on

* server : rework, throw 400 when grammar and ignore_eos are on

* sampling : move the check into the sampler (review suggestion)

* tests : remove redundant tests
…rg#32483)

* qwen4exp : apply the n-gram table scale with ggml_scale

* read the scale once after load

* remove comment
* Added support for Qwen3-TTS ICL/reference mode

Assisted-By: Claude Opus 5.5

* fix correctness

* attach transcription to bitmap

* nits

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.

Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.
* jinja: impl scope-chain key lookup

* add test for call_statement lambda

* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n

* mtmd: add kokoro tts support

* detailed guide about G2P and text norm port

* refine G2P

* clean up

* migrate batch api

* nits
…2480)

* ui: publish @ggml-org/llama-ui from CI when its version moves

ui/lib/ui reaches npm only when someone remembers to publish it by hand, which let 0.3.0
sit a day behind master. The version in its package.json becomes the release: a commit
landing on master with a version npm does not have gets built, linted and published with a
provenance attestation, through OIDC rather than a stored token, as release-publish.yml
already does for GitHub releases. A library change carrying no bump publishes nothing and
says so, on the PR and on the run.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: bump @ggml-org/llama-ui to 0.4.0

The library on npm, published before the design system PR merged, holds no entry points for
the Model UI, the app components or the shared constants and enums, and no dependency on
@ggml-org/llama-brand, whose colours the primitives now build on. Consumers import the new
subpaths and get ERR_PACKAGE_PATH_NOT_EXPORTED.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: name no environment for the Llama UI publish job

Creating a repository environment needs admin rights the UI maintainers do not hold, so a job
that names one cannot be set up by them. The job works without it: npm's trusted publisher
matches on the repository and the workflow file, and its environment field stays blank to
match. Nothing else in the job depends on an environment.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: address review on the Llama UI publish job

- the advisory version check leaves ui.yml for the publish workflow, on a pull_request trigger
  whose path filter keeps it off PRs that touch only Llama Chat: a chat-only PR was told to
  bump a library it did not change
- publish and verify run on master only. The trusted publisher matches on the repository and
  the workflow file, not the branch, so a dispatch from a PR branch - anything with write
  access can start one - could publish code no one reviewed. With no environment, this was the
  only branch protection in place.
- the registry read after a publish retries six times rather than sleeping once, so a release
  the registry is slow to serve is not reported as a failure
- a build metadata suffix (1.2.3+build) is not a prerelease, and versions order by semver's
  prerelease rules, so 0.4.0-beta.1 behind a released 0.4.0 is refused instead of shipped to next
- the summary step passes its values through env like the others, actions: read is gone, and the
  concurrency comment claims only what the setting does
- npm older than 11.5.1 is refused up front, as it cannot publish with an OIDC token

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* ui: note the two trusted-publishing traps in the Llama UI README

A publisher names a workflow file that must already be in .github/workflows, so it is added
after this workflow lands rather than before, and one created now publishes only through
staged publishing until npm publish is permitted for it. Provenance comes free with either,
and only on a GitHub-hosted runner.

Assisted-by: Llama Chat (unsloth/Qwen-3.8-Flash-Next:IQ4_XS)

* Update .github/workflows/ui-lib-publish.yml

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
@mostafafaheem
mostafafaheem marked this pull request as draft November 17, 2026 22:32
@mostafafaheem

Copy link
Copy Markdown
Collaborator Author

Temporarily created a merge commit with upstream to bypass CI issue, will reset when dev branch is rebased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.