Repository navigation
Conversation
This was referenced Sep 29, 2026
Merged
Closed
Contributor
CI performance trends
Krackan - ApplicationsOperators added: Benchmarks that moved by at least 5% and by more than 2x the spread the runs measured:
Phoenix - ApplicationsOperators added: |
hunhoffe
commented
Oct 5, 2026
hunhoffe
marked this pull request as ready for review
October 6, 2026 18:55
hunhoffe
changed the title
…olds
The finish surface finished a producer's tile with unary steps. A step
now enters a finish as a `Link(op, at)`: the tile is its input `at`, and
its other inputs stream into the producer's cores beside the tile, as the
producer's `finish_inputs` (extra `In`s named `finish{i}_{name}`, one
fifo per core, bound like any lane). So SwiGLU's product can finish in
silu's cores, and a step that reads the producer's output twice takes it
beside the folded step's.
- `fold(consumer, at)`: `folded` tries each input of a step, folds a step
into a producer already finished (a chain, `Fold.after`), and places the
result where its producer ran, or where the consumer ran when a step
between them writes the streamed inputs. `replaced` pairs a folded step
with its twin by type and its own inputs.
- A chain streaming inputs of its own has an array of its own
(`finishes=()`): the extra fifos are not shared with designs that do
not read them.
- The shim budget counts the finish's inputs (`shim_columns(operands=)`),
and a fold that would narrow its producer's widths is not taken.
- A core tile has two input channels (`dev.core_dma_channels_in`). A
producer whose cores already read two streams (GEMV's A and B, a binary
Elementwise's a and b) has none left for a finish input, so resolving
one raises `Unresolvable`, which `folded` reads as "not admitted". These
folds wait for a feed through the memtile.
Llama admits no new fold: decode's product follows a GEMV, and the prompt
chunk's buffers are bounded by `rows` (bounded folds are the next
commit). Its table changes only in GEMV keys, which now spell the finish
as `Link`s.
Measured on Strix Halo (npu2):
- Catalog (GEMV, ElementwiseAdd, SiLU, GELU, RMSNorm): 340 pass,
including SiLU finished by ElementwiseMul at either input and by
ElementwiseMul then GELU, and ElementwiseAdd finished by SiLU.
- SiLU finished by ElementwiseMul vs the two run apart (8 rounds,
alternated, medians, each run checked against its reference): at a
prompt chunk's 2048x8192, 1566 us vs 923 + 1483 = 2406 us (-35%); at
decode's 8192, 88.7 us vs 85.0 + 89.5 us, one dispatch saved. The
finished step costs about the product alone: it is the three streams'
DMA, which silu's compute hides behind.
- SwiGLU 30 pass; at 256 rows its product now finishes silu, judged as
the product step against its reference.
- Toolchain narrowing and coresidence 48 pass, infrastructure narrowing
70 (one iteration of the model-prediction test measured 10.2% off its
10% gate, tuned and traced both slow; 5/5 on a rerun),
iron/tests/common 205, rejected_shapes 22, cost table coverage 1.
- costs_npu2.json re-measured for the new GEMV keys.
Co-Authored-By: Claude <noreply@anthropic.com>
The open tile_m that embeddinggemma2 brought in lets a short M take a short tile rather than pad, and resolution picks the widest that splits it. Its construction check waited for a given tile_m, though, so a row count no tile can split (SwiGLU at 300 rows) traced and was refused only at resolution, under the 64-row fallback's rule. - validate() holds M to the kernel's narrowest m (2 r) while tile_m is open, so a graph is refused at trace time again. - resolve() falls back to the kernel's m block rather than 64, so the message names the rule that holds: 64 rows on npu2 under emulation (four rows of 16) and on npu1 (four rows of 4 r, r=4), not 256. Device-free; the shapes this changes are all refused: gemm_tile_divisibility, swiglu_no_padding, rejected_shapes 37 pass. Co-Authored-By: Claude <noreply@anthropic.com>
The finish surface folded only whole, unbounded steps, so a prompt chunk, whose activations are bounded by `rows`, admitted no fold. Its gate GEMM and silu are the obvious pair: GEMM's cores hold each C tile in L1 once the reduction is done, and silu can run there before it drains. This commit admits bounded folds and puts GEMM on the finish surface. Bounded folds (`folded`): - A bounded input folds where the step's output and every other input have its shape and bounds, and the step's only per-call values are the extents those bounds bind. The producer's own bound then sizes the tiles it finishes, and a streamed finish input takes an Extent word like the output's, so it is filled under the same bound. - The bindings of a step folded away are dropped with it. GEMM on the finish surface: - `Out(finish=(tile_m, tile_n))` declares the block one core finishes. GEMM's cores hold it in the matmul kernel's blocked layout, not in row order. A finish there must not depend on the order: `at_line(..., ordered=False)` refuses a Rowwise step (RMSNorm reduces over rows), and `_finish_at` refuses a finish input (it streams in row order) with `Unresolvable`. Unary Elementwise steps take it. - The core finishes its tile after the K loop (after `convert_copy` on the f32-accumulator path). Padding tiles past a bound are passed through unfinished, as they are passed through uncomputed. - C's L1 type is flat, the finish scratch's type, so a core on a shared array can select between the two (the Llama gate and up GEMMs share one array; up runs the empty chain). - `reference()` and `tolerance()` carry the finish. - `declare()` rewrites `finish_block` like the tile, and its names join the array tier. Deferred: streaming a finish input through the memtile (`rt.plan`), which would let GEMM and GEMV take SwiGLU's product, waits for the memtile feed. Measured on Strix Halo (npu2): - SwiGLU's gate (K=2048, N=8192), medians of 8 alternated rounds, each run verified against its reference: | rows | GEMM | GEMM+silu | silu | silu+mul | mul | |------|--------|-----------|-------|----------|-------| | 256 | 2295 | 2288 | 196 | 282 | 268 | | 2048 | 16788 | 16652 | 928 | 1565 | 1470 | (us.) Silu finished in GEMM's cores costs nothing measurable, so the fold the greedy pass now takes (silu into GEMM, then the product) beats the one it took before (the product into silu): 2556 vs 2577 us at 256 rows, 18122 vs 18353 us at 2048. - Llama 3.2 1B shape on random weights, 2 alternated rounds of 3 against the previous commit, each tree's own table: the tuner now takes silu into GEMM in the prompt chunk (16 folds, none before), which runs 1274.5 / 1274.5 ms against 1276.1 / 1279.3 ms; decode, already folded into GEMV, is unchanged (62.45 / 62.28 vs 62.29 / 62.28 ms). Untuned, nothing folds and the two are within noise. Logits of every prompt and decode step are bit-identical between the trees. - The Llama cost table gains the 20 GEMM designs with a finish. - Device tests: GEMM's five finish cases, the catalog (GEMM, SiLU, GELU, ElementwiseMul, RMSNorm, ReLU, ElementwiseAdd; 87) and SwiGLU (30) pass against their references. - Device-free: iron/tests/common and rejected_shapes pass. Co-Authored-By: Claude <noreply@anthropic.com>
The finish surface folds a step into its producer's cores. A norm
feeding a projection has the opposite shape: one producer, the norm, and
one or more readers (q, k and v, or gate and up). This commit adds the
mirror of a finish, a prologue, applied by each reader's cores to each
line of its input as it acquires it, and puts GEMV's B on it.
The prologue surface:
- `In(prepare=True)` names the one input a core prepares (a class
declaring two is refused). `Operator.prefold(producer, at)` prepends
`producer` to `prepare`, as `fold` appends to `finish`; the producer's
other inputs become `prepare_inputs` (`prepare{i}_{name}`), which the
derived sequence fills into the prepared input's stream after its tile.
- `Prepare` (elementwise.py) is `Finish`'s counterpart in an array: it
reads the chain words, acquires the line and its prologue inputs, and
applies each link at the line's length (`at_line`).
- `folded()` runs a prologue pass, last step first, before the finish
pass. A step folds into its readers where its one output is a whole,
unbounded intermediate each of them reads once, as its prepared input,
and nothing between writes the step's other inputs. Each reader must
keep every width it had. `Prologue(producer, consumers, at, after)`
names the fold for the tuner and the cost table, the chain included,
so "ElementwiseAdd, then RMSNorm into GEMV" is one fold.
- RMSNorm's weighted chain is its two kernels in turn, as its two
pipelined cores run them, so a core applying it rounds the normalized
line before the weight as the operator does.
GEMV on it:
- B is `In(..., prepare=True)`. Its fifo is depth 1 and each core's end
takes the prologue's depth: a depth the shim end shared gave each core
one more line than it acquires, which at K=4096 weighted left no room.
- `compatible()` refuses a K whose lines, prologue scratch and stack
pass a core's memory: an unweighted RMSNorm prologue at K=8192 is
refused rather than failing to link. K=4096 weighted now fits.
Measured on Strix Halo (npu2):
- Llama 3.2 1B decode traces to 243 folded steps from 293 with every
fold the graph admits (`compile(fold=True)`): the attention norm into
q, k and v, the feed-forward's into gate and up, the product into down,
and the last residual and final norm into the head.
- Llama 3.2 1B on random weights, 2 alternated rounds of 3 against the
previous commit, each tree's own table. Logits of every prompt and
decode step are bit-identical between the trees, tuned and untuned:
| ms | prompt chunk | decode step |
|-----------------|-----------------|---------------|
| this, untuned | 1289.3 / 1293.0 | 69.47 / 69.66 |
| before, untuned | 1292.4 / 1293.2 | 69.63 / 69.66 |
| this, tuned | 1273.5 / 1280.2 | 63.18 / 63.07 |
| before, tuned | 1273.3 / 1278.7 | 62.31 / 62.47 |
Tuned decode is 0.7 ms slower here. The one prologue the tuner takes,
the last residual and the final norm into the head, costs the head 5 us
measured alone. Dropping those two steps moves the pack search to a
layout with MHA beside RoPE and a Copy, rather than a Copy and the o
projection; the model counts 66 configures against 84 and predicts
300 us saved. Withholding only the prologues from the tuner in this
tree, 3 alternated rounds of 32 decode steps each:
| ms | 16-token context | 2048-token context |
|------------------|---------------------|---------------------|
| every fold | 61.50 / 61.12 / 61.16 | 63.13 / 63.10 / 63.09 |
| no prologue | 61.33 / 61.33 / 61.30 | 62.72 / 62.26 / 62.29 |
The cost grows with the context, which only MHA does: MHA runs slower
in its new pack, and the table, measured a design at a time, cannot
see it. Pricing a design in its pack is the cost model's next step.
- The Llama cost table gains the 20 GEMV designs with a prologue.
- Device tests: the catalog (GEMV, RMSNorm, SiLU, ElementwiseMul,
ElementwiseAdd; 390, GEMV's prologue cases among them) and SwiGLU (30)
pass against their references.
- Device-free: iron/tests/common and rejected_shapes pass (236).
Co-Authored-By: Claude <noreply@anthropic.com>
flm's GEMM already carries an epilogue (an activation, then a clamp) applied to each C tile before it drains. This commit gives it the fold surface's `fold(consumer)`, so a SiLU or Sigmoid step after it becomes its activation and a Clamp after that its clamp, the way the elementwise finish chain folds into GEMM and GEMV. - SiLU and Sigmoid map to the epilogue modes of the same names, where the operator compiled that mode in (`epilogue_modes`) and has no activation yet. Clamp takes its bounds rounded to bf16, as the Clamp operator rounds them. Nothing folds after a clamp, and a consumer carrying a finish or prologue chain of its own is refused. - GELU does not fold: the epilogue's gelu is `x * sigmoid(1.702x)`, not the GELU operator's approximation. - The epilogue is a run-time mode, so a fold changes no xclbin (`config_name` is unchanged) and no build. - flm.GEMM is not traceable in a graph yet (B's Select rank in `infer`), so the fold pass cannot reach it from a graph; the fold is exercised on the operator directly. Tests: - Device-free: the fold's mapping and refusals (iron/tests/common/build.py). - On Strix Halo (npu2): a SiLU then Clamp folded into a 256x512x1024 GEMM, held to SiLU's and Clamp's own references applied to the product, under the GEMM test's accumulated-mass bound: 5 iterations pass. The rest of iron/operators/flm/gemm/test.py: 263 passed. Co-Authored-By: Claude <noreply@anthropic.com>
…as ValueError
halves is named in the output's per= so its lanes can be indexed, which
made it a width the tuner searched over {8, 4, 2, 1}; but the join in
array() bakes two halves, so every other value builds the same array
under another key. Fixing it with init=False leaves cols as the one width.
A tile_n other than 64 was refused with NotImplementedError, which the
tuner's legality filter (a ValueError catch) did not expect. It is a
refusal of the configuration like any other, so it is a ValueError now.
Co-Authored-By: Claude <noreply@anthropic.com>
Every figure calibrate() derives is a time: the dispatch, the reset configure, the switch and the part of a configure no design accounts for (E(A) + E(B) - E(pack)). A negative one means the run was shared with other NPU work or the table's step times came from another session, and the model built on it prices every design's load below zero, so every pack looks like a gain. EmbeddingGemma 2's table carried one (reset -417 us), which left the text versions' exact partition search unbounded. calibrate() now raises before recording such a figure. The device test induces one with a table whose step time is not the session's own. Co-Authored-By: Claude <noreply@anthropic.com>
…gether The table now holds every design of the four text versions at each width `variants` gives (1,367 steps, from 308), beside the vision designs it had. The three configure calibrations tune.py names were measured again in one session on an idle NPU, and agree: D0 46-53 us, R 42-49 us, base 38-41 us. The audio tower's designs are not yet measured (the tune was stopped in its Clamps), so the tuner leaves them at their defaults. Measured on Strix Halo, eight interleaved rounds, every call bit-equal to the untuned encoder (held to the oracle at cosine 0.9999): - query (15 tokens): 141.6 ms untuned, 142.6 ms with the previous table, 71.4 ms with this one (-49.5%) - long (370 tokens): 255.2 ms untuned, 254.4 ms with the previous table, 216.8 ms with this one (-15.0%) A cold text load tunes and places in 174 s; with its fits cached the towers load in 17 s. The packs also bring the audio-mixed and "both" versions under aiebu's section count, so the multimodal and determinism tests pass on the current pin. Co-Authored-By: Claude <noreply@anthropic.com>
A tunable given at the call site, or on the tune CLI, is the caller's decision; until now the tuner widened it like a default. The metaclass now records those names in `Operator.pinned` before a profile is merged, so a profile's values stay starting points a tuning may move and write back. `domains()` leaves a pinned tunable out. `pinned` is compare=False, so two calls of one design share a cost key whether or not one of them pins. The tuner applies one setting to every operator of a design, so `variants()` takes the union of their pins and holds each at the value the design resolves it to, and `Tuning.apply` rebuilds every twin at those held values. The test that widened a call-site column count now gives it through a profile; new tests cover a pinned call, a fully pinned one (one variant) and a design whose second call pins its columns. Co-Authored-By: Claude <noreply@anthropic.com>
The tuner already asks the placer whether a pack fits, and keeps its verdict per recipe. A setting that does not fit alone reached the device build, where one failed compile ended the whole tune. Its check is now `fit_verdict`, shared by the pack search and by `fitting`, which leaves out of `Designs.of` (and the tune CLI) every non-default setting the placer refuses alone and says why in `Designs.refused`; `measure_graph` logs those. A design the generator refuses (a ValueError) is a refusal with its message, so it no longer aborts a pack search either. The default is always kept, so a design that does not build at all still fails loudly. The placer answers in 0.06 s a variant on average; the Llama coverage test takes 7.5 s cold and 2.5 s warm with it. Co-Authored-By: Claude <noreply@anthropic.com>
The placer catches what fails to place, but not what Peano refuses (a kernel's static_assert, program memory, the stack). Such a setting compiled inside `measure_steps` and its RuntimeError ended the whole tune. Now a setting other than the default that does not build is logged, left out of its batch, of the table and of every later descent line, and recorded in the fit cache (`refuse`), so `fitting` leaves it out of the next `Designs.of` without compiling it again. The default and a twin are the graph's own designs, so their failure still raises. Verified on Strix Halo: a GEMM setting with 128x128 B tiles fails aiecc's placement and the search completes on the default. Co-Authored-By: Claude <noreply@anthropic.com>
The tuner has searched widths alone: the column and channel counts a streamed per= names, over the powers of two up to the device's columns. That inference was a good one and stays: a tunable a per= names and that declares nothing is still searched as a Width, so every operator's search is exactly what it was. What it could not express is a tile or a block. auto(domain=) now takes one of three kinds, each asked of the resolved operator so its ladder runs through the value the library would choose: - Width(): the field's value and every power of two up to dev.cols; - Divisors(of, step, cap, span): the resolved value times 2^k that divides `of`, is a multiple of `step` and at most `cap`, within `span` octaves; - Choices(options): a fixed set. Each takes when= naming a bool field, searched only where it is true. Searching stays opt-in per field, so a tunable no one has checked the numerics of is never searched by accident. Operator.domains() is now the one generic rule over these, and widths is the fields it searches as a Width. Co-Authored-By: Claude <noreply@anthropic.com>
The coverage test already knew exactly which settings the checked-in table lacks; it listed them by cost key. Once tiles are searched that list grows to a hundred-odd hashes, so each line now says which operator it is, the default it starts from and which tunables moved, and a calibration is marked as one. Whoever re-measures sees at a glance what the run will be. Co-Authored-By: Claude <noreply@anthropic.com>
…nspose These are the tunables whose every value computes the same thing, so the tuner may move them freely and the gate still judges each one: - GEMV tile_size_input: Divisors of M, capped by the A tiles a core holds; tile_size_output: Divisors of M in steps of tile_size_input, three octaves either side of the resolved tile; - Elementwise tile_size: Divisors of the output in steps of 64 up to tile_cap, three octaves either side (Rowwise reduces over its line, which stays a param()); - Copy tile_size: Divisors of each channel's share up to object_bytes, three octaves either side; - Sample chunk: even Divisors of the slice up to 8192, which at Llama's vocabulary is exactly (8016, 4008, 2004, 1002); - Transpose m and n: Divisors in steps of s within the 8192-element tile, and s itself from (8, 4). RoPE and the other column counts were already searched as widths. Checked on Strix Halo (npu2) against GEMV.reference() and bit for bit against the default: all 44 placeable settings of GEMV(M=2048, K=2048) and all 16 of GEMV(M=2048, K=8192) (the placer refuses the other 4) give the identical output; GELU at every tile from 64 to 2048 likewise. Llama's checked-in costs_npu2.json now lacks 141 settings (GEMV 50, Copy 30, SiLU 23, ElementwiseAdd 17, ElementwiseMul 12, Sample 9), which iron/tests/common/cost_tables.py lists until it is re-measured; the tuner never chooses a setting the table does not hold. Co-Authored-By: Claude <noreply@anthropic.com>
GEMM asked mm.mac_dims for the kernel's block in two places, validate() at aie2p's geometry and resolve() at the device's with aie2's 4r m block. Both now ask one mac_block(dev), which the domains read as well: - tile_m: Divisors of each row of cores' share of M, in m blocks; - tile_n: Divisors of N, in n blocks; - tile_k: Divisors of K, in k blocks, when prio_accuracy is set. Without it C is rounded to bf16 once per K tile, so another tile_k is another result; with it C accumulates in f32 and the gate judges the variant. GEMM(M=K=N=2048) on npu2 has 156 legal settings, of which the placer keeps 52 (11.3 s cold). On Strix Halo the descent line from the default (tile_m 64, 32, 16; tile_n 64, 32, 16; 8, 4, 2, 1 columns) gives C bit for bit identical to the default and within GEMM.reference()'s gate at every setting. Llama's GEMMs each have more than 32 settings, so the cost table needs only their defaults, which it holds; coordinate descent measures the rest as it goes. Co-Authored-By: Claude <noreply@anthropic.com>
MHA's array was written at B_q = B_kv = d = 64, where QK^T and P*V share
one shape, and there it is right. Away from it the module did not
generate: outP streamed P with Q's (B_q, d) blocking, which at B = 32
moves 1024 elements of a 2048-element tile ("outP0 to_stream ... walks a
tensor of shape (32, 64), but each transfer moves 1024 elements"), and
once P streams as P*V's A, the zero kernel, typed for the scores, was
handed O. mha.cc takes one (DIM_M, DIM_K, DIM_N), QK^T's (B_q, d, B_kv),
while matmul_PV reads its key block as DIM_K wide and V as (DIM_K,
DIM_N): P*V is (B_q, B_kv, d). So P*V's core now binds matmul_PV and
rescale_O from mha.cc built at that shape, V and O take their own
(B_kv, d) and (B_q, d) types, and O has its own zero.
At B = 64 the two builds are one object and the module text is
byte-identical to before, at 8 pipelines and at 4; the non-extensive MHA
catalog cases pass on Strix Halo.
Away from 64 nothing new runs correctly, and the operator now says so:
- B = 128 generates and is the placer's to refuse ("buffer outP7_cons_buff_1
needs 32768 bytes and this tile has no room left for it"): P's
(128, 128) tile, double-buffered, does not fit L1 beside V and O.
- B = 32 builds and runs, and fails MHA.reference() (prefill, 1 head,
2048 rows: 6398 of 131072 outputs past the gate). A 128-row run shows
why: in each diagonal block every row but the first comes out zero.
partial_softmax's causal mask loads and stores 64-lane vectors from a
row's start, so on a 32-key row it also masks the whole next row.
B = 16 hangs (ERT_CMD_STATE_TIMEOUT). compatible() now refuses a B_kv
that is not a multiple of 64, which rejected_shapes checks.
build.py checks the B = 128 module's P*V, rescale and zero types without
a device.
Co-Authored-By: Claude <noreply@anthropic.com>
MHA declares its domains: num_pipelines a Width, and B_q and B_kv
Divisors of kv_len in 64-row steps (compatible() refuses a smaller KV
block, and keeps B_q = B_kv). A core holds one (B_q, B_kv) score tile, so
each block is capped by what that tile leaves of core memory at the other
block: 512 on npu2 at the default 64.
The cap is not only a bound on the search. Without it a prefill's B_kv
ladder runs to kv_len, and the placer's fit check generates MHA at a
32768 x 32768 score tile, where the kernel factory's tile layout asks for
8 GiB per variant ("Unable to allocate 8.00 GiB for an array with shape
(1073741824,)"); run without a memory limit, that took the host down.
With it, the device-free suites peak at 315 MB.
At Llama 1B's shapes on npu2 the placer refuses every block past 64 (P's
tile, double-buffered, does not fit beside V and O), so what the tuner
measures of MHA is its pipelines: 4 settings the cost table lacks. The
non-extensive MHA catalog cases pass on Strix Halo.
Co-Authored-By: Claude <noreply@anthropic.com>
flm.GEMM searches tile_ma (divisors of M_TILE in steps of 2R) and
m_chunk (divisors of its row blocks per core). tile_n stays fixed, since
it has to match DequantBFP's. Softmax searches block (divisors of cols
in its 32-element vector step, up to the 4096 a core holds whole) and
streamed. GQAScores/GQAContext search chunk, capped by the
(chunk, head_dim) block a core holds.
Every ladder that follows an extent is capped by core memory. Without
the cap, the fit check generated MHA at a 32768 x 32768 score tile and
asked for 8 GiB; test_a_block_is_no_larger_than_a_core_holds covers it.
Domains on npu2 (8 columns), legal / placer-fit:
- flm.GEMM(2048^3): tile_ma (64, 32, 16), m_chunk (8, 4, 2, 1): 5 / 5
- Softmax(32, 2048): 104 / 104; Softmax(32, 32768): 104 / 104, all streamed
- GQAScores/GQAContext(32 heads, 8 groups, 2048): chunk (512..64): 16 / 8;
the placer refuses chunks 512 and 256
- at 32768 rows, MHA's B_q/B_kv and GQAContext's chunk are (512..64)
Device-free: 276 passed. One failure is expected: the Llama cost table
needs a re-measure. On Strix Halo: flm GEMM 23/23, Softmax and GQA
catalog 15/17. The two failures are the streamed Softmax cases, which
fail to link because mlir_aie 1.4.4.dev88's softmax.cc does not yet
define softmax_{max,sum,scale}_bf16. The kernel source predates this
commit.
Co-Authored-By: Claude <noreply@anthropic.com>
A field that reaches the device only as words the cores read (no transfer, trip count, branch or core follows it) cannot change a step's time, yet the cost key, the design identity, gave each of its values a key of its own: every Clamp bound pair was a design to measure and a table entry to keep. param(probe=v) declares such a field and the value its cost is measured at. Class creation refuses a probe on a field the array reads, on one an operand's shape or presence names, or on one the constructor cannot set. Operator.probed() is the operator with each probe field at its probe (itself when they are there already), and cost_key is the identity of that operator, so designs differing only in probe fields share one key. Designs.of and the one-operator tune CLI measure the probed operator, the key's representative, rather than whichever design was traced first; Tuning.apply still rebuilds each operator with its own fields. No in-tree field declares a probe yet, so every key is unchanged: Llama's decode and prefill calls give 43 designs and 184 variants before and after, and the device-free suites pass as before. Co-Authored-By: Claude <noreply@anthropic.com>
Clamp's low and high reach the device only as two runtime parameter words per core (low_bits, high_bits): generated at two bounds, the modules differ in nothing but the constants their aiex.npu.rtp_write ops take, 64 lines for 16 cores. flm GEMM's clamp is the same: against no clamp it moves words 4 and 5 of each core's parameter block (CLAMP_MIN/MAX) on 32 cores, and the kernel clamps unconditionally (mm_fused.h sends an unclamped call (-inf, +inf)), so no branch follows it either. Both now declare param(probe=): Clamp at (-inf, inf), flm GEMM at None, the identity in each case, so a setting judged exact there is exact at any bound. The shipped flm image keeps clamp without a probe: its parameter block carries a clamp-on word that selects a branch, so its time may follow whether there is a clamp. Folds stay cost-neutral. A Clamp has an array of its own, so it is never a finish or prepare link (Elementwise.at_line refuses it), and a link reaches design_key only through array_key anyway. flm GEMM.fold(Clamp) sets clamp to the bf16-rounded bounds, and the folded GEMM's cost key is the plain GEMM's, so the fold reuses its measurement. probe_identity.py walks each probing operator's module away from and at its probe in lockstep and admits only rtp_write constants as differences (mlir-aie names unnamed flows from a process-wide counter, so flows are renumbered by first appearance). The toolchain narrowing tests check that two Clamps apart only in bounds share one cost key, one Designs entry measured at the probe and one variant list, keep their own bounds after apply, and that the folded flm GEMM costs what the plain one does. Timed on Strix Halo (pmode default), Clamp at 2^23 elements, interleaved over 8 rounds of 50 calls, every output checked against Clamp.reference(): no time follows the bounds. At eight columns a loaded image settles at about 605 or 615 us whichever bounds it holds (A 604 / B 615, then A 614 / B 606); at one column 2132-2157 us, with one image twice at 2447 us that a standalone rerun over the same images (14 trials) never reproduced, every image within 0.6% of the others. That image-level spread is wider than a 1% bound, so the timing stays a measurement here and the module identity test is the gate. No in-tree graph clamps, so Llama stays at 43 designs and 184 variants. Cost tables measured with Clamp or flm GEMM at a real bound are keyed apart from the probe and need measuring again. Co-Authored-By: Claude <noreply@anthropic.com>
…easured it A design with more settings than EXHAUSTIVE is searched by coordinate descent, which measures only the settings on its path. Once GEMV's domains reached 44 settings, the folded GEMV's descent walked to settings whose unfolded twin had never been measured, and measure_steps raised "twin ... is not in the table". It stopped the Llama re-measure after 17 designs. Such a setting is now priced from its own run, like any unpaired design. On Strix Halo, iron/tests/infrastructure/narrowing.py -k "priced_beside or forced_fold": 10 passed. With the filter removed as a negative control, the priced_beside cases fail with the same ValueError. Co-Authored-By: Claude <noreply@anthropic.com>
EmbeddingGemma 2's audio tower computes its log-mel features in model code: a gathered framing, an rfft and a mel filterbank as GEMMs of exact float32 products over Limbs, Magnitude and Log. Those steps were measured and proven there; iron.lm.LogMel holds them as a reusable graph, which a model calls in its body (writing a state's rows it gives, bounded by its per-call frames) or which runs alone, compiled once for every frame count. exact() is the extractor's own float64 computation, and iron.lm.filterbank its HTK triangles. AGENTS.md now says when a capability is an operator (new compute) and when a graph (a composition of operators), and how a graph is called in another. Verified device-free: a graph calling LogMel traces its seven steps into its own, each bound by the caller's frames, its weights named log_mel.dft and log_mel.filterbank, and its reference equals LogMel's own; LogMel's reference (its operators') against exact(): 0.99966 of 3072 frames are its bf16 rounding, every feature within 1 ulp or 4e-6 (nested_graphs.py, 5 passed); a graph calling it lowers on npu1 and npu2 (lowering.py, 2 passed). Its device test (iron/operators/log_mel/test.py: one compile, 200 and 3072 frames) is written and compiled, not yet run: 218's NPU fails every dispatch (DRM_IOCTL_AMDXDNA_GET_INFO, err -22). Co-Authored-By: Claude <noreply@anthropic.com>
The audio tower's log-mel front end (the Hann-windowed rfft and the mel filterbank as exact float32 GEMMs over Limbs, Magnitude and Log) is now the iron.lm.LogMel graph, which keeps every step it proved here. The tower holds one and calls it in its body, writing its features state's rows bounded by the call's frames; the host extractor takes its filterbank from iron.lm.filterbank rather than a second copy. Verified device-free against fixtures captured at 31d81b3: on npu2 the whole audio graph traces to the same 958 steps, each with the same class, design key, slot shapes and roles, and sha256 of every weight it reads (only the weights' names change, to audio.log_mel.dft and audio.log_mel.filterbank); the features graph's reference is bit-identical over all 3072 frames of a 2 s chirp, and the whole tower's reference at full extents is bit-identical; on npu1 the features graph's 9 steps are identical. The host extractor's features are bit-identical for 2, 7.3 and 30 s chirps. Co-Authored-By: Claude <noreply@anthropic.com>
…as a graph block The audio tower's two subsampling convolutions are GEMMs written in model code: the first (one channel, stride 2, padded) as a band over the width applied to three gathered input rows, its output bordered by zero slots that pad the second; the second as a GEMM of gathered 3x3x128 patches with its 32 channels on M. Those layouts were tuned and proven there; iron.lm.Conv2d holds them as a reusable graph with convolution semantics (rows of width x channels, an (out, in, k, k) weight, stride, padding, border), lays its weight out once as it is built, and is called in a model's body bounded by the call's rows, or alone. exact() is the float32 convolution. The channels stay on M, as the tower had them: on npu1 a GEMM of M = 32 needs eg2-npu1's 29a67ddc (a short GEMM over fewer rows of cores). Verified device-free: the reference (its operators') against exact() within 1 bf16 ulp for the band and the patches; a caller's per-call rows bound it, its weight named conv.matrix; padding with more than one channel refused (conv2d_graph.py, 5 passed); two convolutions called in a graph lower on npu1 and npu2 (lowering.py, 2 passed). Its device test (iron/operators/conv2d/test.py, EG2's shapes at two row counts, each row within its GEMM's bound) is written and compiled, not yet run: 218's NPU fails every dispatch. Co-Authored-By: Claude <noreply@anthropic.com>
The tower's two subsampling convolutions were GEMMs over hand-built patterns and weight layouts; it now holds two Conv2d graphs, built from its weights centered over their output channels (so that the layer norm after each is an RMS norm, the one part that is the tower's own), and calls them bounded by its per-call rows and tokens. Verified device-free against fixtures captured at 31d81b3: on npu2 the whole audio graph traces to the same 958 steps, each with the same class, design key, slot shapes and roles, and sha256 of every weight it reads; its reference at full extents is bit-identical. On npu1 it stops at the second convolution's GEMM (M = 32), as it did before; eg2-npu1's 29a67ddc splits that GEMM over two rows of cores. Co-Authored-By: Claude <noreply@anthropic.com>
…h block EmbeddingGemma 2's image processor runs on the NPU as model code: the ResampleTaps tables, Resample into a raster state over two cores a column, PatchPositions' order and a gathering Copy, with the raster's sizing worked out from the patch rows and the device. Those steps and that sizing were proven there; iron.lm.ImagePatches holds them as a reusable graph that owns its raster, called in a model's body with the call's four sizes, or alone, compiled once for every image its buffer holds. pack() lays an image out in whole chunks a row and refuses one a call cannot hold or Resample cannot resize. Verified device-free: called in a graph its six steps trace into the caller's, bound by the caller's sizes, its raster named processor.raster; its reference equals ResampleTaps, Resample and PatchPositions references and a numpy gather computed on their own; an image past the buffer is refused (image_patches_graph.py, 3 passed); a graph calling it lowers on npu1 and npu2 (lowering.py, 2 passed); iron/tests/common, lazy_imports, rejected_shapes and lm_host: 418 passed, 2 skipped. Its device test (iron/operators/image_patches/test.py: one compile, two images, every output exactly its reference) is written and compiled, not yet run: 218's NPU fails every dispatch. Co-Authored-By: Claude <noreply@anthropic.com>
The vision tower's image processor now holds an ImagePatches graph at the tower's patches, windows and positions and calls it in the tower's body; what stays is EG2's own: the size its processor resizes an image to, which inputs() hands to ImagePatches.pack. Verified device-free against fixtures captured at 31d81b3: on npu2 the whole vision graph traces to the same 353 steps, each with the same class, design key, slot shapes and roles, and sha256 of every weight it reads; its reference on a 200x300 image is within 2.1% of the float32 oracle (it runs since 55913b71); the processor's refusals (vision/test.py) pass. Co-Authored-By: Claude <noreply@anthropic.com>
A Dataflow reads the live trace, and for a compiled version its plan, artifacts and tuning, and draws one box per step: the operator, its family's hue, its output shape and the per-call values bound to it, with an edge per buffer it reads, weights as notes, states as cylinders and the inputs, outputs and carried values as ovals. A compiled version's boxes also name the design each step runs, the steps folded into it, the tunables a tuning moved and the packs it made; given a CostTable, the step time the table prices. Steps cluster by layer (layers.N); a picture can keep a few layers alone. dot() is the Graphviz text, svg() lays it out with Graphviz's dot, and html() is one self-contained page of that SVG that pans, zooms and shows a clicked step's operator and tunables. side_by_side() draws a graph as traced beside the same graph tuned. Nothing new is imported; laying out needs the dot program. The tests draw the toy Llama and a folded SwiGLU device-free and check that the picture names every step and exactly the edges the trace has. Co-Authored-By: Claude <noreply@anthropic.com>
A Floorplan reads the lowered module the build kept (the full ELF's, or an xclbin chain design's own) and draws the device a step configures: which tiles run a core, how many bytes of buffers each holds, which shims carry host streams, and the routed stream hops between neighbouring tiles. The hops are mlir-aie's own count per switchbox side, from aie-translate --aie-flows-to-json, found the way aiecc finds it. A pack's tiles are named by the member whose buffers they hold. The toolchain test builds a two-step graph on npu1 and npu2 and checks the picture against the step's own tunables: a core per column and channel, a host stream per operand per core, every hop between neighbours or from a shim to the host. Co-Authored-By: Claude <noreply@anthropic.com>
A tower's weights are named for the attribute holding it (vision.layers.3.q, audio.layers.3.ff.0.up.weight), so a picture of one layer of EmbeddingGemma 2's audio or vision version found none. The layer is now read from layers.N anywhere in a buffer's path. Co-Authored-By: Claude <noreply@anthropic.com>
An Arena draws each scratch buffer the sequence planned as a box from the first step touching it to the last (LiveRange.touching, as the arena plan takes it), at its byte offset and size, coloured by the family of the step that writes it, with the residents' bytes summed beside: where two buffers share bytes is where their boxes stack in one column of steps. The toolchain test builds a two-step graph on npu1 and npu2 and finds its one intermediate drawn where the layout put it, live over both steps. Co-Authored-By: Claude <noreply@anthropic.com>
box() takes the designs, cost keys and moved tunables it shows as parameters rather than a dict, and the per-call values both the boxes and the page's panel show are written once, by values(). The layers argument's docstring states the rule for a head after the last layer. Co-Authored-By: Claude <noreply@anthropic.com>
A tuning lists the designs of one array as one device on an xclbin chain too, where each design is loaded alone, so a tuned npu1 version drew some steps in a "pack" and looked up a pack for the rest. Packs are drawn for a fused image alone, and a step in none stays in its layer. Co-Authored-By: Claude <noreply@anthropic.com>
#241's port configures a full ELF of one design on its first call alone (Fusion.expands), but model_us still charged such a runlist an entry and the parity reset every call, and the pack search weighed each setting's load against its step. On the dev box that overprices a one-design ReLU by about 100 us a call (52 us measured, 159 us reconfigured), and it steers the search toward a cheap load over a fast step. Fusion's rule is now Fusion.expanded(designs, traced=, reconfigure=), which Fusion itself uses. model_us and JointNarrowing._packed ask it whether the image expands: where it does not, a call costs D0 and its steps, with no configures, and a setting is chosen by its step alone. JointNarrowing counts designs from the traced operators' design keys, since two designs that differ only in their probe fields share one cost key (TwoClamps); the tuning record's key includes the answer. A multi-design image, a pack, or an xclbin chain is priced as before. The probe's Standalone still passes reconfigure=, so the calibration's alone = D0 + base + load + R holds. Verified device-free: new tests build real fused sequences (ReLU three times, add+relu, relu+add+relu, add+relu packed) and check that model_us's configures equal the aiex.configure ops the Fusion runs per call (0, 2, 4, 2). The one-design case is priced at D0 plus its steps, a one-design graph is tuned to its fastest step and configures 0, and TwoClamps still configures 2. Both new checks fail on the old model. iron/tests/{infrastructure,toolchain}/narrowing.py and common/cost_tables.py: 96 passed, 137 skipped before; 101 passed after (the 5 new tests). Co-Authored-By: Claude <noreply@anthropic.com>
MEASUREMENT_PROTOCOL.md's ablation (section 11) wants rows 4-6 as one switch each: cost-table widths only, then + folds, then + packs. JointNarrowing always took all three together, so there was no way to measure one decision's gain on its own. Adds NarrowingKinds (three independent bools) and a kinds field on JointNarrowing, every kind on by default, unchanged from today's plan: widths=False keeps every design at its own default setting, folds=False returns the unfolded tuning without running the fold search at all, and packs=False skips the connected-sets/partition search so each design stays its own device. _tuning_record's cache key now includes kinds, so settings do not collide in tuning_cache. Runner.npu gets a matching kinds parameter, threaded through to JointNarrowing, defaulting to None (= NarrowingKinds(), today's plan) so no caller needs to change. Verified device-free (218's NPU is down): 4 new cases in iron/tests/toolchain/narrowing.py, each reusing an existing test's table to show the kind they turn off actually stops happening (widths stay at their default key, groups stay empty, folds stay empty), plus that naming kinds=NarrowingKinds() explicitly matches not naming it at all. The full file, 91 cases including these four, passes: `pytest iron/tests/toolchain/narrowing.py -q -m "not extensive"` (91 passed in 322s, this host under other agents' concurrent load). Co-Authored-By: Claude <noreply@anthropic.com>
Restore the aie_utils import that commit 55be8cd removed as unused in its own history. 35f41f6 (eg2-npu1-v3, stacked earlier in push4) added a use of aie_utils.ensure_current_device() that 55be8cd never saw, so combining the two independently-correct series broke the import. Fold this into 55be8cd before the PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
234f92bc counts a finish's line and LUT tables in GEMM's core memory, so GEMM now narrows an open tile_m, or refuses a given one, at resolution. That is what it set out to do, and it left two narrowing tests built on GEMM refusals that only the placer used to catch. A setting the placer refuses alone is now a Softmax holding a 16384-element row whole (block and streamed given), which every column and channel count leaves past a core's memory: Softmax's resolve does not count it, and the placer says "could not be placed" for each. The EmbeddingGemma 2 gate fold now resolves tile_m=32 and fits, so its test checks that Designs.of measures it under the same key cost_key gives it (the identity a standalone cost_key and the joint search share, traced on both paths: GEMM_1f837af3..., tile_m=32, held 44288 of 65536), with the plain GEMM as its twin. A failed build of its default, recorded with refuse() as the prober records one, then leaves it out with the reason. No real fold's default is refused by the placer alone any more: GEMM and GEMV refuse at resolution, and every elementwise host fold fits at its tile cap on npu1 and npu2. Verified device-free on npu2: both tests pass (295 s, 0.7 s); rejected_shapes.py and gemm_tile_divisibility.py 70 passed. Every pinned GEMM tiling resolve accepts within 12 KB of the cap (22 on npu2, 24 on npu1, plain, prio_accuracy and GELU-finished) is taken by the placer. Co-Authored-By: Claude <noreply@anthropic.com>
A Gather by a call's ids moved every row it was built for, however few the call needed: Llama's prompt chunk moved 2048 rows of 4 KB for a 13-token prompt. `Copy(table[ids[:n]])` now binds the Gather's `valid_rows` extent, and the call moves the first `n` rows alone. A row's control packet now writes its descriptor's length with its address (four words, not three), so the host sets each row past `n` to 16 bytes from the table's first row: a descriptor moves one unit at least, and a drain's per-call length counts 16-byte units. Each feed's drain takes that length from a derived value, `drained0` to `drained3`, so the feeds and their spans of `y` are as before. GatherWords writes its addresses into the new slots, and the trace's encoder now carries the bound, so the host encodes the call's rows alone. Only a full ELF patches the drain length; a bounded gather elsewhere is refused, as is one whose rows are not whole 16-byte units. At Llama 3.2 1B's chunk a 13-row call's descriptors move 85,808 bytes, not 8 MiB (13 rows and 2035 16-byte pads); the control words are 36,480 bytes at any bound. Verified device-free: copy_gather's word-layout and moved-bytes tests (9 passed: the bytes are read from the control words a call writes and checked against each feed's derived drain), rejected_shapes -k gather (5 passed), iron/tests/common (297 passed, 2 skipped), full_elf and lowering_graph -k "prefill or decode or attention" (10 passed), and bounded, unbounded and device-id gathers build as full ELFs. Not yet run on the NPU: 218's device has failed every load since tonight's wedge (DRM_IOCTL_AMDXDNA_GET_INFO, err -22); the new test_the_rows_a_call_bounds_are_gathered is the device check. Co-Authored-By: Claude <noreply@anthropic.com>
The prompt chunk gathers `ids[:rows]`, so a short prompt moves its own embedding rows, and its RoPE angles copy moves its `rows` rows of the chunk's slice of the table rather than all 2048 (256 KB). Every other step of the chunk was already bounded by `rows`. Verified device-free: llama_reference, lm_host and graph.py -k llama (48 passed), the reference gathering the bounded rows against the float32 oracle; the full-ELF prompt builds at Llama 1B's shape (17 of 32 scratchpad words). Not yet run on the NPU (218's device is wedged). Co-Authored-By: Claude <noreply@anthropic.com>
291c4467 made a row's control packet four words (header, length,
address low and high) and sent each batch's packets as one linear
transfer, since they were no longer one size. On NPU2 (211) every gather
by ids then lost all but its first row: short gathers hung
(ERT_CMD_STATE_TIMEOUT), long ones returned row 0 alone, and Llama's
prompt chunk hung. The transfers before it streamed each packet as one
iteration of the transfer ([packets, 1, 1, 3], the preamble [32, 1, 1,
5]). The likely cause: TileControl takes one control packet per stream
packet, and one linear transfer is one stream packet, so it took the
batch's first packet (row 0's) and dropped the rest, pushes included.
The other change, a packet of three data words, which nothing else
sends, is avoided too; the 211 run tells the two apart only if one of
them is fixed alone, which this commit does not try.
A row is now two packets of the size every other batch packet has: its
descriptor's length and address low, then its address high and the
word after, which the preamble writes 0. Every batch transfer is again
[packets, 1, 1, 3]. GatherWords' taps step to the new address words,
and the per-call bound is unchanged: rows past it move 16 bytes.
A new test walks the template and holds every packet after the preamble
to one size, two data words; it fails on 291c4467's layout ({3, 4} !=
{3}). Device-free, with 218's NPU gone: copy_gather's and
rejected_shapes' gather tests (16 passed, 28 device tests skipped). The
device check is copy_gather's bounded, per-call and device-ids gathers,
which 211 runs (patches/llama-npu-gather-repro.patch).
Co-Authored-By: Claude <noreply@anthropic.com>
A Gather bounded per call (table[ids[:n]]) drained a per-call length that
only a full ELF's scratchpad patches, so an xclbin refused it, and with it
NPU1's prompt chunk, whose rows the host then gathered. An xclbin's stream
is generated each call, so each feed's drain there takes its length,
drained{k} 16-byte units, as a dispatch-time size; the control words are
the full ELF's, rows past n moving 16 bytes each.
A new test gathers bounded rows each step its own xclbin, as NPU1 runs a
prompt chunk, against np.take: 15 and 256 rows at bounds from 1 to n,
marked for npu1 and npu2.
Verified device-free (218's NPU is gone): copy_gather (9 passed, the
device tests skipped); the bounded gather builds as an xclbin for npu1
(2 feeds, words n and drained0-1) and npu2 (4 feeds). Not yet run on an
NPU.
Co-Authored-By: Claude <noreply@anthropic.com>
The prompt chunk took its token ids and gathered their embedding rows on the device only on a full ELF; on NPU1 and each step its own xclbin the host gathered them and the chunk took the rows as x, so Llama on Phoenix moved its embeddings on the host. With a Gather, bounded per call, in an xclbin too, the chunk takes its ids on every device and gathers ids[:rows] on the NPU. The x input, shapes()'s image and the model's record of it go; prompt_ids() builds a call's ids, and tune traces the one form. Verified device-free (218's NPU is gone): llama_reference, graph.py, lm_host and copy_gather (142 passed, the device tests skipped). Llama 1B's versions build: npu2 full ELF (prompt 17 scratchpad words), npu2 each step and npu1 each step (xclbins; npu1's prompt Gather on 2 feeds, bound by rows). Not yet run on an NPU: 203 runs TestEachStep on Phoenix. Co-Authored-By: Claude <noreply@anthropic.com>
A bounded call's device buffer is sized to the full n_ids rows; slicing [:n] directly off that flat array before reshaping cut elements, not rows. Reshape to (-1, WIDTH) first, matching what the reference already returns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resample's compatible() only checked patch_columns against the logical columns-a-core bound, so a build that logically fit could still overflow a core's 64KB data memory at link time. held() prices the resize kernel's own tables and row bands plus its image/taps/patches buffers and stack (GEMM's and GEMV's held() do the same for their own arrays), and compatible() now refuses past dev.core_memory_bytes with the byte counts. patch_columns' own default is now device-aware (default_patch_columns(), the same formula padded_rows() already inlined for patch_rows), replacing the flat literal 8. EmbeddingGemma 2's image processor pins patch_columns=8 explicitly, so this is a safety net for it, not a behavior change: verified device-free that held() still fits under core_memory_bytes at patch_columns=8 on both devices (matching real aiecc/ld.lld byte counts to within ~0.1%), and that patch_columns=16 (what a 97x640 image's 126 patch columns need on npu1's 8 cores) is refused at construction with the same numbers ld.lld reports at link time (92128 of kernel statics alone, over the 65536-byte tile, before any ObjectFifo buffer). That shape moves from test_image_processor's matrix (both devices) to test_image_processor_npu2_only (a real device run on npu2, where 126 columns over 16 cores is 8 a core and fits) and test_image_processor_refuses_on_npu1 (construction-only, no NPU needed): npu1's 8 cores need 16 a core for this image, and no patch_columns value makes that fit npu1's per-core memory with today's kernel (coord/resample-npu1-capacity.md has the arithmetic and the real fix, a multi-pass core loop, for iron-next-next). Verified device-free (no NPU; 218's is down): - iron/tests/operators/resample_patch_rows.py: 4 passed - iron/lm/embeddinggemma2/vision/test.py -k test_image_processor_refuses: 4 passed - iron/tests/toolchain/lowering.py -k "resample or image_patches": 6 passed - held()/compatible() checked directly against the real ImageProcessor on npu1 and npu2: identical construction, identical refusal message, for every shape already in the suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
devel's #243 names an operator dropped only against the run before, and iron-next names catalog rows by operator (cf8c459). Both edit the same loop in pretty_trends.py's build_report; the merge keeps devel's skip of undated rows and its previous-run comparison, and iron-next's operator_name over row_key. On three synthetic runs, the merged report names an operator dropped in the run that dropped it and not again after, as devel's does. Co-Authored-By: Claude <noreply@anthropic.com>
Cherry-picked from d66fe703 (branch phi-4-mini-instruct, itself cherry-picked from 4e01ddaf on wt/llama-memory/llama3b), clean cherry-pick, no conflicts. The same 4-GiB host-arena mmap limit (ulimit -l) several ports hit tonight; Phi-4-mini-instruct's own weight+cache arena (~8.6 GiB) is the one that needs it on this branch. Verified: decoder.py/phi4_reference.py (12 tests), the shared image/ graph-version suites this touches (graph_versions.py, full_elf.py, 47 tests), and Llama's device-free suite (21 tests) all pass unchanged. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 87183e4f6f2496178d7ac3f45d3b3b390775b4e8) (cherry picked from commit 6d2d9a340ceed0bcdaca555672488332f351d08c)
MHA never asked for mlir-aie's accurate exp2, so on npu2 the softmax took AIE2P's exp2 instruction, which runs up to 6% high and is the same for every score. Llama 3.2 1B's prompt deep in the cache read KL 0.127 against its oracle; npu1 has no such instruction and passed. The accurate polynomial needs a 4 KiB stack on the softmax core, which aiecc's stack check measures. With the softmax core now setting MHA's time, bf16 products cost what bfp16's shared-exponent ones did (d=64 0.609 vs 0.610 s, d=128 0.875 vs 0.875 s at 2048x32768) at about a third of the error, so MHA defaults to bf16 macs. Measured on npu2 (deep-kl and 220): the deep-prompt test passes at KL 0.063; Llama 1B accuracy at 1024 tokens, max KL 0.056 -> 0.013 and mean down 46%; MHA suites 26/26. Cost: MHA alone 4.8x at d=64 and 2.4x at d=128 (2048x32768); a decode step at position 32700, 97 -> 220 ms; decode at 1024 tokens in turbo, 14.97 -> 13.99 tok/s (-6.6%). On AIE2 the exp2 switch is off and the bfp16 flag is ignored; only the softmax core's stack grows (0xD00 -> 0x1000). Co-Authored-By: Claude <noreply@anthropic.com>
iron.lm keeps what every model shares, in four modules: - base.py: Config and rope_angles, the checkpoint Layout, Checkpoint and the weight trees, the float32 Oracle a model is judged by, Model (a graph whose versions load() compiles up front, in the packaging asked for), and Runner and main; - causal.py: CausalLM, the decoder as one graph; - encoder.py: Encoder, EmbeddingGemma 2's kind of graph, and sandwich_layer, the layer its text encoder and vision tower share; - blocks.py: project, the SwiGLU and GELU-gated feed-forwards, sandwich norms and an encoder's attention. The models are applications over it: iron/lm/llama3 and iron/lm/embeddinggemma2, its audio and vision towers included, become iron/applications/llama3 and iron/applications/embeddinggemma2, each with its test and README, so CI's example suite collects the towers' tests as well. Test function names and parametrize IDs are unchanged, so CI's chart rows follow the directories. Library tests that use Llama import it from its application. Every decision the shared decoder and the EmbeddingGemma 2 port made is kept; what changes is where the code lives and what is shared. The text encoder's layer and the vision tower's ran the same operators in the same order, and are now one function over the same blocks; the text, audio and vision graphs share Model.load, and Multimodal keeps the packaging load() is given for its mixed versions too. Behavior-neutral, checked device-free by rc2/check.py: for Llama 3.2 1B (the decode step and the prompt chunk) and EmbeddingGemma 2 (text, audio, vision and the four multimodal versions), on npu1 and npu2, every traced step with its resolved design and array keys, the bindings, the order the weights and states are met in, and the sha256 of every weight by its path are identical before and after, and so are the float32 oracles' outputs on fixed inputs. pytest --collect-only gives the same node IDs apart from the file paths. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The restructure moved this package from embeddinggemma_2 (underscore) to embeddinggemma2, but one import and two usage examples still named the old path. The import resolved as an empty implicit namespace package rather than failing outright, so it only surfaced when actually run: ImportError: cannot import name 'test' from 'iron.applications.embeddinggemma_2' (unknown location) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The restructure merged what used to be a separate model package directory into the application's own, so "over the model package X" repeated the same path the Location line had just given. Point at the actual shared library instead, matching AGENTS.md's own phrasing for the same section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note, this PR relies on: Xilinx/mlir-aie#3884
PR Merge Checklist
develcommit and pointing todevel.