I work on LLM inference, mostly on Apple silicon with MLX, Core ML and the Neural Engine. A lot of that is measuring runtimes: how well the cache gets reused, what happens under concurrency, whether the same request gives the same output, where the latency actually goes. When I find a bug in something I depend on, I send the fix upstream — including to MLX, oMLX, ROCm AITER, vLLM and Apple's coremltools.

meowcoder.com · Research notes · tc3oliver@gmail.com

Apple coremltools

• Open · #2876: MLModel.predict() held the GIL for the whole native Core ML call, so other Python threads in the process stalled behind it. The patch releases it around the native call, with a threading test.

MLX

• Merged · #4615: The distributed launcher could lose a rank's final output when the process exited before its pipes were drained. The fix reads both pipes to EOF without blocking.

oMLX

• Merged · #3685: The SDPA256 prefill path was picked by how much memory happened to be free, so the same request could give different temperature-0 output in two processes. Now the choice depends only on the input shape.
• Merged · #3840 + #3842: On hybrid attention/recurrent models the SpecPrefill draft cache never hit: a restored cache looked empty, and recurrent state wasn't saved at block boundaries.
• Merged · #3664: The Responses API dropped namespace tool groups, which is how Codex passes MCP tools, so the model never saw them.
• Merged · #4035: After a failed SpecPrefill request, stale state could hang the engine or make the retry prefill the wrong tokens.

ROCm AITER

• Merged · #6693: Qwen3.8-Flash-Next FP8 fused MoE had no tuned rows for MI325X (gfx942, 304 CUs) and ran the default configs. The patch adds eight tuned rows: 3.7–13% less fused-MoE time on those shapes in the tuner's compare report. End to end, cold prefill moved +0.8%; decode mostly doesn't hit the new rows.

vLLM

• Open · #62589: The ROCm sparse-attention indexer scored every row against the full page-table width; in a 32K prefill on MI325X that kernel took 43% of GPU time. The patch skips columns past each request's visible range and tiles the scoring. An independent tester measured 1.61× geometric-mean output throughput over 12 cases on MI355X, on the first revision.

_{Status syncs from GitHub daily. The rest: All upstream pull requests.}

laya-apple: runs Laya on the MLX GPU and the Neural Engine concurrently, with every output checked against upstream Laya. Profiling showed GPU tail latency climbing whenever the Neural Engine ran; tracing it led to the coremltools patch above. laya-apple uses Core ML's async API instead of waiting for the patch, and falls back to its earlier path if it detects the slow state. Median GPU result return on an M4 Max went from 4–9 ms to about 0.04 ms, with no mismatches across 154 validation runs. Research notes · PyPI

llm-inference-systems: experiments on what inference optimizations cost later requests: a sparse prefill that stops the reusable prefix growing, speculative decoding where verify cost matters more than acceptance rate, and rebuilding the reusable state in the background. The oMLX maintainer closed my pull request (#3964) without merging, arguing recovery at best matches SpecPrefill off; I never ran a clean off to compare against. Case study · Engineering notes

Embed2ASR: a small reader trained on a frozen EmbeddingGemma 2 audio encoder transcribes Mandarin, English and code-switched speech. On one MI300X it ran 4.16× faster than Whisper large-v3, which wins on most test sets. Case study · Write-up

qwen3.8-27b-5070ti-eval: a pre-registered evaluation of a 27B model on one 16 GB RTX 5070 Ti, with raw data and graders. It scores 92.1 % on HumanEval+. The first competitor spilled out of VRAM mid-run, so I voided that run and round 1 makes no comparison. Under WSL2 a 128K context loaded without an error but decoded at 5.9 tok/s, apparently because some memory had landed in system RAM. Article · Evidence index

PiShip: lets a company ship Pi as its own coding agent without forking it. Pi keeps the agent loop and tools; PiShip adds OIDC login, short-lived credentials for an internal LLM gateway, policy, MCP rules and a sandbox. If a required sandbox can't start, the command doesn't run. Pre-release; every release archive is attested and built twice to check that the payloads match. TypeScript. Status · Enterprise integration

Smaller things: SignalForge, a self-hosted intelligence pipeline; version-aware-code-mcp, an MCP server that keeps a coding agent's code search on the right commit; deepseek-v4-flash-mi300x, a vLLM serving baseline on AMD MI300X; claude-team-kit, a Claude Code plugin that limits concurrent agent-team workers and shows them in a Mission Control pane (pre-release); Shouri; my coding-agent skills.

I write up research at study.meowcoder.com, in Traditional Chinese. Two good places to start: 當 prefill 變快，agent 反而變慢 and 償還 reusable state 的債.

33 pull requests to projects I don't maintain: 13 merged · 20 open.

Apple coremltools (1 open)

• ○ #2876 Release the GIL during native MLModel prediction

MLX (1 merged)

• ✓ #4615 Fix lost rank output in the distributed launcher

oMLX (6 merged · 7 open)

• ✓ #4035 fix(scheduler): clear SpecPrefill state after failures and cache rejects
• ✓ #3685 fix(attention): keep SDPA256 prefill on the bounded route
• ✓ #3842 feat(specprefill): preserve draft recurrent state at cache boundaries
• ✓ #3840 fix(specprefill): derive draft cache position from attention layers
• ✓ #3746 fix(ane): avoid impossible sequence-length guidance below the ANE minimum
• ✓ #3664 fix(responses): route namespace tool groups through to the model and back
• ○ #4822 fix(unlimited-ocr): don't add a second <image> to a prompt that has one
• ○ #4500 perf(mtp): let greedy chains draft through the coarse lm_head (opt-in)
• ○ #4461 fix(specprefill): keep manual_rope output in the input dtype
• ○ #3962 test: reset the image decode cache between tests
• ○ #3811 fix(specprefill): keep selected tokens at their original positions on mRoPE VLMs
• ○ #3762 feat(anthropic): accept per-request SpecPrefill overrides on /v1/messages
• ○ #3756 fix(specprefill): preserve the full static system/tool prefix

Other projects (6 merged · 12 open)

• ✓ api7/aisix#1348 refactor(bedrock): send Converse request bodies with a stable key order
• ✓ ROCm/aiter#6693 [Config] Qwen3.8-Flash-Next FP8 fused-MoE tuned configs for gfx942 (MI325X, 304 CUs)
• ✓ NandhaKishorM/laya#260 docs: add laya-apple to Community Tools
• ✓ eiei114/pi-model-fallback#64 fix(error-status): parse "Error: <status>" with no colon after the status
• ✓ DrJuChunKoO/TransPal-gemini-transcriber#1 feat: add resume-from-interruption support for long audio transcriptions
• ✓ Baseflow/screenrecorder#9 fix: fix the black background
• ○ ml-explore/mlx-lm#2006 Skip fully masked key tiles in Flint sliding-window prefill
• ○ recursive-oss/rec#27 Raise the HTTP server's listen backlog to 32
• ○ recursive-oss/rec#25 Use the causal mask mode for global-layer prefill attention
• ○ recursive-oss/rec#23 Share one sort between top_p and top_k sampling
• ○ recursive-oss/rec#22 Resolve each KV address once per token in long-context attention
• ○ recursive-oss/rec#21 Skip fully masked key tiles in sliding-window prefill attention
• ○ BerriAI/litellm#47484 fix(least-busy): reserve the picked deployment so a burst of requests spreads out
• ○ BerriAI/litellm#47483 fix(http_handler): keep the connect timeout bounded when a request passes a numeric timeout
• ○ vllm-project/vllm#62590 [ROCm][Perf][Qwen4Exp] Use BLOCK_N=32 for sparse GQA prefill on gfx942
• ○ vllm-project/vllm#62589 [ROCm][Perf][Qwen4Exp] Tile QSA indexer scoring and skip invisible columns
• ○ ROCm/aiter#6692 [CK] [Bugfix] FMoE tuner: use all sources for shapes FlyDSL v2 cannot tune
• ○ ray-project/llmperf#100 feat: Support reasoning_content in OpenAI chat completions streaming response