# MTP + TurboQuant = 🚀

Published 2026-05-29 · by Artjoms Krasnopjorovs (https://krasnopjorovs.com/)

It's nice weather in Zurich, around +30 Celsius (86 Fahrenheit), and it's a fiesta mood. On one of such evenings, after sunset, I decided to do maintenance on my AI cluster. Maintenance sounds boring… but only in a boring environment. With LLMs, it's not the case. So when the regular checks and patches were done, I found an item on the to-do list which seemed interesting for that evening: one LLM was giving strange results and from time to time was going into infinite loops, burning electricity until the context window was filled.

What got my attention to this to-do item was the recent buzz in the AI community: many discussions about LLMs compressed down to Q1, improvements in KV-cache compression with TurboQuant, and the most recent MTP (Multi-Token Prediction) support in llama.cpp for Gemma 4. So I got curious about how all these innovations could improve things. Q1 I decided to put aside — will check it later. The remaining two options to try were MTP and TurboQuant.

🔻 The Bottleneck
LLM inference on local GPUs is bandwidth-bound, not compute-bound. Generating one token means reading all model weights from VRAM. The problematic LLM was giving 35 tokens per second — which is fine for a 31B model on a workstation-class card. But you can feel that it's the software lagging behind — the hardware is willing and able to do better.

So ~34 t/s was taken as the baseline. The LLM is Gemma 4 31B Q4_K_M (~17GB of weights). At typical workstation-GPU memory bandwidth, reading those weights once per token gives a hard theoretical ceiling close to that baseline — so we were already near the limit, with no obvious room to grow on the hardware side. Time to look at the software side.

I asked one of the big cloud LLMs how I could implement MTP and TurboQuant. Below is the result.

🧠 Solution 1: MTP (Multi-Token Prediction)
Classical speculative decoding uses a small "draft" model to guess the next tokens, then verifies them in one parallel pass through the big model. The trick: instead of reading full LLM weights (multiple GBs, depending on model size) to produce one token, you read them once to verify N predictions.

Google ships Gemma 4 with an official MTP "assistant head" — a 470M parameter network trained on the same hidden states as the main 31B model. The acceptance rate on this drafter is much higher than third-party draft models because it literally "knows how the big model thinks." It loads directly into the target via --mtp-head, requiring no separate process.

💾 Solution 2: TurboQuant (Better KV-Cache)
The other usual VRAM hog is the KV-cache, especially at long contexts. I was used to getting the highest possible context per VRAM GB.

TurboQuant (turbo3) is a newer KV-quant scheme that uses a kernel-level Walsh-Hadamard Transform before quantization, aiming to preserve more information in the same bit budget. On Gemma 4 with a 262K context, it saved about 30% VRAM on the KV-cache compared to q4_0. (Note: the README claims no measurable quality loss; I haven't independently verified quality on long-context tasks, just measured throughput.)

🛠️ The Catch: You Have to Recompile llama.cpp
Both optimizations live in a fork — atomic-llama-cpp-turboquant — not in upstream llama.cpp. The gemma4_assistant architecture and the turbo3 KV-cache type are unknown to the official build. So before any benchmarks, you have to recompile it following the README. For those already using llama.cpp, recompiling is a routine process, so not a big deal.

The resulting binary is a drop-in replacement for upstream llama-server — same OpenAI-compatible API, same systemd-friendly behavior. The only "new" flags you actually need in production are --mtp-head, --spec-type mtp, and -ctk turbo3 -ctv turbo3 (plus the -ctkd/-ctvd equivalents for the draft side).

The trade-off is obvious: you're now responsible for tracking upstream merges into the fork. Security fixes and protocol updates in upstream llama.cpp won't arrive automatically. For a homelab this is fine; for a production fleet it requires a bit of discipline.

📊 The Results (Baseline vs. MTP+TurboQuant)
JSON config generation: 34 t/s ➡️ 49 t/s (+45%)
Code (Python quicksort): 35 t/s ➡️ 46 t/s (+33%)
Creative writing: 34 t/s ➡️ 43 t/s (+25%)
Explanation prose: 34 t/s ➡️ 42 t/s (+23%)
Logic puzzle: 34 t/s ➡️ 41 t/s (+20%)

40+ tokens per second feels way better than 30. Sure, it's subjective when you look at the output, but it also saves time when the LLM is used via API. So I was very happy with the improvement. Will monitor how it works — hope no bad surprises appear.

💭 A Broader Thought
This experiment reinforces a view I've been holding for a while: we're nowhere near the ceiling of what current local LLMs can do. The hardware story (more VRAM, faster bandwidth, new architectures) gets most of the attention, but the software and algorithmic side still has plenty of room to grow. MTP, smarter KV-cache schemes like TurboQuant, prompt caching, speculative decoding variants, better quantization formats — each of these squeezes more out of the same silicon. And they stack.

See you next time!

#llm #inference #llamacpp #gemma #speculativedecoding #homelab #AI #machinelearning

#llm #inference #llamacpp #gemma #speculativedecoding #homelab #AI #machinelearning

Source: https://krasnopjorovs.com/articles/mtp-turboquant