Granite 4.0 was IBM's big architectural bet. Most of its attention layers were replaced with Mamba-2 state space layers. IBM said this cut memory use by more than 70%. Granite 4.1 kept the same design. Granite 4.2 arrived on Ollama a few days ago.
We downloaded it and opened the model file. The Mamba-2 layers are gone.
In this blog, we will find out what that costs. We ran Granite 4.2 against Gemma 4, Qwen 3.8 and Ornith 1.5. All three still solve the memory problem that Granite just gave up on.
What We Tested
| Group | Models | VRAM for weights |
|---|---|---|
| Small | Granite 4.2 8B, Gemma 4 12B, Ornith 1.5 9B | 5.3 / 7.4 / 5.8 GB |
| Big | Granite 4.2 30B, Gemma 4 31B, Qwen 3.8 27B | 17.7 / 18.3 / 16.8 GB |
We made the two groups by VRAM, not by parameter count, because what matters is which models fit the same card. Everything ran on one machine in one session.
- GPU: RTX 5090, 32 GB, all layers on the GPU
- Runtime:
llama-serverbuild b10448 with CUDA on Windows 11 - Quantization: Q4_K_M for all six
- KV cache: f16, flash attention on, prompt caching off
- Sampling: each model uses the settings its own card recommends
- Server flags: same for every model, one slot, no warmup, chat templates on
- Grading: plain Python and real test runs, no judge model anywhere
Important
We used Ollama only to download the weights, never to run them. Ollama starts this same server with its own hidden defaults, so its speed numbers measure those defaults, not the model.
What Is Inside the GGUF Files?
A GGUF file is how a model is stored on disk, and it begins with a list of every tensor inside. So we do not have to trust a model card. We can count.
An SSM layer always carries tensors named ssm_, and a draft head carries tensors named nextn. Granite 4.2 has zero of both, in the 8B and in the 30B.
So every Granite block is a full attention block, which keeps a cache that grows with our context. The 8B has 40 such blocks and the 30B has 64.
Gemma 4 instead uses sliding window attention on most blocks. A window block only looks at the last 1024 tokens, so its memory stops growing after that. Qwen 3.8 and Ornith 1.5 use SSM layers, which keep a small state of fixed size instead of a growing cache.
How Much Does the KV Cache Cost per Token?
The KV cache is memory the model keeps for every token in our context, so it grows as the chat grows. This one number decides our real context limit.
We worked this out from the tensor shapes and checked it against what the server really allocates. Both agree exactly.
| Model | Cache per token | Cache at its advertised context |
|---|---|---|
| Granite 4.2 8B | 160 KB | 21.5 GB at 128k |
| Granite 4.2 30B | 256 KB | 34.4 GB at 128k |
| Gemma 4 12B | 16 KB | 4.6 GB at 256k |
| Gemma 4 31B | 80 KB | 22.3 GB at 256k |
| Qwen 3.8 27B | 64 KB | 17.2 GB at 256k |
| Ornith 1.5 9B | 32 KB | 8.6 GB at 256k |
Now let's read the second column. Granite 4.2 30B needs 34.4 GB of cache to reach the 128k printed on its own model page, plus 17.7 GB for its weights. A 32 GB card cannot do this. The 8B has the same problem in smaller form: 160 KB per token, ten times what Gemma 4 12B pays.
How Much Context Can Each Model Really Use?
Caution
On Windows, a CUDA allocation bigger than your VRAM does not fail. The driver quietly uses system RAM instead. The server starts, reports the full context, and never warns you.
So on Windows everything loads, and the real question is how much context the model can hold before it slows down. We call that the usable context: we measure speed at a small context, then grow it until speed falls below 90% of that. The last size that passed is the usable context.
| Model | Advertised | Really usable | Share |
|---|---|---|---|
| Granite 4.2 30B | 128k | 58,368 | 45% |
| Gemma 4 31B | 256k | 155,648 | 59% |
| Qwen 3.8 27B | 256k | 245,760 | 94% |
| Granite 4.2 8B | 128k | 131,072 | 100% |
| Gemma 4 12B | 256k | 262,144 | 100% |
| Ornith 1.5 9B | 256k | 262,144 | 100% |
Two models of almost the same size, and a 4.2 times gap. The drop is sudden too: Granite 30B runs at 78.6 tokens a second at 58,368 tokens, and 15.1 at 61,440. That is 5% more context for 81% less speed, so a size that works today tells us nothing about how close we are to the edge.
Granite 4.2 8B is the good news here. It reaches its full 128k, and Gemma 4 12B and Ornith 1.5 9B reach their full 256k.
Which Model Writes Tokens Fastest?
Every model wrote exactly 512 tokens on five kinds of content, twice: forward and in reverse order. Both runs agree within 0.3%, so the card was not slowing down.
| Model | Tokens per second |
|---|---|
| Granite 4.2 8B | 214 |
| Ornith 1.5 9B | 192 |
| Gemma 4 12B | 144 |
| Granite 4.2 30B | 78.5 |
| Qwen 3.8 27B | 78.3 |
| Gemma 4 31B | 70.5 |
Granite is the fastest model in both groups, and in the small group it is not close. In the big group all three land within 8 tokens a second.
Important
These numbers have the draft head turned off. That is the only setting all six models share. Two of them have a draft head and run about twice as fast with it on.
Is the Granite vs Qwen Speed Tie Real?
Granite 30B at 78.5 against Qwen 3.8 at 78.3 looks like a dead heat. It is not.
A draft head is a small extra layer that guesses the next few tokens. The full model checks all the guesses at once and keeps the ones it agrees with. Good guesses mean a much faster model.
Qwen 3.8 and Ornith 1.5 have one. Granite and Gemma do not, so the fair table above has it off. But nobody runs a model with a free speedup disabled, so we measured both ways, turning the head on with the --spec-type draft-mtp flag.
Qwen 3.8 27B, draft head off and on:
| Content | Off | On | Gain | Guesses accepted |
|---|---|---|---|---|
| counting | 76.4 | 179.6 | 2.35x | 84% |
| json | 76.6 | 181.6 | 2.37x | 86% |
| code | 76.2 | 153.7 | 2.02x | 68% |
| fiction | 76.2 | 103.9 | 1.36x | 39% |
| essay | 76.2 | 93.2 | 1.22x | 31% |
Ornith 1.5 9B, the same test:
| Content | Off | On | Gain | Guesses accepted |
|---|---|---|---|---|
| counting | 208.5 | 351.8 | 1.69x | 81% |
| code | 208.6 | 281.5 | 1.35x | 58% |
| json | 208.1 | 231.2 | 1.11x | 43% |
| fiction | 208.6 | 214.6 | 1.03x | 38% |
| essay | 207.8 | 176.8 | 0.85x | 26% |
With its draft head on, Qwen writes code at 154 tokens a second and JSON at 182, about twice Granite's rate. Granite has no such switch.
The gain tracks the last column: counting and JSON accept 84 to 86% of the guesses and roughly double, while essays accept 26 to 31% and barely gain. Ornith even loses speed on essays, because checking wrong guesses costs time.
Warning
A draft head also changes the text. With it off, two runs give identical output. With it on, the output differed in all ten cases we tested. Treat it as a different setting, not a free speedup.
How Many Tokens Does Each Model Need per Answer?
Writing tokens quickly is not the same as answering quickly.
Our old problem set was too easy: five models scored 39 out of 40 on it. The new set has 13 math problems, 7 problems with no valid answer, and 8 formatting tasks. A brute force program computed every math answer first, so the answer key is not our opinion.
Granite ran past our token limit before it finished a single problem. One hard problem cost Granite 4.2 8B 21,327 tokens, another 33,437. So we raised the limit to 32,000 and recorded how many tokens each model used.
| Model | 2k | 4k | 8k | 16k | 32k |
|---|---|---|---|---|---|
| Granite 4.2 8B | 3/20 | 5/20 | 10/20 | 13/20 | 15/20 |
| Gemma 4 12B | 2/20 | 3/20 | 4/20 | 9/20 | 13/20 |
| Ornith 1.5 9B | 9/20 | 10/20 | 11/20 | 15/20 | 16/20 |
| Granite 4.2 30B | 8/20 | 10/20 | 12/20 | 14/20 | 16/20 |
| Gemma 4 31B | 6/20 | 10/20 | 12/20 | 20/20 | 20/20 |
| Qwen 3.8 27B | 11/20 | 12/20 | 14/20 | 16/20 | 18/20 |
Both Granite models land in the middle the expensive way. Granite 30B solves 16 of 20, close to Qwen's 18, but needs the full 32,000 tokens. Qwen finishes 11 problems inside 2,000. Gemma 4 31B is different: at 16,000 tokens it takes everything, the only clean sweep here.
Let me tabulate the final scores for your better understanding.
| Model | Math | Traps caught | Format | Ran out of room | Median tokens |
|---|---|---|---|---|---|
| Granite 4.2 8B | 9/13 | 6/7 | 8/8 | 3 | 6,533 |
| Gemma 4 12B | 7/13 | 6/7 | 8/8 | 6 | 11,036 |
| Ornith 1.5 9B | 9/13 | 7/7 | 7/8 | 3 | 1,618 |
| Granite 4.2 30B | 9/13 | 7/7 | 8/8 | 3 | 5,240 |
| Gemma 4 31B | 13/13 | 7/7 | 8/8 | 0 | 2,657 |
| Qwen 3.8 27B | 11/13 | 7/7 | 8/8 | 2 | 450 |
"Ran out of room" means the model was still working when it hit the token limit. That is not a wrong answer, so we counted them apart.
Sampling settings are not the cause either. IBM, the Ollama Modelfile, and the Ollama page each suggest different settings, and all three run just as long. Granite 4.2 simply thinks for a long time.
How Well Does Each Model Code?
Nine coding tasks with 99 hidden tests. The model writes a function, and we run the tests against it in a separate Python process.
The tasks are precise specs with edge cases that are easy to miss: Roman numerals that must reject IIII, INI files with line continuations, CSV quoting, negative business day offsets. We wrote our own solutions first and confirmed they pass all 99 tests.
| Model | Tasks passed | Tests passed | Median tokens |
|---|---|---|---|
| Granite 4.2 8B | 9/9 | 100% | 11,947 |
| Gemma 4 31B | 9/9 | 100% | 3,168 |
| Qwen 3.8 27B | 9/9 | 100% | 7,126 |
| Gemma 4 12B | 8/9 | 91% | 11,336 |
| Granite 4.2 30B | 7/9 | 90% | 6,844 |
| Ornith 1.5 9B | 6/9 | 78% | 5,005 |
Granite 4.2 8B passed everything. The 30B did not: it failed 2 of 12 tests on the INI parser and 1 on line wrapping, so the extra 21 billion parameters bought nothing.
Gemma 4 31B and Qwen 3.8 27B also passed all nine, and Gemma did it in 3,168 median tokens against Granite 8B's 11,947. Ornith 1.5 9B is the weak one at 6 of 9, which is where Granite 8B earns its place.
How Well Does Each Model Use Tools?
Eight tasks with real tool calls. One pages through data, one tool rejects a bad key once and explains the fix, and two ask for data that does not exist, where the only right answer is UNAVAILABLE.
| Model | Solvable | Traps | Bad calls | Recovered from error |
|---|---|---|---|---|
| Granite 4.2 8B | 5/6 | 2/2 | 0 | 3/4 |
| Gemma 4 12B | 6/6 | 1/2 | 0 | 3/4 |
| Ornith 1.5 9B | 6/6 | 2/2 | 0 | 5/5 |
| Granite 4.2 30B | 6/6 | 2/2 | 0 | 3/4 |
| Gemma 4 31B | 6/6 | 2/2 | 0 | 4/4 |
| Qwen 3.8 27B | 6/6 | 2/2 | 0 | 4/4 |
Five of six models scored 8 out of 8, Granite 4.2 30B among them, so this test did not separate them. Only two things moved: Granite 4.2 8B alone failed the long task, and Gemma 4 12B alone made up an answer instead of saying UNAVAILABLE.
Across 312 tool calls, not one was broken. Granite writes its calls in an XML style instead of JSON, but so do Qwen and Ornith. Gemma is the odd one out.
Does Granite's Reasoning Effort Setting Work?
Granite 4.2 says we can adjust its reasoning effort. Claims like this have fooled us before, so instead of judging the output, we asked the server to build the prompt and compared the bytes. We sent the same question five times, changing only the effort setting.
| Effort setting | Prompt size | Difference |
|---|---|---|
| not set | 102 bytes | this is the baseline |
medium |
102 bytes | identical to the baseline |
high |
102 bytes | identical to the baseline |
max |
102 bytes | identical to the baseline |
low |
127 bytes | adds {reasoning effort: low} to your message |
Only low changes anything, and the template explains why: exactly one line reads the setting, and all it checks is whether the value equals low. Other values are accepted in silence, which is worse than an error, because a config file full of reasoning_effort: high looks like it is working.
To Ollama's credit, its model page does say two levels. The phrase "effort level" is what misleads.
Which Model Should We Run?
Granite 4.2 8B is worth running: every coding task passed, the fastest in its group, and a full 128k context. Two catches: that 128k needs 21.5 GB of cache, and hard problems cost tens of thousands of tokens.
Granite 4.2 30B is for short work only. It gets 58k of usable context against Qwen's 246k at the same size, and it lost the coding test to its own 8B.
Qwen 3.8 27B is the model to beat, and Granite does not beat it: a 450 token median answer, 94% of its advertised context, and a draft head that doubles its code speed.
Gemma 4 31B is the most accurate, the only model to get all 13 math problems right, paid for with a 16,000 token budget and the slowest decode here.
Ornith 1.5 9B gives short answers, a cheap cache, and a full 256k context. It only fell down on coding.
| Model | Tokens per second | Usable context | Best at |
|---|---|---|---|
| Granite 4.2 8B | 214 | 131,072 | Careful coding on a small card |
| Granite 4.2 30B | 78.5 | 58,368 | Short context work only |
| Ornith 1.5 9B | 192 | 262,144 | Short answers, cheap cache |
| Gemma 4 12B | 144 | 262,144 | The cheapest context here, at 16 KB per token |
| Qwen 3.8 27B | 78.3 | 245,760 | Long context, and anything you pay tokens for |
| Gemma 4 31B | 70.5 | 155,648 | Hard math, if you can wait |
So, back to the question we started with. Granite 4.0 promised 70% less memory. Granite 4.2 gave that back, and the numbers above are the price.
What We Got Wrong
Two of our own test items were wrong, and the models were right: a train problem we had marked unanswerable, which all six solved, and Granite's fair answer that an impossible event has probability 0. We fixed both graders and rescored the saved answers. No model was re-run.
Conclusion
The story of this benchmark is memory. Granite 4.2 writes tokens fast, but its cache now grows faster than any other model here, and that decides how much context it can really hold.
Key takeaways:
- Every Granite 4.2 block now pays for a full attention cache: 256 KB per token on the 30B, four times Qwen 3.8.
- That bill caps Granite 30B at 58k usable context on a 32 GB card; Qwen reaches 246k at the same size.
- Granite 4.2 8B is the bright spot: fastest in the small group, and a clean pass on all nine coding tasks.
- Speed per token is not speed per answer. Qwen's median answer is 450 tokens against thousands for Granite.
- Only the
lowreasoning effort setting does anything;medium,highandmaxchange nothing.
Next steps:
- The Ornith 1.5 9B vs 35B benchmark runs these same measurements on a new model family.
- Qwen 3.8 27B llama.cpp speed settings shows how to turn the draft head on and tune the other flags.
- The local LLMs technical reference guide matches a model and context size to our own GPU.
This is how Granite 4.2 measures up. We started inside the GGUF files where the Mamba layers used to be, we saw the cache bill that decision creates, and we finished with the numbers that say which model our card should run.
