How Gemma 4 Got 4× Better Without Getting Bigger

Introduction

Google released the Gemma 4 weights on 2 April 2026 and published the technical report three months later. The numbers in it are the reason anyone opened the document. AIME 2026 goes from 20.8 to 89.2 against Gemma 3 27B. Codeforces Elo goes from 110 to 2150. τ²-bench telecom, one of the harder agentic evaluations in circulation, goes from 3.1 to 69.3.

The parameter count went from 27B to 31B. That is a fifteen percent increase. And the report describes both its pre-training and its post-training as following a similar approach to Gemma 3, which removes the other obvious explanation.

So the report has to account for the gap somewhere, and it does. The instruction-tuning section states that including subsets of data encouraging in-context attribution, hedging and refusal improved factuality metrics without degrading performance on other metrics. That is a claim about data composition, with a measured result attached and an explicit absence of trade-off. Elsewhere, §4.2 reports that the smallest model in the family roughly matches Gemma 3 27B at about a tenth the parameter count.

Then you reach section 2.4. The pre-training corpus gets one sentence naming modalities, a January 2025 cutoff, and no token count for any of the five sizes — the first Gemma generation to publish none. Both statements live in the same seventeen pages: data composition explains the results, and the data will not be described. That gap is the story, and it has direct consequences for anyone planning to build on the model.

What Did Google Actually Change in Gemma 4?

The suite spans five sizes: dense models at 2.3B effective, 4.5B, 12B and 31B parameters, plus a Mixture-of-Experts variant, the 26B-A4B, carrying 26B total parameters with 3.8B activated. All Gemma models in the family ship under Apache 2.0 with a 256k context window, and all come with quantization-aware-trained checkpoints alongside the raw weights.

The models' architecture changed in three ways that carry most of the weight. Thinking mode lets the model emit a reasoning trace before answering. Long-context efficiency comes from a 5:1 ratio of local sliding-window to global attention combined with p-RoPE and KV cache sharing, cutting the global KV cache by 37.5%. And the 12B abandons its encoders entirely: instead of a 550M vision encoder, it projects raw 48×48×3 image patches and 40ms audio chunks straight into the embedding space through a 35M projection, discarding the 305M audio encoder in the process.

Human evaluation metrics come from LMArena, where the 31B sits at 1451 ± 8 Elo, ranked 43rd overall and the strongest dense open model on the board. Forbes framed the release as frontier-class scores on a single GPU, which is roughly why enterprise teams are evaluating it at all. Epoch AI's measurement of the open-to-closed capability gap puts open-weight large language models (LLMs) about four months behind the closed frontier over the first five months of 2026. Four months is a lag most teams can live with when the licence is Apache 2.0.

What Do the Gemma 4 Technical Report's Benchmark Numbers Actually Measure?

Two caveats apply to every figure below. The benchmark evaluation metrics all come from Google's own technical report; no independent replication existed at the time of writing. And the report says it filtered its data to decontaminate benchmarks without describing how, so decontamination has to be taken on trust. One point in Google's favour: AIME 2026 postdates the January 2025 pre-training cutoff, so that particular jump cannot be contamination.

τ²-bench puts the agent in a dual-control customer service environment: both the agent and a simulated user hold tools and can modify shared state. The agent has to follow a written domain policy, and the evaluator checks the resulting database state rather than the final message. Telecom is the hardest of its three domains because the agent has to guide a user who has their own tools and their own partial view of the world. In the original paper, GPT-4.1's pass^1 fell from 74% on retail to 34% on telecom, and pass^k declined faster on telecom too, meaning less consistency across repeated trials.

That makes it a test of policy adherence under coordination pressure, which is closer to enterprise agent work than a competition maths problem is.

One note on the τ²-bench row: the Gemma 4 report does not state which pass metric or which user simulator it used, so its 86.4 and 69.3 are not directly comparable to the original benchmark paper's figures. Treat them as strong, and treat the like-for-like comparison as unavailable.

Why Did Gemma 4 Improve So Much Without Getting Much Bigger?

The 31B is fifteen percent larger than the Gemma 3 27B it is compared against. AIME quadruples. τ²-bench retail moves from 6.6 to 86.4. No account of parameter scaling closes that gap, and the report does not ask you to believe one: §2.4 says pre-training follows a similar approach to Gemma 3, and §3 says the same of post-training, with thinking mode named as the significant difference. What is left is the training data, which significantly influences the model's performance in ways parameter count here does not.

§4.2 reports that the E2B roughly matches Gemma 3 27B across the board with about ten times fewer parameters. Nothing in the models' architecture accounts for a tenfold size difference at equal performance; the model training recipe and the data feeding it have to.

The instruction-tuning section describes the post-training data as carefully optimized to maximize model performance, then names the filtering methods applied. Data filtered out of the mixture includes examples containing certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicates. Deduplication and sensitive data filtering are table stakes. Mistaken self-identification is a category you only strip out after someone has gone looking for it.

Google then reports that including subsets of data which encourage in-context attribution, hedging and refusal improved performance on factuality metrics without degrading performance on other metrics. Adding a targeted category of examples to the mixture bought a measured capability gain with no observed trade-off elsewhere. Attribution, hedging and refusal are also behaviours that require human judgment to demonstrate. You cannot scrape a corpus of well-calibrated hedging; somebody has to decide what appropriate uncertainty looks like in a given context and write it down consistently enough that a model can generalize from it.

Gemma 2 described setting its data mixture by ablation, treating composition as a variable to tune rather than a bucket to fill. Across three generations the Gemma reports have located their gains in the composition of the training material more than in how big the model got.

Which of These Numbers Should Change Your Decision?

Two findings change how the rest of the table should be read.

The first is that every Gemma 4 column in the main results table is a thinking-mode number, and the Gemma 3 27B comparison column is not. The caption says so plainly. Models in thinking mode emit a reasoning trace before answering, which costs tokens and latency, partly offset by the drafter head for speculative decoding. If you are building a cost-per-task comparison against a non-reasoning baseline, you are not comparing like with like. If you plan to deploy without thinking mode, none of these numbers describe your deployment.

The second sits in the appendix. The vision results in the main table use the maximum supported resolution of 1120 vision tokens. Table 12 reports the same benchmarks at 280 tokens. On InfographicVQA, the 12B falls from 88.4 to 58.7. On OmniDocBench, where lower is better, it degrades from 0.164 to 0.408, and the E2B from 0.290 to 0.496. The 31B holds up best at 92.0 down to 82.8.

Resolution is a deployment cost decision, not a model property. The same weights answer a document-understanding question competently or incompetently depending on a serving parameter you chose for throughput reasons. Any conclusion you draw about whether Gemma 4 is fit for document work is a conclusion about a configuration.

Add the pass^k point from τ²-bench and the shape of a sensible evaluation becomes clear. A model that succeeds once in three attempts cannot go in front of customers, and Terminal Bench Hard at 36.0 is a large relative gain sitting on a low absolute score. Published evaluation metrics tell you how a model performed against similarly sized models on someone else's configuration. As our guide to AI benchmarks in 2026 documents, the gap between lab benchmark scores and real-world deployment performance runs to 37%. The only evaluation that answers your question is one built on your data, at your deployment configuration, with your task distribution and your consistency threshold.

Can You Fine-Tune Gemma 4, and What Does That Actually Take?

Yes, and more cheaply than most teams expect. Unsloth's fine-tuning documentation reports LoRA training for the E2B at 8–10GB of VRAM, E4B at 17GB, and QLoRA on the full 31B at 22GB. The Gemma 4 26B-A4B needs more than 40GB. From the report's own memory table, quantization takes the 31B from 64.0GB to 19.2GB in weights. Those tooling figures are self-reported and move with sequence length, batch size and gradient checkpointing, so treat them as indicative. Google's model card documents support for full SFT, LoRA, QLoRA, DPO and RLHF across 140+ languages.

One detail worth catching before your first run: Gemma pre-trained models and instruction-tuned variants terminate generation with different control tokens (PT models emit <eos>, IT models emit <turn|>), so fine-tuning either one requires adding its respective end token.

The compute barrier is a single consumer-class GPU. What stays binding is the data.

Meta's own fine-tuning guidance names three quality principles for instruction training datasets, and the one it names first is consistent annotation, followed by freedom from errors and mislabelled data and a representative distribution. The LIMA result it reports — strong alignment from roughly a thousand curated examples, beating the same base model tuned on tens of thousands of machine-generated ones — has held up well enough that recent work on token-level supervision in SFT still treats small curated sets as the reference point. On the multimodal side, DataComp-VLM found across 160 datasets and 6T tokens that mixture composition mattered more than filtering.

When your target is a few hundred to a few thousand examples rather than a few hundred thousand, every individual example is load-bearing, and inter-annotator agreement stops being a process nicety and becomes a training-data variable. The lever available to you is the same one Google described pulling: decide what goes into the mixture and in what proportion, then measure whether it helped. The difference is that you have to prove it to someone. That makes measurement the practical requirement — consensus scoring to quantify whether annotators actually agree on the judgments you are about to train on, gold-standard testing to catch accuracy drift before it propagates into a mixture, and multi-stage review with enforced step separation so nobody signs off on their own work. Kili's asset coverage across image, video, PDF, text and geospatial lines up with Gemma 4's multimodal surface, which matters if the fine-tune spans modalities.

One caveat for anyone reaching straight for preference data: Kili's dedicated LLM and RLHF project types, including pairwise response comparison and model integrations, are in private beta and available on request rather than by default. The quality machinery described above is generally available and documented.

What Does Fine-Tuning Do to the Safety Work You Didn't Do Yourself?

Google ran extensive safety evaluation on the model it shipped, without safety filters, and published the results in §5.3. Those results describe a model you stop running the moment you fine-tune.

A 2026 preprint measured this directly on Gemma 3 4B. Narrow fine-tuning eroded safety alignment, and misalignment increased monotonically with LoRA rank: the more expressive the adapter, the more damage. A mixture containing only 10% harmful data was enough to induce substantial degradation. The harmful behaviours occupied a low-dimensional subspace, with most of the misalignment signal concentrated in roughly ten principal components.

Text-only safety evaluation processes substantially understated the damage relative to multimodal evaluation of the same fine-tuned models. Deploy a vision-language model, evaluate its safety on text prompts, and you will under-report the problem you created. That study was run on Gemma 3 4B rather than Gemma 4, so the mechanism transfers by architecture and family rather than by direct measurement.

The mitigation literature converges on data composition rather than technique. Work on LoRA-based safety alignment in reasoning models and on how fine-tuning disrupts evaluation consistency both report the same finding from Bianchi et al.: injecting a few hundred safety examples into an instruction-tuning mixture measurably improves post-fine-tuning safety. He et al. add a stranger result, that among benign data, formatted content such as lists, bullet points and mathematical layouts does disproportionate alignment damage. Both deserve retrieval as primaries before anyone quotes their figures.

So the composition of your fine-tuning mixture is a curation decision with a measurable safety consequence, and your evaluation set has to cover every modality you deploy or it will understate the damage.

What Does the Gemma 4 Technical Report Not Tell You About Its Training Data?

Section 2.4 describes the pre-training corpus as "a large-scale, diverse collection of data from a wide range of domains and modalities", names diverse data sources at the level of category — web documents, code, images and audio — gives a January 2025 cutoff, and stops. There is no token count for any of the five sizes. The v2 revision posted on 24 July leaves the section unchanged.

The one data preprocessing detail the report does document is the tokenizer: SentencePiece with split digits, preserved whitespace and byte-level encodings, over a 262k-entry vocabulary. That tells you how text was segmented. It tells you nothing about what text there was.

This is a departure from the two previous generations. Gemma 2 published 13T tokens for the 27B, 8T for the 9B and 2T for the 2B, and described setting its mixture by ablation. Gemma 3 published 14T, 12T, 4T and 2T by size. Gemma 4 publishes none of it, and the mixture-ablation and quality-reweighting discussion is gone too. Google has not commented publicly on the change, so what follows describes what happened rather than guessing at why.

Gemma 4 is typical here rather than singular. Stanford CRFM's Foundation Model Transparency Index recorded a mean score drop from 58 to 41 year over year, with training data the weakest domain of all: an average of 14 on Data Properties, where eight of the assessed companies scored zero. Some of that drop reflects changed indicators, which the report says itself. The same opacity showed up in our data story on MiniMax M3, where a claimed 100T-token pipeline shipped with no published breakdown of domains, languages or filtering methodology.

The regulatory picture moved in the opposite direction. Article 53(1)(d) of the EU AI Act requires GPAI providers to publish a training-content summary on the AI Office template, and enforcement powers commenced on 2 August 2026 — a deadline covered in our introduction to the EU AI Act. The open-source exemption in Article 53(2) covers points (a) and (b) only, so an Apache 2.0 licence does not carry an exemption from point (d). The template asks for overall training data size and the latest acquisition date; the report supplies the date and not the size. Peer-reviewed work presented at FAccT '26 found that as of January 2026 most major providers had published no such summary at all, and that the ones which existed were hard to find. This reading is a reading of the Article text and the commentary around it; it is not legal advice, and we are not asserting that any named provider is out of compliance.

The practical consequences are simpler than the compliance debate. The limitations the training data section leaves you with are concrete ones. You cannot check the corpus against your own evaluation set for contamination. You cannot assess domain coverage before committing engineering time to a fine-tune. And nothing that happened after January 2025 is in the base model, which for a model whose weights shipped in April 2026 is a fifteen-month blind spot you have to fill yourself.

What Gemma 4 Tells Us About the State of Open-Weight Training Data

Gemma 4 is an unusually good architecture paper and an unusually thin data paper. The architecture detail is specific enough to plan a deployment around. The evaluation detail is useful precisely because the appendix undercuts the main table, which is a form of honesty worth crediting. The data section gives you almost nothing to reason with, in a document whose own account of its gains keeps pointing back at data.

A fifteen percent parameter increase produced a fourfold jump on competition maths and a model at a tenth the size matching the previous flagship. Google names the mechanism in its instruction-tuning section: choose what goes into the mixture, measure what it does, keep what improves factuality without costing you elsewhere. Data quality is the explanation the report offers for its own results.

That asymmetry sets the boundary of what you can be accountable for. The pre-training corpus is Google's; its composition, its coverage and its contamination profile are outside your visibility and will stay there. Your fine-tuning mixture and your evaluation set are yours. They are the part of the stack you can specify, measure, audit and defend to a customer or a regulator, and under Article 53 the regulator is now asking. Whatever a fine-tune of Gemma 4 does to safety alignment, and it will do something, the mixture that caused it is a composition decision somebody made deliberately or by default.

The teams that get the most out of open weights over the next year will be the ones treating the model as the commodity and the composition of their own labelled data as the differentiated asset. Google has published that argument about its own model while withholding the evidence for it. You will have to show yours.

Resources

Gemma Model Reports and Documentation

• Gemma 4 Technical Report — architecture, benchmarks, memory footprints, and the two-sentence data section
  • https://arxiv.org/abs/2607.02770
• Gemma 3 Technical Report — per-size token budgets for the previous generation
  • https://arxiv.org/abs/2503.19786
• Gemma 2 — named data sources and mixture set by ablation
  • https://arxiv.org/abs/2408.00118
• Gemma 4 model card — supported fine-tuning methods, context length, language coverage
  • https://ai.google.dev/gemma/docs/core/model_card_4
• Gemma 4 launch announcement — Google DeepMind, 2 April 2026
  • https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/

Benchmarks and Evaluation

• τ²-bench: Evaluating Conversational Agents in a Dual-Control Environment — Barres, Dong, Ray, Si & Narasimhan (2025), arXiv
  • https://arxiv.org/abs/2506.07982
• Epoch AI: open models lag closed models by four months — Epoch Capabilities Index, 2026
  • https://epoch.ai/data-insights/open-closed-eci-gap

Fine-Tuning and Data Quality

• Unsloth Gemma 4 fine-tuning guide — VRAM figures for LoRA and QLoRA across the family (vendor documentation)
  • https://unsloth.ai/docs/models/gemma-4/train
• How to fine-tune: focus on effective datasets — Meta AI engineering (2024)
  • https://ai.meta.com/blog/how-to-fine-tune-llms-peft-dataset-curation/
• Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority — arXiv (2026)
  • https://arxiv.org/abs/2602.01227
• DataComp-VLM: Improved Open Datasets for Vision-Language Models — arXiv (2026)
  • https://arxiv.org/abs/2606.28551

Safety and Alignment Under Fine-Tuning

• Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents — arXiv (2026), measured on Gemma 3 4B
  • https://arxiv.org/abs/2602.16931
• LoRA Is All You Need for Safety Alignment of Reasoning LLMs — arXiv (2025)
  • https://arxiv.org/abs/2507.17075
• Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency — arXiv (2025)
  • https://arxiv.org/abs/2506.17209

Transparency and Regulation

• The 2025 Foundation Model Transparency Index — Wan, Klyman, Kapoor, Maslej, Longpre, Xiong, Liang & Bommasani, Stanford CRFM
  • https://crfm.stanford.edu/fmti/December-2025/index.html
• EU AI Act, Article 53: Obligations for providers of general-purpose AI models
  • https://artificialintelligenceact.eu/article/53/
• Quality assessment of public summaries of training content required by Article 53(1)(d) — Blankvoort, Pandit & Gahntz, FAccT '26
  • https://doi.org/10.1145/3805689.3806755

Journalism and Analysis

• Google's Gemma 4 runs frontier AI on a single GPU — Janakiram MSV, Forbes (April 2026)
  • https://www.forbes.com/sites/janakirammsv/2026/04/04/googles-gemma-4-runs-frontier-ai-on-a-single-gpu/

Kili Technology

• AI Benchmarks 2026: The Top Evaluations and Why They're Not Enough
  • https://kili-technology.com/blog/ai-benchmarks-guide-the-top-evaluations-in-2026-and-why-theyre-not-enough
• Data Story: How MiniMax M3 Reversed Its Own Engineering Decision
  • https://kili-technology.com/blog/data-story-minimax-m3
• Introduction EU AI Act: What Every AI Team Needs to Know Before August 2026
  • https://kili-technology.com/blog/introduction-eu-ai-act-what-every-ai-team-needs-to-know-before-august-2026

‍

Frequently Asked Questions

What is Gemma 4?

Gemma 4 is Google DeepMind's open-weight model family, released on 2 April 2026 with the technical report following on 2 July. It spans five sizes: dense models at 2.3B effective, 4.5B, 12B and 31B parameters, plus the 26B-A4B Mixture-of-Experts variant with 3.8B activated parameters. All ship under Apache 2.0 with a 256k context window, quantization-aware-trained checkpoints, and support for 140+ languages.

Is Gemma 4 open source?

Gemma 4 is open-weight rather than open-source in the strict sense. The parameters are downloadable and the licence is Apache 2.0, which is more permissive than the community licences some competitors use. But the training code and the training data are not published, and the technical report does not give a token count or a data mixture breakdown. You can run, modify and redistribute the model. You cannot reproduce it or audit what went into it.

What training data was Gemma 4 trained on?

The report names the categories — web documents, code, images and audio — and gives a January 2025 cutoff. It publishes no token count for any of the five sizes, which breaks with Gemma 2 (13T/8T/2T by size) and Gemma 3 (14T/12T/4T/2T). The mixture-ablation and quality-reweighting discussion present in earlier reports is also absent. The v2 revision posted on 24 July did not add these figures.

Are Gemma 4's benchmark scores comparable to Gemma 3's?

Not directly. Every Gemma 4 column in the main results table is a thinking-mode number, meaning the model emits a reasoning trace before answering, while the Gemma 3 27B comparison column is non-thinking. That affects both the scores and the cost per task. Vision results add a second complication: the main table uses 1120 vision tokens, and the appendix shows the 12B falling from 88.4 to 58.7 on InfographicVQA at 280 tokens.

Can you fine-tune Gemma 4, and what hardware does it need?

Yes. Unsloth's documentation reports LoRA training for the E2B at 8–10GB of VRAM, E4B at 17GB, and QLoRA on the full 31B at 22GB, with the 26B-A4B needing more than 40GB. Those figures are self-reported and shift with sequence length, batch size and gradient checkpointing. Google's model card documents full SFT, LoRA, QLoRA, DPO and RLHF. One practical detail: pre-trained and instruction-tuned variants use different end-of-generation tokens, so fine-tuning either requires adding the right one.

Does fine-tuning Gemma 4 affect its safety alignment?

The best available evidence says yes, though it was measured on Gemma 3 4B rather than Gemma 4. A 2026 preprint found misalignment increasing monotonically with LoRA rank, with a mixture containing only 10% harmful data sufficient to cause substantial degradation. Text-only safety evaluation substantially understated the damage compared to multimodal evaluation of the same models. The mitigation literature points at mixture composition: a few hundred safety examples in the instruction-tuning set measurably improves post-tuning safety.

Why did Gemma 4 improve so much over Gemma 3?

Not through scale. Parameters grew fifteen percent, from 27B to 31B, and the report describes both pre-training and post-training as following a similar approach to Gemma 3. What the report does credit is data: the instruction-tuning section states that adding subsets encouraging in-context attribution, hedging and refusal improved factuality metrics without degrading other metrics. Section 4.2 reinforces the point, reporting that the smallest model roughly matches Gemma 3 27B at about a tenth the parameter count.

Does Gemma 4 comply with the EU AI Act's training data transparency rule?

Article 53(1)(d) requires GPAI providers to publish a training-content summary on the AI Office template, and enforcement powers commenced on 2 August 2026. The Article 53(2) open-source exemption covers points (a) and (b) only, so an Apache 2.0 licence does not exempt a provider from the summary requirement. The template asks for overall training data size and latest acquisition date; the technical report supplies the date and not the size. Whether a separate summary has been published elsewhere is a question for Google, and this is not legal advice.

‍