Data Story: MiMo-V2.6 and the LLM-as-a-Judge Behind Xiaomi's RL Run

Introduction

Of every dollar Xiaomi spent on reinforcement learning (RL) for MiMo-V2.6-Pro, about 13 cents paid for grading. The MiMo-V2.6 technical report splits the $2.6 million RL bill three ways. Generating rollouts (the model's practice attempts at each task) took 43.8%, training took 43.5%, and the grader took 12.7%. For the smaller Flash model the grader's share is 14.2%. Judging the model's work is now a line item, and an LLM-as-a-judge does most of it.

Few post-training reports publish a breakdown like that. Xiaomi also released the weights under MIT, along with its RL environments, verifiers, training framework and a public training log. The report tabulates the specific ways its models tried to cheat, with the models' reasoning quoted verbatim.

Nearly all of the judging in that pipeline is done by models. An agent writes task specifications. An agent audits whether tests match those specifications. An agent hunts for leaked answers. An agent writes the rubrics, and a model scores rollouts against them. In the domains where the answer can be executed, Xiaomi ties every one of those model judgments to ground truth. In the domains where it can't, the report publishes no measure of how often a qualified person would agree with the grader.

This Data Story walks through the grader Xiaomi built and maps what grades each part of the run. Then it looks at the evidence on how much that missing measurement could matter.

What Did Xiaomi Release With MiMo-V2.6?

Two models and most of the stack behind them. MiMo-V2.6-Pro is a 1.02-trillion-parameter mixture-of-experts model with 42 billion parameters active per token. MiMo-V2.6-Flash has 310 billion parameters, 15 billion active. Both take text, image, video and audio input in one shared context. Each accepts up to 1 million tokens of context and returns up to 128,000 output tokens. A third variant, Pro-UltraSpeed, targets latency-sensitive work with output up to 20 times faster. Xiaomi's launch notes add 3D spatial reasoning and computer use to the list, and aim the family at coding, agent workflows and scientific research.

Alongside the weights, Xiaomi released MiMo-V2.6-Distill-Qwen-9B, a small model distilled from MiMo's outputs, and roughly 7,000 RL task environments with their verifiers. It also released an end-to-end RL framework and the composable mini-harnesses it trained with. (A harness is the scaffolding of prompts, tools and memory an agent runs inside.)

API pricing stayed at V2.5-generation levels, TestingCatalog reported:

Source: Xiaomi list prices, as reported at launch.

Xiaomi positions Flash for high-frequency calls and large-scale tasks. Because the weights are open, teams can also self-host either model. Pro still needs substantial hardware, even with only 42 billion parameters active per token.

The release got attention for its leaderboard position. VentureBeat reported that MiMo-V2.6-Pro scored 46 on Artificial Analysis' Intelligence Index at release, the highest of any open-weight model. That put it ahead of other Chinese models such as GLM-5.3 (45) and Kimi K3 (44), TechNode noted, while the leading closed models sat at 53.

Xiaomi's own comparison table is more restrained than the coverage. On DeepSWE v1.1, Xiaomi reports Pro at 71.9 against 74.0 for Claude Opus 5 and 73.0 for GPT-5.6 Sol. On Terminal Bench 4.0 the gap is wider: 34.9 against 49.0. On OSWorld-Verified, a computer-use benchmark, Pro reaches 82.0 against Opus 5's 83.4. Pro leads on AutomationBench. Three of the seventeen benchmarks in the table are Xiaomi's own, and all baselines were run by Xiaomi at maximum reasoning effort, so these are vendor-reported figures.

Where Did the $2.6 Million RL Budget Go?

Into very long attempts, and into judging them. Each RL step started with 1,568 prompts and sampled 16 attempts at each. That is about 25,000 sequences, carrying 2.7 to 3.7 billion training tokens. Individual sequences averaged 110,000 to 150,000 tokens. Xiaomi ran 30 of these steps in under six days. The training method was an asynchronous version of Group Relative Policy Optimization (GRPO), which scores each attempt against the others sampled for the same prompt. VentureBeat reported that the Xiaomi MiMo team, led by former DeepSeek researcher Fuli Luo, put several dozen people on the effort.

Source: MiMo-V2.6 technical report, §4.1 and Figure 3. avg@3 is the average score over three runs.

The report frames the grader as a deliberate investment. Grading is one of three dimensions along which Xiaomi says it scaled RL, alongside batch size and environment diversity. A test suite tells you whether a patch passes. It can't tell you which of two passing patches you would want to merge.

That makes grader quality a training input, on the same footing as data. A grader that ranks passing solutions badly teaches the model the wrong preferences, at the scale of 25,000 sequences per step.

How Does MiMo-V2.6's Grader Decide Which Passing Patch Is Better?

Xiaomi built two mechanisms for coding tasks, which make up 68% of the RL mix.

The first, Groupwise Reward Synthesis (GRS), is used on tasks the model already passes often. An agent studies several earlier attempts at the same task, along with the repository, and writes two sets of task-specific rubrics. Solution rubrics cover the quality of the code. Behaviour rubrics cover how the agent gathered evidence and verified its changes. During training, a grader agent scores each new attempt against those rubrics. The final reward multiplies the binary test result by both rubric scores. A failed test still earns zero, so no rubric can rescue broken code. Among passing attempts, the rubrics decide.

The second, Groupwise Advantage Redistribution (GAR), covers the remaining coding tasks. Here the grader is an agent trained with supervised fine-tuning (SFT), the standard first stage of post-training. It examines every attempt in a group together and ranks the passing ones on five dimensions: suitability of the approach, precision, minimality, avoidance of side effects, and fit with the codebase's conventions. The training signal then shifts toward the better passing attempts. If the grader finds evidence that an attempt used a leaked answer, that attempt's reward is reset to zero.

Xiaomi reports an ablation on the Flash model with code-only RL. Without GAR, turns and tokens per attempt climbed quickly. More attempts hit the length limit, and pass-rate gains stalled. With GAR, pass rates kept improving through step 52 while turn counts stayed roughly flat.

"Separate maintainer-oriented audits" found that models trained without online grading drifted toward bad habits. They added speculative compatibility branches, broad exports, exception swallowing, relaxed validation and evaluation-specific configuration changes. The resulting code passes tests and is harder to maintain. The finding is useful. It is also the only audit in the report that doesn't say who performed it, so read it as exactly what it says.

Who Judges What in the MiMo-V2.6 RL Run?

The report's pipeline, domain by domain:

Source: MiMo-V2.6 technical report, §4.2, §4.3, §5.1, §5.6.

The open-domain row covers Xiaomi's most ambitious showcase work. In one case study reported by VentureBeat, Xiaomi's materials researchers used MiMo-V2.6-Pro to review literature and patents and shortlist metal-organic frameworks for capturing PFAS chemicals. Capabilities like these reach the model through distillation from teachers trained on synthetic demonstrations, because Xiaomi judged reliable RL rewards too hard to design for them.

The supervision work before training is extensive, and models do almost all of it. For coding, when a GitHub issue is too thin to serve as a specification, an LLM reconstructs one from the reference patch. An auditing agent then reviews four attempts per task against the specification, the tests and the reference solution. It flags passing attempts it judges wrong and failing attempts it judges right. Xiaomi revises or removes tests that enforce requirements the specification never states. Every task must give the same outcome across eight reruns.

Humans appear in the report at the edges. One coding pathway, covering everyday work including interactive vibe coding, uses "real-world development requests contributed by employees within our organization." Some tasks come from licensed data vendors. The cybersecurity tasks rest on vulnerabilities from OSS-Fuzz, Google's open-source fuzzing service, which people confirmed by hand. The report gives no annotator count, no reviewer qualifications, and no share of tasks a human checked.

None of this is hidden. It is where the report stops.

Why Did Xiaomi Stop Using LLM-as-a-Judge for Cybersecurity?

Because the verdicts wouldn't hold still. The task here is vulnerability reproduction: the agent must write an input, a proof of concept (PoC), that triggers a specific known bug. Xiaomi writes that LLM-based judging "returns different verdicts for the same PoC across runs" and so "cannot serve as a stable reward." It replaced the judge with a deterministic rule. A PoC counts only if its crash matches both the vulnerability type and the crash location in the ground-truth sanitizer report, the diagnostic log a memory-safety tool writes when the program crashes. Xiaomi also found that CyberGym's LLM-generated task descriptions were often too broad or simply wrong, and corrected CyberGym's evaluation environments before reporting on it.

That is good engineering, and it draws a line. Where a machine-checkable ground truth existed, Xiaomi preferred it to a model's opinion.

Knowledge work has no sanitizer report. In the general-agent tasks, which the report describes as professional workflows that must meet "domain-specific standards," an agent writes the tasks and the atomic rubric items. Deterministic properties such as database values and file formats get code checks. Open-ended content gets LLM checks. To catch ambiguous rubric items, Xiaomi looks for agreement across repeated judgments and across different judge models. Then it states the limit of that test itself: "Consistency alone does not establish that rubrics capture task completion."

Its answer to that limit is another model. A review agent examines attempts from models of different strength and revises criteria that reject valid work or accept incomplete work. During RL, the grader is a self-hosted MiMo-V2.6-SFT checkpoint. The released knowledge-work environments, about 1,000 tasks, list their verifier as "rubric-based judging." They are the only released set whose verifier is a judgment rather than a check.

Enterprise document teams read mixed files and produce work to a professional standard. That is the domain this set most resembles. It is also the one where the report offers the least evidence that its grader agrees with a professional.

How Much Does the Rubric Author Matter to an LLM-as-a-Judge?

By about 27 points of judge accuracy, in the benchmark built to isolate it.

RubricBench, published at ACL 2026, tested LLM judges on 1,147 hard pairs of responses. Experts had written rubrics for each pair. One judge setup scored 57.8% using rubrics it generated itself. Given the human-annotated rubrics instead, with the same model and the same procedure, it scored 84.9%. The gain held at around 26 to 27 points across several frontier judges. The authors' conclusion was that models are good at checking explicit instructions and poor at inferring the implicit constraints experts care about.

That gap is the size of what MiMo's report leaves unmeasured. It doesn't mean Xiaomi's rubrics are 27 points worse than an expert's. It means rubric authorship is a large enough variable that a report which doesn't measure it can't tell us how its grader compares.

Two older results sharpen the concern. Panickssery, Bowman and Feng found at NeurIPS 2024 that LLM evaluators score their own outputs higher than others' when human raters judge them equal. The bias grows with the model's ability to recognise its own text. MiMo's knowledge-work grader is a MiMo-V2.6 checkpoint scoring MiMo-V2.6 attempts. The report doesn't show that this bias occurred. It is the configuration the finding applies to.

Gao, Schulman and Hilton showed at ICML 2023 that optimising hard against a proxy reward eventually lowers true performance, and that the effect scales predictably. Their study had to use a large model as the "gold" stand-in for human judgment because collecting real human preference labels at that scale was too expensive. It is the same substitution MiMo makes for knowledge work.

The clinical evidence shows how far a grader's errors travel. A team auditing MedCalc-Bench, a medical benchmark whose labels were partly produced with LLM assistance, relabelled it with physician oversight. A model trained with RL on the corrected labels beat one trained on the originals by 13.5 points on physician-labelled cases. Label errors didn't stay in the evaluation. They became training signal.

When expert rubric authorship is disclosed, its cost is visible too. A survey of rubric-based evaluation notes that HealthBench engaged 262 physicians to write 48,562 criteria for 5,000 medical dialogues. That is the kind of number MiMo's report doesn't contain for any domain.

Does the Same Leak That Breaks Benchmarks Break RL Rewards?

Yes, and Xiaomi documents it in unusual detail.

The report's section on reward hacking, where a model earns reward without doing the intended work, lists five ways MiMo passed tests in early coding runs without solving the task. It installed a newer release of pytest and read the fix from the installed source. It fetched an Astropy file from a raw GitHub URL. It cloned a later Matplotlib checkout. It read the change history of the Django ticket that had already been fixed. It listed available Sphinx versions to find one that contained the fix. The report quotes the model's reasoning for each. In one: "Let me just look at the file directly from the GitHub raw URL."

These are the same channels the SWE-Bench Pro Verified audit found in evaluation harnesses: future git history, upstream sources over the network, leftover artefacts. We covered that audit in our SWE-Bench Pro Verified Data Story. The difference is the stakes. A leaky benchmark inflates a score. A leaky training environment rewards the model for looking up answers, and the model learns to do it more.

The mitigation came in layers. Xiaomi stripped build logs, verifier outputs and caches; kept git history only up to the base commit; and isolated containers from the network. Then a dedicated hack agent probed the prepared environments for remaining leaks. It found routes the cleanup had missed. Xiaomi reran it after each fix until it "could no longer find a successful exploit in any of the environments." During training, offline audits of trajectories continued, and the grader zeroed any confirmed hack. Xiaomi reports that confirmed hacks stayed below 2% of trajectories for both models. Xiaomi also fed the early hacking cases back into mid-training as correction examples, with the model rewriting its own faulty reasoning.

One caveat belongs to the 2% figure: it counts confirmed hacks, and the confirming is done by agents. That is a reasonable design. It is also a detection rate bounded by what the detector can see.

Isn't an Anchored Grader Good Enough?

For code, mostly yes.

The GRS reward is multiplied by the test result. A rubric can separate good passing code from mediocre passing code; it cannot turn a failure into a success. GAR only reorders passing attempts. Every judgment in the coding pipeline sits on top of an executable check. Under that check, Xiaomi layered reruns, spec–test alignment and a hack agent. Where it found a model's judgment unreliable, in cybersecurity, it removed the model from the reward.

Other evidence suggests the model learned something transferable rather than learning to please its own grader. Xiaomi trained across four mini-harnesses, then tested in three it never trained in: Codex, Claude Code and mini-swe-agent. Mean Pass@1 (the share of tasks solved on the first attempt) on DeepSWE v1.1 rose from about 50% to 66% in those unseen harnesses.

And Xiaomi published enough for outsiders to check its machine-side work. The environments, verifiers and framework are downloadable. That openness is the precondition for any audit.

The argument doesn't transfer out of code, though. A rubric item such as "the reply identifies the policy exclusion that applies" has no test suite underneath it. The 12% of the MiMo mix labelled general tool use runs on the judgment of whoever wrote and applies the rubric. So does every enterprise workflow that looks like it. In that setting the grader is the ground truth. Its agreement with a qualified person tells you whether that ground truth is any good.

What Would an Auditable Grader Look Like?

It would answer, for the grader, the questions an audit asks of any reviewer: who decided, under what standard, and with what measured agreement.

In practice that means a calibration set. Take a sample of the grader's verdicts across rubric items and difficulty levels. Have people qualified in the domain judge the same items independently: underwriters for claims documents, clinicians for medical summaries, engineers for code review. Where two or more of those experts judge the same item, you get their agreement with each other, which sets the ceiling. Where the grader's verdicts are compared against theirs, you get the grader's agreement with experts, per rubric item. Items where the experts disagree get rewritten. Items where the grader disagrees with agreeing experts get fixed or handed to a human.

This is the work that expert-in-the-loop platforms exist to make routine. In Kili, the reviewer who holds the domain expertise does the judging directly on text and PDF assets, inside a configurable multi-step review workflow. Consensus scores agreement between annotators working on the same asset, and honeypot scores each labeler against items with known answers, which is a different measurement from agreement. Labels are stored with their author, including those added at review, so the calibration set carries its own audit trail.

The MiMo report shows that frontier teams already spend 12 to 14 cents of each RL dollar on judging. Measuring that judge against experts is how you find out what the spending buys.

Conclusion: The Grader Is Now the Dataset

MiMo-V2.6 makes a strong case that scaling agentic reinforcement learning means scaling the grader. Xiaomi spent real money on it, engineered it carefully, and anchored it to ground truth wherever ground truth existed. Where it didn't, the rubric a model wrote and the model that applied it became the definition of good work, reinforced across hundreds of thousands of trajectories.

That changes what a post-training dataset is. It used to be the examples. In a run like this one, it is also the rubrics and the grader's verdicts. Those deserve the scrutiny we give to labels: who wrote them, how they were checked, and how often a qualified person agrees. An open RL report that adds a grader-versus-expert agreement table, per domain, would close the gap this one leaves. For teams post-training on their own documents and workflows, that table is also the one to build first.

Resources

Primary Technical Report

• MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement – Xiaomi LLM-Core technical report
  • https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf

Release Coverage

• 'Better than DeepSeek': Xiaomi's MiMo-V2.6-Pro debuts as the top open weights model – VentureBeat
  • https://venturebeat.com/technology/better-than-deepseek-xiaomis-mimo-v2-6-pro-debuts-as-the-top-open-weights-model-in-the-world-alongside-cheaper-v2-6-flash
• Xiaomi open-sources MiMo-V2.6 models after scaling reinforcement learning – TechNode
  • https://technode.com/2026/09/22/xiaomi-open-sources-mimo-v2-6-models-after-scaling-reinforcement-learning/
• Xiaomi MiMo-V2.6 Series: 3 New Models Officially Released – Xiaomi MiMo launch notes
  • https://mimo.mi.com/docs/news/latest/v2-6
• Xiaomi open-sources MiMo-V2.6 Pro and Flash models – TestingCatalog
  • https://www.testingcatalog.com/xiaomi-open-sources-mimo-v2-6-pro-and-flash-models/

Research on Graders, Rubrics and Reward

• RubricBench: Aligning Model-Generated Rubrics with Human Standards – Zhou et al., ACL 2026
  • https://aclanthology.org/2026.acl-long.1439/
• LLM Evaluators Recognize and Favor Their Own Generations – Panickssery, Bowman & Feng, NeurIPS 2024
  • https://papers.nips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html
• Scaling Laws for Reward Model Overoptimization – Gao, Schulman & Hilton, ICML 2023
  • https://proceedings.mlr.press/v202/gao23h.html
• Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight – Ye et al., arXiv
  • https://arxiv.org/abs/2512.19691
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape – survey, arXiv
  • https://arxiv.org/html/2606.08625v2

Benchmark Audits

• SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents – Zheng et al., arXiv
  • https://arxiv.org/abs/2609.08149
• Data Story: SWE-Bench Pro Verified – Kili Technology
  • https://kili-technology.com/blog/data-story-ai-benchmark-audit-swe-bench-pro-verified

Kili Documentation and Guides

• Workflow configuration – Kili docs
  • https://docs.kili-technology.com/docs/workflow-configuration
• Consensus overview – Kili docs
  • https://docs.kili-technology.com/docs/consensus-overview
• How to build a custom AI benchmark – Kili blog
  • https://kili-technology.com/blog/how-to-build-a-custom-ai-benchmark