MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

0:00Welcome to Daily Papercast. Today's episode features a paper from the Hugging Face Daily Paper List of October 9th, 2026, with 43 upvotes. The title of the paper is MimoV2.6, Scaling reinforcement learning towards self-improvement. The first two authors are Zongming Chiao, Zi Yue Hua, and the corresponding author is Fuli Luol from Xiaomi. Let's dive into the introduction section of this paper. So, Ashley, what's the background here? Sure, Evan, the paper focuses on recursive self-improvement, or RSI, which envisions models that expand their capabilities through sustained exploration and feedback. And how do they propose to achieve this? They suggest that achieving this vision requires coupling models with interactive environments, allowing them to generate multi-step trajectories, and receive the feedback signals critical for self-improvement. But Scaling reinforcement learning on complex tasks sounds challenging. What are the specific hurdles?

1:06That's right. They identify two major challenges. First, reinforcement learning, or RL for foundation models, requires suitable architectures and a sufficiently rich exploration space for agents. And the second challenge? Second is the need for sophisticated solutions in infrastructure, environments, and grading mechanisms to effectively scale RL. Interesting. So, what does the MIMO V2.6 series aim to accomplish? The MIMO V2.6 series, including MIMO V2.6 Pro, and MIMO V2.6 Flash, aims to bridge these gaps. MIMO V2.6 Pro is a 1.02 trillion parameter model with 42 billion active parameters, and MIMO V2.6 Flash is a 310 billion parameter model with 15 billion active parameters. That's massive! What kind of architecture and training do they use to support this?

2:07To support these huge models, they use a hybrid sparse mixture of experts transformer backbone. It interleaves local sliding window attention with global attention, and includes specialized encoders for visual and audio data. Interesting combination. What's the strategy behind their pre-training and mid-training? Pre-training first builds strong foundational language capabilities using text-only data, then integrates visual and audio data. In the mid-training phase, they focus on expanding the exploration space for agentic tasks. What about scaling the RL compute? How do they handle that? They scale RL compute systematically in three dimensions. Firstly, by increasing batch size and training throughput through fully asynchronous training, processing thousands of rollouts and billions of tokens per step. And the second dimension? Secondly, they increase the diversity and complexity of RL environments across various domains like coding, general workflows, visual tasks, and cybersecurity.

3:12And the third dimension of scaling? The third dimension is enhancing the grading mechanisms. They do this through groupwise agentic grading, providing more precise reward signals beyond binary test outcomes. It seems they've put a lot of thought into maintaining training stability. What measures have they taken? To keep training stable at scale, they freeze the mixture of experts router and establish multi-layer defenses against reward hacking. Sounds comprehensive. How do they handle agentic interactions and task representations? They implement a unified trajectory representation and a high concurrency multi-frame work rollout to manage agent interactions. Additionally, control and data planes are decoupled to handle the massive trajectories effectively. Do they provide resources for others to build on this work? Yes, they open source the training dynamics, RL environments, and RL framework to aid reproduction and further research in scaled RL and model self improvement.

4:14Fantastic. That covers the introduction section of the paper. Alright, moving on. Let's delve into the methodology behind MIMO V2.6. Ashley, give us an overview of their approach to scaling the reinforcement learning. Of course. The methodology section of the paper primarily covers three main areas, scaling RL training computation, environments and agent harnesses, and their unique grading system. How do they approach scaling RL training computation? They scale RL training computation by leveraging thousands of GPUs in a single training run. From MIMO V2.6 Pro, they spent approximately $2.6 million, and from MIMO V2.6 Flash, it was $900,000. That's a significant investment. What about the RL training batches? Yes, they use large training batches. Specifically, for each step, the system rolls out 1,568 prompts with a group size of 16, resulting in 25,000 sequences total, processing between 2.7 to 3.7 billion tokens.

5:26What are the benefits they've observed with this approach? They've seen steady improvements in the benchmarks, such as the Deep SWE benchmark, which reflects continual performance gains as cumulative RL cost increases. Interesting. What about the design of the RL environments and harnesses? They've invested heavily in developing a diverse set of RL environments. This includes domains such as coding, general professional workflows, visual tasks, and cybersecurity. What stands out in their approach to coding tasks? Well, coding tasks are particularly suited to reinforcement learning because they can be executed and evaluated through automated tests. They use various synthesis pathways to create tasks. Such as? They utilize GitHub-based synthesis, real-world development workflows, specification-driven tasks, and source code-driven synthesis. This allows them to cover a broad range of programming languages and development contexts.

6:26But ensuring reliable supervision must be challenging. How do they handle that? They align task specifications with unit tests to accurately accept implementations that satisfy the specified requirements. They also include rollout-based auditing to ensure the consistency of these alignments. What about their method to maintain the robustness of their supervision? They perform checks for reward stability through repeated test executions to detect flakiness and verify reward stability, ensuring more robust supervision outcomes. How do they manage the environments for general agent tasks? For general agent tasks, they design environments that typically consist of a workspace and a set of software tools accessible through interfaces like APIs, CLIs, and GUIs. They balance realism with execution needs by using real-world files and software mocks. Could you provide a bit more detail on how tasks are synthesized and verified?

7:28Sure. They survey common professional tasks and generalize them into seed tasks, agents then explore these environments and construct tasks, which are evaluated using atomic, binary rubric items. And how do they handle visual agent tasks? For visual tasks, they curate a suite where agents work with a range of artifacts like websites, interactive applications, games, and 3D scenes. They focus on both open-ended design and high-fidelity visual replication. That sounds comprehensive. What about the cybersecurity tasks? For cybersecurity, agents are trained on real-world vulnerability reproduction. This involves constructing inputs that trigger specific vulnerabilities in projects, a critical skill underlying many cybersecurity applications. How do they verify the correctness of these tasks? They use oracles derived from vulnerability reports, matching the crash location and type to ensure that agents are reproducing specific vulnerabilities and not just causing random crashes.

8:31What is their approach to handling various agent harnesses? They employ a multi-haroness training approach using many harnesses with minimal agent loops. This allows for safe and controlled recombination of components needed for task completion. And what about the challenge of reward hacking? They combine mid-training alignment data, environment preparation, and monitoring throughout training. They also have a dedicated hack agent that continuously probes for leaks and exploits. It seems they've significantly robustified their training mechanisms. How do they ensure consistent scaling and deployment of these massive models? They optimize both their training and inference pathways. On the training side, they use a quantized, de-quantized mechanism to maintain consistency in mixture of experts routing and probability normalization. For inference, cached states and speculative decoding improve efficiency.

9:31And how do they handle the data generated through these large batches? Their infrastructure disaggregates the data plane from the control plane, enabling efficient handling of tens of thousands of sequences. They pack payload data once to reduce communication and processing overhead. What measures do they take from multimodal data? They carefully manage multimodal payloads by handling only the delta between requests during rollouts and performing post-processing based on task requirements to minimize redundant data movement. This complex orchestration of tasks, harnesses, and data is fascinating. It sounds like they've developed an extremely well-thought-out system. Exactly. It's a systematic and thoroughly refined approach to scaling reinforcement learning with real-world applicability in mind. And that wraps up the method section of the paper. Now let's move on to the experiments and results section of the paper.

10:33Ashley, can you walk us through the experimental setup and the key findings? Evan, the experiments conducted were quite exhaustive, covering a range of domains including coding, general workflows, visual design, and cybersecurity tasks. They aim to validate the performance of MIMO V2.6 across these diverse, agentic tasks. What were the main benchmarks they used for evaluation? They used several benchmarks, both public and internal. For instance, they used deep-stwe version 1.1, program bench, and their proprietary MIMO code bench for coding tasks. Cybersecurity, they used benchmarks like CyberGym, Exploite Bench, and MIMO CyberBench. And what about general workflows and visual tasks? For general workflows, they used AutomationBench, Toolathlon Verified, and GDPVAL-AA among others. For visual tasks, they primarily evaluated using MIMO visual coding, which involves tasks such as web development and image-to-code translations.

11:39That's a wide array of tasks. How did MIMO V2.6 perform across these benchmarks? The MIMO V2.6 series showed substantial improvements across nearly all benchmarks compared to previous generation models. For example, on the Deepest WE version 1.1 benchmark, MIMO V2.6 Pro improved from 58.4 to 72.6 by the end of RL training. That's quite an improvement. What about the Cybersecurity tasks? In Cybersecurity, MIMO V2.6 models also excelled. On CyberGym, for instance, MIMO V2.6 Pro scored 94.0, indicating strong capabilities and vulnerability reproduction. They even outperformed existing models across a suite of benchmarks. What were some of the specific improvements noted in coding tasks? On coding tasks, MIMO V2.6 models achieved notable enhancements.

12:43For instance, on the SWE Bench Pro, the performance jumped significantly, showing nearly a 15-point increase over baseline models, which is a testament to effective RL and agentic capabilities. And how did the model fare in visual tasks, especially considering the complexity of such tasks? For visual tasks, the performance was equally impressive. On MIMO visual coding, which involves tasks such as generating web pages from text descriptions, MIMO V2.6 Pro consistently showed higher scores, indicating superior execution of high fidelity visual replication tasks. It seems like MIMO V2.6 had a comprehensive approach to handle different types of environments and scenarios. Were there any significant challenges or failures noted during these experiments? Yes, they did observe certain challenges. One notable issue was related to router stability in the mixture of experts model, which sometimes led to imbalance in expert load.

13:47However, they addressed this by freezing the router during reinforcement learning. That's interesting. How did freezing the router impact the training stability and performance? Freezing the router proved to be effective. It helped maintain a balanced load across the experts, thereby ensuring stable training dynamics. This was evident as the fraction of cold experts remained low and stable throughout the training process. Given the scale of the experiments, how did they manage the massive amounts of data generated during training? To manage the large-scale data, they used a payload porter system to handle tens of thousands of sequences efficiently. By disaggregating the data plane from the control plane, they were able to buffer and manage data without overburdening a single node, thereby maintaining high throughput. It's clear they developed a sophisticated framework for these experiments. What were the final conclusions drawn regarding the performance and capabilities of MIMO V2.6?

14:49The final results indicated that MIMO V2.6 models are extremely capable across diverse tasks. They not only demonstrated improvements in benchmarks, but also set a solid foundation for further research in recursive self-improvement through large-scale reinforcement learning. And that concludes the experiments and results section of the paper. Next, we'll delve into the related work section of the MIMO V2.6 paper. Ashley, how do the authors position their work within the broader landscape of current research? The authors provide an extensive overview of how their approach compares to and builds upon existing research in the field. They discuss several key areas, starting with the foundational models and the mixture of experts, or MOE, paradigm. How does their use of MOE compare to other implementation models? Great question. They cite the original work on MOE by Shazir Adal from 2017, which introduced the idea of employing multiple experts within a single model to enhance efficiency and performance.

15:57MIMO V2.6 extends this concept by incorporating advancements in sparse MOE transformers, specifically tailored for large-scale reinforcement learning. And what about reinforcement learning itself? How do they situate their work in the context of recent developments in RL? In the realm of reinforcement learning, they reference several pivotal studies, including the works by Silver Adal on AlphaGo and Alpha0, which demonstrated the potential for RL to master complex tasks. However, MIMO V2.6 takes these principles further by integrating a diverse array of environments and incorporating mixed task RL, which is less common. That's intriguing. How does their approach to task diversity and multi-domain training stand out? They emphasize the uniqueness of their approach by discussing the breadth and complexity of the environments they've synthesized. Unlike many previous works that focus on a single domain, MIMO V2.6 spans multiple domains, including coding, general workflows, visual, and cybersecurity tasks.

17:02This breadth of application is a significant step forward. What sources do they cite to highlight the novelty of their environment synthesis and task diversity? They refer to contemporary benchmarks and data sets from various sources. For example, they leverage the well-known OpenAI Jim environments for baseline comparisons, while augmenting these with their own environments like MIMO Code Bench and MIMO CyberBench. They also mention recent advancements in automated environment synthesis by weighing it all and shepherd and saliments. They've certainly built a comprehensive framework. What about the grading mechanisms? How does their groupwise, agentech grading compare to traditional approaches? In grading mechanisms, they draw comparisons with standard reward structures in RL, specifically citing works like the proximal policy optimization, PPO from Schulman at all. Their groupwise agentech grading, which includes groupwise reward synthesis and groupwise advantage redistribution, is positioned as a more nuanced method, improving the granularity and precision of rewards.

18:06And how does their novel approach to reward synthesis and advantage redistribution fit in with existing literature? They highlight how their approach is an evolution of existing methods, combining ideas from both supervised and reinforcement learning. They mention dong ed al's work on agentech grading and the practical improvements it brings by addressing common pitfalls in traditional RL reward structures, such as reward sparsity and binary outcomes. It's clear that MIMO V2.6 builds on a solid foundation of past research while introducing significant innovations. Anything else worth noting from the related work? Yes. They also delve into the infrastructural and methodological advancements that support their large-scale training. They reference asynchronous training frameworks like Asyncflow and the role of quantization techniques, drawing on Han ed al's work to optimize the training efficiency of their large models. Quite a thorough examination of related work. That wraps up the related work section of the paper.

19:10Now, let's wrap up our discussion by summarizing the key contributions and takeaways of the MIMO V2.6 paper. Certainly, the MIMO V2.6 series represents a significant advance in scaling reinforcement learning towards self-improvement. With models like MIMO V2.6 Pro and MIMO V2.6 Flash, Xiaomi has pushed the boundaries of what's possible with large foundation models, integrating capabilities across text, visual and audio data. Yes, their use of a hybrid sparse mixture of experts transformer architecture and meticulous pre-training and mid-training phases provide a robust foundation. They've continuously scaled RL compute along three dimensions, increasing batch size and throughput, diversifying and complicating RL environments, and enhancing reward signals with innovative grading mechanisms.

20:10In terms of infrastructure, their efforts to ensure stable training at scale, like freezing the MoE router and implementing system level defenses against reward hacking, are particularly notable. Exactly. Plus, the infrastructure investments, like the unified trajectory representation, high concurrency rollouts, and decoupled control and data planes, demonstrate a strong commitment to practical, scalable RL. Their open sourcing of tools and environments adds a layer of community involvement, ensuring that other researchers can build on their work. The experimental results speak for themselves, with MIMO V2.6 models outperforming prior generations across diverse benchmarks from coding to cybersecurity to visual tasks. This highlights the comprehensive and versatile nature of their approach. It's clear that MIMO V2.6 lays a robust groundwork for future advancements in recursive self-improvement and efficient reinforcement learning.

21:12We hope you found today's deep dive into MIMO V2.6 insightful and informative. As always, we aim to bring you the latest and most impactful research each day. Thank you for tuning in to Daily Papercast. Be sure to join us again tomorrow as we explore another exciting paper from the world of AI, NLP, CV, and more. Stay curious, stay informed, and we'll see you next time. Goodbye, everyone!