AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks

0:00Welcome to Daily Papercast. Today's paper comes from the Hugging Face Daily Paper List of October 1st, 2026. It's garnered quite a bit of interest with 64 uproats. That's right. The paper we're diving into is titled AREX2, advancing self-improving agents through long horizon reflective tasks. The first two authors are Hongjin Kian and Chao Fan Li, with correspondence to Chan-Kian and Jung-Lu. They're all affiliated with the Beijing Academy of Artificial Intelligence. Okay, let's jump right into the introduction section. What problem is AREX2 aiming to solve? AREX2 addresses the challenge of self-improving large language model agents. The main goal is to enable these models to iteratively refine solutions at test time. Iteratively refine solutions. Can you break that down for us? The idea is that instead of generating a single solution and stopping there,

1:03the agent revisits and improves its solution over multiple rounds based on feedback. I see, and the paper mentions two critical capabilities for this. Reflection and long horizon execution. What do these mean? Reflection is essentially producing a better solution in response to feedback. Long horizon execution is about maintaining effectiveness over many rounds. But how does that scale across different domains? Great question. The authors hypothesize that both reflection and long horizon execution are domain agnostic, meaning these skills can be learned in controlled environments and then applied broadly. Interesting. So what domains did they choose to train these capabilities? They used machine learning and algorithmic programming tasks. These areas provide verifiable feedback and reward sustained iteration, making them ideal for training a self-improving agent. And the results? How does AREX2 perform on benchmarks?

2:05AREX2. Built on QN3827B shows strong results. For instance, it scores 81.8 on MLA bench light and surpasses other models on several benchmarks like Browse Comp and DeepSearch QA. And does it continue to improve with more rounds? Yes, it does. The paper reports that AREX2 continues to improve its scores as the number of rounds increases, which is a hallmark of effective long horizon execution. That's a solid background. How is this different from what we've seen before in iterative learning or self-improvement agents? Most existing systems implement the iterative loop externally, relying on external scaffolding to decide when to retry and what to keep. AREX2 aims to move this loop inside the model, making the agent itself responsible for self-improvement. I see. So it's about embedding the iterative process directly into the agent's training and functionality.

3:07Exactly. The agent learns to use feedback effectively and retains useful information across multiple iterations, which leads to continuous improvement without needing an external system to manage it. That's quite comprehensive. Is there anything else about the contributions of this paper we should highlight? The key takeaway is that AREX2 provides a proof of concept that long horizon reflective data can significantly enhance the self-improving capabilities of large language models. It opens the door to more efficient and effective model training. And that wraps up the introduction section. All right, Ashley, let's dive into the method section of this paper. How did the authors approach creating a self-improving agent like AREX2? The authors took a structured approach. They framed self-improvement as test-time scaling, which means improving the agent's performance through multiple rounds of task execution. So how exactly do they define self-improvement in this context?

4:11Self-improvement here is defined by the agent's ability to turn more rounds on a given task into progressively better solutions. The idea is to scale the agent's performance by effectively using the allocated rounds for optimization. Got it. And this involves reflection and long horizon execution, right? Yes, exactly. Reflection involves improving the current solution based on feedback, while long horizon execution ensures that the process remains effective across many rounds. These concepts were central to how they designed and trained AREX2. But how did they train AREX2 to have these capabilities? Let's break that down. To train AREX2, the authors constructed specialized environments from GitHub repositories and online judges, which provided machine learning and algorithmic programming tasks respectively. Can you explain what these environments entail? Sure. For machine learning tasks, the environment includes the code, data,

5:15a specific metric to improve, and a sandbox for safe execution. For algorithmic programming tasks, it includes a problem statement, a compiler, some sample test cases, and a scoring function that isn't just binary but graded. Interesting. And these were used to teach AREX2 through actual interactive tasks. Exactly. The agent engages with these environments over many rounds, refining its solutions based on the feedback it receives. This forms long horizon reflective trajectories, where the path to the solution is as important as the final score. But how do these trajectories get selected and used for training? Great question. The trajectories are selected based on two primary criteria. The final score achieved and adherence to the task's requirements. Both successful and unsuccessful rounds are included to teach the model how to recover from setbacks. So they capture the entire process, not just the successful attempts.

6:16Yes, this way the model learns to deal with failures and refine its approach continuously. There's no filtering to remove the less successful attempts, because those are crucial for teaching recovery and resilience. I'm curious about the feedback mechanism. How does the agent learn what to improve in each round? Feedback comes after each round in the form of scores and additional logs or errors from the environment. The agent uses this information to decide what changes to make next, fostering a reflective process. That sounds comprehensive. What about the operational knowledge the model needs? Operational knowledge is critical here. The agent doesn't start with all the answers. It acquires necessary knowledge on the fly by reading documentation, searching for relevant papers, and experimenting within the environment. So it's constantly learning about the environment as it goes. Exactly. This kind of learning is essential for adapting to new tasks and for the self-improving agents to remain versatile and effective. What about the round budget you mentioned earlier?

7:21How does this factor into the training? The round budget is the total number of rounds and agent is allowed to perform on a given task. This budget is much larger than what is typically required to solve the task in one go, allowing the agent to iteratively improve its solution. So the budget essentially gives the agent room to experiment and refine its solutions over time. Precisely. The larger round budget enables the agent to establish a baseline, measure its performance, iterate on potential improvements, and thus demonstrate how to turn more rounds into cumulative improvements. It's fascinating how they structured this system. Were there any tools or abilities provided to help the agent? Yes, the agent was given skills, which are essentially compact documents containing useful operational knowledge. These are either general skills prepared in advance from common practices or task-specific skills designed to assist the agent in understanding new tasks. And these skills remain available even during the testing phase? Yes, they're available

8:23to help the agent make informed decisions and refine its solutions better, even during the test time. It sounds like a very well-rounded workflow. Do they describe how well this all worked in practice? They provided some compelling evidence. A REX2 not only performed exceptionally well on benchmarks like MLE, Benchlight, and Frontier CS, but also continued to improve its performance with additional rounds. And this improvement with more rounds is a key finding, right? Yes, it's a major indication that the model is effectively utilizing its long horizon reflective capabilities, something that sets AREX2 apart from other models that plateau after a certain point. That's quite an impressive feat. Rapping up this section, is there anything else notable about their methodological approach? The most notable aspect is how the authors managed to embed this feedback and iteration process directly into the agent, making it independent of external systems for self-improvement. This is a significant step forward in the field of self-improving agents.

9:29And that concludes our discussion on the Method section. All right, Ashley, let's get into the Experiment and Result section. How did AREX2 perform when tested in various benchmarks? The authors evaluated AREX2 on six benchmarks that span four different capabilities, algorithmic programming, machine learning engineering, deep research, and general agentic reasoning. Can you walk us through these benchmarks and what each of them measures? Of course, the benchmarks include Frontier CS for algorithmic programming, MLE Benchlight for machine learning engineering, and deep research benchmarks like Browse Comp, HLE, Gaia, and deep search QA. Each of these evaluates different dimensions of an agent's capabilities, from solving open-ended problems to deep research tasks requiring multi-step reasoning. What scores did AREX2 achieve on these benchmarks?

10:30Sure. On MLE Benchlight, AREX2 scores an any metal rate of 81.8%, significantly higher than other models. In Frontier CS, it achieves 70.7%, making it competitive against closed-weight models. On Browse Comp, AREX2 scores 84.0, on HLE 52.6, on Gaia 92.2, and on deep search QA 93.8. Those are impressive numbers, especially for a model with 27 billion parameters. How does AREX2 compare with other larger models in these benchmarks? Indeed, despite having a smaller parameter count, AREX2 compares favorably with much larger models. For example, it surpasses models with up to 2.8 trillion parameters in certain tasks. This shows that AREX2's design for long horizon reflection is quite effective.

11:30How do the authors demonstrate continued improvement with more rounds during the experiments? The authors provide two key analyses, round-scaling with feedback and round-scaling without feedback. For instance, in the Frontier CS agent track, AREX2 keeps improving its score over a span of 5 hours, showing 4.8 points of gain in the final hour, which indicates a long-effective horizon. What about round-scaling without feedback? How does that work? On Browse Comp, the agent doesn't receive feedback indicating whether its answers are correct during each run. Despite this, AREX2 improves from 64.8% accuracy at 47 turns to 84.0% at 143 turns. This demonstrates that the agent can reassess and refine its solutions based on its own judgment, not just based on external correctness signals. That's remarkable. So AREX2 is capable of self-improvement with minimal external guidance.

12:34How do the stagewise ablation results contribute to our understanding of AREX2's performance? The stagewise ablation was conducted on MLE Benchlight to measure the contribution of operational knowledge, skills, training, and a larger round budget. The base model without any skills scored 28.8%, while adding general skills raised it to 41.2%. Tasks-specific skills with more rounds increased it to 68.2%, but with training, AREX2 achieved 81.8%, showing how each element adds value. It's noteworthy that training alone added significant improvement. Any other detailed results from the experiments worth mentioning? The experiments confirm that AREX2 turns a larger budget into better solutions with sustained improvement over time. These improvements are observed in various tasks and benchmarks, showcasing the model's ability to effectively use long horizon reflection across different domains. And do the authors provide evidence that long horizon reflection transfers across

13:41domains? Yes, the cross-domain results are quite telling. AREX2 showed substantial gains in deep research benchmarks over previous AREX models, even though no new deep research data was added for training. It supports the hypothesis that long horizon reflection is indeed a meta-skill that transfers well between different domains. Rapping up the experiment section, what's the overall takeaway from these results? Overall, AREX2 demonstrates that embedding the iterative improvement loop within the agent and equipping it with capabilities for long horizon reflection can lead to significant performance gains. This approach not only outperforms larger models in several tasks, but also proves effective across different domains, affirming the power of test time scaling. And that wraps up the experiment and results section. All right, Ashley, let's explore the related work section. How does AREX2 compare to other efforts in the field? Sure, the paper situates AREX2

14:47within a broader landscape of research on iterative learning and self-improving agents. Several lines of work are particularly relevant here. That makes sense. Can you walk us through these lines of work and how they relate to AREX2? One prominent area is the use of scaffolded learning systems where the iterative loop is managed externally rather than being embedded within the model itself. Examples include recent studies by Giang Edal in 2025 and Novikov Edal in the same year, who designed frameworks that focus on external verifiers and schedulers to manage the learning process. Okay, and how does AREX2 differ from these scaffolded learning systems? AREX2 distinguishes itself by embedding the iterative improvement loop directly into the model. This integrated approach enables the agent to autonomously decide when to retry and what aspects to refine, making it more efficient and potentially more effective. Got it. What about

15:50works that focus on feedback mechanisms? How do those compare? There's a lot of interesting work in feedback mechanisms. Some notable examples are the rest approach by Golcherry Edal in 2023 and the Star Framework by Zellik Menadal in 2022. These methods leverage reinforced feedback to continually improve the agent's performance. And how does AREX2's feedback mechanism fit into this landscape? AREX2 uses a reflective feedback mechanism where each round's output is evaluated and the feedback is used to enhance the next iteration. Unlike rest and star, which may require specific types of reinforced feedback, AREX2 can handle a broader range of feedback types, including logs and error messages. Interesting. What about Long Horizon Execution Models? How does AREX2 align with or differ from them? Long Horizon Execution is another critical area.

16:51Models like those proposed by Loo Edal in 2024 and Yang Edal in 2026 incentivize agents to maintain effectiveness over extended tasks by leveraging historical performance. But AREX2 builds on this by not only ensuring sustained performance but also refining solutions iteratively to continuously improve outcomes over long periods. I see. So AREX2 is truly integrating the best aspects of these earlier efforts but embedding them within the model. Exactly. The key innovation here is moving these capabilities inside the model, allowing the agent itself to manage iteration and improvement autonomously, which is a significant departure from externally managed systems. Are there any other related areas of work mentioned? Yes, the paper also touches on the importance of task-agnostic skills and the development of generalizable capabilities. Studies by Chan Edal in 2025 and Mang Edal in the same year emphasize creating agents that can apply learned skills across

17:56diverse tasks. AREX2 aligns well with this by showing that Long Horizon Reflection Skills learned in one domain can transfer effectively to others. So it's not just about improving within a single domain but being able to generalize improvements across multiple domains, right? Precisely. This ability to generalize is crucial for the practical deployment of self-improving agents across various fields and AREX2's results demonstrate promising advancements in this direction. That's fascinating. How do the authors see the future development of AREX2 in light of this related work? The authors suggest expanding the training domains and lengthening the horizons to further enhance the agents' capabilities. They also propose that allowing the models' own trajectories to become its next training data could create a more self-sustaining loop of improvement. Interesting. So the model would essentially be training itself in a more autonomous way. Yes, that's the idea. By continuously

18:58learning from its own experience, the model could potentially achieve more sophisticated levels of self-improvement. It's incredible how AREX2 builds upon and advances the body of existing research. Anything else from the related work section that we should highlight? That about covers the main connections and differentiators. AREX2 stands out by integrating reflection and Long Horizon Execution within the model, leveraging broader and more flexible feedback and demonstrating effective skill transfer across domains. And that wraps up the related work section. All right, Ashley, let's wrap up by summarizing the key contributions and takeaways from this paper. What stands out to you? AREX2 makes significant strides in the realm of self-improving agents. The core innovation is embedding both reflection and Long Horizon Execution directly into the model. This approach contrasts with previous methods that rely on external scaffolding for iterative

19:59improvement. Embedding these capabilities within the model itself is a big leap. And reflecting on the results, AREX2 performs impressively across multiple benchmarks. Yes, definitely. It not only scores high on benchmarks like MLE Bench Lite and Frontier CS, but also shows strong performance in deep research tasks like Browse Comp and Deep Search QA. The ability to continue improving with more rounds further highlights its Long Horizon reflective capabilities. Another important aspect is its demonstration of effective skill transfer across domains. How does this impact the broader field of AI? This shows that skills learned in one domain can be effectively applied to others, making the model highly versatile. It opens up possibilities for deploying such agents in varied real-world scenarios where continuous improvement is key. That's a powerful capability. Anything else that the authors suggest for future work?

21:00The authors propose expanding the training domains and allowing the model's own iterative trajectories to be used as further training data. This could lead to even more autonomous and sophisticated levels of self-improvement. It's exciting to think about the future potential of these advancements. All right, thanks Ashley for breaking down AR-EX2 for us today. Thank you, Evan. It's been great discussing this fascinating paper. And thank you to our listeners for tuning in to Daily Papercast. We hope you found today's episode enlightening. Remember to join us again tomorrow as we explore another cutting-edge paper from the Hugging Face Daily Paper List. Until next time, stay curious and keep learning. Goodbye, everyone.