Andrej Karpathy’s AutoResearch

0:00So I actually spent like three weeks trying to perfectly tweak my Wi-Fi router settings manually. Oh wow, three weeks. Yeah, literally three weeks. I was moving antennas by millimeters, changing channels, running speed tests. Well, let me guess, failing miserably. Failing completely miserably. And I was just sitting there, you know, wishing I could just run 100 automated experiments while I was asleep. I mean, we've all wanted to just hand off the tedious stuff to a machine. Exactly. But what if you could do exactly that, but instead of fixing a router, you were building advanced artificial intelligence. That is, that's a massive leap. It really is. Today's deep dive explores this amazing project released by Andrej Carpathy, called Auto Research. The mission here is to engineer AI agents that make the fastest research progress possible on their own indefinitely. Without any human involvement. Right. Yeah, it's a completely novel approach to automated model tuning. We're essentially looking at an autonomous AI agent swarm designed specifically to train large language models.

1:01And looking at their repository, it's, it's surprisingly stripped down like there are really just a couple of main files driving the whole thing. That's the elegant part of it, honestly, the simplicity. As the human in the loop, your only real job is to iterate on the instructions in a single text file. The program.md file. Exactly. That text file contains your high level directives. The AI agent reads that and its job is to continually rewrite the actual neural network code, which lives in the train.py file. You got it. So you're basically programming the organization's overarching goals and the agent acts as the researcher doing all the heavy lifting. Precisely. The agent comes up with an idea, maybe tweaking the neural network architecture or adjusting hyperparameters. It edits the Python script, commits the change, and just kicks off a training experiment. And I noticed there's a very strict constraint on those experiments. Every single training run lasts exactly five minutes. Right. But playing devil's advocate here isn't a five minute limit way too short to evaluate truly complex architectural changes.

2:05That is the core challenge of automated tuning for sure. But the strict five minute budget is completely deliberate. It guarantees a perfectly fair comparison of radically different ideas. To make sure those quick reads are actually accurate. Right. And to evaluate the results, the agent checks a metric called Val underscore BPB. Validation bits per byte. Yes. Why use bits per bytes specifically rather than say standard accuracy metrics. Because it removes dependencies on variables like vocabulary size, it gives you this pure standardized measure of how well the model is compressing and predicting the data. I see. So even in just five minutes, if that val B B scores lower, the model definitively improved. Exactly. And the agent saves that code change as a git commit. Yeah. The score goes up. It just discards it rolls back the script and tries a new hypothesis. It optimizes perfectly for the exact hardware it's running on. Completing about 12 structured experiments every single hour while you sleep. It's essentially

3:06building a custom autonomous feedback loop for the code itself. Figuring out how to automate those tedious bottlenecks is such a massive advantage. Huge advantage. And speaking of automating progress, this deep dive is actually sponsored by Embersilk. Need help with AI training, automation, software development, or uncovering where agents could make the most impact for your business or personal life. They are definitely the ones to look into. Check out Embersilk.com for your AI needs. But getting back to the big picture of auto research, it really changes the paradigm of how we work. We are moving past the era of humans acting as slow meat computers. Meat computers. I love that. Yeah. Trying to manually run research between eating and sleeping. By structuring the program file to guide the train file, the AI accumulates these small wins, getting incrementally smarter five minutes at a time. And every dot on the project's progress chart is another successful accumulation of knowledge. That's an incredibly optimistic way to look at it. It represents a massive leap forward for humanity. We are looking at the promise of exponential

4:10autonomous progress. We are literally building the tools that will rapidly generate solutions to some of our most complex problems. It's an unconstrained engine of discovery. Absolutely. So what does this all mean for you listening right now? I want to leave you with a final provocative thought. Think about the broader applications. Exactly. If an AI can autonomously evolve its own underlying code in five minute loops, imagine applying this exact same autonomous research method to other fields you're curious about. Like a swarm of agents iterating on clean energy design or discovering new medicines five minutes at a time. The future is looking incredibly bright. If you enjoyed this deep dive, please subscribe to the show. Hey, leave us a five star review if you can. It really does help get the word out. Thanks for tuning in.