What Is Autoresearch? Inside Andrej Karpathy's Self-Running ML Experiment Loop

The Basic Idea

Autoresearch is an open source Python tool that lets an AI coding agent run machine learning experiments on its own, usually on a single GPU, without anyone watching each step. You describe a research direction in a markdown file, point a coding agent at your repository, and walk away. By the next morning, you have a git history full of validated improvements and a full log of everything the agent tried along the way.

The loop is simple to describe: the agent proposes an idea, edits the code, runs a short experiment, checks whether a chosen metric improved, and keeps the change only if it did. Then it does it again. And again. It is a very literal version of the scientific method, just handed off to a machine that never gets tired or bored.

The Three File Architecture

Part of why autoresearch caught on is how minimal it is under the hood. The whole system revolves around a small set of files with clearly separated jobs, generally covering data preparation and evaluation, the training script itself, and a log or ledger that tracks every attempt, its outcome, and whether it beat the current best result.

Each run competes inside the same fixed time window, often around five minutes, so results stay comparable no matter what the agent changes. That constraint keeps the “edit surface” narrow enough for a human to inspect afterward, even though the agent technically has the freedom to rewrite arbitrary parts of the code.

How It’s Different From AutoML or Hyperparameter Tuning

It is worth being clear about what autoresearch is not. Tools like Optuna or Ray Tune search within a hyperparameter space that a human defines up front. AutoML and neural architecture search frameworks are similarly bounded, precise, but limited to a fixed search space.

Autoresearch throws that boundary out. The agent is not selecting values from a predefined menu. It can rewrite training code, swap out approaches, and try things nobody explicitly programmed it to try, as long as the result still runs and produces a score. The search space becomes whatever the underlying language model can think of, which puts it in a different category from traditional optimization tools.

Google DeepMind’s AlphaEvolve takes a related evolutionary approach for algorithm discovery, but it is closed source and out of reach for most teams. General purpose coding agents such as SWE-Agent, OpenHands, and Aider can write arbitrary code too, but they were not built specifically for this experiment and evaluate loop.

The Results That Got People’s Attention

Karpathy reportedly used the tool to squeeze meaningful speed gains out of GPT-2 style training runs. Shopify CEO Tobi Lütke has talked about training a smaller model overnight that ended up outperforming a larger one he had trained previously by hand.

Stories like these are part of what fueled the excitement. Training runs that used to take a researcher months of manual tuning were compressed into an overnight, unattended run.

Beyond Model Training

One of the more interesting threads to come out of the autoresearch conversation is that the technique is not only useful for training neural networks. Once you frame it as “give an agent a script and a metric, let it try things, keep what works,” the same loop can apply to other measurable problems, like shrinking a slow build pipeline or improving the runtime of an existing system. Engineers at companies like Shopify have written about experimenting with this broader application, using the same propose, test, and keep or discard cycle on infrastructure problems rather than model weights.

Where It Hits Its Limits

Autoresearch is not magic, and it is not really a replacement for a research team. It works best on problems that can be reduced to a fast, cheap, and reliable evaluation signal. If your metric is expensive to compute or noisy, the five minute style budget and rapid iteration loop stop being useful, since the agent cannot get enough signal quickly enough to know what actually helped.

It is also a tool for narrowing search, not a source of genuinely novel scientific insight. The agent tries variations and keeps what scores better, but a human still needs to interpret why something worked and decide what larger direction to pursue next.

The Takeaway

Autoresearch is a compact, disciplined way to automate the repetitive parts of ML experimentation: try something, measure it, keep only what earns its place, and repeat. It does not replace researchers, but it does compress a process that once took weeks of manual tuning into something that can run quietly overnight. Given how quickly it spread and how many people are already experimenting with variations on the idea, it is likely we will keep seeing this pattern applied to more corners of software and research over the coming months.