AI agents with $3,000 budget flunk open-ended AI research assignment

Will AI soon be able to build AI? Maybe not quite as soon as some pundits have expected, based on a recent preprint from researchers at Princeton, Stanford, UC Berkeley, Johns Hopkins, the University of Toronto, Georgetown and the UK AI Security Institute.

One important driver of the current AI wave is the expectation that AI models may soon be able to build their successors. Executives at companies ranging from Anthropic and OpenAI to Meta have espoused the idea that recursive self-improvement (RSI) was nigh. And have cited evidence that AI models were already accelerating AI development inside frontier labs, from writing production code to helping post-train new models.

The setup for the preprint was straightforward. Agents powered by Anthropic’s Claude Opus 4.8, with a separate robustness run on OpenAI’s GPT-5.6 Sol Ultra, received six days, a $3,000 API budget. They also got GPU credits and a Linux virtual machine and open web access.

Each agent took the central research question from an unpublished NeurIPS 2026 submission, which is one of the most influential events for AI research. After the agents had completed the work, the original authors of those submissions graded the output as conference reviewers. The researchers call this a “shadow evaluation,” since the agent works the same question as the authors without access to their findings.

‘Dense and heavily hedged’

Both agent papers were rejected, scoring 2 out of 6 (reject), and 1 out of 6 (strong reject). Reviewers citing weak experimental design, unsupported conclusions, shallow engagement with prior work and impenetrable prose. “The experiments and methodological choices were bizarre, and hard to understand. The results seem clearly a result of post hoc choices,” wrote David Africa, one of the authors and reviewers of the work. “The experiments and methodological choices were bizarre, and hard to understand,” he said.

Both Africa and the other reviewer Viet Nguyen criticized the agents’ writing. It is “impossible to quickly distill what is noise and what is important,” wrote Nguyen while Africa found the AI prose “dense and heavily hedged, often to the point of obscuring what was actually done and found.”

Agents handled the parts of research that can be specified in advance, including reading the literature, standing up GPU environments, running hundreds of experiments and producing LaTeX without a human in the loop. Tasks involving judgment were a different story. The agents struggled to decide which hypothesis deserved a week of compute, to recognize when a line of attack had died, and to assess whether the finished product cleared a publication bar.

The agents struggled to course correct

The team instructed the agents to review their own drafts through a subagent and external tools including the Stanford Agentic Reviewer, the CMU Paper Reviewer and refine.ink.  “Across dozens of rounds of revision, the agent’s self-review never once returned an acceptance (see Figure 3). These reviews surfaced many of the issues that the human reviewers later raised,” the preprint notes. While the review tools surfaced some of the issues human reviewers later did, the agents largely failed to address it and “continued to pursue unpromising research directions,” as the researchers note.

Across ten rounds of revision on one paper, the agent’s self-review never once returned an acceptance, and it mostly returned “weak reject” on work the human experts rejected outright. The agents also mishandled disagreement among the review tools, overweighting the most lenient one, which recommended acceptance on early drafts, and citing it in their final reports.

The abstract of the preprint noted the AI agents exhibited “ineffective backtracking from dead ends.” While the agents had a budget of $3,000, neither used even half of that amount. The runs ended with $1,130 and $1,235, respectively. One agent declared the project complete seven hours before its deadline shortly after its own reviewer returned another reject. Both agents could check their remaining tokens, compute and clock at any moment. The GPT-5.6 Sol run failed the same test in reverse, burning the full $3,000 in just over two days and leaving roughly 100 hours of wall-clock time it could no longer use.

Would a better harness help?

One objection is that a better scaffold would change the outcome in the research. Two-thirds of the coauthors surveyed before the experiment said a failed run might be explained by scaffold limitations. The team ran two checks. Rerunning one experiment on Codex with GPT-5.6 Sol Ultra reproduced nearly every failure mode. They also handed the pilot logs to Claude Fable 5, one of the most powerful models currently available, and asked it to fix the scaffold. The model exhibited the same problems it was meant to solve, weighting a single sample heavily and swapping heuristics without addressing the underlying failures.

The coauthors were torn on whether the failures were a result of “a lack of creativity, poor judgment, or epistemic lock-in.”

The authors acknowledge the study’s limits, including the small sample size. The reviewers had spent months on the same questions and knew the papers were machine-generated. The agents were told they were being evaluated against a NeurIPS rubric. The setup also required a paper and offered no option to abstain. The authors further disclose that members of the core team are publicly identified with the position that imminent recursive self-improvement is unlikely.

The team could not test Fable 5 in full, as it is is deliberately limited on frontier AI R&D tasks. External evaluators measuring AI research capability are working from a narrowed slate, a safety decision with a real cost to public measurement.

The AI industry has been warm on recursive self-improvement

The concept of recursive self-improvement predates the current genAI wave by decades, but the ChatGPT’s November 2022 launch and steady gains in coding and other benchmarks have added ammunition to the argument, leading prominent AI executives to hint that RSI is a near-term reality, potentially impacting the timing of when to take frontier AI labs public. “The faster the potential RSI takeoff looks like it could be, the more it could be advantageous to delay an IPO,” because the “technology and the world may change in surprising ways, and there might be good reasons to be a private company during that time,” OpenAI CEO Sam Altman said, according to a report from The Information.

Last month, Anthropic, in a post titled “When AI builds itself,” cited evidence from within and outside the company to argue that recursive self-improvement appeared to be on the horizon. It suggested that frontier AI companies consider “the option to slow or temporarily pause frontier AI development” to grapple with the ramifications. The preprint takes direct aim at that post’s evidence base, noting that Anthropic cites Claude’s rising success rate on LLM-judged open-ended coding sessions as a self-improvement signal. Roughly at the same time as announcing the post, the company had also filed paperwork for an IPO.

The idea of recursive-self improvement traces to British mathematician I.J. Good, who in 1965 described an “intelligence explosion” producing a machine that could far surpass human capabilities. Nine economists revisited the question in “The Economics of Recursive Self-Improvement,” published through the Elasticity Institute on July 13 with METR support.