Inside AutoLabs: PNNL’s self-correcting AI still needs an expert in the loop

Researchers from Pacific Northwest National Laboratory (PNNL) developed a generative agentic AI that can translate experimental goals into instructions for a laboratory robot. Their paper was published in Scientific Reports.  

Gihan Panapitiya, a data scientist at PNNL and the lead author of the paper, said the researchers had plans to extend AutoLabs to literature review, persistent memory and other autonomous lab equipment besides the automated lab workstation Big Kahuna from Unchained Labs. 

The self-checks catch errors before instructions reach the hardware. The system can spot errors before instructions reach the physical hardware. “We are also developing additional feedback mechanisms that will allow us to continuously refine and introduce new guardrails over time,” Panapitiya, said. Designing experiments for automated laboratory instruments like Big Kahuna traditionally requires equal understanding between scientists and engineers, whose expertise often does not overlap. The team built AutoLabs to close that gap using natural-language dialogue instead of manual translation of intent into code.  

AutoLabs follows a string of LLM-driven lab automation systems such as Carnegie Mellon’s Coscientist, the Chinese University of Hong Kong and Zhejiang Lab’s Chemist-X, and the University of Toronto’s ORGANA. Carnegie Mellon published Coscientist in Nature in December 2023.  

Still a case study  

The researchers tested 20 distinct agent architecture configurations, each run ten times. The agents were tested on five benchmark experiments of increasing complexity, from a simple naphthalene calibration sample to a multi-plate, timed esterification synthesis. The full physical robot was able to fully execute two of the five experiments.  

The paper notes that, in its final form, AutoLabs achieved 100% hardware-instruction-loading success for experiments one through four, with “occasional issues” for experiment five due to multi-plate complexity.  

This evaluation was confined to a single lab and a single instrument. The paper described its results as a “case study” on Big Kahuna. The authors note that the underlying architecture could theoretically extend to other liquid handling platforms.  

How AutoLabs checks its own work  

AutoLabs has a self-correction mechanism that “identifies and fixes errors in a protocol before it is sent to the robot,” Panapitiya said in an email to R&D World.  

“We pass the agent-generated protocol to the LLM along with a checklist of specific criteria to verify,” he said. “This self-check process can be repeated multiple times, iterating until the system is confident that the protocol contains no remaining errors.”  

The paper describes guided and unguided self-checks. Guided checks improve procedural correctness, while unguided checks are more effective at catching chemical amount errors, but can introduce inefficiencies such as redundant water-addition steps.  

The self-check process checks unit consistency, minimal-step chemical additions, correct handling of transfers between plates, delay and timing checks, plate array validation, solvent specification review and chemical addition tagging structure.  

“We are also developing additional feedback mechanisms that will allow us to continuously refine and introduce new guardrails over time,” Panapitiya said.  

Guardrails with gaps  

The paper provides a list of the guardrails: self-correction and validation, tool-based stoichiometric calculations grounded in a curated chemical database, prompts asking users to confirm ambiguous details, validation against hardware-specific constraints, human review before execution and logging for post-hoc traceability.  

To strengthen the safeguards, the researchers tested retrieval-augmented generation (RAG) to reduce errors by grounding the understand and refine agent in prior validated procedures, with mixed results. In one configuration, RAG improved outcomes across nearly all experiments, but in another, it made results worse. The paper attributes the inconsistency to retrieved information, sometimes introducing “noise, irrelevant information or conflicting instructions,” rather than clarity.  

When an expert systems engineer worked with AutoLabs, results improved significantly over both the fully automated system and non-expert users. But even the expert missed some errors, such as omitted stir-rate commands, cap values assigned to the wrong plate and an unintended delay step that disabled a timed experiment.  

Non-expert users who answered AutoLabs’ clarifying questions produced less accurate results than letting the system proceed on its own judgement.  

Multiple agents 

AutoLabs was initially developed as a single agent before it was expanded to a multi-agent tool to prevent the AI from losing track of its instructions, Panapitiya said.  

“We observed that as a conversation grows longer, the LLM tends to lose track of the instructions it was originally given,” he said. “To address this limitation, we moved to a multi-agent architecture… Because every sub-agent begins each task from its own instruction set, the risk of the model ‘forgetting’ its guidance is greatly reduced.” 

The final version of AutoLabs features multiple specialized agents, each responsible for a specific task and equipped with its own dedicated set of instructions, Panapitiya said. A supervisor agent delegates tasks to the sub-agents. According to the paper, AutoLabs features five sub-agents: understand and refine, which clarifies the request; chemical calculations, which performs stoichiometry calculations via tool-calling; vial arrangement; processing steps, which is in charge of heating, stirring and timing; and final steps.  

“We are actively working to expand the agent’s capabilities to include literature review, hypothesis generation, persistent memory and continuous self-improvement,” Panapitiya said.