Argonne’s ChemGraph opens quantum chemistry to non-experts, tracks AI model performance 

ChemGraph, an open-source AI framework from scientists at Argonne National Laboratory, lets scientists without computational chemistry expertise run quantum chemistry simulations through natural language prompts. Its creators also made a model-by-model reliability leaderboard that ranks models based on various criteria and tasks. 

In a paper published early this year, the team evaluated ChemGraph across 13 benchmark experiments involving six integrated tools, organized into three categories: molecule names, SMILES strings and chemical reactions, totaling 360 independent evaluations. They tested the open-weight Qwen2.5-14B, and proprietary GPT-4o-mini, GPT-4o, and Claude-3.5-haiku models.  

“Sometimes some smaller models do not call the tool and try to answer the question through their memory, through their training, and that can make them hallucinate. For smaller molecules, they can do the correct job, even without calling the tool. But when the molecule is larger or more complicated, they make a mistake, but they confidently give the user the wrong structure.” said Murat Keçeli, a computational scientist at Argonne.  

For example, on simpler tasks (name-to-SMILES conversion through file generation), Claude-3.5-haiku and GPT-4o-mini both exceeded 80% accuracy, while Qwen2.5-14B was more variable, dropping as low as 60% on the name2gibbs task, which asked the model to calculate Gibbs free energy given a molecule name and temperature. On the two most complex tasks, calculating reaction enthalpy and reaction Gibbs free energy, Qwen2.5-14B scored below 20% accuracy, GPT-4o-mini achieved accuracies of 40% and 49%, while Claude-3.5-haiku managed 67% and 69% respectively, and GPT-4o was the strongest performer at above 83% on both.  

“We want a reliable tool, so we want high accuracy; we don’t want that 1 percent mistake, because it can be very costly down the line if you start with a wrong answer or a wrong plan for the original work,” Keçeli said.  

More, smaller agents 

ChemGraph began as a single-agent tool, but the team found that it struggled with complicated workflows. Instead, they pivoted to using multiple smaller models.  

“If we can manage to get our workflows to run flawlessly with smaller models, that is a kind of a win-win situation because we spend less tokens, we spend less money, and also we don’t need to rely on the frontier model that runs on the cloud… this is very helpful, especially for workflows where privacy is the concern, security is the concern.” Keçeli said.  

For the reaction enthalpy task, moving to the multi-agent design raised GPT-4o-mini’s accuracy from 40% to 87% across three runs, while Claude-3.5-haiku improved from 67% to 87%, surpassing GPT-4o’s single-agent performance. For the reaction Gibbs free energy task, GPT-4o-mini improved from 49% to 87%, and Claude-3.5-haiku from 69% to 93%.  

The reliability leaderboard 

To help scientists using ChemGraph gauge model accuracy, the team created a reliability leaderboard users can access.  

“We evaluate all the models that we work with… Whenever we have access to new models, we try to rerun the scripts to run the evaluation benchmark, and within this leaderboard, one can clearly see which tasks, which models are failing, which models are succeeding,” said Keçeli.  

“In some models we see 100% reliability. They always answer the questions correctly because they call the right tools, so the users can have more confidence in those models. But for some smaller models, we see that they are still not there, and they can still make mistakes. So, we suggest our users check this leaderboard and pick the right models for the tasks that they are interested in,” he added.  

“When a simulation fails unexpectedly, in ways that aren’t common, the LLM can hallucinate the output and make up something. So there’s still human intervention needed to look at the logs and see what’s failing,” said Thang Duc Pham, an Argonne postdoctoral fellow.   

Keçeli and Pham’s team is already building toward a more autonomous version of ChemGraph, with visions of a fully autonomous lab. Pham emphasized that people will still be necessary to check the models’ work for errors.  

“The LLM helps with bridging the gap from a question to running a simulation, but how to run it and how to interpret from the results still needs the human oversight,” he said.