Discovery of the last neutrino mixing angle — one of Science magazine’s top 10 breakthroughs of the year 2012 — was announced in March 2012, just a few months after the Daya Bay Neutrino Experiment’s first detectors went online in southeast China. Collaborating scientists from China, the United States, the Czech Republic and Russia were thrilled that their experiment was producing more data than expected, and that a positive result was available so quickly.
But that result might not have been available so quickly without the NERSC Global Filesystem (NGF) infrastructure, which allowed staff at the U.S. Department of Energy’s (DOE’s) National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory (Berkeley Lab) to rapidly scale up disk and node resources to accommodate the surprisingly large influx of data.

Coping with Success
The Daya Bay experiment had been expected to produce large data sets — after all, the nuclear reactors of the China Guangdong Nuclear Power Group at Daya Bay and nearby Ling Ao produce millions of quadrillions of electron antineutrinos every second. However, neutrinos and antineutrinos are very difficult to detect — they pass through most materials, even our bodies, without leaving a trace. Nevertheless, the Daya Bay physicists expected the eight massive detectors buried in the nearby mountains to produce enough high-quality data to measure, for the first time, Θ13 (pronounced “theta-one-three”) — the last unobserved parameter of neutrino oscillation, and a key to understanding the difference between matter and antimatter.

Craig Tull, head of Berkeley Lab’s Science Software Systems Group and U.S. manager of the Daya Bay experiment’s overall software and computing effort, had been working with NERSC and Energy Sciences Network (ESnet) staff to ensure that data coming from Daya Bay could be processed, analyzed and shared in real time, providing scientists immediate insight into the quality of the physics data and the performance of the detectors.

The plan was this: Shortly after the experimental data was collected in China, it would travel via the National Science Foundation’s GLORIAD network and DOE’s ESnet to NERSC in Oakland, CA. At NERSC, the data would be processed automatically on the PDSF and Carver systems, stored on the PDSF file system (a dedicated resource for high energy and nuclear physics), and shared with collaborators around the world via the Daya Bay Offline Data Monitor, a Web-based “science gateway” hosted by NERSC. NERSC is the only U.S. site where all of the raw, simulated, and derived Daya Bay data were to be analyzed and archived.

The first six detectors to go online performed beyond expectations. Between December 24, 2011, and February 17, 2012, they recorded tens of thousands of interactions of electron antineutrinos, about 50 to 60 percent more than anticipated, according to Tull — about 350 to 400 gigabytes (GB) of data per day. “There were three things that caught us by surprise,” he says. “One was that we were taking more data than projected. Two was that the initial processing of the data did not reduce the volume as expected. And three was that the ratio of I/O [input/output] operations to CPU operations was much higher for Daya Bay analysis than, say, a simulation on a NERSC computer. The combination of those three things created larger than anticipated I/O needs.”

In short, Tull was worried that the PDSF file system did not have the space and bandwidth to handle all the incoming data. So, in January 2012, he met with Jason Hick, leader of NERSC’s Storage Systems Group, to find a solution.
“NGF was already running at a petabyte scale,” Hick says, “and Craig’s requirements were largely satisfied by simply scaling up the current system. It’s exciting to see NERSC’s vision for NGF realized when we can directly help scientists like Craig achieve their aims.”

Quick Access, Fast Analysis
In less than a week, the Storage Systems team, including Greg Butler and Rei Lee, added more disk space to NGF and updated its configuration, then worked with Tull to reroute the daily Daya Bay data runs to NGF. As a result, processed data was made available to Daya Bay collaborators within two hours after it was collected at the detectors. Only two months later, in March 2012, the Daya Bay Collaboration announced that Θ13 equals 8.8°, outpacing other neutrino experiments in France, South Korea, Japan and the U.S.

By December 2012, the project had accumulated half a petabyte of data, and it was time to reprocess all of the data using the latest refinements in software and calibrations, in preparation for a January collaboration meeting in Beijing. Tull consulted with Iwona Sakrejda of NERSC’s Computational Systems group and received a temporary loan of 1,600 cores for processing the data.

“We were predicting that it would take three to four weeks to do the whole analysis,” Tull says, “but with NERSC’s help, we were able to do it in eight days instead of three to four weeks. It was the combination of NGF and the extra cores which shortened the time to finish production.”

The Daya Bay story is typical of what NERSC expects to see more of in the future: data-intensive science that requires high-bandwidth, high-volume, and low-response-time data systems.