Celebrating its 30th anniversary this November, the San Diego Supercomputer Center likes to refer to itself — with tongue loosely in cheek — as an HPC resource “for the 99 percent.” Its mission: support thousands of scientific end users, most of them running jobs that typically occupy only a small amount of the processing power contained in the Center’s world-class systems.
SDSC occupies a unique niche among supercomputing sites. Scientists come to the SDSC not so much for compute power as they do for a cyberinfrastructure designed to support high numbers of end-users and to store and move massive datasets. Ah, data. When you talk to a SDSC senior manager you quickly realize you’re dealing with an HPC site that is data-intensive, very data-intensive.
“Every center has to look at the emerging trends in science and computing and choose which ones to be part of their strategy,” said Rick Wagner, SDSC’s High Performance Computing Systems Manager. “SDSC has primarily focused on two areas: We put the data in the middle, knowing that hardware lasts only a few years, but the data lives on. And that data needs to be accessible to many systems to enable complex workflows, rather than tying the data to a single system that may run only a portion of the application. We’ve scaled our storage system, and we want users to access the data from whichever system is most appropriate for the science they’re doing.”
The result: SDSC has been an industry leader in data-intensive computing, much of which has been driven by the explosion in data generated by such pursuits as genomic research. The Center’s mission is to support the “long tail” of science, the idea that a large number of modest-sized, computationally-based research projects represent, in aggregate, a tremendous amount of research and resulting scientific impact and advance.
SDSC was established as one of the nation’s first supercomputer centers under an agreement by the National Science Foundation (NSF) in collaboration with the University of California, San Diego, and GA Technologies. And while the Center supports thousands of scientific end-users, most of them involved in smaller-scale projects, make no mistake about the SDSC’s self-effacing “computing for the 99 percent” tag. For the past 30 years, users of SDSC systems have achieved major scientific breakthroughs spanning many domains, from earth sciences and biology to astrophysics, bioinformatics, and health IT. A few noteworthy milestones:
1987: Scientists take a major step in the new arena of rational drug design, determining the relative free energies of binding for different chemical inhibitors. The result is significant, since what makes drugs effective is that, at the molecular level, binding at the site where it acts.- 1989: Scientists construct a 3-D model of the terrestrial carbon dioxide cycle using data collected since the 1950s by University of California, San Diego’s Scripps Institution of Oceanography. It is the first to confirm the importance of fossil fuel combustion in loading the atmosphere with CO2.
- 1999: Molecular dynamics simulations provide new insights into attacking integrase, which helps the HIV virus hijack the body’s cells. The simulations lead to Isentress, hailed as the most important new AIDS drug in a decade.
- 2004: Michael Norman, professor of physics at the Center for Astrophysics and Space Sciences at UC San Diego (and now Director of SDSC), with colleagues runs the largest simulation of the evolution of the universe. Using SDSC’s IBM Blue Horizon supercomputer, the team tracks the formation of galaxies and gas clouds after the Big Bang.
- 2014: Researchers investigating the genome of a 115-year-old woman discover many somatic mutations that arose during the woman’s lifetime, concluding that few of them mapped to genomic regions that code for proteins, whereas most were in regions predicted to have little impact on genetic fitness.
Just as SDSC users have been active at the headwaters of scientific discovery, computer scientists at SDSC have developed systems at the leading edge of HPC technology. The Center’s first system was the state-of-the-art supercomputer of its day: a Cray X-MP/48, Serial No. 6, with four CPUs and a peak performance of 840 MFLOPS, 64 MB of memory (roughly equivalent to a smart phone today), and a cost of $14 million. By the early 1990s, the Center was using Intel iPSC/860 and Intel Paragon systems along with newer Cray systems, and later acquired IBM systems as well.
By 2005, SDSC had adopted adopting a three-pronged HPC architectural strategy:
- an emphasis on data storage as at least as important as compute power
- the ability to support high numbers of users who typically run relatively smaller scientific jobs
- a build-your-own systems building expertise that integrates commodity components into clusters custom designed to support the Center’s unique mission
For example: the Gordon supercomputer, the result of a $20 million NSF grant that entered operations in early 2012 as one of the 50 fastest supercomputers in the world. With 300 trillion bytes of flash memory, 64 I/O nodes and 1,024 compute nodes with Intel Xeon processors E5. Gordon was designed to be many times faster than conventional HPC systems for data-intensive analyses. By 2014, 1,098 research projects using Gordon were awarded among 762 principal investigators at U.S. academic institutions.
“Gordon was configured as a data-intensive system rather than a traditional compute-intensive resource,” said Wayne Pfeiffer, SDSC Distinguished Scientist and Gordon Project Manager. “We had significant latitude to explore alternate modes of allocation, one of which was to carve out small fractions of the system and offer them as dedicated resources to projects. A typical allocation involves a single I/O node plus some fraction of the compute nodes connected to the same switch.”
Likewise, SDSC’s Comet system, launched in April 2015, is one of the world’s most powerful HPC systems for data-intensive computing. With 1,944 nodes utilizing Intel Xeon processors E5, 36 nodes accelerated by NVIDIA K80 GPUs, and four large-memory nodes each with 1.5 TB of DRAM, Comet is designed to support up to 10,000 users.
At the heart of SDSC’s HPC resources is the high-performance, scalable parallel file system called Data Oasis that features an innovative integration of the Lustre parallel file system with OpenZFS storage management and data protection software, which is supported by Intel’s High Performance Data Division. With 13 petabytes of storage capacity and speeds of up to 300+ GB/s, Data Oasis is integrated with all SDSC compute and network resources, including the new Comet system, and significantly cuts the time required to retrieve, analyze, store and share extremely large datasets.
Wagner said Intel’s support in the development of Lustre-OpenZFS integration was essential to the success of the Comet launch.
“Most people see Intel as a maker of processors, but from my view the place where Intel helped us the most during the Comet deployment was around Lustre,” he said. “We needed their support, and we got it, and Comet hit its performance numbers as a result. Intel’s High Performance Data Division is a hardware-agnostic team, they’re willing to work in support of Lustre, they’re really there to keep the Lustre community alive and active. Intel is much broader than just processors, and that team in particular was critical to the success of Comet.”
Wagner said SDSC’s technology strategy will remain in force while its scientific mission is to seek new scientific endeavors and communities to support.
“We know the data is going to continue to be critical to the Center and to our users, and getting it close to the compute is essential for driving a lot of the science work our users do,” Wagner said, “so all of our systems have to be architected around where the data will be and how it will flow.”
The other priority for SDSC: “Identifying the emerging science communities that are unexpected and are going to need the Center’s help,” said Wagner, citing the explosion of data generated by bioinformatics. “We’re always on the lookout for how science is changing and what we can do to help it accelerate. A big part of that is continually developing the expertise and experience in taking applications and making them run faster and more efficiently. That’s what we’re all about.”
Doug Black is a communications professional who has been involved in the HPC industry for close to 20 years.
