The Cathedral of Learning, listed in the National Register of Historic Places, is the centerpiece of the University of Pittsburgh’s main campus.
As a center of learning, the University of Pittsburgh empowers researchers who are on the forefront of scientific innovation. Enabling their discoveries requires the assistance of advanced computing power. The University of Pittsburgh’s Center for Research Computing (CRC) has the responsibility of supporting these researchers, and their cutting-edge endeavors, with a high-performance computing (HPC) system optimized to enable those projects.
As the second early adopter of the newest Intel Xeon Scalable processors, CRC researchers have experienced up to three times the performance of the previous system. This increase in performance makes it possible to accommodate more research projects and to allow a broad range of existing engineering, biomedical, scientific, and other researchers to expand their work to the next level of discovery.
Enabling Innovative Research
With diverse fields of study conducted at University of Pittsburgh, the need for CRC’s computing hardware to remain adaptable and flexible is paramount. To help faculty and students make the most of these computing resources, CRC offers training for managing system resources, increasing code performance, and optimizing computational workflow. Additional CRC-provided training opportunities help users learn best practices for input-output strategies, select ideal algorithms, and adopt parallelization techniques to speed computational research.
Kenneth Jordan, Associate Director of CRC states that “Our team is committed to enhance our HPC system to meet the computational needs of over 330 research groups from 37 Departments. With enhanced computing power provided by the new cluster, some complex computational problems that required a full day in the past can now yield results in as little as an hour. Faster results inform us quickly if we need to re-strategize our approach.” Continuing, he notes, “The new Intel Xeon Scalable processors and the added cores they provide give us tools to address increasingly complex computing challenges even more efficiently.”
Modern applications
With the support of CRC’s supercomputer, studies at the University of Pittsburgh have led to real-world, breakthrough applications. John Keith, a Richard King Mellon Faculty Fellow in Energy, Dept. of Chemical and Petroleum Engineering, alongside CRC-affiliated faculty members Peng Liu of the Department of Chemistry, and Giannis Mpourmpakis and Christopher Wilmer of Chemical and Petroleum Engineering, have each won National Science Foundation 2017 CAREER awards in recognition of their innovative work, which is facilitated by the CRC. The newest iteration of Intel technologies added to the CRC’s HPC system will advance these projects at a rate not possible before.
Keith’s research seeks to reduce carbon emissions through the identification of chemical catalysts that convert greenhouse gases like carbon dioxide into carbon-neutral liquid fuels like methanol. Elaborated Keith, “If we can identify and create a chemical catalyst to facilitate the molecular conversion, we can use the gases that were previously discarded by factories into the atmosphere, and make a useable, clean fuel.” Added Keith, “Nature has optimized catalysts that perform all kinds of functions in the world today. We seek to create a human-made catalyst to carry out the task we need it to.”
The straightforward goal, though, is a highly challenging proposition. Keith and his team must perform thousands of complex simulations using quantum chemical methods to identify and test various catalysts, and to anticipate the amount of energy each catalyst releases during the carbon gas conversion process. With Intel’s Intel Xeon Scalable Processors in place, his team can complete in one day the advanced calculations which may have required weeks before. “Our team is thrilled to be at the University of Pittsburgh, where we have access to the advanced HPC technology that many research institutions lack,” noted Keith. “With CRC’s support, we have an incredibly powerful HPC system, alongside much human brainpower, to help us do simulations not possible before.”
HPC systems also have substantial impacts in the field of epidemiology. Epidemiologists at the University analyze large data sets to identify factors which have an impact on human health. Ultimately, these studies shape national policy decisions for preventative healthcare. Ashley Naimi, Assistant Professor, Graduate School of Public Health at the University of Pittsburgh, is currently conducting research to determine if a small, daily dose of aspirin may help women to more easily achieve pregnancy and to carry a baby to term. The study is a randomized trial with 1,200 volunteers, some of whom are taking the aspirin while others are not. The detailed information collected from each participant over the course of a year-and-a-half study produces large datasets.
Reflects Naimi, “Through our research, we want to help improve lives of pregnant women. CRC’s HPC system and programming expertise are critical to us because our data-intensive research relies on machine learning algorithms to interpret the data we collect. Before we had access to the latest processors from Intel, each simulation might take a week. With the new processors in place, we can obtain meaningful insights up to four times faster than before. The sooner we have a clear understanding of data, the faster we can make recommendations that shape health policies, potentially helping millions of people.”
Technology behind CRC’s system
Because the CRC funding comes from a combination of University funds, faculty researchers, grants from organizations like the National Science Foundation, and other sources, the HPC infrastructure is architected with future scalability in mind. Advanced technology must work seamlessly to enable the HPC performance CRC requires.
Ralph Roskies, Associate Vice Provost for Research Computing, reinforces the nature of the challenge. “Researchers need a wide variety of architectures. Some need to use large numbers of tightly coupled compute nodes in parallel, some need a multitude of independent processors, some need large shared memory, some need GPUs, and almost all need storage. Architecting a system to meet these requirements with a finite budget is challenging.” Roskies also notes an additional barrier his team must overcome. “Another big challenge surrounds awareness on campus about the expertise CRC can provide for advanced computing needs. Our greatest reward comes from the successful application of these resources to solve research problems.”
The CRC computing clusters are housed at an enterprise-class Network Operations Center (NOC), part of the Computing Services and Systems Development (CSSD) group at the University and linked to the main campus by 100 Gb/s fiber optic networking. CSSD maintains the system’s power, cooling system, networking, operating systems, and backups, and provides expertise in Linux administration.
As budget permits, the CRC and CSSD teams complete upgrades to offer their researchers and students improved performance. The new computing cluster added 104 compute nodes provided by Supermicro, each fueled by a pair of the latest 12-core 2.6 GHz Intel Xeon Scalable processors
While the new processors offer a huge leap in performance, the CRC team also needed to evaluate the rest of the HPC system for bottlenecks limiting system performance. They refer to one challenge they faced as a “memory wall.” Without memory optimization for rapid data transfer, the gains in processor speed would prove moot. This problem is addressed by the Xeon Scalable processors’ much greater bandwidth to memory. Each compute node has 192 GB of high-speed memory and Intel Solid State Drives. The global storage includes several elements including a 130 TB Isilon home space, and a 450 TB Lustre parallel filesystem, and 1PB ZFS file system for archival purposes
Disparate nodes are connected utilizing low latency Intel Omni-Path fabric to maximize the efficiency of distributed parallel workloads.
What do they see as the future?
Jordan notes, “With University of Pittsburgh’s support of computational-based research, and the most advanced technology from Intel, we look to the future with great optimism. The CRC team is working hard to help our faculty, students, and other researchers apply the latest computing resources to projects that have the potential to make meaningful impacts on the world.”
Rob Johnson spent much of his professional career consulting for a Fortune 25 technology company. Currently, Rob owns Fine Tuning, LLC, a strategic marketing and communications consulting company based in Portland, Oregon.
