When the National Science Foundation (NSF) announced the first-ever direct observation of gravitational waves, they confirmed a prediction Albert Einstein made when he published his General Theory of Relativity 100 years ago.

Einstein never expected gravitational waves to be observed. These ripples in the fabric of spacetime are triggered by events occurring so far away and so long ago that, by the time they reached earth, he figured their signals would be too weak to measure.

But Einstein didn’t foresee the development of the Laser Interferometer Gravity-Wave Observatory (LIGO) detectors — instruments so sensitive they can measure signals that change a fraction of the width of an atom’s nucleus over a distance of more than two miles. He didn’t anticipate Moore’s Law and the enormous computational resources that would enable scientists to extract and analyze these signals.

“This discovery is the first time we’ve been able to hear the cosmos communicating to us using gravitational waves. I am quite confident that it won’t be the last time. Even more exciting, we will see, or rather hear, something completely unexpected from the universe that will revolutionize our understanding of the universe. We cannot wait to get there.”

With LIGO’s success, scientists have yet again confirmed Einstein’s genius. They’ve also opened a window on a whole new field of study — gravitational astrophysics — that will provide exciting new ways to light up the dark aspects of the universe. The LIGO team’s discovery and its ongoing work will deepen our understanding of phenomena such as black holes, neutron stars and, potentially, even the big bang. And it will lead to new insights we can’t yet imagine. 

“This discovery is the first time we’ve been able to hear the cosmos communicating to us using gravitational waves,” Dr. David Reitze, Executive Director of LIGO Laboratory told the U.S. House Committee on Science, Space and Technology.¹ “I am quite confident that it won’t be the last time. Even more exciting, we will see, or rather hear, something completely unexpected from the universe that will revolutionize our understanding of the universe. We cannot wait to get there.”

Precise measurements from a vast universe

The LIGO experiment is awe-inspiring at both ends of the size spectrum. The gravitational waves captured by the LIGO detectors resulted from two black holes orbiting each other, colliding and merging approximately 1.3 billion years ago and 1.3 billion light-years away. The black holes, which were about 29 and 36 times the mass of our sun, collided at nearly half the speed of light. Their collision converted mass to the energy equivalent of three suns, producing the gravitational wave that LIGO detected. In the fraction of a second before the merger, they radiated with an energy 50 times greater than all the other stars in the visible universe.  

The LIGO equipment that observed the waves are the most sensitive measuring devices ever made, according to David Shoemaker, MIT LIGO Laboratory Director and Leader of the Advanced LIGO upgrade. Located at LIGO observatories in Louisiana and Washington State, each detector has an L-shaped interferometer that splits laser light, beams it in a vacuum down the 2.5 mile sides of the L, bounces it off mirrors at the ends of the L, measures how long each beam takes to travel and return, recombines it, and analyzes the “interference pattern.” Distances are measured over the 2.5 miles at a precision of 1/10¹⁹ of a meter, or one ten-thousandth the diameter of a proton.

“The LIGO experiment gets to very fundamental physics,” says Stuart B. Anderson, Senior Research Scientist at Caltech and head of computing for LIGO. “When this gravitational wave passed through our solar system, it effectively changed the distance between the earth and the sun by the width of one atom. To be able to make this exquisitely precise measurement about something that’s so fundamental to our universe, in such a high-risk, high-impact area — it’s incredibly exciting.”

Collaborative, transformational science

The LIGO success is a lesson in visionary leadership, commitment and collaboration. “NSF showed foresight and determination in pursuing the goal of detecting gravitational waves and learning what these waves will be able to tell us,” Shoemaker says. “As with a lot of transformational science, you’re dealing with large-scale challenges. You have to develop new technologies and new workplace skills. You have to think in terms of longer time scales for development. NSF was far-sighted, and it made those commitments, including supporting non-teaching academic research scientists like Stuart and me.”

“NSF showed foresight and determination in pursuing the goal of detecting gravitational waves and learning what these waves will be able to tell us. As with a lot of transformational science, you’re dealing with large-scale challenges. You have to develop new technologies and new workplace skills. You have to think in terms of longer time scales for development. NSF was far-sighted, and it made those commitments.”

International collaboration has been a central element. MIT and Caltech have been the pathfinders, beginning with NSF-funded projects since the 1970s to explore interferometer research. A broader, self-governing organization, the LIGO Scientific Collaboration (LSC), formed in 1997 to exploit the scientific potential of the LIGO instruments that were then under construction. Caltech and MIT play leading roles in the LSC, with responsibility for designing the equipment, creating the experiment, and operating the hardware. The LSC today includes more than 1,000 scientists at 90 universities and research institutions in 15 countries.

“LSC is a big happy laboratory family where people work in collaboration and synergy across multiple labs and multiple universities,” says Shoemaker. “It’s a wonderfully distributed system with a great richness to the collaboration.”

Massive computing to validate the signal and advance the science

Recording a gravitational signal is one thing. LIGO teams also use sophisticated algorithms and massive computing resources to identify and validate the signal and explore its scientific significance. “Large-scale computing is absolutely key,” Anderson says. “We have very complicated workflows to execute, including one that has anywhere from 100,000 to 1 million compute tasks that need to be organized and executed. We ran something like 50 million core hours total to extract the signal and pull the science out of the data stream. We use a highly distributed computing environment with approximately 20,000 cores, and 90 percent are Intel processors.”

“We have very complicated workflows to execute, including one that has anywhere from 100,000 to 1 million compute tasks that need to be organized and executed. We ran something like 50 million core hours total to extract the signal and pull the science out of the data stream. We use a highly distributed computing environment with approximately 20,000 cores, and 90 percent are Intel processors.”

The gravitational wave reading came on September 14, shortly after a major upgrade to the next-generation interferometer known as Advanced LIGO or ALIGO. “We had ramped up to take three months of readings and, almost immediately, we saw this very loud, very bright and highly statistically significant signal,” Anderson recalls. “It was a loud, unambiguous signal, but because it came so early in the run, it seemed almost too good to be true. For an extraordinary claim, there’s an extraordinary burden of proof, so we worked very hard and went through the data extremely carefully to try to prove ourselves wrong.”

Between the September signal capture and NSF’s February 11, 2016, announcement, LSC researchers performed three major computational tasks.

The first involves sifting the data from 100,000 other channels that are recorded simultaneously with gravitational wave channel. Scrutinizing these channels helps eliminate the effects of something like a minor earthquake or even a truck driving into a parking lot contaminating the data.

In the second task, researchers conduct computationally intensive pseudo-experiments to evaluate the statistical significance of their signal capture. These experiments build confidence in the signal’s validity by allowing scientists to see how much louder it is compared to false alarms and noise fluctuations. 

With the signal validated, the third task focuses on determining what events produced the signal and how far it traveled. This work extracts the detailed parameters of the signal and matches them against models that have been developed based on relativity theory.

The end result was the NSF announcement, a peer-reviewed paper in Physical Review Letters, celebrations by scientists and citizens around the world, and even appearances on popular television shows. Many observers say the team’s science leaders can also clear space on their shelves for a Nobel Prize.  

Distributed, high-throughput computing

Reflecting the LSC’s broad network of participants, much of LIGO’s computing infrastructure is distributed around the world and performed by LSC members. Low-latency computing, which must process incoming data and keep pace as it is received, is performed on dedicated systems at the observatories, Caltech, and redundant sites in the US and Europe. Other workflows are less time-sensitive, and many are embarrassingly parallel and loosely coupled. This makes it possible to distribute the data, matching the workload to the most suitable platform and having different systems examine different portions of the parameter space.

In addition to dedicated resources at Caltech and other LSC institutions, LIGO codes run on national multi-petaflops systems funded through NSF’s Extreme Science and Engineering Discovery Environment (XSEDE) program, such as the Stampede supercomputer at Texas Advanced Computing Center — a Dell PowerEdge cluster equipped with Intel Xeon Phi coprocessors — and the Comet system at the San Diego Supercomputer Center — a Dell-integrated cluster using the Intel Xeon processor E5-2600 v3 family  as well as 36 NVIDIA GPU nodes. LIGO also captures unused cycles from campus grids using NSF and the Department of Energy’s Open Science Grid (OSG), as well as from volunteer computing from the Einstein at Home project.  

Compute clusters access cached copies of the data over a variety of file systems. The central data archive, housed at Caltech, holds 5.3 PB of observational data and 1.1 PB of analysis results. The archive uses Oracle Hierarchical Storage Management (HSM) to manage storage tiering between disk and tape.

The software infrastructure includes the HTCondor workload management system, Pegasus Workflow Management System, and Open Science Grid Software. The Python open source programming language, which Intel software experts have contributed to developing, is in heavy use.

Rising requirements

Even with such extensive resources, the LIGO workloads are compute-constrained — and rising. “Our science is computation-bound,” Anderson says. “If I had access to all the Intel computers around the world right now, I would use them. There are codes that would benefit from being run with greater sensitivity. As it is, we make compromises to get the most science we can out of the available computing, and we try to get as much computing as we can.”

Much of the work involves single-precision, floating-point operations, with heavy use of numerical libraries including the Intel Math Kernel Library (Intel MKL) and the FFTW open source C subroutine library for computing discrete Fourier transforms. LIGO scientists gained a tenfold speedup for parameter extraction by optimizing LIGO codes for Intel MKL. They used Stampede for the optimization work, and Anderson says the improvements transferred well to other systems. “Because of those improvements, we could search all the parameter space we wanted to,” he comments. “Without that, we would not have been able to analyze all the data and keep up with it as it was coming in.”

Moving to the Advanced LIGO instrument increased the project’s computational load, in both the number of tasks and total number of floating-point operations, by more than an order of magnitude. Among the contributing factors, the new instrument is more sensitive at lower frequencies, bringing potential events into the observable band for longer and providing a larger and more data-rich parameter space to search.

Caltech upgraded its supercomputer to keep pace, adding Intel Xeon processor E3 and E5 v3 families and Intel Solid-State Drives (Intel SSDs). Workflows that need local temporary databases to manage their results got a particular boost from using the Intel SSDs as a local cache, according to Anderson. Caltech also uses some Intel Xeon Phi coprocessors, and Anderson is exploring using more of them going forward. 

“I’m excited that Intel Xeon Phi processors code name Knight’s Landing will be a cost-effective way to get access to lots of floating-point engines and run the same code as the Intel Xeon processor, so we don’t have to spend a lot of human resources to port it over and get it working,” Anderson says. “We’ll be well-positioned to take advantage of the Intel AVX-512 instruction set.”

An open window

As work goes forward, LIGO’s need for compute cycles will climb even more sharply.  The instrumentation team will continue to tune the ALIGO detector for increased sensitivity, expanding the parameter space further. Additional gravitational wave detectors are coming on line around the world and in space, offering greater opportunities for collaboration and further insights into the workings of the universe. Astrophysicists have a long list of questions they want to answer and ways they want to tie the study of gravitational waves to other observational techniques and other fields of astronomy. 

Then there are the surprises. “When you open a brand new window, you see things you didn’t expect,” says Anderson. “That’s the really exciting aspect to this. We have to be prepared to discover things we don’t know about, and we really don’t know what their computational costs will be. From the perspective of software and middleware, we have to be agile enough to follow up on new things. We need to do it as efficiently as possible and at reasonable costs.”

“In addition to the large high-throughput computing needs to search LIGO data for gravitational waves, high-performance computing is critical to solving Einstein’s equations numerically to provide information on both what LIGO ought to look for and in extracting the detailed information available in the waveforms recorded by LIGO.”

Programs such as the National Strategic Computing Initiative (NSCI), which aims to accelerate the development of exascale computers, may prove key to building on LIGO’s discoveries. “As the Advanced LIGO instruments are tuned up over the next few years to probe an order of magnitude larger volume of the universe, the corresponding computational challenge to search those data will also grow by more than an order of magnitude,” Anderson says. “In addition to the large high-throughput computing needs to search LIGO data for gravitational waves, high-performance computing is critical to solving Einstein’s equations numerically to provide information on both what LIGO ought to look for and in extracting the detailed information available in the waveforms recorded by LIGO. This HPC task will become significantly more challenging when LIGO detects other types of systems than the initial discovery of two merging black holes. For example, it will require world-class HPC resources to model the complicated physics of matter under extreme conditions, such as relativistically accurate simulations of merging neutron stars as they are ripped apart by gravitational forces, or to accurately simulate a three-dimensional supernova explosion.”

Technology transfer, workforce development, and awe

At a total investment of $1.1 billion over the last three decades, LIGO is delivering benefits beyond the scientific impact.

A major area of impact is in technology innovation and technology transfer to the private sector. “To make LIGO work, we had to develop the world’s most stable lasers, the world’s best mirrors, some of the world’s largest vacuum systems, as well as push the frontiers of quantum science and high performance computing,” Reitze told the Science, Space and Technology Committee. “LIGO advances the state-of-the-art in every technology it uses, and it uses lots of technology. We have partnered with many commercial technology firms in the U.S. and abroad to produce the incredible technology that LIGO uses.”

LIGO is also an engine of workforce development, educating scientists who have either stayed with the program or carried their advanced skills to national laboratories, Silicon Valley, and companies in sectors such as biotechnology, aerospace and telecommunications. The excitement generated by the LIGO discoveries is helping to inspire a new generation of students, raise appreciation of science, and offer each of us a deeper sense of awe at the complexity of the universe.

“Einstein was a fascinating figure,” Shoemaker says in conclusion. “The LIGO announcement once again shows that he was visionary in an almost unimaginable way. I hope taxpayers and organizations who helped support this process feel a sense of satisfaction from their participation in the endeavor. They’ve been part of something important.”

1. Testimony of Reitze and other LIGO team members: www.ligo.caltech.edu/news/ligo20160218

Jan Rowell is a technical writer in Portland Oregon.