The average number of years before returns started was 1.9

Not long ago, I wrote in this publication that the end of the Cold War inaugurated a multi-year shift in government funding rationales for HPC. The historical heavy tilt toward national security and advanced science/engineering is slowly being counterbalanced by arguments for returns-on-investment (ROI). In the emerging era, ROI expectations are extending beyond traditional IT progress metrics, such as counts of published papers, patents and advanced degrees. Newer measurements target the ROI achievements themselves: scientific and engineering innovations, cost savings, job creation and private-sector revenues and profits.

Studies IDC conducted earlier on for the Washington, DC-based Council on Competitiveness firmly established the link between HPC and competitiveness. In one study, 97 percent of corporate participants that had adopted HPC said they could no longer compete or survive without it. A few years later, in the midst of the global economic recession, political leaders around the world were increasingly persuaded that HPC could accelerate scientific progress, economic competitiveness and perhaps also economic recovery.

U.S. presidents Bush and Obama praised HPC in state-of-the-union speeches, Russian President Medvedev said that without more investment in HPC, in five years no one would be interested in buying Russian exports, and the economic argument bolstered the successful argument for an EU-wide HPC plan that doubled annual budgets. HPC has long contributed to Japanese industry, South Korea will be investing more in HPC, and China’s HPC investments have been skyrocketing.

Interest in quantifying ROI from HPC investments has existed for at least 20 years, but has typically been limited to showcasing a handful of shining examples. In November 2012, the U.S. Department of Energy’s Office of Science and National Nuclear Security Agency awarded IDC an eight-month grant to create economic models that could be used to track HPC ROI and to help predict, in the aggregate, the returns from investments in HPC. Under this pilot study, which is now complete, IDC tested the economic models by populating them with a limited number (208) of real-world examples from throughout the world. The next step, if funding for the proposal is approved, is to conduct a full-out, multi-year study whose goals include refining the models to advance from correlation toward causation, and populating the models with large numbers of examples from around the world.

The Capture-and-Measurement Tools
The pilot study created and tested two correlation-based economic models and a new innovation index:

• Model 1 aimed to capture how HPC investments result in financial ROI: revenues, profits/cost savings and job creation.
• Model 2 targeted how HPC investments generate non-financial innovations, both basic and applied innovations in the public and private sectors.
• The new innovation index asked respondents to rank the importance of their innovations on a 1-10 scale that also considers how many organizations use the innovation and its impact on the larger societies. This self-grading exercise was followed by assessments from domain experts, which on the whole did not differ markedly from the results of the self-grading. IDC refined this index four times before concluding that it was appropriate for the requirements of the pilot study.

Key Findings of the Pilot Study
The 208 real-world examples collected and analyzed as part of the pilot study included 67 financial ROI examples and 141 innovation ROI examples. Seventy-nine of the innovations were in the public sector and 62 were in the private sector.
IDC was able to collect the required data across a broad set of organizations with enough detail to populate the two economic models and the innovation index. (If the responses did not make use of the entire index, numbers 1 through 10, the index would not have been useful.)

• The data in the pilot study strongly imply that investments in HPC typically deliver very substantial returns. In this study, each dollar invested in HPC returned, on average, $356 in revenue and $38 in profits or cost savings.
• The average number of years before returns started was 1.9 years, a shorter period than we expected.
• The average HPC investment needed to create a new job was $93,000. Forty-two sites reported job creation. On average, each of these sites created 29.8 jobs from their HPC investment. A total of 1,251 jobs were created within this group.
• As a group, the respondents invested about $500 million to generate 141 innovations, or about $3.1 million per innovation on average. Academic and industrial sites contributed the bulk of the innovations in the sample. Most of the basic research innovations were in academia, while most of the applied research innovations were in industry.
• Government innovations had higher average scores on the innovation index (7.0), while innovations in industry averaged 5.7 and academic innovations averaged 3.9. Government sites tended to invest more per HPC innovation ($4.4). Industrial sites invested $3.8 million on average, and academic sites invested $2.5 million per innovation.

Conclusions and Next Steps
By directly capturing and quantifying ROI achievements, the pilot study fortified the argument that investments in HPC are strongly associated with strong returns. The pilot study also shed light on differences in returns that might be expected from HPC-related investments in basic and applied research, as well as differences by major sector (government, academia, industry) and by scientific and engineering domain.
In future research phases, IDC plans to refine the economic models and populate them far more heavily, as noted earlier. The ultimate goal is to be able to present planners and funders with tools they can reliably use to help support their important decisions. Our hope is that these tools will allow decision-makers to project the impacts of HPC-related investment decisions over as long as a decade. We want to advance decision-support for important questions such as these:

• What returns can be expected from investing money in HPC, compared with investing the same amounts to address other national/regional priorities?
• How much of the HPC investment should be directed toward basic research? Applied research? Industrial/commercial R&D?
• Which scientific and engineering domains are most in need of additional investments? (This presumably will differ by region and nation.)

Success Stories and More Information
Another outcome of the pilot study research is the expansive list of HPC success stories IDC collected from the survey respondents. DOE has encouraged IDC to share summaries of the success stories, along with the other research findings from the ROI pilot study. Anyone interested in more information can contact the author. 

Steve Conway is Research VP, HPC at IDC. He can be reached at editor@ScientificComputing.com.