In the past year, collaborations between HPC vendors and the open source community to co-develop future software have grown in number and intensity. These collaborations generally follow the model pioneered by the Linux community, where participating vendors contribute to and use open source software, then create enhanced, supported commercial versions for HPC customers needing these extras.

Vendor-catalysts for these expanded collaborations with the open source community are sometimes chipmakers focused mainly on the HPC software stack, including

• Intel (OpenHPC)
• IBM (OpenPOWER)
• AMD (GPUOpen)
• ARM Holdings (multiple projects)

In other cases, the catalyzing vendors are independent software vendors wanting help with their applications products. In both cases, the rationale is similar. Developing competent HPC software has never been easy, but mounting requirements have now made this task so daunting that vendors increasingly welcome assistance from the open community.

Even an abbreviated list of user requirements includes

• applications compatibility
• larger system sizes (soon to reach exascale)
• mixes of compute-intensive and data-intensive workloads
• heterogeneous processing elements
• newer environments, such cloud computing
• improved energy efficiency and reliability/resiliency
• deeper memory hierarchies
• better command-and-control capabilities to manage all this concurrent complexity

And, of course, these elements also need to be integrated, tested and validated before users can exploit them as a coherent resource.

The Linux model has demonstrated the compelling benefits for vendors of the collaborative, “open software ecosystem” approach:

• The collaborations enable vendors to select the best open source components for baseline functions, while focusing most of their R&D budgets and efforts on software development targeted at providing competitive differentiation. In other words, open source software can act as an offset to vendors’ R&D expenses.
• These partnerships often result in two versions of the co-developed software — an open version that’s free to use, and a paid version for users who need special added capabilities and vendor support.
• Last, but not least, the open version seeds the community with potential new users of the paid version. A portion of users who become accustomed to employing the free version will later on want the paid version. Think, for example, of students who are later employed by HPC sites needing vendor-supported software.

Of course, this is all easier said than done. Among the challenges vendors face are identifying the most promising open source software components from among thousands of candidates, and accurately estimating the amount of work needed to “harden” them for use in commercial and other production environments. The early years of the “CRADA movement” (Cooperative Research and Development Agreements) taught the HPC community that the path from experimental code to commercial-grade software can be long and expensive.

For vendors collaborating with the open source community today, piecing together commercial solutions is made somewhat easier by the availability of heavily used baseline components, such as Open MPI, OpenSFS, CentOS, Rocks Cluster Distribution, OpenFOAM, OpenStack, and others, including Linux itself.

In sum, the trend toward collaborative development of open software ecosystems is a good and necessary thing. That said, the fact that there are multiple open software ecosystem initiatives in progress, rather than just one, is pressing vendors to make choices. IDC research shows that some HPC system makers (OEMs) are huddling internally to decide how many of the open ecosystems to participate in and how much time and money to devote to each collaboration.

The decisions are especially important for these OEMs, because most are accustomed to developing and offering their own HPC software stacks and now need to inflect that strategy to reap the benefits of collaborative development.

Hypothetically, OEMs, ISVs and other vendors who want access to the entire global HPC market should participate in all of the major open ecosystem initiatives. Why cut yourself off from any initiative that represents a sizeable chunk of the market? But practical considerations of time and money will likely force many vendors to choose a limited number of dance partners and to prioritize these involvements, some of which offer multiple levels of financial and hands-on participation.

OEMs IDC recently talked with note that each of the major open ecosystem initiatives is pushing in a somewhat different direction. That’s natural, given that the initiatives are focused on different base processors. For now, the OEMs hope there will be enough flexibility to substitute some software components — that will allow the OEMs to continue differentiating heavily from competitors. They also hope there will be enough overlap of components to reduce the burden of participating in multiple open ecosystems.

It’s still very early in the lives of HPC open software ecosystems that are applying the Linux community model, especially to the software stack and ISV applications. It’s far too early to know how much success each initiative will achieve. What’s clear, though, is that burgeoning requirements have prompted a new, more collaborative model for HPC software development, one that’s likely to be around for many years to come.

Steve Conway is Research VP, HPC at IDC. He may be reached at editor@ScientificComputing.com.

 

R&D 100 AWARD ENTRIES NOW OPEN: Establish your company as a technology leader! For more than 50 years, the R&D 100 Awards have showcased new products of technological significance. You can join this exclusive community! Learn more.