As one of Europe’s leading aerospace research institutions, it’s no surprise that France’s Office National d’Etudes et Recherches Aérospatiales (ONERA) has a huge appetite for HPC processing power and that it is constantly looking for ways to push its research capabilities to the next level. Currently, as part of its priorities for 2030, ONERA is working to update and optimize one of its key computational and fluid dynamics (CFD) codes, known as elsA. The work will not only help to vastly improve efficiency, but it is putting ONERA on a path to supporting next-generation aerodynamics simulations on its new Top 500 NEC system.

A broad mandate

Since its founding in 1946, ONERA has made steady advancements, along with many breakthroughs in aeronautics, space and defense research. Its original mission included conducting aeronautical research, advancing French and European industry interests, running experimental facilities, providing specialized analysis and services to industry and the government, and training researchers and engineers. Today, as a multidisciplinary institution, ONERA employs nearly 2,000 people across six scientific departments that support a wide range of research and development activities, including modelling, advanced simulations, and in-flight and wind-tunnel based tests and experimentation. 

CFD in the spotlight

While most of ONERA’s departments rely on the organization’s HPC systems, the demands from the computational and fluid dynamics (CFD) teams are especially high, accounting for upwards of 95% of CPU usage. Two of ONERA’s most important CFD applications are elsA and CEDRE. elsA, which ONERA first developed in 1997, is aerodynamic simulation software that is designed to support multi-disciplinary applications involving aerodynamics. And CEDRE is reference simulation software for energy and propulsion. Over the years, elsA and CEDRE have been instrumental in much of ONERA’s CFD research, in addition to the work of industry partners, including Airbus, Airbus Helicopter, Safran, Ariane and other aerospace manufacturers.

 “These codes, which exceed one million lines, are tools for capitalizing ONERA’s know-how,” explains Alain Refloch, special advisor for HPC at ONERA. “They have helped our industry partners with everything from finding ways to reduce drag to improve fuel consumption to increasing the efficiency of engines and reducing the number of blades to complying with international standards for noise.”

Aligning HPC approaches with research priorities

The ONERA philosophy holds that a combination of modeling and real-world testing produces the best research results, so the lab continuously seeks better ways to fuse tests from its world-class wind tunnel facilities with HPC simulations. As ONERA confronts a long list of challenges and priorities that it is aiming to address by 2020 and 2030, the evolution of its proprietary CFD software will play a central role. Both the elsA and CEDRE applications feature prominently in ONERA’s goal of “developing a comprehensive simulation strategy based on building modular models and simulations, chained or coupled, validated by experiments at all levels, using very high-performance computing capabilities.” One of the key initial steps to realizing this goal is to develop a new architecture that can address interoperability challenges and improve simulations.

To advance its efforts, ONERA applied to the Intel Parallel Computing Center (Intel PCC) program for help building an HPC foundation optimized for the new code design.

Refloch explained that development challenges behind the desire to work with Intel: “Today, on one hand, the development time for large codes is longer than the lifetime of the machines. On the other hand, there are many unknowns in anticipating electrical consumption reduction and other factors associated with processor evolution. That’s why we need as early access as possible to new processors and it was one of our main motivations for becoming an Intel PCC.”

Partnering for FAST optimization

The Intel PCC partnership gave birth to a study of one of ONERA’s most important new kernels, dubbed Flexible Aerodynamic Solver Technology (FAST), on Intel Xeon Scalable and Intel Xeon Phi processors. FAST is actually a prototype for a forthcoming many-core version of elsA. It performs on-the-fly post-processing of structured and unstructured grids for compressible flows with unsteady simulation of turbulence. ONERA began work on the FAST prototype in January 2016, with a series of milestones set through 2019. For the first milestone in 3Q16, the ONERA applied aerodynamics department tested a prototype version of FAST on a challenging use case. ONERA then released an optimized version for structured mesh in 4Q17 for evaluation by its industry partners, including Safran where it is now installed. A second optimized release for structured and unstructured mesh is scheduled for 2Q18, with the final release with the architecture for hosting elsA development scheduled for 2Q19.

According to Refloch, the collaboration with the IPCC has been invaluable for ONERA’s FAST optimization efforts. He noted that in addition to getting early access to new processors, ONERA can turn to Intel for advice on topics ranging from how to overcome technical difficulties with new software releases and features (including beta) to how to do in-depth analysis on performance issues.

“For instance, Intel showed us how to use padding in the solver to alleviate cache eviction due to associativity. This insight enabled us to speedup processing by a factor of six on Intel Xeon Phi processors, in some cases,” explained Refloch. “One of the most valuable aspects of the IPCC partnership is that we now have a better understanding of the opportunities for making performance gains along with whether or not we are executing properly. That’s significant since with code modernization efforts it can be so difficult to estimate if a modification will lead to sufficient performance improvements compared to the man-hours required to change the code.” The ONERA team also has monthly meetings with Intel France where it has the opportunity to ask any questions that come up about optimization. “These exchanges significantly increase our productivity,” said Refloch.

Refloch added that the Intel Advisor tool has also been important to the success of ONERA’s code optimization efforts. “Profiling is one thing, but interpreting the results and above all modifying the code accordingly is something else. The work done on Advisor, particularly with respect to Roofline Analysis, has really helped our developers stay focused on what matters most to getting the best possible performance from our software,” said Refloch.

A window into the future

According to Ivan Mary, the project leader for FAST, ONERA expects that the new software architecture for elsA will be game changing based on its ability to:

• Easily achieve efficiency on the new hardware by automatically generating code based on python“templates” for the code hotspots
• Automatically source code differentiation for adjoint problems
• Provide interoperable Python components for data pre-, post-, and co-treatment, and multiphysics simulations

“Having these solver characteristics will help up us run multidisciplinary CFD simulations with more parameters and higher fidelity in a shorter amount of time,” explained Mary. “Over time that will help ONERA and our partners design much safer, cleaner and quieter aircraft.”

Seeing great scalability on PRACE and TOP500 systems

While researchers at ONERA have several HPC systems at their disposal, in September 2017 the lab replaced its SGI ICE 8200 supercomputer with a powerful new NEC system that will be instrumental to ONERA’s CFD research moving forward. The NEC system includes 17,360 cores comprised of Intel Xeon processors each with 14 cores each. The interconnect is 100Gbps Intel Omni-Path architecture. As of June 2017, the new system was ranked number 341 on the TOP500 list, with a Linpack Performance of 579.2 TFlops and a theoretical peak of 667 TFlops. (Sator moved to 473 in the November 2017 edition of the TOP500 list).  “Right now, we also have the option to add a new computer at the end of 2018 with Intel Xeon Scalable processors. If we add that system, before the end of 2018 ONERA researchers will have access to 1Pflops in house,” explained Refloch.

Based on an initial test of FAST on the new NEC system, as well as on a PRACE (Partnership for Advanced Computing in Europe) system with Intel Xeon Scalable processors, ONERA’s code optimization work is paying off as expected. “FAST was one of the first applications we tested on our new system, and we showed 95% scalability on more than 17,000 cores.  We also showed 95% scalability on the MareNostrum PRACE system [through the Preparatory Access program] on an even higher number of cores,” said Nicolas Alferez, a CFD optimization expert.

While ONERA won’t start fully merging the elsA capabilities into the new architecture until 2019, the new NEC system should help speed up the work of CFD researchers considerably in the meantime. “The transition from the old system to the new one is very fast, since both are based on the Intel processor and Intel compiler,” explained Refloch. “Once everyone switches over, most of our users should have 2x more cores for their usual workloads moving forward. That means we should be able to complete jobs at least two times faster, which is a welcome improvement as we wait for even bigger and better things to come at the completion of the next-generation code” he concludes.