The Met Office, the United Kingdom’s national weather service, has been at the forefront of global weather and climate science for 150 years. They are continuing this leadership in weather prediction and climate modeling with the installation of one of the world’s most advanced HPC systems. According to Dave Underwood, Met Office, Deputy Director of High Performance Computing Programme, “The new supercomputer will enable the UK to lead the world in weather, climate and environmental science high performance computing. In addition, the power of the computer will be a catalyst for regional growth supporting collaboration and partnerships between science, business and academia.” The Met Office estimates that the system will deliver £2bn socio-economic benefits through enhanced UK resilience to severe weather and related hazards.
When compared to the previous IBM Power 775 supercomputer, the Cray system using Intel processors will weigh nearly three times as much. It will have 12 times as many processing cores, 25 times more memory for calculations, eight times more storage and be able to perform calculations 13 times faster. |
In a typical day, the Met Office processes over 106 million observations with more than 95 percent of the observations from space. They use their advanced modeling to deliver 3,000 tailored weather forecasts and briefings each day to customers ranging from government, businesses, the general public, armed forces and other organizations and also provide training, multi-media and consulting services to a variety of industries. They are recognized as one of the world’s most accurate forecasters in the world. The work performed by the Met Office requires tremendous computing power. The Met Office could not perform their work without the use of supercomputers and HPC-related software.
Selecting a new Met Office supercomputer
When the Met Office decided to upgrade their existing supercomputer, they performed an exhaustive analysis of vendors and their products over a two year period of time. In October 2014, the Met Office awarded Cray the $128 million contract to install the new supercomputer. The Cray systems, using Intel Xeon processors, is being installed in phases. Phase1a was installed in September 2015 “five weeks ahead of schedule with no major issues,” according to Underwood. This will be the first winter where weather forecasts are used with the new system. Phase1b will be installed in 2016 and the final system upgrades will be installed in 2017.
Introducing the new Phase1a Cray supercomputer using Intel processors
Cray deployed two Cray XC40 supercomputer systems and four Cray Sonexion 2000 storage systems to meet the Met Office’s Phase 1a requirements. Each Cray XC40 system comprises four Cray XC40 cabinets, and is equipped with Intel Xeon processors E5-2698v3. The supercomputers are supported by over 13PB of Cray Sonexion 2000 integrated Lustre storage systems providing almost 1TB/s of I/O bandwidth. According to Philip Brown, Cray Segment Leader for Earth Sciences, “The Cray XC40 supercomputers delivered in Phase 1a were designed as a like-for-like replacement of the Met Office’s previous generation of supercomputers, delivering equivalent application performance while consuming three times less power and allowing for the installation of future upgrades to the Cray systems.” The Phase1b HPC system is being installed now and will have six times the performance of the Phase1a system.
The new supercomputer will be one of the fastest in the world. At peak performance (available from March 2017), it can do more than 3 million calculation per second for every man, woman and child on the planet. (This is 16 petaflops – 23,000 trillion calculations per second). Here is a comparison of the previous Met Office supercomputer with the new Cray supercomputer at the end of Phase 1c.
|
Previous system |
New system |
Number of cores (CPUs) |
40,000 |
480,000 — more processing power than 100,000 PlayStation 4 machines |
Memory for calculations |
80,000 gigabytes (0.08 petabytes) |
2 million gigabytes (2 petabytes) — more memory than 120,000 iPhones |
Storage for data |
2 million gigabytes |
17 million gigabytes (17 petabytes) — enough to store over 100 years of HD movies |
Calculations at peak |
1200 trillion per second |
23,000 trillion per second |
Independently reviewed studies determined that there are six distinct areas which would benefit from the improved weather and climate advice made possible by the new HPC system.
Benefits of the Met Office Supercomputer
|
The Cray XC40 supercomputer systems include the Cray Linux Environment (CLE), which provides a highly optimized environment to support the efficient execution of the Unified Model, the Met Office’s massively parallel numerical weather prediction software, as well as features to ensure predictable runtime and extreme reliability required to meet the operational needs of the Met Office. The Cray Linux Environment also includes a range of industry-leading tools to support application development, optimization and debugging. To support the compilation of a wide range of software on the Cray XC40 platform, Cray has installed the Cray Compiler Environment and the Intel Parallel Studio Composer XE for Linux compilers, in addition to the open-source GCC compilers, as well as optimized scientific libraries for all compiler suites. Flexible workload management is provided by Altair’s PBS Professional job scheduler, working in concert the Cray’s ALPS resource manager. Parallel debugging is supported by Allinea’s DDT debuggers.”
The Met Office uses a sophisticated Unified Model to predict weather and climate. This system can predict global, European and United Kingdom weather. Its flexibility also allows it to be used to model a wide range of time-scales, from daily weather forecasts to decadal climate predictions. The model uses deterministic and ensemble modeling techniques and can be combined with ocean and atmospheric chemistry models to make predictions for the full earth system. Because of the use of HPC and the Unified Model, “We have seen a steady improvement in the skill of weather predictions over the last 25 years, with a 2% reduction in errors year on year. This has been achieved through a combination of increased resolution, better weather observations and improved modeling.” according to Dr. Paul Selwood, who leads the work done by the Met Office to optimize codes for High-Performance Computers.

The Met Office uses complex mathematical calculations in weather modeling. The dynamical core in the Met Office’s Unified Model, was changed from New Dynamics to ENDGame in 2014. The new Met Office Cray-Intel HPC system, uses the ENDGame dynamical core which provides better weather modeling results over the older version. New Dynamics suffered from scalability issues which limited the ability of the model to best utilize modern computers. Selwood states, “The ENDGame dynamical core is considerably more scalable and so solves calculations faster while also providing more accuracy and stability than New Dynamics. ENDGame achieves this by providing improved grid staggering at the poles, fewer polar communications, a better Helmholtz matrix structure and a mixed precision solver.”
So what does this mean in terms of weather forecasting? According to Helen Chivers, Head of News and Social Media: “When you think about a weather forecast model, it is almost like looking at a picture from a digital camera because it is made up of many pixels. We can currently make the pixels in a globe weather image model about 12 kilometers square and in the UK they are approximately 1.5 kilometers square. So the finer you can get the pixels in the weather image then the more detail you can see and the more accurate your forecast can be. The increase in HPC computing power and software improvements with the new Met Office system will provide us with more detailed images to help improve our weather forecasting ability.”
This is important not only to transportation and aircraft but to the general public in terms of planning and safety. An example of this is shown in Figure 2, where you can see the increased visual resolution detail that ENDGame also provides which aids in weather visualization and forecasting in areas such as orographic or stratospheric gravity waves.
How HPC will aid weather prediction and climate modeling in the future
The Met Office envisions being able to better predict weather and climate models as the final phases of the HPC system is installed. “We hope to be able to deliver more accurate short-term weather forecasts in terms of what can be delivered in the new few hours and days. Currently, we look at weather forecasts going out 5 days. As we move into the final installation of the HPC systems, we hope to be able to generate more detailed weather forecast 10 days out — to give accurate forecasts at a much earlier stage. As we move further into the second and third phases of the HPC installation, we will be able to better predict the impacts of climate change that can help businesses and governments make informed decisions for our expected future climate,” states Underwood.
In Phase 1c, there will be one large system with its own storage that will be housed in a new IT hall. “Future deliveries of Cray systems will allow the Met Office to increase the resolution of their high resolution regional and global deterministic models, to increase the resolution and scope of the probabilistic ensemble forecasting systems and to expand collaborative research activities with UK and international organizations. All generations of the system share the same tightly integrated Cray software stack, giving Met Office users and application developers, continuity across all three Phases and maximizing both developer productivity and the productivity of the supercomputer systems,” says Brown.
According to Robert Maskell, Director of High Performance Computing, Intel UK, “Increasingly sophisticated and powerful Intel Xeon processors will be installed in the final phases of the Met Office HPC system. In Phase 1b which will be completed in early 2016, Cray will install Intel Xeon processors formerly known as Broadwell which will run at six PFlops with a processing speed six times faster than the Phase 1a system. In Phase 1c to be installed in 2017, Intel Xeon processors will move to 12 PFlops which will allow the system to be used for scientific research in areas such as climate modelling. The UK Met Office is also researching using Intel Xeon Phi coprocessors along with optimized HPC code in Phase 1c systems to further increase the speed and accuracy of Met Office weather and climate modelling processing. Experiments performed on the system using Intel Xeon Phi coprocessors will include lexicographic array ordering, explicit red-black memory ordering, standard variable bounds arrays, strip-mined arrays for Single instruction, multiple data (SIMD) and using a solver mini application to track performance on the CPU and Intel Xeon Phi coprocessor.“
Paul Selwood is the expert overseeing optimizing HPC code on the new Met Office system. Selwood states, “Software tools lag hardware for HPC. For example, Fortran 2003 support is still immature, and hand strip-mining over three million lines of legacy code to vectorize it for Xeon Phi processors is not feasible.” Selwood recommends using a layered architecture and moving some of this low-level optimization away from the core scientific algorithms. This is the approach being taken by the Met Office, working in collaboration with the Natural Environment Research Council (NERC) and a partnership involving Imperial College, Science and Technology Facilities Council (STFC), Universities of Bath, Exeter, Manchester, Leeds, Reading and Warwick, to develop the next generation system. This system will contain a dynamical core that is scientifically as good as ENDGame but that also scales on future computer architectures.
“Operational HPC is hard but worthwhile. Big HPC can pay for itself many times over as we develop tools that allow scientific staff to use it efficiently,” Selwood concluded.
Linda Barney is the founder and owner of Barney and Associates, a technical/marketing writing, training and web design firm in Beaverton, OR.