For the past 60+ years, the frozen continent of the Antarctic has been the scene of ongoing scientific field research conducted by the British Antarctic Survey (BAS). In the years I have been at BAS, data collection is exploding, creating a new challenge for us and our researchers. We’ve had to upgrade our storage infrastructure, since massive amounts of data and scientific research has become ever more reliant on an improved storage system.
As one of the world’s leading environmental research centers and a component of the Natural Environment Research Council, BAS is responsible for massive data-collection expeditions in the Antarctic that provide critical insights for ongoing scientific research regarding the future of our climate and environment. The current research strategy, called “Polar Science for Planet Earth,” will help shed light on major environmental issues, such as the impact of global warming on the Antarctic ice shelf.
As part of this mission, our researchers at BAS combine advanced science programs with essential logistics to carry out complicated and sophisticated scientific field research. This is no small feat, as BAS employs more than 400 staff and operates three research stations in the Antarctic, as well as two stations on South Georgia. We use ice-strengthened ships, numerous aircraft and a multitude of sensors and data collection systems to drive advancements in oceanographic and climate research.
Recently, along with a small, dedicated IT team, I participated in a 2.5-month Antarctic expedition, during which we set out to measure as much of the Antarctic environment as possible. This included dozens of individual and highly-specialized probes, sensors, floating buoys and GPS systems, measuring everything from air and water temperature, water turbulence, wind information and the amount of radiation from sunshine falling on the ocean, to the types and amounts of marine life moving in the water.
The incoming data was massive — the probes and sensors would collect up to 10GB of data an hour. To give you an example of data size, one six-week expedition produces at least 500GBs of data. This is a dramatic increase from past years. In fact, we are collecting 10 times the amount of data we gathered just 10 years ago, and we will collect twice as much data this year as last.
All of this information needs to be secured for subsequent analysis and scientific modeling. After it is collected, it is sent to the BAS high-performance computing (HPC) center in Cambridge, where it is fed into a variety of models used by hundreds of scientists around the world.
Our challenge was clear: with limited datacenter space, limited power and budget parameters, we needed to determine a solution that would provide high-capacity, high performance storage, while allowing for in-place growth at our space-constrained data center.
Additionally, it can take up to two to three months to process code, so we needed to make sure that high-performance storage would be available to support long-term projects and that everything including power supplies, controllers and connections were redundant. We also needed a solution that would seamlessly support a growing VMware environment, as we expect to grow well beyond our current cadre of 100 virtual machines.
In our initial review of vendor options, we found most solutions were too expensive, too large physically, too heavy or required too much power.
Since we had previously used storage from DataDirect Networks (DDN) and were familiar with price-performance benefits, we took interest in its SFA770X platform, which features “pay-as-you-grow” scalability. The platform’s hybrid flash storage appliance is purpose-built to tackle big data requirements, and enables us to combine the power of Flash technology to meet our performance needs with the economy of HDDs to lower our overall total cost ownership. The ability to mix and match storage types in one enclosure meant we could deploy SSD drives for the first time, and blend some of the fastest online storage with near-line storage.
Our results have been excellent! By utilizing DDN’s SFA7700X hybrid flash storage, we were able to effectively double our capacity versus last year’s levels, and our current storage capacity should last us another two years of aggressive growth before we need to consider expansion. By moving to DDN’s hybrid flash SFA770X, our IT team is now supporting BAS scientists and researchers with a powerful storage system that takes advantage of a mix of flash and rotating media to cost-effectively speed performance of different applications and models. And thanks to DDN’s high-density configuration, we’ve been able to cut data center space dedicated to storage by almost half, while increasing capacity by over 10x.
In the past month, BAS has set off on its latest Antarctic expedition — and the nature of the research, combined with the harsh environmental conditions, means the data collected is precious — we can only collect it once. With DDN’s hybrid flash solution, we can extract maximum performance from both spinning disk and flash media — so for the precious data collected, once is enough!
Jeremy Robst is IT Support Engineer and Head of Unix Systems for the British Antarctic Survey.
