Research & Development World

  • Home Page
  • Topics
    • Aerospace
    • Archeology
    • Automotive
    • Biotech
    • Chemistry
    • COVID-19
    • Environment
    • Energy
    • Life Science
    • Material Science
    • R&D Market Pulse
    • R&D Management
    • Physics
  • Technology
    • 3D Printing
    • A.I./Robotics
    • Battery Technology
    • Controlled Environments
      • Cleanrooms
      • Graphene
      • Lasers
      • Regulations/Standards
      • Sensors
    • Imaging
    • Nanotechnology
    • Scientific Computing
      • Big Data
      • HPC/Supercomputing
      • Informatics
      • Security
      • Software
    • Semiconductors
  • 2021 R&D 100 Award Winners
    • R&D 100 Awards
    • 2020 Winners
    • Winner Archive
  • Resources
    • Digital Issues
    • Podcasts
    • Subscribe
  • Global Funding Forecast
  • Webinars

Handling Trillions of Supercomputer Files Just Got Simpler

By Los Alamos National Laboratory | March 18, 2019

Gary Grider, left, and Brad Settlemyer discuss the new Los Alamos and Carnegie Mellon software product, DeltaFS, released to the software distribution site GitHub. (Credit: Los Alamos National Laboratory)

A new distributed file system for high-performance computing distributed March 14 via the software collaboration site GitHub provides unprecedented performance for creating, updating and managing extreme numbers of files. 

“We designed DeltaFS to enable the creation of trillions of files,” said Brad Settlemyer, a Los Alamos computer scientist and project leader. Los Alamos National Laboratory and Carnegie Mellon University jointly developed Delta FS. “Such a tool aids researchers in solving classical problems in high-performance computing, such as particle trajectory tracking or vortex detection.” 

DeltaFS builds a file system that appears to the user just like any other file system, doesn’t require specialized hardware, and is exactly tailored to assisting the scientist in new discoveries when using a high-performance computing platform. 

“One of the foremost challenges, and primary goals of DeltaFS, was scaling across thousands of servers without requiring a portion of them be dedicated to the file system,” said George Amvrosiadis, assistant research professor at Carnegie Mellon University and a coauthor on the project. “This frees administrators from having to decide how to allocate resources for the file system, which will become a necessity when exascale machines become a reality.”

Carnegie Mellon team members, from left: Chuck Cranor, Qing Zheng and George Amvrosiadis.

The file system brings about two important changes in high-performance computing. First, DeltaFS enables new strategies for designing the supercomputers themselves, dramatically changing the cost of creating and managing files. In addition, DeltaFS radically improves the performance of highly selective queries, dramatically reducing time to scientific discovery. 

DeltaFS is a transient, software-defined service that allows data to be accessed from a handful up to hundreds of thousands of computers based on the user’s performance requirements.

“The storage techniques used in DeltaFS are applicable in many scientific domains, but we believe that by alleviating the metadata bottleneck we have really shown a way for designing and procuring much more efficient HPC storage systems,” Settlemyer said.

GitHub link: https://github.com/pdlfs/deltafs/

Video link: https://youtu.be/GIIyzcZusUw

Related Articles Read More >

Q-CTRL introduces quantum sensing division to meet market demands
SwRI develops automated shuttle for campus tours, research
R&D 100 winner of the day: AI-Rad Companion Organs RT
R&D 100 winner of the day: WEC-Sim
2021 R&D Global Funding Forecast

Need R&D World news in a minute?

We Deliver!
R&D World Enewsletters get you caught up on all the mission critical news you need in research and development. Sign up today.
Enews Signup

R&D World Digital Issues

February 2020 issue

Browse the most current issue of R&D World and back issues in an easy to use high quality format. Clip, share and download with the leading R& magazine today.

Research & Development World
  • Subscribe to R&D World Magazine
  • Enews Sign Up
  • Contact Us
  • About Us
  • Drug Discovery & Development
  • Pharmaceutical Processing
  • 2022 Global Funding Forecast

Copyright © 2022 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search R&D World

  • Home Page
  • Topics
    • Aerospace
    • Archeology
    • Automotive
    • Biotech
    • Chemistry
    • COVID-19
    • Environment
    • Energy
    • Life Science
    • Material Science
    • R&D Market Pulse
    • R&D Management
    • Physics
  • Technology
    • 3D Printing
    • A.I./Robotics
    • Battery Technology
    • Controlled Environments
      • Cleanrooms
      • Graphene
      • Lasers
      • Regulations/Standards
      • Sensors
    • Imaging
    • Nanotechnology
    • Scientific Computing
      • Big Data
      • HPC/Supercomputing
      • Informatics
      • Security
      • Software
    • Semiconductors
  • 2021 R&D 100 Award Winners
    • R&D 100 Awards
    • 2020 Winners
    • Winner Archive
  • Resources
    • Digital Issues
    • Podcasts
    • Subscribe
  • Global Funding Forecast
  • Webinars