Hermes: a Multi-Tiered Distributed I/O Buffering System for HDF5
TimeThursday, November 15th8:30am - 5pm
DescriptionHigh-Performance Computing (HPC) systems’ increasing ability to run data-intensive problems at larger scale and resolution has driven the evolution of modern storage technologies. In addition, extreme amounts of data are collected by large scientific instruments and sensor network is resulting in a push for more capable storage systems. Hierarchical Data Format (HDF) technologies address the problems of how to organize, access, analyze, and preserve data in the face of enormous growth in size and complexity. To mitigate this I/O performance bottleneck, modern storage subsystems are going through extensive changes, by adding additional levels of memory and storage in a hierarchy. In this work, we present Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system for HDF5. Hermes enables, manages, and supervises I/O buffering in DMSH. We tested our solution on Cori and show Hermes can accelerate applications by 62x than the state of the buffering platform.