Hermes: a Multi-Tiered Distributed I/O Buffering System for HDF5

Authors: Hariharan Devarajan (Illinois Institute of Technology, HDF Group)

Abstract: High-Performance Computing (HPC) systems’ increasing ability to run data-intensive problems at larger scale and resolution has driven the evolution of modern storage technologies. In addition, extreme amounts of data are collected by large scientific instruments and sensor network is resulting in a push for more capable storage systems. Hierarchical Data Format (HDF) technologies address the problems of how to organize, access, analyze, and preserve data in the face of enormous growth in size and complexity. To mitigate this I/O performance bottleneck, modern storage subsystems are going through extensive changes, by adding additional levels of memory and storage in a hierarchy. In this work, we present Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system for HDF5. Hermes enables, manages, and supervises I/O buffering in DMSH. We tested our solution on Cori and show Hermes can accelerate applications by 62x than the state of the buffering platform.

Best Poster Finalist (BP): no

Poster: pdf
Poster summary: PDF


Back to Poster Archive Listing