<span class="var-sub_title">Evaluation of HPC Application I/O on Object Storage Systems</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

PDSW-DISCS: Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems


Evaluation of HPC Application I/O on Object Storage Systems

Authors: Jialin Liu (Lawrence Berkeley National Laboratory)

Abstract: POSIX-based parallel file systems provide strong consistency semantics, which many modern HPC applications do not need and do not want. Object store technologies avoid POSIX consistency and are designed to be extremely scalable, for use in cloud computing and similar commercial environments. In this work, we evaluate three object store systems: Intel DAOS, Ceph RADOS, and Openstack Swift, and evaluate them with three HPC applications: VPIC, H5Boss, and BDCATS. We have developed virtual object layer (VOL) plugins for HDF5 that can redirect the applications' HDF5 calls to the underlying object storage systems' APIs, with minimum application code change. Through our evaluation, we found that object stores have better scalability in many cases than POSIX file systems, but are not optimized for common HPC use cases, such as collective I/O. Understanding current object store I/O details and limitations will enable us to better design object stores for future HPC systems.

Archive Materials


Back to PDSW-DISCS: Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems Archive Listing

Back to Full Workshop Archive Listing