<span class="var-sub_title">Toward Understanding I/O Behavior in HPC Workflows</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

PDSW-DISCS: Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems


Toward Understanding I/O Behavior in HPC Workflows

Authors: Jakob Luettgau (German Climate Computing Center, Argonne National Laboratory)

Abstract: Scientific discovery increasingly depends on complex workflows consisting of multiple phases and sometimes millions of parallelizable tasks or pipelines. These workflows access storage resources for a variety of purposes, including preprocessing, simulation output, and postprocessing steps. Unfortunately, most workflow models focus on the scheduling and allocation of computational resources for tasks while the impact on storage systems remains a secondary objective and an open research question. I/O performance is not usually accounted for in workflow telemetry reported to users.

In this paper, we present an approach to augment the I/O efficiency of the individual tasks of workflows by combining workflow description frameworks with system I/O telemetry data. A conceptual architecture and a prototype implementation for HPC data center deployments are introduced. We also identify and discuss challenges that will need to be addressed by workflow management and monitoring systems for HPC in the future. We demonstrate how real-world applications and workflows could benefit from the approach, and we show how the approach helps communicate performance-tuning guidance to users.


Archive Materials


Back to PDSW-DISCS: Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems Archive Listing

Back to Full Workshop Archive Listing