Stacker: An Autonomic Data Movement Engine for Extreme-Scale Data Staging-Based In Situ Workflows

Authors: Pradeep Subedi (Rutgers University), Philip Davis (Rutgers University), Shaohua Duan (Rutgers University), Scott Klasky (Oak Ridge National Laboratory), Hemanth Kolla (Sandia National Laboratories), Manish Parashar (Rutgers University)

Abstract: Data staging and in situ workflows are being explored extensively as an approach to address data-related costs at very large scales. However, the impact of emerging storage architectures (e.g., deep memory hierarchies and burst buffers) upon data staging solutions remains a challenge. In this paper, we investigate how burst buffers can be effectively used by data staging solutions, for example, as a persistence storage tier of the memory hierarchy. Furthermore, we use machine learning based prefetching techniques to move data between the storage levels in an autonomous manner. We also present Stacker, a prototype of the proposed solutions implemented within the Data\-Spaces data staging service, and experimentally evaluate its performance and scalability using the S3D combustion workflow on current leadership class platforms. Our experiments demonstrate that Stacker achieves low latency, high volume data-staging with low overhead as compared to in-memory staging services for production scientific workflows.




Back to Technical Papers Archive Listing