Stacker: An Autonomic Data Movement Engine for Extreme-Scale Data Staging-Based In Situ Workflows
Event Type
Registration Categories
Data Management
File Systems
State of the Practice
System Software
TimeThursday, November 15th4pm - 4:30pm
DescriptionData staging and in situ workflows are being explored extensively as an approach to address data-related costs at very large scales. However, the impact of emerging storage architectures (e.g., deep memory hierarchies and burst buffers) upon data staging solutions remains a challenge. In this paper, we investigate how burst buffers can be effectively used by data staging solutions, for example, as a persistence storage tier of the memory hierarchy. Furthermore, we use machine learning based prefetching techniques to move data between the storage levels in an autonomous manner. We also present Stacker, a prototype of the proposed solutions implemented within the Data\-Spaces data staging service, and experimentally evaluate its performance and scalability using the S3D combustion workflow on current leadership class platforms. Our experiments demonstrate that Stacker achieves low latency, high volume data-staging with low overhead as compared to in-memory staging services for production scientific workflows.
Back To Top Button