Dac-Man: Data Change Management for Scientific Datasets on HPC Systems

Authors: Devarshi Ghoshal (Lawrence Berkeley National Laboratory), Lavanya Ramakrishnan (Lawrence Berkeley National Laboratory), Deborah Agarwal (Lawrence Berkeley National Laboratory)

Abstract: Scientific data is growing rapidly and often changes due to instrument configurations, software updates, or quality assessments. These changes in datasets can result in significant waste of compute and storage resources on HPC systems as downstream pipelines are reprocessed. Data changes need to be detected, tracked, and analyzed for understanding the impact of data change, managing data provenance, and making efficient and effective decisions about reprocessing and use of HPC resources. Existing methods for identifying and capturing change are often manual, domain-specific, and error-prone and do not scale to large scientific datasets. In this paper, we describe the design and implementation of Dac-Man framework, which identifies, captures, and manages change in large scientific datasets, and enables plug-in of domain-specific change analysis with minimal user effort. Our evaluations show that it can retrieve file changes from directories containing millions of files and terabytes of data in less than a minute.


Presentation: file


Back to Technical Papers Archive Listing