<span class="var-sub_title">PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

ISAV 2018: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization


PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows

Abstract: This paper presents PaDaWAn, an infrastructure written in Python to provide loosely coupled in situ capabilities to accelerate file-based simulation workflows. It provides services for in-memory data exchange between applications and a simple configuration model to switch from a file-based workflow to a loosely coupled in situ workflow. The infrastructure is currently based on CEA-DAM Hercule parallel I/O library by providing an ABI-compatible library to intercept simulation data in a transparent way and to facilitate integration into existing simulation codes and tools. PaDaWAn implements a producer-consumer pattern with buffering of data in an in-memory staging service with automatic memory management and running on dedicated resources.

We describe the key design decisions and main architectural features, and share the lessons learned from the development of the infrastructure and from test runs on two production-like workflow cases. We conclude on the perspectives for our infrastructure.


Archive Materials


Back to ISAV 2018: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization Archive Listing

Back to Full Workshop Archive Listing