Leveraging Scalable Event Distribution to Enable Data-Driven In Situ Scientific Workflows
TimeMonday, November 12th12:10pm - 12:15pm
DescriptionNovel event-driven workflow systems have been effectively used to increase the performance of large-scale scientific applications by removing most of the implicit synchronization required to orchestrate distributed tasks. However, these event-driven workflow systems, by focusing only on events related to the completion of tasks and data transfers, fail to address the dynamic and irregular workflows that require fine adaptation of the execution to the environment, faults, and to partial results from the application itself.
In this article, we explore the idea of a programming model for irregular and dynamic workflows that is not only based on task-related events, but also on the intermediate data produced the tasks. We contend that compared to traditional workflow execution systems this technique will ease development, increase flexibility and performance by removing implicit synchronization and automating previously tedious tasks related to workflow steering. We identify the classes of workflows that will benefit the most from this model and discuss design considerations for future implementations. In particular, we discuss how novel in-situ analysis techniques can be leveraged to implement a workflow system based on events of various natures and origins, from the infrastructure to the intermediate data while a workflow is running.