<span class="var-sub_title">DagOn*: Executing Direct Acyclic Graphs as Parallel Jobs on Anything</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

WORKS 2018: 13th Workshop on Workflows in Support of Large-Scale


DagOn*: Executing Direct Acyclic Graphs as Parallel Jobs on Anything

Authors: Raffaele Montella (Parthenope University of Naples)

Abstract: The democratization of computational resources, thanks to the advent of public, private, and hybrid clouds, changed the rules in many science fields. For decades, one of the effort of computer scientists and computer engineers was the development of tools able to simplify access to high-end computational resources by computational scientists. However, nowadays any science field can be considered “computational” as the availability of powerful, but easy to manage workflow engines, is crucial. In this work, we present DagOn* (Direct acyclic graph On anything), a lightweight Python library implementing a workflow engine able to execute parallel jobs represented by direct acyclic graphs on any combination of local machines, on-premise high performance computing clusters, containers, and cloud-based virtual infrastructures. We use a real-world production-level application for weather and marine forecasts to illustrate the use of this new workflow engine.

Archive Materials


Back to WORKS 2018: 13th Workshop on Workflows in Support of Large-Scale Archive Listing

Back to Full Workshop Archive Listing