<span class="var-sub_title">Managing Python in HPC Environments</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

8th Workshop on Python for High-Performance and Scientific Computing

Managing Python in HPC Environments

Abstract: Python has seen a rapid adoption in the weather and climate modeling science communities. This swift rise has taken HPC system administrators by surprise, leading to inadequate support. These trends, like those in other sciences, led to the development and widespread adoption of user managed binary distributions. An example being Anaconda in 2012, which comes with security risks. We present a system for mirroring Anaconda Python that embeds PyRats, a dependency analyzer and logger descended from Blais’ Snakefood into the Anaconda installers. We show that Anaconda performance, reliability, security, and availability can be improved dramatically and enable timely integration into automated test environments. The dependency logging yields insights into which packages users rely on most. It can help prioritize optimization efforts such as building packages for the CPU families used in a given HPC environment. We also discuss related work, including a complimentary automated Python provisioning effort by Oak Ridge National Laboratory that they call PythonEnv-noaa.

Archive Materials

Back to 8th Workshop on Python for High-Performance and Scientific Computing Archive Listing

Back to Full Workshop Archive Listing