Getting Scientific Software Installed

Authors: Davide Vanzo (Vanderbilt University), Kenneth Hoste (Ghent University), Robert McLay (University of Texas)

Abstract: We intend to provide a platform for presenting and discussing tools to deal with the ubiquitous problems that come forward when building and installing scientific software, which is known to be a tedious and time consuming task. Several user support tools for allowing scientific software to be installed and used will briefly be presented, for example (but not limited to) EasyBuild, Lmod, Spack, Singularity, etc. We will bring various experienced members of HPC user support teams and system administrators as well as users together for an open discussion on tools and best practices.

Long Description: Providing scientists with the scientific software they require is a crucial aspect of supercomputer systems. However, it is well known that building and installing scientific software, and implementing user-friendly access to it for end-users can be a tedious and time consuming task.

Compilation from source is preferred in an HPC setting where performance is critical, because it allows using compilers, libraries and compilation options best suited for the hardware platform and architecture of the targeted system. Scientists tend to focus on the functional aspect of the software they write, and typically spend little time in making sure their tools are simple to get built and installed. Examples of common scientific software packages that are notoriously challenging to get built include NWChem, OpenFOAM, QuantumESPRESSO, TensorFlow, WRF, etc.

Consequently, expert knowledge with regard to both building and installing software from source is indispensable when deploying a collection of scientific software. Likewise, keeping the deployment up to date can be a tedious process as new versions of compilers, libraries and the software itself are being released. These tasks are known to consume significant amounts of time from HPC user support teams for various reasons, e.g., manually stepping through documentation, platform/compiler/library dependent compilation issues, linking issues with well established libraries, runtime problems because of dynamic resolving of libraries, sub-optimal application performance due to poor choice of build parameters, etc.

Moreover, these efforts to obtain correct and (near) optimal builds of scientific software packages and provide them to end users are duplicated all over the world by numerous HPC teams. Several sites and consortia publicly document their `best practices', but documentation is often outdated, effort duplication remains, manually performing installation procedures is still error-prone, etc. Although most sites suffer from the same problems, there was little collaboration among them (until recently thanks to projects like EasyBuild and Spack).

Once installed, the software packages must be provided to the users. A common and adequate solution for this is providing environment modules, but many open problems remain to be dealt with, e.g., module files are often constructed manually -- making it difficult to keep them consistent. Additionally, various design choices further impact the user experience, e.g., module naming schemes, structure of the module files tree, tools for consuming modules, etc.

This Birds-of-a-Feather session is a follow-up to earlier BoF sessions held at ISC'13, SC13, ISC'14, SC14 and ISC'15 with three main goals. First, we will bring various experienced members of HPC user support teams, system administrators and end users together for an open discussion on how to (further) improve the current situation. For this, we provide a platform for presenting tools and best practices that help alleviate or resolve current problems. Second, we want to continue building a community, collaborate and streamline the work that is being done. Finally, we will compare the results from this session with the information we gathered during the former BoF sessions. This will allow us to evaluate the evolution of the state of practice with respect to getting scientific software installed.

URL: https://github.com/easybuilders/easybuild/wiki/SC18-BoF-session-Getting-Scientific-Software-Installed



Back to Birds of a Feather Archive Listing