<span class="var-sub_title">SaNSA - the Supercomputer and Node State Architecture</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Workshop on Fault-Tolerance for HPC at Extreme Scale (FTXS)


SaNSA - the Supercomputer and Node State Architecture

Authors:

Abstract: In this work, we present SaNSA, the Supercomputer and Node State Architecture, a software infrastructure for historical analysis and anomaly detection. SaNSA consumes data from multiple sources including system logs, the resource manager, scheduler, and job logs. Furthermore, additional context such as scheduled maintenance events or dedicated application run times for specific science teams can be overlaid. We discuss how this contextual information allows for more nuanced analysis. SaNSA allows the user to apply arbitrary attributes, for instance, positional information where nodes are located in a data center. We show how using this information we identify anomalous behavior of one rack of a 1,500 node cluster. We explain the design of SaNSA and then test it on four open compute clusters at LANL. We ingest over 1.1 billion lines of system logs in our study of 190 days in 2018. Using SaNSA, we perform a number of different anomaly detection methods and explain their findings in the context of a production supercomputing data center. For example, we report on instances of misconfigured nodes which receive no scheduled jobs for a period of time as well as examples of correlated rack failures which cause jobs to crash.

Archive Materials


Back to Workshop on Fault-Tolerance for HPC at Extreme Scale (FTXS) Archive Listing

Back to Full Workshop Archive Listing