Workloads and Benchmarks for System Acquisition

Authors: Neil Bright (Georgia Institute of Technology), Laura Brown (US Army Engineer Research and Development Center), Henry Neeman (University of Oklahoma), Alex Younts (Purdue University)

Abstract: New HPC systems must balance storage, networking, and computational power to meet the different needs of its user base and intended usage. Government agencies, universities, and companies invest significant time, effort, and money when evaluating and acquiring new HPC systems. Unfortunately, many of these evaluations are duplicated from site-to-site. This BoF aims to start a best-practices collaboration between HPC sites to reduce the effort required to make good system evaluations. This session’s leaders will engage with the audience to discuss practical solutions to system benchmarking and evaluation for acquisition purposes.

Long Description: Government agencies, universities, and companies invest significant time, effort, and money when evaluating and acquiring new HPC systems. The final system architecture must balance storage, networking, and computational power to meet the different needs of its userbase and research thrusts. Achieving a balanced system that enables users and maximizes productivity for the life of the system is a difficult and arduous task. CORAL has developed a standardized set of benchmarks and workloads to evaluate leadership-class systems like Summit. Some of these tests are applicable and relevant to small and medium-sized HPC sites, but some are not. The session leaders of this BoF will give brief overviews of how they solve the problems of system benchmarking for acquisition at their sites. Different presenters will give insight into the acquisition details like when to start developing the benchmarking workload, how to balance the workload, and how many applications and tests should be contained. Each site may have a different answer to Questions like “Is it appropriate to use synthetic benchmarks like STREAM as a part of the workload?”, “Should the workload consist solely of real applications (WRF, VASP, LAMMPS, etc.)?”, and “Do we run our workload in the cloud?”. Presenters will cover techniques for obtaining current performance and resource utilization, and then estimating and codifying expected performance of the next system. Fair system evaluation is becoming more difficult now that accelerators and different CPU architectures are common. Tuning the workload for fair evaluation with emphasis on apples/apples comparison will be discussed. Smaller teams may not have the personnel nor time to evaluate each possible architecture, and organizations with different numbers of staff will address these major points in different ways. Each session leader will have the opportunity to discuss how their team works. The tradeoffs made by each team, and the insights gained from the process of evaluation are valuable to the wider community. At the end of the session, attendees should grasp the overarching picture of workload characterization and benchmarking for system acquisition. The major desired result of this BoF is to start a multi-institution collaboration for developing and curating benchmarks and workloads that small and medium sized HPC groups can use to evaluate new systems for acquisition. A concrete result of the BoF will be a published document giving an overview of the steps of effective acquisitional benchmarking with advice and setups from the different session leaders.




Back to Birds of a Feather Archive Listing