<span class="var-sub_title">Comparison of the HPC and Big Data Java Libraries Spark, PCJ and APGAS</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

PAW-ATM: Parallel Applications Workshop - Alternatives to MPI


Comparison of the HPC and Big Data Java Libraries Spark, PCJ and APGAS

Abstract: Although Java is rarely used in HPC, there are a few notable libraries. Use of Java may help to bridge the gap between HPC and big data processing.

This paper compares the big data library Spark, and the HPC libraries PCJ and APGAS, regarding productivity and performance. We refer to Java versions for all libraries. For APGAS, we include both the original version and an own extension by locality-flexible tasks. We consider three benchmarks: Calculation of pi from HPC, Unbalanced Tree Search (UTS) from HPC, and WordCount from the big data domain.

In performance measurements with up to 144~workers, the extended APGAS library was the clear winner. With 144 workers, APGAS programs were up to a factor of more than two faster than Spark programs, and up to about 30% faster than PCJ programs. Regarding productivity, the extended APGAS programs consistently needed the lowest number of different library constructs. Spark ranged second in productivity, and PCJ third.


Website: http://sourceryinstitute.github.io/PAW/

Archive Materials


Back to PAW-ATM: Parallel Applications Workshop - Alternatives to MPI Archive Listing

Back to Full Workshop Archive Listing