search-icon
Paper
:
Large-Scale Hierarchical K-Means for Heterogeneous Many-Core Supercomputers
Event Type
Paper
Registration Categories
TP
Tags
Algorithms
Architectures
Data Analytics
Deep Learning
Networks
Scientific Computing
Visualization
TimeTuesday, November 13th1:30pm - 2pm
LocationC146
DescriptionThis paper presents a novel design and implementation of k-means clustering algorithm targeting the Sunway TaihuLight supercomputer. We introduce a multi-level parallel partition approach that not only partitions by dataflow and centroid, but also by dimension. Our multi-level (nkd) approach unlocks the potential of the hierarchical parallelism in the SW26010 heterogeneous many-core processor and the system architecture of the supercomputer.

Our design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability, significantly improving the capability of k-means over previous approaches. The evaluation shows our implementation achieves performance of less than 18 seconds per iteration for a large-scale clustering case with 196,608 data dimensions and 2,000 centroids by applying 4,096 nodes (1,064,496 cores) in parallel, making k-means a more feasible solution for complex scenarios.
Archive
Back To Top Button