<span class="var-sub_title">Big Data Challenge - How to Engage with Large Scale Facilities?</span> SC18 Proceedings

The International Conference for High Performance Computing, Networking, Storage, and Analysis

Big Data Challenge - How to Engage with Large Scale Facilities?


Authors: John Towns (National Center for Supercomputing Applications), Florian Berberich (Partnership for Advanced Computing in Europe (PRACE))

Abstract: In recent years there has been a rapid growth in the number of large scale facilities (LSF) – that produce large amounts of data. The classical examples are the Large Hadron Collider and SKA. In the context of a set of concrete examples, this BoF will discuss both current and projected requirements for processing, storing, sharing, and securing of data as well as data quantities, processing needs, and expected data transfer needs with the goal of collecting an aggregate set of requirements from the LSFs allowing for gap analysis of these requirements against the roadmaps of international HPC data centers.

Long Description: In recent years there had been a rapid growth in the number of international research infrastructures – large scale facilities (LSF) – that produce large amounts of data. The classical examples are the Large Hadron Collider in high-energy physics and SKA in astronomy, but there are many more in various scientific domains, including physics, astronomy and life sciences. Processing, storing, sharing and securing of data are part of data life cycle management and will need to be carefully considered in the context of these large-scale, global collaborations. Insight into data volumes, processing needs (from the edge to the HPC centers) and expected data transfer requirements is needed in order to prepare the next generation of resources and services at HPC centers around the world. In this BoF the full chain from the production of data at the LSF (e.g. CERN / SKA / HBP …), the transport and storage of data, and finally the data processing and sharing of data products will be analyzed. Collaborating Research infrastructures, PRACE, XSEDE, Pawsey, and CHPC will provide some concrete examples to show the current work flow and possible solutions. PRACE (Europe), XSEDE (USA), Pawsey (Australia) and CHPC (South Africa) will jointly organize this BoF and invite collaborating LSF and network providers e.g. GEANT to this BoF. This BoF was not held before and the ultimate goal of the BoF session will be to collect and compare all of the requirements spanning these LSFs. A white paper based on the collected information will be developed and published to share the results with the community.




Back to Birds of a Feather Archive Listing