Co-Presenters/lead authors: Stefan Gary, Alvaro Vidal Torreira
Co-Authors: Mike Wilde, Matthew Shaxted
Abstract: There is widespread commercial and academic interest in democratizing supercomputing by porting user applications to workflows that run on scalable and elastic high-performance computing (HPC) resources in the Cloud. One open-source framework that helps to address this aspiration in Python is the Parsl parallel scripting package (http://parsl-project.org). However, configuring Parsl (or other workflow tools) to safely navigate the Cloud is extremely challenging for non-experts. To bridge this gap, Parallel Works, a small Chicago-based startup, has wrapped Parsl into a streamlined user interface that leverages Parsl's large-scale computational orchestration while also providing users with the ability to intuitively customize resources on nearly any cloud. Here, we present recent case studies of real-world, cloud-based machine learning and HPC workflows from physical oceanography, fluvial biogeochemistry, and industrial design optimization to highlight the process of deploying an application at a large scale in the cloud.
Bios:
Dr. Stefan Gary is a physical oceanographer and climate scientist with 15 years of experience observing and modeling the North Atlantic Ocean. In his current role at Parallel Works, Stefan works with platform users to develop workflows in Parsl and other frameworks for earth and environmental science applications, including machine learning. He has also taught undergraduates and graduate students in the US and Scotland.
Dr. Alvaro Vidal Torreira is an early career engineering scientist. He has extensive computational fluid dynamics and finite element analysis experience and has spent the past 5 years with Parallel Works, building machine learning workflows and extending the workflow programming fabric. He is an expert in the structure of computational science and engineering workflow patterns, and in benchmarking and tuning on HPC and cloud resources.
- Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
