Bootstrapping Dask on 1000 Cores with AWS Fargate

Bootstrapping Dask on 1000 Cores with AWS Fargate

In this tutorial, let's see Bootstrapping Dask on 1000 cores with AWS Fargate. Harness the power of the cloud to provision and deploy 1000 cores to process large amounts and get stuff done

As a data engineer, I found myself many times in need of a system where I can prototype heavy big data tasks. My team traditionally used EMR with Apache Spark for such tasks but with news about Apache retiring hadoop projects and with the common paradigm that python is the go-to tool for data analysis, I was in pursuit of a more modern, python-native framework.

My goal was to produce a one-click deployment of a 1000-core cluster that can be run to prototype processing of ~1TB of data and then torn down. Another condition is that this resources need not to interfere with the rest of my team’s stack. Therefore the one-click-eks-fargate-dask tool was born!

kubernets

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes vs Docker

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Typical flow for deploying applications to Kubernetes

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Microsoft Announces General Availability Of Bridge To Kubernetes

Microsoft announced the general availability of Bridge to Kubernetes, formerly known as Local Process with Kubernetes.