Pandas on the Cloud with Dask

Pandas on the Cloud with Dask

Pandas on the Cloud with Dask. Scaling your Pythonic data science and machine learning to the cloud using Dask. All from the comfort of your own laptop.

When doing data science and/or machine learning, it is becoming increasingly common to need to scale up your analyses to larger datasets.

When working in Python and the PyData ecosystem, Dask is a popular tool for doing so. There are many reasons for this, one being that Dask composes well with all of the PyData tools. It’s a simple system designed to parallelize any PyData library.

When beginning to work with a larger dataset, you’ll first want to scale up your analysis to leverage all the cores of a single workstation.

After this, you may need to scale out your computation to leverage a cluster on the cloud (AWS, Azure, or Google Cloud Platform, for example).

In this post, we

  • demonstrate a common pattern in data science workflows using pandas,
  • show how we can scale it up using Dask to harness the cores of a single workstation, and
  • Show how we can scale it out to the cloud using Coiled Cloud. There are many solutions for scaling out to the cloud but I’m particularly excited about Coiled as we just launched our Cloud product.

You can also find all the code here on Github.

Note: you should always try to reason about whether you actually need to scale out your computation. For example, before doing so, perhaps you could make your pandas code more efficient. If you’re doing machine learning, plot learning curves to make sure that including more data will actually result in improving your model.

big-data python cloud-computing machine-learning data-science

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.