Dask vs Vaex: Experience of a Data Point in Large Data Processing

Dask vs Vaex: Experience of a Data Point in Large Data Processing

I would like to share my experience as a data point working with my new managers, Dask and Vaex, as well as some tips to have a good working relationship with them.

Hello there! Nice to meet you! 😄 I’m Data N (you can call me N) and today, I would like to share my experience as a data point working with my new managers, Dask and Vaex, as well as some tips to have a good working relationship with them (wink).

Background Story

The background story goes like this… Recently, our company had a little restructuring and our ex-manager, Pandas 🐼, was taken over by two new hires. The official reason given was that Pandas moved on to new opportunities but all of us insiders knew what happened.

Well, the truth is that the top level management was not pleased with Pandas’ performance lately. Our company had grown quickly and business increased exponentially. Pandas was initially doing great but gradually find himself unable to cope with increasing data. When the full truckload of us data points arrives, we prove to be too much for Pandas to cope. Usually, we will sit in a large warehouse called hard disk, but when we need to be processed, there’s this temporary storage room called Random-Access Memory (a.k.a. RAM) where we will be transported to for further processing. Here’s where the problem lies: there’s not enough space for all of us to fit into RAM.

dask data-processing vaex machine-learning data-science

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

How To Build A Data Science Career In 2021

In Conversation With Dr Suman Sanyal, NIIT University,he shares his insights on how universities can contribute to this highly promising sector and what aspirants can do to build a successful data science career.

Hire Machine Learning Engineer | Offshore Machine Learning Experts

We are a Machine Learning Services provider offering custom AI solutions, Machine Learning as a service & deep learning solutions. Hire Machine Learning experts & build AI Chatbots, Neural networks, etc. 16+ yrs & 2500+ clients.

The Difference between Data Science, Machine Learning and Big Data!

Many professionals and 'Data' enthusiasts often ask, “What's the difference between Data Science, Machine Learning and Big Data?”. Let's clear the air. If you are still wondering about it then this article is for you.

5 stages of learning Data Science

5 stages of learning Data Science and how to ace each of them

More Resources in AI, Data Science, and Machine Learning; Speeding up Scikit-Learn

More Resources for Women in AI, Data Science, and Machine Learning; Speeding up Scikit-Learn Model Training; Dask and Pandas: No Such Thing as Too Much Data; 9 Skills You Need to Become a Data Engineer; 8 Women in AI Who Are Striving to Humanize the World. It's a pity if you miss this great article.