Hello there! Nice to meet you! 😄 I’m Data N (you can call me N) and today, I would like to share my experience as a data point working with my new managers, Dask and Vaex, as well as some tips to have a good working relationship with them (wink).

Background Story

The background story goes like this… Recently, our company had a little restructuring and our ex-manager, Pandas 🐼, was taken over by two new hires. The official reason given was that Pandas moved on to new opportunities but all of us insiders knew what happened.

Well, the truth is that the top level management was not pleased with Pandas’ performance lately. Our company had grown quickly and business increased exponentially. Pandas was initially doing great but gradually find himself unable to cope with increasing data. When the full truckload of us data points arrives, we prove to be too much for Pandas to cope. Usually, we will sit in a large warehouse called hard disk, but when we need to be processed, there’s this temporary storage room called Random-Access Memory (a.k.a. RAM) where we will be transported to for further processing. Here’s where the problem lies: there’s not enough space for all of us to fit into RAM.

#dask #data-processing #vaex #machine-learning #data-science