Have you worked on big data or “big data”?

Petabytes. Not Terabytes, Petabytes. That’s how much storage space the Hadoop File System (HDFS) had when I first started working with big data.

We have all heard of the term big data, but I believe we never fully understand it before getting a hands-on experience.

I first learnt about it during my sophomore year in the university and even worked on “big data” in my first job.

I was responsible for setting up a pipeline to process 10 MB of generated data everyday. Which would amount to around 3.65 GB of data per year.

We even decided to not use Hadoop since the overhead of loading the data from HDFS is much longer than simply reading the data directly through the file system.

Now that I have came across a big data, I realised what I did that time not come remotely close to what I would call big data.

Here are 5 things I learned from my first experience with big data.

#learning #software-development #lessons-learned #data-science #big-data #things i learned from working with big data