If you’ve ever struggled with setting up pipelines for extracting, transforming, and loading data (so-called ETL jobs), managing different databases, and scheduling workflows – know that there’s an easier way to automate these data engineering tasks. In this article, you’ll learn how to build an n8n workflow that processes text, stores data in two databases, and sends messages to Slack.
A few months ago, I completed a Data Science bootcamp, where one week was all about data engineering, ETL pipelines, and workflow automation. The project for that week was to create a database of tweets that use the hashtag #OnThisDay, along with their sentiment score, and post tweets in a Slack channel to inform members about historical events that happened on that day. This pipeline had to be done with Docker Compose and included six steps:
1. Collect tweets with the hashtag #OnThisDay
2. Store the collected tweets in a MongoDB database
3. Extract tweets from the database
4. Process the tweets (clean the text, analyse sentiment)
5. Load the cleaned tweets and their sentiment score in a Postgres database
6. Extract and post tweets with positive sentiment in a Slack channel
This is a fun project that offers lots of learning opportunities about different topics: APIs, text processing with Natural Language Processing libraries, both relational and non-relational databases, social media and communication apps, as well as workflow orchestration. If you’re wondering, like I did, why we had to use two different databases, the answer is simple: for the sake of learning more. Postgres and MongoDB represent not only different database providers, but different kinds of database structures – relational (SQL) vs non-relational (NoSQL) – and it’s useful to be familiar with both.
Though our use case is just for fun, this pipeline can support most common data engineering tasks (e.g. aggregating data from multiple sources, setting up and managing the data flow across databases, developing and maintaining data pipelines).
I was really excited, though also a bit overwhelmed by all the things I had to set up for this project. In total, I spent five days learning the tools, debugging, and building this pipeline with Python (including libraries like Tweepy, TextBlob, VADER, and SQLAlchemy), Postgres, MongoDB, Docker, and Airflow (most frustrating part…). If you’re interested to see how I did this, you can check out the project on GitHub and read this blog post.
But in this article, I’ll show you an easier way to achieve the same result in as much as an hour – with n8n!
#automation #data-engineering #databases #slack-bot #sentiment-analysis #twitter-api #natural-language-processing #no-code
