Build a Rust app to consume from the Twitter Streaming API.

Welcome to this series of blog posts that covers  Redis Streams with the help of a practical example. We will use a sample application to make Twitter data available for search and query in real-time.  RediSearch and  Redis Streams serve as the backbone of this solution that consists of several co-operating components, each of which will be covered in a dedicated blog post.

The code is available in this GitHub repo -  https://github.com/abhirockzz/redis-streams-in-action

In this part, we look at the service which interacts with the Twitter Streaming API to consume tweets and move them on to the next part in the processing pipeline.

Our end goal is to be able to process tweets and make them available for search and queries via RediSearch. One could write a “do it all” service to consume tweets and directly store them in RediSearch. But, in order to scale to handle the volume of tweets, we need a service to act as a buffer and decouple our producer (the application we will focus in this blog) and consumer (covered in next blog).

This is exactly what our first component facilitates - it consumes streaming Twitter data and forwards it to Redis Streams. We will deploy it to  Azure Container Instances, validate it’s functionality and also walk-through how it works along with the code.

As you will see in the later parts of this series, this also provides a foundation for scale-out architecture.

All in all, this blog post is short and simple! It lays down the ground work for other parts of the solution which will be covered in subsequent posts. Please don’t worry about the fact that the service is written in Rust (in case you don’t know it already). The logic can be easily ported over to your favorite programming language.

Pre-requisites

Start by getting a  free Azure account if you don’t have one already and  install the Azure CLI as well.

We will be deploying the tweets consumer application to  Azure Container Instances using regular Docker CLI commands. This capability is enabled by  integration between Docker and Azure. Just ensure that you have Docker Desktop version 2.3.0.5 or later, for  Windows,  macOS, or install the  Docker ACI Integration CLI for Linux.

To use the Twitter Streaming API, you will also need a Twitter developer account. If you don’t have one already, please  follow these instructions on how to set it up.

#java #database #cloud #tutorial #azure #serverless #redis #golang #rust #redis streams